Loading ...
Sorry, an error occurred while loading the content.

Postscreen status script, take two

Expand Messages
  • Mike.
    I made some changes to the script based upon the excellent feedback I received here. The script no longer wanders beyond the postscreen log records in order to
    Message 1 of 6 , Jan 30, 2013
    • 0 Attachment
      I made some changes to the script based upon the excellent feedback I
      received here.

      The script no longer wanders beyond the postscreen log records in order
      to gather the information needed to determine the postscreen rejection
      rate. So that removes the problems caused by multiple-recipient
      messages.

      There is now the need to tell the script whether deep protocol testing
      is being done. There's an easy way to do this in the script. The
      default setting for this is the same as postscreen's default - deep
      protocol testing is disabled.

      Also, there is also the ability in the script to adjust the mktemp
      template according to the OS being used.



      You can download version 1.2 of the script from here:
      http://archive.mgm51.com/sources/pslogscan.html


      Here is the sample output that pslogscan.sh produces (the 158MB file
      was processed on 4 seconds):

      Scanning /var/log/maillog

      CONNECT log records: 116340
      PASS NEW log records: 8190
      PASS OLD log records: 37002
      WHITELISTED log records: 2289
      BLACKLISTED log records: 0

      rejected: 77049 (66%)


      Protocol errors:
      HANGUP log records: 62580
      PREGREET log records: 3927
      BARE NEWLINE log records: 21
      COMMAND TIME LIMIT log records: 168
      COMMAND PIPELINING log records: 21

      DNS black lists log records:
      b.barracudacentral.org: 57939
      dnsbl.sorbs.net: 28098
      zen.spamhaus.org: 66654

      DNSBL blocked log records: 50610
      DNSBL rank 3: 10353
      DNSBL rank 4: 0
      DNSBL rank 5: 0
      DNSBL rank 6: 19698
      DNSBL rank 7: 0
      DNSBL rank 8: 0
      DNSBL rank 9+: 20559

      DNSBL blocks by domain:
      example.com: 8253
      example.net: 1449
      example.info: 35679
      example.bix: 2268
    • Sahil Tandon
      ... Be careful with grep(1) patterns. You overstate CONNECTs by including NOQUEUE: reject: CONNECT in the count. Meanwhile, the script ... That bracket
      Message 2 of 6 , Feb 2 6:52 AM
      • 0 Attachment
        On Wed, 2013-01-30 at 14:23:19 -0500, Mike. wrote:

        > I made some changes to the script based upon the excellent feedback I
        > received here.
        >
        > The script no longer wanders beyond the postscreen log records in
        > order to gather the information needed to determine the postscreen
        > rejection rate. So that removes the problems caused by
        > multiple-recipient messages.
        > ...

        Be careful with grep(1) patterns. You overstate CONNECTs by including
        'NOQUEUE: reject: CONNECT' in the count. Meanwhile, the script
        understates total DNSBL rejections, which you measure with:

        | grep -c "DNSBL rank [3-99]"

        That bracket expression matches on a _single_ character, and does not
        capture double-digit ranks. A similar mistake occurs in the attempt to
        aggregate 9+ ranks:

        | grep -c "DNSBL rank [9-99] "

        This only counts appearances of "DNSBL rank 9" in the log, as
        illustrated below:

        | % grep -c "DNSBL rank [9-99] " maillog
        | 4494

        | % grep -c "DNSBL rank 9 " maillog
        | 4494

        Review the re_format(7) and grep(1) manuals to improve understanding of
        regular expressions. In case it helps you, last year I had cobbled
        together a slower (it is Python rather than a set of grep(1)
        expressions) script[1] to collect similar statistics. No promises that
        it is error-free.

        [1] http://people.freebsd.org/~sahil/scripts/mailstats.py.txt

        --
        Sahil Tandon
      • Mike.
        ... I ... to ... of ... that ... ============= Thanks for the feedback.
        Message 3 of 6 , Feb 2 9:01 AM
        • 0 Attachment
          On 2/2/2013 at 9:52 AM Sahil Tandon wrote:

          |On Wed, 2013-01-30 at 14:23:19 -0500, Mike. wrote:
          |
          |> I made some changes to the script based upon the excellent feedback
          I
          |> received here.
          |>
          |> The script no longer wanders beyond the postscreen log records in
          |> order to gather the information needed to determine the postscreen
          |> rejection rate. So that removes the problems caused by
          |> multiple-recipient messages.
          |> ...
          |
          |Be careful with grep(1) patterns. You overstate CONNECTs by including
          |'NOQUEUE: reject: CONNECT' in the count. Meanwhile, the script
          |understates total DNSBL rejections, which you measure with:
          |
          || grep -c "DNSBL rank [3-99]"
          |
          |That bracket expression matches on a _single_ character, and does not
          |capture double-digit ranks. A similar mistake occurs in the attempt
          to
          |aggregate 9+ ranks:
          |
          || grep -c "DNSBL rank [9-99] "
          |
          |This only counts appearances of "DNSBL rank 9" in the log, as
          |illustrated below:
          |
          || % grep -c "DNSBL rank [9-99] " maillog
          || 4494
          |
          || % grep -c "DNSBL rank 9 " maillog
          || 4494
          |
          |Review the re_format(7) and grep(1) manuals to improve understanding
          of
          |regular expressions. In case it helps you, last year I had cobbled
          |together a slower (it is Python rather than a set of grep(1)
          |expressions) script[1] to collect similar statistics. No promises
          that
          |it is error-free.
          |
          |[1] http://people.freebsd.org/~sahil/scripts/mailstats.py.txt
          |
          |--
          |Sahil Tandon

          =============


          Thanks for the feedback.
        • Mike.
          ... I tightened up that regex to include only the CONNECT occurrences I want. ... ============= I fixed both of those. Version 1.4 of the pslogscan.sh script,
          Message 4 of 6 , Feb 3 9:06 AM
          • 0 Attachment
            On 2/2/2013 at 9:52 AM Sahil Tandon wrote:

            |Be careful with grep(1) patterns. You overstate CONNECTs by
            |including 'NOQUEUE: reject: CONNECT' in the count.

            I tightened up that regex to include only the CONNECT occurrences I
            want.



            | Meanwhile, the script
            | understates total DNSBL rejections,...
            |
            |A similar mistake occurs in the attempt to
            |aggregate 9+ ranks:
            =============

            I fixed both of those.


            Version 1.4 of the pslogscan.sh script, incorporating the above fixes,
            is available at:
            http://archive.mgm51.com/sources/pslogscan.html


            Thanks again for your feedback.
          • Steve Jenkins
            ... Hey, Mike. It s a cool idea - and almost works on a CentOS 6 box, but I m getting ambiguous redirect errors in a couple of cases. When it s first run
            Message 5 of 6 , Feb 6 9:41 AM
            • 0 Attachment
              On Sun, Feb 3, 2013 at 9:06 AM, Mike. <the.lists@...> wrote:
              Version 1.4 of the pslogscan.sh script, incorporating the above fixes,
              is available at:
              http://archive.mgm51.com/sources/pslogscan.html

              Hey, Mike. It's a cool idea - and almost works on a CentOS 6 box, but I'm getting "ambiguous redirect" errors in a couple of cases. When it's first run (/tmp/pslogscan does not exist yet) I get:

              # ./pslogscan.sh /var/log/maillog
              Scanning /var/log/maillog
              mktemp: cannot create temp file /tmp/pslogscan: File exists

                CONNECT log records:      1106
                PASS NEW log records:     50
                PASS OLD log records:     25
                WHITELISTED log records:  717
                BLACKLISTED log records:  0

                        rejected:         314  (28%)


                Protocol errors:
                              HANGUP log records:  223
                            PREGREET log records:  62
                        BARE NEWLINE log records:  0
                  COMMAND TIME LIMIT log records:  0
                  COMMAND PIPELINING log records:  0

                DNS black lists log records:
              ./pslogscan.sh: line 140: ${TmpFile}: ambiguous redirect
                            zen.spamhaus.org:

              Then it freezes and I have to CTRL+C out.

              On all subsequent attempts (if /tmp/pslogscan already exists) I get:

              # ./pslogscan.sh /var/log/maillog
              Scanning /var/log/maillog
              mktemp: cannot create temp file /tmp/pslogscan: File exists
              mktemp: cannot create temp file /tmp/pslogscan: File exists

              ./pslogscan.sh: line 78: ${PostscreenLog}: ambiguous redirect

              Thanks for your efforts! I'm happy to test out future versions on my system. Feel free to email me directly and I'll test them out.

              SteveJ 
            • Steve Jenkins
              ... And.... ignore all that. Commenting mktempTemplate=pslogscan and uncommenting #mktempTemplate=pslogscan.XXX did the trick. :) Thx again! SteveJ
              Message 6 of 6 , Feb 6 9:43 AM
              • 0 Attachment
                On Wed, Feb 6, 2013 at 9:41 AM, Steve Jenkins <stevejenkins@...> wrote:
                Hey, Mike. It's a cool idea - and almost works on a CentOS 6 box, but I'm getting "ambiguous redirect" errors in a couple of cases. When it's first run (/tmp/pslogscan does not exist yet) I get:

                # ./pslogscan.sh /var/log/maillog
                Scanning /var/log/maillog
                mktemp: cannot create temp file /tmp/pslogscan: File exists

                  CONNECT log records:      1106
                  PASS NEW log records:     50
                  PASS OLD log records:     25
                  WHITELISTED log records:  717
                  BLACKLISTED log records:  0

                          rejected:         314  (28%)


                  Protocol errors:
                                HANGUP log records:  223
                              PREGREET log records:  62
                          BARE NEWLINE log records:  0
                    COMMAND TIME LIMIT log records:  0
                    COMMAND PIPELINING log records:  0

                  DNS black lists log records:
                ./pslogscan.sh: line 140: ${TmpFile}: ambiguous redirect
                              zen.spamhaus.org:

                Then it freezes and I have to CTRL+C out.

                On all subsequent attempts (if /tmp/pslogscan already exists) I get:

                # ./pslogscan.sh /var/log/maillog
                Scanning /var/log/maillog
                mktemp: cannot create temp file /tmp/pslogscan: File exists
                mktemp: cannot create temp file /tmp/pslogscan: File exists

                ./pslogscan.sh: line 78: ${PostscreenLog}: ambiguous redirect

                Thanks for your efforts! I'm happy to test out future versions on my system. Feel free to email me directly and I'll test them out.

                And.... ignore all that. Commenting mktempTemplate=pslogscan and uncommenting #mktempTemplate=pslogscan.XXX did the trick. :)

                Thx again!

                SteveJ

              Your message has been successfully submitted and would be delivered to recipients shortly.