Loading ...
Sorry, an error occurred while loading the content.

Postscreen status script

Expand Messages
  • Mike.
    I implemented the postscreen capability on a small MTA I run for friends and family. Once I got postscreen configuration producing the results I wanted, I
    Message 1 of 12 , Jan 29, 2013
    • 0 Attachment
      I implemented the postscreen capability on a small MTA I run for
      friends and family. Once I got postscreen configuration producing the
      results I wanted, I soon tired of watching the detailed maillog to see
      how postscreen was operating. So I wrote a quick shell script to
      summarize the log file and give me an overview of how well postscreen
      is working.

      I offer the script to anyone who would like to use it. One company I
      worked for would not allow open source software into the company unless
      there was an explicit license on the software, so I put the BSD license
      on the script.

      You can download the script from here:
      http://archive.mgm51.com/sources/pslogscan.html


      Here is the sample output that pslogscan.sh produces:

      Scanning /var/log/maillog

      All "incoming" log records: 5789
      All "status=sent" log records: 1873
      All "status=deferred" log records: 10
      rejected: 3906 (67%)

      PASS NEW log records: 390
      PASS OLD log records: 1762

      WHITELISTED log records: 109
      BLACKLISTED log records: 0

      Protocol errors:
      HANGUP log records: 2980
      PREGREET log records: 187
      BARE NEWLINE log records: 1
      COMMAND TIME LIMIT log records: 8
      COMMAND PIPELINING log records: 1

      DNS black lists log records:
      zen.spamhaus.org: 3174
      dnsbl.sorbs.net: 1338
      b.barracudacentral.org: 2759

      DNSBL blocked log records: 2410
      DNSBL rank 3: 493
      DNSBL rank 4: 0
      DNSBL rank 5: 0
      DNSBL rank 6: 938
      DNSBL rank 7: 0
      DNSBL rank 8: 0
      DNSBL rank 9+: 979

      DNSBL blocks by domain:
      example.com: 393
      example.biz: 69
      example.net: 1699
      example.info: 108
    • Brian Evans
      ... Fails without modification on my Gentoo mailserver: Scanning /var/log/maillog mktemp: too few X s in template ‘mailqscan’ All incoming log records:
      Message 2 of 12 , Jan 29, 2013
      • 0 Attachment
        On 1/29/2013 1:07 PM, Mike. wrote:
        > I implemented the postscreen capability on a small MTA I run for
        > friends and family. Once I got postscreen configuration producing the
        > results I wanted, I soon tired of watching the detailed maillog to see
        > how postscreen was operating. So I wrote a quick shell script to
        > summarize the log file and give me an overview of how well postscreen
        > is working.
        >
        > I offer the script to anyone who would like to use it. One company I
        > worked for would not allow open source software into the company unless
        > there was an explicit license on the software, so I put the BSD license
        > on the script.
        >
        > You can download the script from here:
        > http://archive.mgm51.com/sources/pslogscan.html
        >
        Fails without modification on my Gentoo mailserver:
        Scanning /var/log/maillog
        mktemp: too few X's in template ‘mailqscan’

        All "incoming" log records: 10121
        ./pslogscan.sh: line 51: ${TmpFile}: ambiguous redirect

        Changing mailqscan to mailqscan.XXX works.

        Brian
      • Mike.
        ... the ... see ... postscreen ... I ... unless ... license ... ============= Thanks for the feedback. I only run FreeBSD, so I figure there may be some minor
        Message 3 of 12 , Jan 29, 2013
        • 0 Attachment
          On 1/29/2013 at 1:14 PM Brian Evans wrote:

          |On 1/29/2013 1:07 PM, Mike. wrote:
          |> I implemented the postscreen capability on a small MTA I run for
          |> friends and family. Once I got postscreen configuration producing
          the
          |> results I wanted, I soon tired of watching the detailed maillog to
          see
          |> how postscreen was operating. So I wrote a quick shell script to
          |> summarize the log file and give me an overview of how well
          postscreen
          |> is working.
          |>
          |> I offer the script to anyone who would like to use it. One company
          I
          |> worked for would not allow open source software into the company
          unless
          |> there was an explicit license on the software, so I put the BSD
          license
          |> on the script.
          |>
          |> You can download the script from here:
          |> http://archive.mgm51.com/sources/pslogscan.html
          |>
          |Fails without modification on my Gentoo mailserver:
          |Scanning /var/log/maillog
          |mktemp: too few X's in template �mailqscan�
          |
          |All "incoming" log records: 10121
          |./pslogscan.sh: line 51: ${TmpFile}: ambiguous redirect
          |
          |Changing mailqscan to mailqscan.XXX works.
          |
          |Brian

          =============


          Thanks for the feedback.

          I only run FreeBSD, so I figure there may be some minor issues like the
          one you mention when running on other OS's.
        • Brian Evans
          ... Also, your expressions don t count real postscreen numbers for connects and rejects. Take into account the following lines. Jan 28 12:47:57 mx1
          Message 4 of 12 , Jan 29, 2013
          • 0 Attachment
            On 1/29/2013 1:29 PM, Mike. wrote:
            >
            > On 1/29/2013 at 1:14 PM Brian Evans wrote:
            >
            > |On 1/29/2013 1:07 PM, Mike. wrote:
            > |> I implemented the postscreen capability on a small MTA I run for
            > |> friends and family. Once I got postscreen configuration producing
            > the
            > |> results I wanted, I soon tired of watching the detailed maillog to
            > see
            > |> how postscreen was operating. So I wrote a quick shell script to
            > |> summarize the log file and give me an overview of how well
            > postscreen
            > |> is working.
            > |>
            > |> I offer the script to anyone who would like to use it. One company
            > I
            > |> worked for would not allow open source software into the company
            > unless
            > |> there was an explicit license on the software, so I put the BSD
            > license
            > |> on the script.
            > |>
            > |> You can download the script from here:
            > |> http://archive.mgm51.com/sources/pslogscan.html
            > |>
            > |Fails without modification on my Gentoo mailserver:
            > |Scanning /var/log/maillog
            > |mktemp: too few X's in template ‘mailqscan’
            > |
            > |All "incoming" log records: 10121
            > |./pslogscan.sh: line 51: ${TmpFile}: ambiguous redirect
            > |
            > |Changing mailqscan to mailqscan.XXX works.
            > |
            > |Brian
            >
            > =============
            >
            >
            > Thanks for the feedback.
            >
            > I only run FreeBSD, so I figure there may be some minor issues like the
            > one you mention when running on other OS's.
            >
            >
            >
            Also, your expressions don't count real postscreen numbers for connects
            and rejects.
            Take into account the following lines.

            Jan 28 12:47:57 mx1 postfix/error[19363]: 3Yvy410c1Mz8GKk:
            to=<someuser@...>, relay=none, delay=2332, delays=2331/1.2/0/0.07,
            dsn=4.7.0, status=deferred (delivery temporarily suspended: host
            mta7.am0.yahoodns.net[66.94.238.147] refused to talk to me: 421 4.7.0
            [TS01] Messages from xx.xx.xx.xx temporarily deferred due to user
            complaints - 4.16.55.1; see http://postmaster.yahoo.com/421-ts01.html)
            Jan 28 12:48:26 mx1 postfix/smtp[19336]: 3Yvy4D6lG7z8GL8:
            to=<someuser@...>, relay=none, delay=2350,
            delays=2319/0.05/31/0, dsn=4.4.1, status=deferred (connect to
            yahoo.com.com[216.239.120.187]:25: Connection timed out)

            Because of that, I have skewed numbers:
            All "incoming" log records: 10187
            All "status=sent" log records: 7506
            All "status=deferred" log records: 3302
            rejected: -621 (-6%)

            It is not a simple math of "A minus B minus C" to find out how much
            postscreen is rejecting in its current state.

            Brian
          • Mike.
            ... producing ... to ... company ... the ... connects ... delays=2331/1.2/0/0.07, ... ============= Yup. When there are a lot of deferrals, then things get
            Message 5 of 12 , Jan 29, 2013
            • 0 Attachment
              On 1/29/2013 at 1:43 PM Brian Evans wrote:

              |On 1/29/2013 1:29 PM, Mike. wrote:
              |>
              |> On 1/29/2013 at 1:14 PM Brian Evans wrote:
              |>
              |> |On 1/29/2013 1:07 PM, Mike. wrote:
              |> |> I implemented the postscreen capability on a small MTA I run for
              |> |> friends and family. Once I got postscreen configuration
              producing
              |> the
              |> |> results I wanted, I soon tired of watching the detailed maillog
              to
              |> see
              |> |> how postscreen was operating. So I wrote a quick shell script to
              |> |> summarize the log file and give me an overview of how well
              |> postscreen
              |> |> is working.
              |> |>
              |> |> I offer the script to anyone who would like to use it. One
              company
              |> I
              |> |> worked for would not allow open source software into the company
              |> unless
              |> |> there was an explicit license on the software, so I put the BSD
              |> license
              |> |> on the script.
              |> |>
              |> |> You can download the script from here:
              |> |> http://archive.mgm51.com/sources/pslogscan.html
              |> |>
              |> |Fails without modification on my Gentoo mailserver:
              |> |Scanning /var/log/maillog
              |> |mktemp: too few X's in template �mailqscan�
              |> |
              |> |All "incoming" log records: 10121
              |> |./pslogscan.sh: line 51: ${TmpFile}: ambiguous redirect
              |> |
              |> |Changing mailqscan to mailqscan.XXX works.
              |> |
              |> |Brian
              |>
              |> =============
              |>
              |>
              |> Thanks for the feedback.
              |>
              |> I only run FreeBSD, so I figure there may be some minor issues like
              the
              |> one you mention when running on other OS's.
              |>
              |>
              |>
              |Also, your expressions don't count real postscreen numbers for
              connects
              |and rejects.
              |Take into account the following lines.
              |
              |Jan 28 12:47:57 mx1 postfix/error[19363]: 3Yvy410c1Mz8GKk:
              |to=<someuser@...>, relay=none, delay=2332,
              delays=2331/1.2/0/0.07,
              |dsn=4.7.0, status=deferred (delivery temporarily suspended: host
              |mta7.am0.yahoodns.net[66.94.238.147] refused to talk to me: 421 4.7.0
              |[TS01] Messages from xx.xx.xx.xx temporarily deferred due to user
              |complaints - 4.16.55.1; see http://postmaster.yahoo.com/421-ts01.html)
              |Jan 28 12:48:26 mx1 postfix/smtp[19336]: 3Yvy4D6lG7z8GL8:
              |to=<someuser@...>, relay=none, delay=2350,
              |delays=2319/0.05/31/0, dsn=4.4.1, status=deferred (connect to
              |yahoo.com.com[216.239.120.187]:25: Connection timed out)
              |
              |Because of that, I have skewed numbers:
              |All "incoming" log records: 10187
              |All "status=sent" log records: 7506
              |All "status=deferred" log records: 3302
              |rejected: -621 (-6%)
              |
              |It is not a simple math of "A minus B minus C" to find out how much
              |postscreen is rejecting in its current state.
              |
              |Brian

              =============

              Yup.

              When there are a lot of deferrals, then things get complicated,
              requiring one to following individual messages through the process to
              eliminate multiple deferrals, etc., e.g., a single message "incoming"
              can get deferred many times leading to the numbers you cite.

              I wanted to keep things simple, so I made some compromises on the
              accuracy. I've been flipping back and forth between keeping the count
              of deferrals in there, or taking it out. Perhaps I should report the
              deferrals, but not count them in the expression.... hmmm...
            • Brian Evans
              ... Furthermore, the script assumes that connect to sent ratio is (1:1). This is almost never the case with multi-recipient mail or clients that can send more
              Message 6 of 12 , Jan 29, 2013
              • 0 Attachment
                On 1/29/2013 1:43 PM, Brian Evans wrote:
                > Because of that, I have skewed numbers:
                > All "incoming" log records: 10187
                > All "status=sent" log records: 7506
                > All "status=deferred" log records: 3302
                > rejected: -621 (-6%)
                >
                > It is not a simple math of "A minus B minus C" to find out how much
                > postscreen is rejecting in its current state.

                Furthermore, the script assumes that connect to sent ratio is (1:1).
                This is almost never the case with multi-recipient mail or clients that
                can send more than one message in a single transaction.

                Brian
              • Mike.
                ... that ... ============= Version 1.1, now uploaded to http://archive.mgm51.com/sources/pslogscan.html has removed the deferrals from the rejected
                Message 7 of 12 , Jan 29, 2013
                • 0 Attachment
                  On 1/29/2013 at 2:01 PM Brian Evans wrote:

                  |On 1/29/2013 1:43 PM, Brian Evans wrote:
                  |> Because of that, I have skewed numbers:
                  |> All "incoming" log records: 10187
                  |> All "status=sent" log records: 7506
                  |> All "status=deferred" log records: 3302
                  |> rejected: -621 (-6%)
                  |>
                  |> It is not a simple math of "A minus B minus C" to find out how much
                  |> postscreen is rejecting in its current state.
                  |
                  |Furthermore, the script assumes that connect to sent ratio is (1:1).
                  |This is almost never the case with multi-recipient mail or clients
                  that
                  |can send more than one message in a single transaction.
                  |
                  |Brian

                  =============

                  Version 1.1, now uploaded to

                  http://archive.mgm51.com/sources/pslogscan.html

                  has removed the deferrals from the rejected calculation.


                  Multi-recipients handling would involve some very detailed processing,
                  which is beyond the stated goal of this script.

                  I use the script to watch day-to-day trends, not for detailed analysis.
                  In that capacity, it works fine for me. YMMV

                  Thanks again for your feedback.
                • lconrad@...
                  ... I suggest you simplify and use only postscreen log lines. sent and deferred are not postscreen actions. and sent double counts when postfix sends to
                  Message 8 of 12 , Jan 29, 2013
                  • 0 Attachment
                     
                     
                    On Tuesday 29/01/2013 at 1:37 pm, Mike. wrote:


                    On 1/29/2013 at 2:01 PM Brian Evans wrote:

                    |On 1/29/2013 1:43 PM, Brian Evans wrote:
                    |> Because of that, I have skewed numbers:
                    |> All "incoming" log records: 10187
                    |> All "status=sent" log records: 7506
                    |> All "status=deferred" log records: 3302
                    |> rejected: -621 (-6%)
                    |>
                    |> It is not a simple math of "A minus B minus C" to find out how much
                    |> postscreen is rejecting in its current state.
                    |
                    |Furthermore, the script assumes that connect to sent ratio is (1:1).
                    |This is almost never the case with multi-recipient mail or clients
                    that
                    |can send more than one message in a single transaction.
                    |
                    |Brian

                    =============

                    Version 1.1, now uploaded to

                    http://archive.mgm51.com/sources/pslogscan.html

                    has removed the deferrals from the rejected calculation.


                    Multi-recipients handling would involve some very detailed processing,
                    which is beyond the stated goal of this script.

                    I use the script to watch day-to-day trends, not for detailed analysis.
                       In that capacity, it works fine for me. YMMV

                    Thanks again for your feedback.



                    I suggest you simplify and use only postscreen log lines.

                    "sent" and "deferred" are not postscreen actions.  

                    and "sent" double counts when postfix sends to content filter  AND sends to next hop, in a relay-only gateway.

                    "incoming" should be "SMTP connections"

                    you should automatically detect RBL servers rather than looking for defined, eg sorbs, RBL server, which I don't use

                    awk '/dnsblog/{print $11}' /var/log/maillog | sort -f | uniq -ic
                    290700 b.barracudacentral.org
                    209424 zen.spamhaus.org

                    good work

                    I think I'll write my own in python  :)

                    Len

                     


                  • Mike.
                    ... ============= Yes, after pondering the helpful pointers that Brian gave me, I have started to think about using only the Postscreen log lines, that way I
                    Message 9 of 12 , Jan 29, 2013
                    • 0 Attachment
                      On 1/29/2013 at 2:06 PM lconrad@... wrote:

                      |On Tuesday 29/01/2013 at 1:37 pm, Mike. wrote:
                      |>
                      |I suggest you simplify and use only postscreen log lines.
                      |
                      |"sent" and "deferred" are not postscreen actions.
                      |
                      |and "sent" double counts when postfix sends to content filter AND
                      |sends to next hop, in a relay-only gateway.
                      |
                      |"incoming" should be "SMTP connections"
                      |
                      |you should automatically detect RBL servers rather than looking for
                      |defined, eg sorbs, RBL server, which I don't use
                      |
                      |awk '/dnsblog/{print $11}' /var/log/maillog | sort -f | uniq -ic
                      |290700 b.barracudacentral.org
                      |209424 zen.spamhaus.org
                      |
                      |
                      |good work
                      |
                      |
                      |I think I'll write my own in python :)
                      |
                      |
                      |Len
                      =============

                      Yes, after pondering the helpful pointers that Brian gave me, I have
                      started to think about using only the Postscreen log lines, that way I
                      can avoid the multiplication of messages due to multi-recipient
                      messages and other messes, such as the double count you note. I backed
                      myself into a corner when I tried to track the flow of messages without
                      tracking the details thereof.

                      I'll leave the auto-detect to those who are more adventurous in that
                      area than I. :)

                      "incoming" currently also includes "pickup". But that may be removed
                      when I go to postscreen-only log messages.


                      If I sparked an idea for someone else, all the better.

                      Thanks for the comment.
                    • Eliezer Croitoru
                      ... Thanks Mike. The concept is really good but I must say it s a script for very small logs but in a system that the logs are in sizes of more then 100MB I
                      Message 10 of 12 , Jan 29, 2013
                      • 0 Attachment
                        On 1/29/2013 8:07 PM, Mike. wrote:
                        >
                        > I implemented the postscreen capability on a small MTA I run for
                        > friends and family. Once I got postscreen configuration producing the
                        > results I wanted, I soon tired of watching the detailed maillog to see
                        > how postscreen was operating. So I wrote a quick shell script to
                        > summarize the log file and give me an overview of how well postscreen
                        > is working.
                        >
                        > I offer the script to anyone who would like to use it. One company I
                        > worked for would not allow open source software into the company unless
                        > there was an explicit license on the software, so I put the BSD license
                        > on the script.
                        >
                        > You can download the script from here:
                        > http://archive.mgm51.com/sources/pslogscan.html

                        Thanks Mike.

                        The concept is really good but I must say it's a script for very small
                        logs but in a system that the logs are in sizes of more then 100MB I
                        assume your script will be very slow.

                        How are you in other scripting languages?
                        I have been working with Ruby\Perl\Python\Bash and for me Ruby is the
                        most intuitive and seems like capable of doing this task easily.

                        Regards,
                        --
                        Eliezer
                      • Mike.
                        ... the ... see ... postscreen ... I ... unless ... license ... ============= I ve tried it on logs up to 40MB, and it ran to completion in around five
                        Message 11 of 12 , Jan 30, 2013
                        • 0 Attachment
                          On 1/30/2013 at 3:55 AM Eliezer Croitoru wrote:

                          |On 1/29/2013 8:07 PM, Mike. wrote:
                          |>
                          |> I implemented the postscreen capability on a small MTA I run for
                          |> friends and family. Once I got postscreen configuration producing
                          the
                          |> results I wanted, I soon tired of watching the detailed maillog to
                          see
                          |> how postscreen was operating. So I wrote a quick shell script to
                          |> summarize the log file and give me an overview of how well
                          postscreen
                          |> is working.
                          |>
                          |> I offer the script to anyone who would like to use it. One company
                          I
                          |> worked for would not allow open source software into the company
                          unless
                          |> there was an explicit license on the software, so I put the BSD
                          license
                          |> on the script.
                          |>
                          |> You can download the script from here:
                          |> http://archive.mgm51.com/sources/pslogscan.html
                          |
                          |Thanks Mike.
                          |
                          |The concept is really good but I must say it's a script for very small

                          |logs but in a system that the logs are in sizes of more then 100MB I
                          |assume your script will be very slow.
                          |
                          |How are you in other scripting languages?
                          |I have been working with Ruby\Perl\Python\Bash and for me Ruby is the
                          |most intuitive and seems like capable of doing this task easily.
                          |
                          |Regards,
                          |--
                          |Eliezer

                          =============


                          I've tried it on logs up to 40MB, and it ran to completion in around
                          five seconds. However, for that test, I copied the log file off the
                          production mail server and on to a lightly loaded box.
                        • Eliezer Croitoru
                          ... It s a pretty decent speed. I have wrote a script to analyze squid apache logs before and it s more complicated then just match a line to a string. grep in
                          Message 12 of 12 , Jan 30, 2013
                          • 0 Attachment
                            On 1/30/2013 4:32 PM, Mike. wrote:
                            > =============
                            >
                            >
                            > I've tried it on logs up to 40MB, and it ran to completion in around
                            > five seconds. However, for that test, I copied the log file off the
                            > production mail server and on to a lightly loaded box.

                            It's a pretty decent speed.
                            I have wrote a script to analyze squid\apache logs before and it's more
                            complicated then just match a line to a string.

                            grep in general is faster for exact matches in most cases I have seen
                            yet and it's amazing.

                            The same lookup on any other scripting lang will take *3-4 or more.

                            --
                            Eliezer Croitoru
                          Your message has been successfully submitted and would be delivered to recipients shortly.