Loading ...
Sorry, an error occurred while loading the content.

Log file checking

Expand Messages
  • Mark Scholten
    Hello, I am looking for a solution to get the following information from postfix: - Ignore connections from 127.0.0.1 or process only connections from
    Message 1 of 9 , Jul 31 5:15 AM
    • 0 Attachment
      Hello,

      I am looking for a solution to get the following information from postfix:
      - Ignore connections from 127.0.0.1 or process only connections from
      127.0.0.1 (with another flag/option set)
      - What is done with the connection (mail accepted/mail rejected (if rejected
      what was the reason, for example helo check failed/greylisted/recipient
      doesn't exist/blacklisted))

      Or if the above isn't available something that outputs the following to a
      file or the commandline (so I can grep at it and use wc -l):
      - Create a single line with information about a message
      (time/sender/recipient/helo/sending server/action (including error
      code/error information if available)

      I did check and didn't find it here (or I didn't look good enough)
      http://www.postfix.org/addon.html#logfile

      Is something like that available for postfix or should I create something
      for it?

      With kind regards,

      Mark Scholten
    • Jeroen Geilman
      ... I have no idea what you re talking about - do you want to make postfix DO something, or do you want to analyze log files ? In the former case, you can
      Message 2 of 9 , Jul 31 6:15 AM
      • 0 Attachment
        On 07/31/2010 02:15 PM, Mark Scholten wrote:
        > Hello,
        >
        > I am looking for a solution to get the following information from postfix:
        > - Ignore connections from 127.0.0.1 or process only connections from
        > 127.0.0.1 (with another flag/option set)
        > - What is done with the connection (mail accepted/mail rejected (if rejected
        > what was the reason, for example helo check failed/greylisted/recipient
        > doesn't exist/blacklisted))
        >
        > Or if the above isn't available something that outputs the following to a
        > file or the commandline (so I can grep at it and use wc -l):
        > - Create a single line with information about a message
        > (time/sender/recipient/helo/sending server/action (including error
        > code/error information if available)
        >
        > I did check and didn't find it here (or I didn't look good enough)
        > http://www.postfix.org/addon.html#logfile
        >
        > Is something like that available for postfix or should I create something
        > for it?
        >


        I have no idea what you're talking about - do you want to make postfix
        DO something, or do you want to analyze log files ?

        In the former case, you can restrict pretty much anything.

        In the latter case, there are no limitations, since they're presumably
        your log files - go wild.

        J.
      • Mark Scholten
        ... I am looking at analyzing the log files. I want to get certain information from the log files and I want to know if there is something available to limit
        Message 3 of 9 , Jul 31 9:00 AM
        • 0 Attachment
          > -----Original Message-----
          > From: owner-postfix-users@... [mailto:owner-postfix-
          > users@...] On Behalf Of Jeroen Geilman
          > Sent: Saturday, July 31, 2010 3:16 PM
          > To: postfix-users@...
          > Subject: Re: Log file checking
          >
          > On 07/31/2010 02:15 PM, Mark Scholten wrote:
          > > Hello,
          > >
          > > I am looking for a solution to get the following information from
          > postfix:
          > > - Ignore connections from 127.0.0.1 or process only connections from
          > > 127.0.0.1 (with another flag/option set)
          > > - What is done with the connection (mail accepted/mail rejected (if
          > rejected
          > > what was the reason, for example helo check
          > failed/greylisted/recipient
          > > doesn't exist/blacklisted))
          > >
          > > Or if the above isn't available something that outputs the following
          > to a
          > > file or the commandline (so I can grep at it and use wc -l):
          > > - Create a single line with information about a message
          > > (time/sender/recipient/helo/sending server/action (including error
          > > code/error information if available)
          > >
          > > I did check and didn't find it here (or I didn't look good enough)
          > > http://www.postfix.org/addon.html#logfile
          > >
          > > Is something like that available for postfix or should I create
          > something
          > > for it?
          > >
          >
          >
          > I have no idea what you're talking about - do you want to make postfix
          > DO something, or do you want to analyze log files ?
          >
          > In the former case, you can restrict pretty much anything.
          >
          > In the latter case, there are no limitations, since they're presumably
          > your log files - go wild.

          I am looking at analyzing the log files. I want to get certain information
          from the log files and I want to know if there is something available to
          limit it to a single line per email (as that is easier to process and to
          find the last action).

          Any ideas if there are ready to use scripts for this part?

          Regards, Mark

          >
          > J.
        • Stan Hoeppner
          ... If you give us your exact requirement, instead of the vague I want to get certain information , one of us might be able to hack up a simple shell script,
          Message 4 of 9 , Jul 31 3:25 PM
          • 0 Attachment
            Mark Scholten put forth on 7/31/2010 11:00 AM:

            > Any ideas if there are ready to use scripts for this part?

            If you give us your exact requirement, instead of the vague "I want to get
            certain information", one of us might be able to hack up a simple shell
            script, or even a single bash line, to do what you want. Keep in mind
            however, that you're probably not going to get "everything" on a single line.
            If you do it won't be legible.

            In the mean time, take a look at pflogsumm, a simple log summary generator for
            Postfix:
            http://jimsun.linxnet.com/postfix_contrib.html

            Debian distros have a pflogsumm package, other distros may as well.

            --
            Stan
          • Mark Scholten
            ... To be as clear as possible: I want the following information (per day or per hour, it should be possible to exclude email addresses or to only get
            Message 5 of 9 , Jul 31 4:53 PM
            • 0 Attachment
              > -----Original Message-----
              > From: owner-postfix-users@... [mailto:owner-postfix-
              > users@...] On Behalf Of Stan Hoeppner
              > Sent: Sunday, August 01, 2010 12:26 AM
              > To: postfix-users@...
              > Subject: Re: Log file checking
              >
              > Mark Scholten put forth on 7/31/2010 11:00 AM:
              >
              > > Any ideas if there are ready to use scripts for this part?
              >
              > If you give us your exact requirement, instead of the vague "I want to
              > get
              > certain information", one of us might be able to hack up a simple shell
              > script, or even a single bash line, to do what you want. Keep in mind
              > however, that you're probably not going to get "everything" on a single
              > line.
              > If you do it won't be legible.

              To be as clear as possible:

              I want the following information (per day or per hour, it should be possible
              to exclude email addresses or to only get information for certain email
              addresses):
              - Number of email attempts made by other systems
              - Number of messages blocked based on the HELO requirements (I have a few
              regexp lines with blocked HELOs (botnets/spammers))
              - Number of connections greylisted (we use postgrey)
              - Number of attempts for an invalid recipient
              - Number of messages blocked based on blacklists
              - Number of messages blocked by content filter (not really important)
              - Number of messages accepted (not blocked at any stage)

              I now have a few commands that I use to get something like this (however
              based on the actual numbers I think something is wrong).

              Currently used commands:
              cat /var/log/mail.log | grep -v
              "double-bounce@..." | grep -v 127.0.0.1 | grep
              "Jul 31" | grep "Helo command rejected" | wc -l
              cat /var/log/mail.log | grep -v
              "double-bounce@..." | grep -v 127.0.0.1 | grep
              "Jul 31" | grep -v "Helo command rejected" | grep "action=greylist" | wc -l
              cat /var/log/mail.log | grep -v
              "double-bounce@..." | grep -v 127.0.0.1 | grep
              "Jul 31" | grep -v "Helo command rejected" | grep -v "action=greylist" |
              grep 550 | grep -i "recipient address rejected" | wc -l
              cat /var/log/mail.log | grep -v
              "double-bounce@..." | grep -v 127.0.0.1 | grep
              "Jul 31" | grep -v "Helo command rejected" | grep -v "action=greylist" |
              grep -vi "recipient address rejected" | grep 550 | grep -i "Your MTA is
              listed in too many DNSBLs" | wc -l
              cat /var/log/mail.log | grep -v
              "double-bounce@..." | grep "Jul 31" | grep
              "relay=127.0.0.1\[127.0.0.1\]\:10024" | grep -v SPAM | wc -l
              cat /var/log/mail.log | grep -v
              "double-bounce@..." | grep "Jul 31" | grep
              "relay=127.0.0.1\[127.0.0.1\]\:10024" | grep SPAM | wc -l

              There is probably a better/faster way to get this information I guess. These
              lines probably have a few mistakes in it (at least I guess they have).

              >
              > In the mean time, take a look at pflogsumm, a simple log summary
              > generator for
              > Postfix:
              > http://jimsun.linxnet.com/postfix_contrib.html

              I did check pflogsumm, however most information isn't provided by pflogsumm
              (same for awstats). At least not with the package debian provides.
              >
              > Debian distros have a pflogsumm package, other distros may as well.
              >
              > --
              > Stan
              --
              Mark
            • Sahil Tandon
              ... I use postfix-logwatch. See: http://logreporters.sourceforge.net/ If it doesn t meet your exact needs, then hack it to do so. If your changes would
              Message 6 of 9 , Jul 31 5:32 PM
              • 0 Attachment
                On Sun, 2010-08-01 at 01:53:42 +0200, Mark Scholten wrote:

                > I want the following information (per day or per hour, it should be possible
                > to exclude email addresses or to only get information for certain email
                > addresses):
                > - Number of email attempts made by other systems
                > - Number of messages blocked based on the HELO requirements (I have a few
                > regexp lines with blocked HELOs (botnets/spammers))
                > - Number of connections greylisted (we use postgrey)
                > - Number of attempts for an invalid recipient
                > - Number of messages blocked based on blacklists
                > - Number of messages blocked by content filter (not really important)
                > - Number of messages accepted (not blocked at any stage)

                I use postfix-logwatch. See: http://logreporters.sourceforge.net/

                If it doesn't meet your exact needs, then hack it to do so. If your
                changes would benefit a wider audience, then share them with Mike. And
                unless you have a Postfix problem, we are veering off-topic.

                --
                Sahil Tandon <sahil@...>
              • Stan Hoeppner
                ... /usr/sbin/pflogsumm.pl --smtpd_stats /var/log/mail.log /var/log/mail.log.1 Grand Totals ... messages 3658 received 5323 delivered 0 forwarded 480
                Message 7 of 9 , Jul 31 6:49 PM
                • 0 Attachment
                  Mark Scholten put forth on 7/31/2010 6:53 PM:

                  > I want the following information (per day or per hour, it should be possible
                  > to exclude email addresses or to only get information for certain email
                  > addresses):

                  /usr/sbin/pflogsumm.pl --smtpd_stats /var/log/mail.log /var/log/mail.log.1

                  Grand Totals
                  ------------
                  messages

                  3658 received
                  5323 delivered
                  0 forwarded
                  480 deferred (2631 deferrals)
                  1 bounced
                  1740 rejected (24%)
                  0 reject warnings
                  0 held
                  0 discarded (0%)

                  25387k bytes received
                  49655k bytes delivered
                  825 senders
                  728 sending hosts/domains
                  19 recipients
                  18 recipient hosts/domains

                  > - Number of email attempts made by other systems

                  smtpd

                  5304 connections
                  1399 hosts/domains
                  10 avg. connect time (seconds)
                  14:54:24 total connect time


                  > - Number of messages blocked based on the HELO requirements (I have a few
                  > regexp lines with blocked HELOs (botnets/spammers))

                  If these are done with something like "check_helo_access
                  regexp:/etc/postfix/helo.regexp" then you'd see something like this, but with
                  "Helo command rejected: ". I don't do any custom HELO checks, only client
                  checks, but the output is otherwise the same in pflogsumm.

                  Client host rejected: Dynamic - Please relay via ISP (chello.nl) (total: 1)
                  1 dhcp-077-248-074-059.chello.nl
                  Client host rejected: Dynamic - Please relay via ISP (embarqhsd.net)
                  (total: 1)
                  1 embarqhsd.net
                  Client host rejected: Dynamic - Please relay via ISP (eunet.rs) (total: 1)
                  1 dynamic-78-30-138-239.adsl.eunet.rs

                  ** I have separate rejection messages for each expression in my regexp table.
                  Pflogsumm counts each one as distinct, and gives a total for each one,
                  instead of a total for all "custom HELO checks" If you want a singular total
                  for yours, you probably don't want to specify rejection text for each, but use
                  the Postfix default. Doing so should give you the total you want.

                  > - Number of connections greylisted (we use postgrey)

                  Recipient address rejected: Greylisted (total: 30)
                  30 stan@...

                  ** greylisting here is used as a last ditch bot blocker. Some call this "very
                  selective greylisting".

                  > - Number of attempts for an invalid recipient

                  Recipient address rejected: User unknown in local recipient table (total: 24)
                  21 4050505@...
                  1 4C4F0705.2050005@...
                  1 4c4f17db.7010101@...
                  1 4c20361c.7090309@...

                  > - Number of messages blocked based on blacklists

                  message reject detail
                  ---------------------
                  RCPT
                  Client host rejected: Access denied (total: 262)
                  22 annaeyes.com
                  ...
                  Client host rejected: Email not accepted from Africa (total: 34)
                  3 41.140.254.160
                  ...
                  Client host rejected: Mail not accepted from Belarus (total: 4)
                  3 93.85.201.97
                  ...
                  Client host rejected: Mail not accepted from China (total: 23)
                  6 60.190.77.242
                  ...
                  Client host rejected: Mail not accepted from Hungary (total: 1)
                  1 www.imac.hu
                  Client host rejected: Mail not accepted from Indonesia (total: 14)
                  6 118.96.252.201
                  ...
                  Client host rejected: Mail not accepted from Korea (total: 32)
                  3 61.105.220.135
                  ...
                  Client host rejected: Mail not accepted from Malaysia (total: 1)
                  1 110.74.129.155
                  ...
                  Client host rejected: Mail not accepted from Romania (total: 10)
                  3 81.181.221.62
                  ...
                  Client host rejected: Mail not accepted from Russia (total: 34)
                  3 77.34.255.9
                  ...
                  Client host rejected: Mail not accepted from Thailand (total: 6)
                  3 113.53.213.186
                  ...
                  Client host rejected: Mail not accepted from Ukraine (total: 11)
                  3 79.135.202.145

                  > - Number of messages blocked by content filter (not really important)

                  Here neither. I don't use content filters. If you saw my entire A/S Postfix
                  config and my user base you'd understand why.

                  > - Number of messages accepted (not blocked at any stage)

                  This is a gripe of my own. Once you get an accurate method for counting this
                  via the mail log, please share it with the pflogsumm dev. My guess is that
                  it's not at all straightforward, due to the multiple delivery methods available.

                  > I did check pflogsumm, however most information isn't provided by pflogsumm
                  > (same for awstats). At least not with the package debian provides.

                  All of the above snippets are from Version: 1.1.0-3 (Lenny)

                  It appears pflogsumm meets all of your requirements but one. Maybe not in the
                  exact mode of operation you'd like, but this is open source code. Change it
                  as you see fit to meet your needs. Just share your patches. :)

                  --
                  Stan
                • Mark Scholten
                  ... Getting it in a single number is important for me, however looking at the http://logreporters.sourceforge.net/ link you did give I see that all but one
                  Message 8 of 9 , Aug 1, 2010
                  • 0 Attachment
                    > -----Original Message-----
                    > From: owner-postfix-users@... [mailto:owner-postfix-
                    > users@...] On Behalf Of Stan Hoeppner
                    > Sent: Sunday, August 01, 2010 3:50 AM
                    > To: postfix-users@...
                    > Subject: Re: Log file checking
                    >
                    > Mark Scholten put forth on 7/31/2010 6:53 PM:
                    >
                    > > I want the following information (per day or per hour, it should be
                    > possible
                    > > to exclude email addresses or to only get information for certain
                    > email
                    > > addresses):
                    >
                    > /usr/sbin/pflogsumm.pl --smtpd_stats /var/log/mail.log
                    > /var/log/mail.log.1
                    >
                    > Grand Totals
                    > ------------
                    > messages
                    >
                    > 3658 received
                    > 5323 delivered
                    > 0 forwarded
                    > 480 deferred (2631 deferrals)
                    > 1 bounced
                    > 1740 rejected (24%)
                    > 0 reject warnings
                    > 0 held
                    > 0 discarded (0%)
                    >
                    > 25387k bytes received
                    > 49655k bytes delivered
                    > 825 senders
                    > 728 sending hosts/domains
                    > 19 recipients
                    > 18 recipient hosts/domains
                    >
                    > > - Number of email attempts made by other systems
                    >
                    > smtpd
                    >
                    > 5304 connections
                    > 1399 hosts/domains
                    > 10 avg. connect time (seconds)
                    > 14:54:24 total connect time
                    >
                    >
                    > > - Number of messages blocked based on the HELO requirements (I have a
                    > few
                    > > regexp lines with blocked HELOs (botnets/spammers))
                    >
                    > If these are done with something like "check_helo_access
                    > regexp:/etc/postfix/helo.regexp" then you'd see something like this,
                    > but with
                    > "Helo command rejected: ". I don't do any custom HELO checks, only
                    > client
                    > checks, but the output is otherwise the same in pflogsumm.
                    >
                    > Client host rejected: Dynamic - Please relay via ISP (chello.nl)
                    > (total: 1)
                    > 1 dhcp-077-248-074-059.chello.nl
                    > Client host rejected: Dynamic - Please relay via ISP
                    > (embarqhsd.net)
                    > (total: 1)
                    > 1 embarqhsd.net
                    > Client host rejected: Dynamic - Please relay via ISP (eunet.rs)
                    > (total: 1)
                    > 1 dynamic-78-30-138-239.adsl.eunet.rs
                    >
                    > ** I have separate rejection messages for each expression in my regexp
                    > table.
                    > Pflogsumm counts each one as distinct, and gives a total for each one,
                    > instead of a total for all "custom HELO checks" If you want a singular
                    > total
                    > for yours, you probably don't want to specify rejection text for each,
                    > but use
                    > the Postfix default. Doing so should give you the total you want.
                    >
                    > > - Number of connections greylisted (we use postgrey)
                    >
                    > Recipient address rejected: Greylisted (total: 30)
                    > 30 stan@...
                    >
                    > ** greylisting here is used as a last ditch bot blocker. Some call
                    > this "very
                    > selective greylisting".
                    >
                    > > - Number of attempts for an invalid recipient
                    >
                    > Recipient address rejected: User unknown in local recipient table
                    > (total: 24)
                    > 21 4050505@...
                    > 1 4C4F0705.2050005@...
                    > 1 4c4f17db.7010101@...
                    > 1 4c20361c.7090309@...
                    >
                    > > - Number of messages blocked based on blacklists
                    >
                    > message reject detail
                    > ---------------------
                    > RCPT
                    > Client host rejected: Access denied (total: 262)
                    > 22 annaeyes.com
                    > ...
                    > Client host rejected: Email not accepted from Africa (total: 34)
                    > 3 41.140.254.160
                    > ...
                    > Client host rejected: Mail not accepted from Belarus (total: 4)
                    > 3 93.85.201.97
                    > ...
                    > Client host rejected: Mail not accepted from China (total: 23)
                    > 6 60.190.77.242
                    > ...
                    > Client host rejected: Mail not accepted from Hungary (total: 1)
                    > 1 www.imac.hu
                    > Client host rejected: Mail not accepted from Indonesia (total: 14)
                    > 6 118.96.252.201
                    > ...
                    > Client host rejected: Mail not accepted from Korea (total: 32)
                    > 3 61.105.220.135
                    > ...
                    > Client host rejected: Mail not accepted from Malaysia (total: 1)
                    > 1 110.74.129.155
                    > ...
                    > Client host rejected: Mail not accepted from Romania (total: 10)
                    > 3 81.181.221.62
                    > ...
                    > Client host rejected: Mail not accepted from Russia (total: 34)
                    > 3 77.34.255.9
                    > ...
                    > Client host rejected: Mail not accepted from Thailand (total: 6)
                    > 3 113.53.213.186
                    > ...
                    > Client host rejected: Mail not accepted from Ukraine (total: 11)
                    > 3 79.135.202.145
                    >
                    > > - Number of messages blocked by content filter (not really important)
                    >
                    > Here neither. I don't use content filters. If you saw my entire A/S
                    > Postfix
                    > config and my user base you'd understand why.
                    >
                    > > - Number of messages accepted (not blocked at any stage)
                    >
                    > This is a gripe of my own. Once you get an accurate method for
                    > counting this
                    > via the mail log, please share it with the pflogsumm dev. My guess is
                    > that
                    > it's not at all straightforward, due to the multiple delivery methods
                    > available.
                    >
                    > > I did check pflogsumm, however most information isn't provided by
                    > pflogsumm
                    > > (same for awstats). At least not with the package debian provides.
                    >
                    > All of the above snippets are from Version: 1.1.0-3 (Lenny)
                    >
                    > It appears pflogsumm meets all of your requirements but one. Maybe not
                    > in the
                    > exact mode of operation you'd like, but this is open source code.
                    > Change it
                    > as you see fit to meet your needs. Just share your patches. :)

                    Getting it in a single number is important for me, however looking at the
                    http://logreporters.sourceforge.net/ link you did give I see that all but
                    one thing is given the way I want it. This last option isn't given the way I
                    like it, but that can be done by parsing the output from postfix-logwatch to
                    combine the last information. Thank you for giving the link.

                    Regards, Mark
                    >
                    > --
                    > Stan
                  • Stan Hoeppner
                    ... Actually Sahil deserves that credit--I didn t know of logwatch until he mentioned it. And just like you, I prefer the logwatch output over pflogsumm. I ve
                    Message 9 of 9 , Aug 1, 2010
                    • 0 Attachment
                      Mark Scholten put forth on 8/1/2010 5:46 AM:

                      > Getting it in a single number is important for me, however looking at the
                      > http://logreporters.sourceforge.net/ link you did give I see that all but
                      > one thing is given the way I want it. This last option isn't given the way I
                      > like it, but that can be done by parsing the output from postfix-logwatch to
                      > combine the last information. Thank you for giving the link.

                      Actually Sahil deserves that credit--I didn't know of logwatch until he
                      mentioned it. And just like you, I prefer the logwatch output over pflogsumm.
                      I've already converted everything over to it here.

                      Thanks again Sahil.

                      --
                      Stan
                    Your message has been successfully submitted and would be delivered to recipients shortly.