Loading ...
Sorry, an error occurred while loading the content.

Realtime log reporting when postfix delivers mails

Expand Messages
  • Ram
    I have a postfix server sending out mails and we are creating reports by parsing the maillogs using a couple of perl cron scripts (linux machine with mysql )
    Message 1 of 6 , Mar 14, 2013
    • 0 Attachment
      I have a postfix server sending out mails and we are creating reports by
      parsing the maillogs using a couple of perl cron scripts
      (linux machine with mysql )

      Now the requirement is of realtime reporting.
      I tried using rsyslog with a mysql table. But the performance is far too
      bad. Rsyslog seems to have some memory leak and it brings down the machine.

      I guess realtime logging should be a very common requirement. What is
      the best way for this


      Thanks
      Ram
    • Stan Hoeppner
      ... Actually it s not. ... Postfix logs to the UNIX syslog facility. What you do with it from there is outside the scope of Postfix. What you should probably
      Message 2 of 6 , Mar 15, 2013
      • 0 Attachment
        On 3/15/2013 1:59 AM, Ram wrote:
        > I have a postfix server sending out mails and we are creating reports by
        > parsing the maillogs using a couple of perl cron scripts
        > (linux machine with mysql )
        >
        > Now the requirement is of realtime reporting.
        >
        > I tried using rsyslog with a mysql table. But the performance is far too
        > bad. Rsyslog seems to have some memory leak and it brings down the machine.
        >
        > I guess realtime logging should be a very common requirement.

        Actually it's not.

        > What is
        > the best way for this

        Postfix logs to the UNIX syslog facility. What you do with it from
        there is outside the scope of Postfix.

        What you should probably be looking for is a generic log watching daemon
        that can capture appends on the fly.

        --
        Stan
      • Robert Schetterer
        ... real time reporting is easy done with over ssh and tail -f /var/log/mail.log without ssh you may use webmin over https, then create some unprivileged
        Message 3 of 6 , Mar 15, 2013
        • 0 Attachment
          Am 15.03.2013 07:59, schrieb Ram:
          > I have a postfix server sending out mails and we are creating reports by
          > parsing the maillogs using a couple of perl cron scripts
          > (linux machine with mysql )
          >
          > Now the requirement is of realtime reporting.
          > I tried using rsyslog with a mysql table. But the performance is far too
          > bad. Rsyslog seems to have some memory leak and it brings down the machine.
          >
          > I guess realtime logging should be a very common requirement. What is
          > the best way for this
          >
          >
          > Thanks
          > Ram
          >
          >
          >
          >
          >

          real time reporting is easy done with over ssh and tail -f /var/log/mail.log

          without ssh you may use webmin over https, then create some unprivileged
          webmin user , do local login in the webmin gui, give the user enough
          permission to read tail -f /var/log/mail.log

          you may also use webmins inbuild read syslog log stuff,
          or write some own scripts for grep/tail etc

          there are some other solutions with http guis
          but as i dont like them after testing, but it would be nice to here some
          more

          xymon has limited stuff to read log files, but it may enough to create
          alarms to special events in mail log, guess nagios etc have equal

          you may also try some other syslog compatible servers with guis
          for real time search and display







          Best Regards
          MfG Robert Schetterer

          --
          [*] sys4 AG

          http://sys4.de, +49 (89) 30 90 46 64
          Franziskanerstraße 15, 81669 München

          Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
          Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer
          Aufsichtsratsvorsitzender: Joerg Heidrich
        • Abhijeet Rastogi
          Have a look into logstash project. I have just started using it for mail logs and it s awesome. ... -- Regards, Abhijeet Rastogi (shadyabhi)
          Message 4 of 6 , Mar 16, 2013
          • 0 Attachment
            Have a look into logstash project. I have just started using it for mail logs and it's awesome.

            On Fri, Mar 15, 2013 at 12:29 PM, Ram <ram@...> wrote:
            I have a postfix server sending out mails and we are creating reports by parsing the maillogs using a couple of perl cron scripts
            (linux machine with mysql )

            Now the requirement is of realtime reporting.
            I tried using rsyslog with a mysql table. But the performance is far too bad. Rsyslog seems to have some memory leak and it brings down the machine.

            I guess realtime logging should be a very common requirement. What is the best way for this


            Thanks
            Ram








            --
            Regards,
            Abhijeet Rastogi (shadyabhi)
            https://plus.google.com/107316377741966576356/
          • Reinaldo Gil Lima de Carvalho
            We need a structured log to avoid parsing. I talk with Wietse in the year 2011 at FISL conference (Porto Alegre/Brasil). The second problem is load this data
            Message 5 of 6 , Mar 16, 2013
            • 0 Attachment
              We need a structured log to avoid parsing. I talk with Wietse in the year 2011 at FISL conference (Porto Alegre/Brasil).

              The second problem is load this data to a database. Rsyslog put the data in a single column, and use full text search is inevitable.

              While don't have a better solution, I wrote a daemon to parse and insert the data on a database. I will search this code and put on github on next week.

              []'s

              Reinaldo Gil Lima de Carvalho

              Em 15/03/2013, às 03:59, Ram <ram@...> escreveu:

              > I have a postfix server sending out mails and we are creating reports by parsing the maillogs using a couple of perl cron scripts
              > (linux machine with mysql )
              >
              > Now the requirement is of realtime reporting.
              > I tried using rsyslog with a mysql table. But the performance is far too bad. Rsyslog seems to have some memory leak and it brings down the machine.
              >
              > I guess realtime logging should be a very common requirement. What is the best way for this
              >
              >
              > Thanks
              > Ram
              >
              >
              >
              >
              >
            • Florian Schaal
              ... I use something similar for apache with syslog-ng. It can adapted for each message-string: http://blog.schaal-24.de/?p=769&lang=en Using syslog-ng you can
              Message 6 of 6 , Mar 19, 2013
              • 0 Attachment
                Am 16.03.2013 22:11, schrieb Reinaldo Gil Lima de Carvalho:
                > We need a structured log to avoid parsing. I talk with Wietse in the year 2011 at FISL conference (Porto Alegre/Brasil).
                >
                > The second problem is load this data to a database. Rsyslog put the data in a single column, and use full text search is inevitable.
                >
                > While don't have a better solution, I wrote a daemon to parse and insert the data on a database. I will search this code and put on github on next week.
                >
                > []'s
                >

                I use something similar for apache with syslog-ng. It can adapted for
                each message-string: http://blog.schaal-24.de/?p=769&lang=en

                Using syslog-ng you can easily split each log-message in different parts
                and than store each part of the message in a different column. Maybe
                this is possible with ryslog, too.

                regards
                Florian
              Your message has been successfully submitted and would be delivered to recipients shortly.