Loading ...
Sorry, an error occurred while loading the content.
 

New Postfix log analyzer tool, statistics, grapher, ... PostgreSQL DB 9.2.x based

Expand Messages
  • Nicolas HAHN
    Dear Postfix Community, I m writing for the first time there but working in the area of SMTP messaging since a long time. With Postfix, that I really love. The
    Message 1 of 7 , Apr 11, 2013
      Dear Postfix Community,

      I'm writing for the first time there but working in the area of SMTP messaging since a long time. With Postfix, that I really love.

      The goal of my e-mail today is just to let you know that I'm working since some time on my open source GPLv3 project dedicated to realtime postfix log analysis, but not only log analysis.

      This project need several components to work fine, primarily a Linux server running Apache, Php, Rsyslog, postgreSQL 9.2, SNMPD, ... A lot of things are described in the INSTALL file.

      If you are interested by such project, you can find it on Sourceforge there: https://sourceforge.net/projects/x-itools/files/X-Itools%20releases/E-mail%20Log%20Search%20Engine/

      The archive I just uploaded this morning deal with Postfix version
      >=2.8.x logs.

      This tool is used in the United Nations datacenters, for Messaging Services, where I'm currently working as a messaging architect consultant. Depending of the processing power of the server, it is able to work with a mail flow of 1 million e-mails a day in real time. The version deployed in the UN also process Exchange servers logs in real time. The version I've packaged on sourceforge is a little bit in late (I need time to commit all my code) and is able to process Postfix logs only as of today.
      Some (and me too :-)) say it is much more powerfull than what Postini from Google is offering, especially if we consider it is working in real time.

      Version available on sourceforge in the tar.gz archive is 0.9.10.
      Version starting to process Exchange Servers logs is 0.9.11. It is comming...

      Also, I kept my tool "secret" since 2004 despite the fact I decided to make it under GPL, using it for my own needs as a small provider myself. I decided to publish it on sourceforge in 2011, when UN shown a big interest in it, and then I restarted the development. It means the Wiki is empty, the doc is enclosed in my brain, ... and all of this needs to be publicly available. that will take time...

      So, if you're interested, I can answer questions and provide help. It can be quite complex to install because of the dependencies needed.
      This project also need, as you may know, volunteers and talents, people to debug, ... I'm not the most talented coder of the world of course :) PHP code I produce as a PHP newbie for example, could be greatly enhanced, secured, and so on. I'm learning PHP the same time I'm coding this tool and it's not easy as I'm an old school C/C++ coder.

      In brief, a continuous effort is needed as usual.

      NOTE: I've removed the attached screenshot because e-mail size is limited to  40000 characters.

      Thanks for your attention

      Best regards,
      Nicolas
    • Abhijeet Rastogi
      Any screenshots would be highly appreciated. I m currently using Elasticsearch to store all my logs and Kibana for search. I was wondering how this tool will
      Message 2 of 7 , Apr 12, 2013
        Any screenshots would be highly appreciated. I'm currently using Elasticsearch to store all my logs and Kibana for search. I was wondering how this tool will help more.


        On Thu, Apr 11, 2013 at 6:59 PM, Nicolas HAHN <hahnn@...> wrote:
        Dear Postfix Community,

        I'm writing for the first time there but working in the area of SMTP messaging since a long time. With Postfix, that I really love.

        The goal of my e-mail today is just to let you know that I'm working since some time on my open source GPLv3 project dedicated to realtime postfix log analysis, but not only log analysis.

        This project need several components to work fine, primarily a Linux server running Apache, Php, Rsyslog, postgreSQL 9.2, SNMPD, ... A lot of things are described in the INSTALL file.

        If you are interested by such project, you can find it on Sourceforge there: https://sourceforge.net/projects/x-itools/files/X-Itools%20releases/E-mail%20Log%20Search%20Engine/


        The archive I just uploaded this morning deal with Postfix version >=2.8.x logs.

        This tool is used in the United Nations datacenters, for Messaging Services, where I'm currently working as a messaging architect consultant. Depending of the processing power of the server, it is able to work with a mail flow of 1 million e-mails a day in real time. The version deployed in the UN also process Exchange servers logs in real time. The version I've packaged on sourceforge is a little bit in late (I need time to commit all my code) and is able to process Postfix logs only as of today.
        Some (and me too :-)) say it is much more powerfull than what Postini from Google is offering, especially if we consider it is working in real time.

        Version available on sourceforge in the tar.gz archive is 0.9.10.
        Version starting to process Exchange Servers logs is 0.9.11. It is comming...

        Also, I kept my tool "secret" since 2004 despite the fact I decided to make it under GPL, using it for my own needs as a small provider myself. I decided to publish it on sourceforge in 2011, when UN shown a big interest in it, and then I restarted the development. It means the Wiki is empty, the doc is enclosed in my brain, ... and all of this needs to be publicly available. that will take time...

        So, if you're interested, I can answer questions and provide help. It can be quite complex to install because of the dependencies needed.
        This project also need, as you may know, volunteers and talents, people to debug, ... I'm not the most talented coder of the world of course :) PHP code I produce as a PHP newbie for example, could be greatly enhanced, secured, and so on. I'm learning PHP the same time I'm coding this tool and it's not easy as I'm an old school C/C++ coder.

        In brief, a continuous effort is needed as usual.

        NOTE: I've removed the attached screenshot because e-mail size is limited to  40000 characters.

        Thanks for your attention

        Best regards,
        Nicolas



        --
        Regards,
        Abhijeet Rastogi (shadyabhi)
        http://blog.abhijeetr.com
      • LuKreme
        ... It requires Linux? Why would it require Linux? -- you d think you could trust a horde of hungarian barbarians
        Message 3 of 7 , Apr 13, 2013
          On Apr 11, 2013, at 7:29, Nicolas HAHN <hahnn@...> wrote:
          > a Linux server

          It requires Linux? Why would it require Linux?

          --
          you'd think you could trust a horde of hungarian barbarians
        • Nicolas HAHN
          Mmhhh... What can I say??? If somebody can make the full product working on another platform, then I would be happy to propose the corresponding archive on
          Message 4 of 7 , Apr 13, 2013
            Mmhhh... What can I say???

            If somebody can make the full product working on another platform, then
            I would be happy to propose the corresponding archive on Sourceforge
            available files project page...

            But for now the best answer is simply "it's like that". ;)


            Le 13/04/2013 11:31, LuKreme a écrit :
            > On Apr 11, 2013, at 7:29, Nicolas HAHN <hahnn@...> wrote:
            >> a Linux server
            > It requires Linux? Why would it require Linux?
            >
          • /dev/rob0
            [ Top-posting fixed ] ... I think the point is that none of the software you mention are Linux-specific. Postfix, PostgreSQL, rsyslog, apache (Apache httpd),
            Message 5 of 7 , Apr 13, 2013
              [ Top-posting fixed ]
              > Le 13/04/2013 11:31, LuKreme a écrit :
              > >On Apr 11, 2013, at 7:29, Nicolas HAHN <hahnn@...> wrote:
              > >>a Linux server
              > >It requires Linux? Why would it require Linux?

              On Sat, Apr 13, 2013 at 11:38:07AM +0200, Nicolas HAHN wrote:
              > Mmhhh... What can I say???
              >
              > If somebody can make the full product working on another
              > platform, then I would be happy to propose the corresponding
              > archive on Sourceforge available files project page...
              >
              > But for now the best answer is simply "it's like that". ;)

              I think the point is that none of the software you mention are
              Linux-specific. Postfix, PostgreSQL, rsyslog, "apache" (Apache
              httpd), and php all work and are commonly seen on other Unix and
              Unix-like systems. It doesn't sound likely that you have done
              something to restrict this to Linux-only.
              --
              http://rob0.nodns4.us/ -- system administration and consulting
              Offlist GMX mail is seen only if "/dev/rob0" is in the Subject:
            • Larry Stone
              ... My first thought was he thinks Linux and Unix are just different words for the same thing. Or he knows Linux and has never heard of Unix. Wouldn t be the
              Message 6 of 7 , Apr 13, 2013
                On Apr 13, 2013, at 8:17 AM, /dev/rob0 <rob0@...> wrote:

                >
                > I think the point is that none of the software you mention are
                > Linux-specific. Postfix, PostgreSQL, rsyslog, "apache" (Apache
                > httpd), and php all work and are commonly seen on other Unix and
                > Unix-like systems. It doesn't sound likely that you have done
                > something to restrict this to Linux-only.

                My first thought was he thinks Linux and Unix are just different words for the same thing. Or he knows Linux and has never heard of Unix.

                Wouldn't be the first. I've run into people (although less technical) who have heard of Linux since it's been a "cool" buzz-word but have no idea what Unix is.

                --
                Larry Stone
                lstone19@...
                http://www.stonejongleux.com/
              • Nicolas HAHN
                ... You are totally right. In theory it is not restricted. My point is that I m coding, testing, and using the tool ONLY on a bunch of RHEL 6.x servers. So in
                Message 7 of 7 , Apr 13, 2013
                  Le 13/04/2013 15:17, /dev/rob0 a écrit :
                  > I think the point is that none of the software you mention are
                  > Linux-specific. Postfix, PostgreSQL, rsyslog, "apache" (Apache httpd),
                  > and php all work and are commonly seen on other Unix and Unix-like
                  > systems. It doesn't sound likely that you have done something to
                  > restrict this to Linux-only.

                  You are totally right. In theory it is not restricted.
                  My point is that I'm coding, testing, and using the tool ONLY on a bunch
                  of RHEL 6.x servers.
                  So in practice, I never used the tool on another OS (even another LINUX
                  distrib), and this means it's not validated for other OS than RHEL 6.

                  But again, if people in the community want to deploy it, test it,
                  install it, debug it, validate it on other platforms, then that will be
                  a win-win deal for sure :)

                  There are so much to be done...

                  Kind regards,
                  Nicolas
                Your message has been successfully submitted and would be delivered to recipients shortly.