Loading ...
Sorry, an error occurred while loading the content.

Re: frequent qmgr crashes with postfix-2.10-20120630

Expand Messages
  • Ralf Hildebrandt
    ... Indeed it was! ... -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin
    Message 1 of 16 , Jul 4, 2012
    • 0 Attachment
      * Viktor Dukhovni <postfix-users@...>:

      > If your previous snapshot is 20120520

      Indeed it was!

      > or earlier, then the issue is likely:
      >
      > 20120527-8
      >
      > Infrastructure: limited support to shrink VSTREAM buffers.
      > The change takes place when reading from (a stream for the
      > first time | an empty buffer) or when writing to (a stream
      > for the first time | a full buffer). TODO: the change should
      > also happen after purging or flushing a buffer. File:
      > util/vstream.c.

      --
      Ralf Hildebrandt
      Geschäftsbereich IT | Abteilung Netzwerk
      Charité - Universitätsmedizin Berlin
      Campus Benjamin Franklin
      Hindenburgdamm 30 | D-12203 Berlin
      Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
      ralf.hildebrandt@... | http://www.charite.de
    • Wietse Venema
      ... I am puzzled. There is code that changes the vstream buffer size in: - haproxy handlers (smtpd, postscreen) - code that tweaks VSTREAM read/write buffer
      Message 2 of 16 , Jul 4, 2012
      • 0 Attachment
        Viktor Dukhovni:
        > On Wed, Jul 04, 2012 at 11:53:46PM +0200, Ralf Hildebrandt wrote:
        >
        > > I'm encountering frequent qmgr crashes with postfix-2.10-20120630:
        > >
        > > Jul 4 19:20:27 mail postfix/qmgr[25825]: panic: vstream_buf_alloc: attempt to shrink non-empty buffer
        > > Jul 4 19:20:28 mail postfix/master[2383]: warning: process /usr/lib/postfix/qmgr pid 25825 killed by signal 6
        > >
        > > Jul 4 19:39:15 mail postfix/qmgr[6531]: panic: vstream_buf_alloc: attempt to shrink non-empty buffer
        > > Jul 4 19:39:16 mail postfix/master[2383]: warning: process /usr/lib/postfix/qmgr pid 6531 killed by signal 6
        > >
        > > Jul 4 21:53:21 mail postfix/qmgr[25007]: panic: vstream_buf_alloc: attempt to shrink non-empty buffer
        > > Jul 4 21:53:22 mail postfix/master[2383]: warning: process /usr/lib/postfix/qmgr pid 25007 killed by signal 6
        >
        > If your previous snapshot is 20120520 or earlier, then the issue is likely:
        >
        > 20120527-8
        >
        > Infrastructure: limited support to shrink VSTREAM buffers.
        > The change takes place when reading from (a stream for the
        > first time | an empty buffer) or when writing to (a stream
        > for the first time | a full buffer). TODO: the change should
        > also happen after purging or flushing a buffer. File:
        > util/vstream.c.
        >
        > So revert to the 0520 snapshot.

        I am puzzled. There is code that changes the vstream buffer size in:

        - haproxy handlers (smtpd, postscreen)
        - code that tweaks VSTREAM read/write buffer sizes for TCP sockets

        None of which would apply to qmgr.

        Without further details I suspect that Ralf made a build error of some kind.
        Does the error happen after sending a message to the qmgr FIFO?

        Things have been running error-free with qmgr configured in master.cf
        to use FIFOs and with qmgr configured for UNIX-domain sockets.

        Wietse
      • Ralf Hildebrandt
        ... Oh come on :) This is what I ve been using for quite some time make tidy CCARGS= -Wl,--as-needed pkg-config --cflags openssl libpcre libcdb -DUSE_TLS
        Message 3 of 16 , Jul 5, 2012
        • 0 Attachment
          * Wietse Venema <wietse@...>:

          > I am puzzled. There is code that changes the vstream buffer size in:
          >
          > - haproxy handlers (smtpd, postscreen)
          > - code that tweaks VSTREAM read/write buffer sizes for TCP sockets
          >
          > None of which would apply to qmgr.
          >
          > Without further details I suspect that Ralf made a build error of some kind.

          Oh come on :) This is what I've been using for quite some time

          make tidy
          CCARGS="-Wl,--as-needed pkg-config --cflags openssl libpcre libcdb -DUSE_TLS -DHAS_PCRE -DHAS_CDB -DHAS_LDAP" \
          AUXLIBS="-Wl,--as-needed pkg-config --libs openssl libpcre libcdb -lldap -llber -lnsl" make makefiles &&
          make && postfix stop; make upgrade && postfix start && sleep 2 &&
          postfix flush

          > Does the error happen after sending a message to the qmgr FIFO?

          How would I find out?


          --
          Ralf Hildebrandt
          Geschäftsbereich IT | Abteilung Netzwerk
          Charité - Universitätsmedizin Berlin
          Campus Benjamin Franklin
          Hindenburgdamm 30 | D-12203 Berlin
          Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
          ralf.hildebrandt@... | http://www.charite.de
        • Ralf Hildebrandt
          ... cut & paste broke the `` s, sorry: make tidy CCARGS= -Wl,--as-needed `pkg-config --cflags openssl libpcre libcdb` -DUSE_TLS -DHAS_PCRE -DHAS_CDB
          Message 4 of 16 , Jul 5, 2012
          • 0 Attachment
            * Ralf Hildebrandt <Ralf.Hildebrandt@...>:
            > * Wietse Venema <wietse@...>:
            >
            > > I am puzzled. There is code that changes the vstream buffer size in:
            > >
            > > - haproxy handlers (smtpd, postscreen)
            > > - code that tweaks VSTREAM read/write buffer sizes for TCP sockets
            > >
            > > None of which would apply to qmgr.
            > >
            > > Without further details I suspect that Ralf made a build error of some kind.
            >
            > Oh come on :) This is what I've been using for quite some time
            >
            > make tidy
            > CCARGS="-Wl,--as-needed pkg-config --cflags openssl libpcre libcdb -DUSE_TLS -DHAS_PCRE -DHAS_CDB -DHAS_LDAP" \
            > AUXLIBS="-Wl,--as-needed pkg-config --libs openssl libpcre libcdb -lldap -llber -lnsl" make makefiles &&
            > make && postfix stop; make upgrade && postfix start && sleep 2 &&
            > postfix flush

            cut & paste broke the ``'s, sorry:

            make tidy
            CCARGS="-Wl,--as-needed `pkg-config --cflags openssl libpcre libcdb` -DUSE_TLS -DHAS_PCRE -DHAS_CDB -DHAS_LDAP" \
            AUXLIBS="-Wl,--as-needed `pkg-config --libs openssl libpcre libcdb` -lldap -llber -lnsl" make makefiles
            make

            --
            Ralf Hildebrandt
            Geschäftsbereich IT | Abteilung Netzwerk
            Charité - Universitätsmedizin Berlin
            Campus Benjamin Franklin
            Hindenburgdamm 30 | D-12203 Berlin
            Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
            ralf.hildebrandt@... | http://www.charite.de
          • Wietse Venema
            ... In master.cf, add two -v options at the end of the qmgr service. This may produce a lot of output that is better exchanged off-list. Wietse
            Message 5 of 16 , Jul 5, 2012
            • 0 Attachment
              Ralf Hildebrandt:
              > > Does the error happen after sending a message to the qmgr FIFO?
              >
              > How would I find out?

              In master.cf, add two -v options at the end of the qmgr service.

              This may produce a lot of output that is better exchanged off-list.

              Wietse
            • Ralf Hildebrandt
              ... I ll recompile the latest snapshot and will enable verbose logging. -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité -
              Message 6 of 16 , Jul 5, 2012
              • 0 Attachment
                * Wietse Venema <wietse@...>:
                > Ralf Hildebrandt:
                > > > Does the error happen after sending a message to the qmgr FIFO?
                > >
                > > How would I find out?
                >
                > In master.cf, add two -v options at the end of the qmgr service.
                >
                > This may produce a lot of output that is better exchanged off-list.

                I'll recompile the latest snapshot and will enable verbose logging.

                --
                Ralf Hildebrandt
                Geschäftsbereich IT | Abteilung Netzwerk
                Charité - Universitätsmedizin Berlin
                Campus Benjamin Franklin
                Hindenburgdamm 30 | D-12203 Berlin
                Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
                ralf.hildebrandt@... | http://www.charite.de
              • Wietse Venema
                ... What OS version? I may be able to set up a VM for some tests, once you have some logging to focus the search. Wietse
                Message 7 of 16 , Jul 5, 2012
                • 0 Attachment
                  Ralf Hildebrandt:
                  > * Wietse Venema <wietse@...>:
                  > > Ralf Hildebrandt:
                  > > > > Does the error happen after sending a message to the qmgr FIFO?
                  > > >
                  > > > How would I find out?
                  > >
                  > > In master.cf, add two -v options at the end of the qmgr service.
                  > >
                  > > This may produce a lot of output that is better exchanged off-list.
                  >
                  > I'll recompile the latest snapshot and will enable verbose logging.

                  What OS version? I may be able to set up a VM for some tests, once
                  you have some logging to focus the search.

                  Wietse
                • Chris
                  ... I suspect Ubuntu 12.04 LTS. ... -- Chris
                  Message 8 of 16 , Jul 5, 2012
                  • 0 Attachment
                    2012/7/5 Wietse Venema <wietse@...>:
                    > Ralf Hildebrandt:
                    >> * Wietse Venema <wietse@...>:
                    >> > Ralf Hildebrandt:
                    >> > > > Does the error happen after sending a message to the qmgr FIFO?
                    >> > >
                    >> > > How would I find out?
                    >> >
                    >> > In master.cf, add two -v options at the end of the qmgr service.
                    >> >
                    >> > This may produce a lot of output that is better exchanged off-list.
                    >>
                    >> I'll recompile the latest snapshot and will enable verbose logging.
                    >
                    > What OS version? I may be able to set up a VM for some tests, once
                    > you have some logging to focus the search.

                    I suspect Ubuntu 12.04 LTS.

                    > # uname -a
                    > Linux mail.charite.de 3.2.0-26-generic #41-Ubuntu SMP Thu Jun 14 17:49:24 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

                    --
                    Chris
                  • Ralf Hildebrandt
                    ... Ubuntu 12.04 LTS AMD64. but the same thing also happened on Debian/testing 32 bit -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité -
                    Message 9 of 16 , Jul 5, 2012
                    • 0 Attachment
                      * Wietse Venema <wietse@...>:
                      > Ralf Hildebrandt:
                      > > * Wietse Venema <wietse@...>:
                      > > > Ralf Hildebrandt:
                      > > > > > Does the error happen after sending a message to the qmgr FIFO?
                      > > > >
                      > > > > How would I find out?
                      > > >
                      > > > In master.cf, add two -v options at the end of the qmgr service.
                      > > >
                      > > > This may produce a lot of output that is better exchanged off-list.
                      > >
                      > > I'll recompile the latest snapshot and will enable verbose logging.
                      >
                      > What OS version? I may be able to set up a VM for some tests, once
                      > you have some logging to focus the search.

                      Ubuntu 12.04 LTS AMD64.
                      but the same thing also happened on Debian/testing 32 bit

                      --
                      Ralf Hildebrandt
                      Geschäftsbereich IT | Abteilung Netzwerk
                      Charité - Universitätsmedizin Berlin
                      Campus Benjamin Franklin
                      Hindenburgdamm 30 | D-12203 Berlin
                      Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
                      ralf.hildebrandt@... | http://www.charite.de
                    • Ralf Hildebrandt
                      ... Yep ... -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 |
                      Message 10 of 16 , Jul 5, 2012
                      • 0 Attachment
                        * Chris <xchris89x@...>:

                        > I suspect Ubuntu 12.04 LTS.

                        Yep

                        > > # uname -a
                        > > Linux mail.charite.de 3.2.0-26-generic #41-Ubuntu SMP Thu Jun 14 17:49:24 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

                        --
                        Ralf Hildebrandt
                        Geschäftsbereich IT | Abteilung Netzwerk
                        Charité - Universitätsmedizin Berlin
                        Campus Benjamin Franklin
                        Hindenburgdamm 30 | D-12203 Berlin
                        Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
                        ralf.hildebrandt@... | http://www.charite.de
                      • Wietse Venema
                        ... I have no errors on 64 bit FC or Ubuntu. Wietse
                        Message 11 of 16 , Jul 6, 2012
                        • 0 Attachment
                          Ralf Hildebrandt:
                          > * Wietse Venema <wietse@...>:
                          > > Ralf Hildebrandt:
                          > > > * Wietse Venema <wietse@...>:
                          > > > > Ralf Hildebrandt:
                          > > > > > > Does the error happen after sending a message to the qmgr FIFO?
                          > > > > >
                          > > > > > How would I find out?
                          > > > >
                          > > > > In master.cf, add two -v options at the end of the qmgr service.
                          > > > >
                          > > > > This may produce a lot of output that is better exchanged off-list.
                          > > >
                          > > > I'll recompile the latest snapshot and will enable verbose logging.
                          > >
                          > > What OS version? I may be able to set up a VM for some tests, once
                          > > you have some logging to focus the search.
                          >
                          > Ubuntu 12.04 LTS AMD64.
                          > but the same thing also happened on Debian/testing 32 bit

                          I have no errors on 64 bit FC or Ubuntu.

                          Wietse
                        • Wietse Venema
                          ... This was the result of false alarms in a new safety check (this was for new VSTREAM code that I added to implement haproxy support). Such safety checks are
                          Message 12 of 16 , Jul 15, 2012
                          • 0 Attachment
                            Ralf Hildebrandt:
                            > I'm encountering frequent qmgr crashes with postfix-2.10-20120630:
                            >
                            > Jul 4 19:20:27 mail postfix/qmgr[25825]: panic: vstream_buf_alloc: attempt to shrink non-empty buffer

                            This was the result of false alarms in a new safety check (this was
                            for new VSTREAM code that I added to implement haproxy support).

                            Such safety checks are all over the place in Postfix, and their
                            purpose is to pull the emergency brake before something bad happens.
                            In this case the safety check was at fault, not the code that it
                            was trying to keep out of trouble.

                            Thanks to Ralf for bearing with me as I simplified the code and
                            made the problem go away.

                            Postfix snapshots 20120713 and later are OK.

                            Wietse
                          • Ralf Hildebrandt
                            ... -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203
                            Message 13 of 16 , Jul 16, 2012
                            • 0 Attachment
                              * Wietse Venema <wietse@...>:

                              > Thanks to Ralf for bearing with me as I simplified the code and
                              > made the problem go away.
                              >
                              > Postfix snapshots 20120713 and later are OK.

                              :)

                              --
                              Ralf Hildebrandt
                              Geschäftsbereich IT | Abteilung Netzwerk
                              Charité - Universitätsmedizin Berlin
                              Campus Benjamin Franklin
                              Hindenburgdamm 30 | D-12203 Berlin
                              Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
                              ralf.hildebrandt@... | http://www.charite.de
                            Your message has been successfully submitted and would be delivered to recipients shortly.