Loading ...
Sorry, an error occurred while loading the content.
 

Re: Connection caching/SMTP piggybacking and postfix?

Expand Messages
  • Wietse Venema
    ... Well, how does one migrate **AN OPEN TLS SESSION** from one process into the other? I am not aware an OpenSSL API for doing that. Wietse
    Message 1 of 14 , Feb 27, 2013
      Thorsten Glaser:
      > Wietse Venema <wietse <at> porcupine.org> writes:
      >
      > > deliveries. Proper SMTP connection caching is not done by the SMTP
      > > clients but by a separate process that is queried by SMTP clients.
      >
      > If you don?t manage to do that with TLS, this statement is plainly wrong.

      Well, how does one migrate **AN OPEN TLS SESSION** from one process
      into the other? I am not aware an OpenSSL API for doing that.

      Wietse
    • Thorsten Glaser
      ... Then just don’t do that… keep it in the other process. (I think OpenSSH does that, though probably not standard TLS.) bye, //mirabilos -- tarent
      Message 2 of 14 , Feb 28, 2013
        On Wed, 27 Feb 2013, Wietse Venema wrote:

        > Well, how does one migrate **AN OPEN TLS SESSION** from one process
        > into the other? I am not aware an OpenSSL API for doing that.

        Then just don’t do that… keep it in the other process.

        (I think OpenSSH does that, though probably not standard TLS.)

        bye,
        //mirabilos
        --
        tarent solutions GmbH
        Rochusstraße 2-4, D-53123 Bonn • http://www.tarent.de/
        Tel: +49 228 54881-393 • Fax: +49 228 54881-314
        HRB 5168 (AG Bonn) • USt-ID (VAT): DE122264941
        Geschäftsführer: Boris Esser, Sebastian Mancke
      • Reindl Harald
        ... and how do you imagine that? did you try to understand how postfix works? i guess no
        Message 3 of 14 , Feb 28, 2013
          Am 28.02.2013 10:38, schrieb Thorsten Glaser:
          > On Wed, 27 Feb 2013, Wietse Venema wrote:
          >
          >> Well, how does one migrate **AN OPEN TLS SESSION** from one process
          >> into the other? I am not aware an OpenSSL API for doing that.
          >
          > Then just don’t do that… keep it in the other process

          and how do you imagine that?
          did you try to understand how postfix works? i guess no
        • Thorsten Glaser
          ... Hrm. Okay, lowering default_destination_concurrency_limit is certainly a viable workaround. I’m not the Postfix expert here, and he didn’t suggest it
          Message 4 of 14 , Feb 28, 2013
            On Wed, 27 Feb 2013, Viktor Dukhovni wrote:

            > No, connection caching has nothing to do with concurrency. Whether
            > connections are cached, or not, Postfix will attempt parallel
            > delivery up to the configured concurrency limit:

            Hrm. Okay, lowering default_destination_concurrency_limit is certainly
            a viable workaround. I’m not the Postfix expert here, and he didn’t
            suggest it to me.

            > of the MX hosts for a destination is unreachable. The load on the
            > remote server is identical with and without connection caching.

            That’s untrue because it certainly makes a difference whether you
            send mails in parallel or serial, especially when the receiving
            host then spawns an LDAP connection, a shell command and an MSA
            to forward the mail further.

            > Sendmail lacks a queue-manager, if you really want to avoid Sendmail
            > melt-downs under load, deploy Postfix.

            Sendmail does not “melt down”, it just refuses connections until
            the load gets lower again. I sure hope Postfix does the same if
            the system load is high? (The load is not necessarily from sendmail;
            this is a spamfilter setup which is tightly coupled to several
            system components, one of which is sendmail, but the LDAP search
            and two sed invocations as well as calling bmf are the CPU hogs
            here.)

            > Connection caching without the ability to re-use the connection in
            > a different delivery-agent process *is* abusive of the remote
            > system, since you're keeping a remote process occupied and idle, while
            > making new connections. This will not be implemented.

            You can just close them immediately if there’s no mail in the
            queue for the remote system any more. Or after a second.

            > When disgruntled, especially due to problems with system "A", don't
            > rant on the mailing list for system "B".

            My problem here is with system "B" as system "A" is working
            just fine and by the specs.

            bye,
            //mirabilos
            --
            tarent solutions GmbH
            Rochusstraße 2-4, D-53123 Bonn • http://www.tarent.de/
            Tel: +49 228 54881-393 • Fax: +49 228 54881-314
            HRB 5168 (AG Bonn) • USt-ID (VAT): DE122264941
            Geschäftsführer: Boris Esser, Sebastian Mancke
          • Thorsten Glaser
            ... If the way something works makes it impossible to do something that is a requirement it says something about the way that particular something works, not
            Message 5 of 14 , Feb 28, 2013
              On Thu, 28 Feb 2013, Reindl Harald wrote:

              > > Then just don’t do that… keep it in the other process

              > did you try to understand how postfix works? i guess no

              If the way something works makes it impossible to do
              something that is a requirement it says something about
              the way that particular something works, not about the
              requirement.

              But Viktor provided me with a viable workaround I did not
              know (considering I’m not the Postfix expert here).

              bye,
              //mirabilos
              --
              tarent solutions GmbH
              Rochusstraße 2-4, D-53123 Bonn • http://www.tarent.de/
              Tel: +49 228 54881-393 • Fax: +49 228 54881-314
              HRB 5168 (AG Bonn) • USt-ID (VAT): DE122264941
              Geschäftsführer: Boris Esser, Sebastian Mancke
            • Reindl Harald
              ... after years on this mailing-list in 99 out of 100 cases the opposite is true because mostly people are coming up with a solution not working as expected
              Message 6 of 14 , Feb 28, 2013
                Am 28.02.2013 10:45, schrieb Thorsten Glaser:
                > On Thu, 28 Feb 2013, Reindl Harald wrote:
                >
                >>> Then just don’t do that… keep it in the other process
                >
                >> did you try to understand how postfix works? i guess no
                >
                > If the way something works makes it impossible to do
                > something that is a requirement it says something about
                > the way that particular something works, not about the
                > requirement

                after years on this mailing-list in 99 out of 100 cases
                the opposite is true because mostly people are coming up
                with a solution not working as expected instead to describe
                or understand the problem

                if you have a VERY HIGH amount of outgoing mails to send
                you can not expect that this works in burst with only
                one outgoing server - nothing will change this
              • Thorsten Glaser
                ... Yes well, we have three, but I’m running into firewall admins currently so in practice there’s only one at the moment. I hope that this part of the
                Message 7 of 14 , Feb 28, 2013
                  On Thu, 28 Feb 2013, Reindl Harald wrote:

                  > if you have a VERY HIGH amount of outgoing mails to send
                  > you can not expect that this works in burst with only
                  > one outgoing server - nothing will change this

                  Yes well, we have three, but I’m running into firewall
                  admins currently so in practice there’s only one at
                  the moment. I hope that this part of the issue will
                  also resolve eventually…

                  I set the concurrency limit to 4 for now.

                  bye,
                  //mirabilos
                  --
                  tarent solutions GmbH
                  Rochusstraße 2-4, D-53123 Bonn • http://www.tarent.de/
                  Tel: +49 228 54881-393 • Fax: +49 228 54881-314
                  HRB 5168 (AG Bonn) • USt-ID (VAT): DE122264941
                  Geschäftsführer: Boris Esser, Sebastian Mancke
                • Wietse Venema
                  ... I welcome you to propose a design that is as scalable as the rest of Postfix. A centralized process that proxies all TLS sessions for Postfix does not meet
                  Message 8 of 14 , Feb 28, 2013
                    Thorsten Glaser:
                    > On Wed, 27 Feb 2013, Wietse Venema wrote:
                    >
                    > > Well, how does one migrate **AN OPEN TLS SESSION** from one process
                    > > into the other? I am not aware an OpenSSL API for doing that.
                    >
                    > Then just don?t do that? keep it in the other process.
                    >
                    > (I think OpenSSH does that, though probably not standard TLS.)

                    I welcome you to propose a design that is as scalable as the rest
                    of Postfix. A centralized process that proxies all TLS sessions for
                    Postfix does not meet that requirement.

                    Wietse
                  • Wietse Venema
                    Thorsten Glaser: [ Charset UTF-8 unsupported, converting... ] ... Connection caching has nothing to do with concurrency. Sheesh. Wietse
                    Message 9 of 14 , Feb 28, 2013
                      Thorsten Glaser:
                      [ Charset UTF-8 unsupported, converting... ]
                      > On Wed, 27 Feb 2013, Viktor Dukhovni wrote:
                      >
                      > > No, connection caching has nothing to do with concurrency. Whether
                      > > connections are cached, or not, Postfix will attempt parallel
                      > > delivery up to the configured concurrency limit:
                      >
                      > Hrm. Okay, lowering default_destination_concurrency_limit is certainly
                      > a viable workaround. I?m not the Postfix expert here, and he didn?t
                      > suggest it to me.
                      >
                      > > of the MX hosts for a destination is unreachable. The load on the
                      > > remote server is identical with and without connection caching.
                      >
                      > That?s untrue because it certainly makes a difference whether you
                      > send mails in parallel or serial, especially when the receiving

                      Connection caching has nothing to do with concurrency. Sheesh.

                      Wietse
                    • Thorsten Glaser
                      ... I was assuming that, when you already have a connection to a receiving MX, you’d not open another one but reuse the existing one, so it would, of course,
                      Message 10 of 14 , Feb 28, 2013
                        On Thu, 28 Feb 2013, Wietse Venema wrote:

                        > > That?s untrue because it certainly makes a difference whether you
                        > > send mails in parallel or serial, especially when the receiving
                        >
                        > Connection caching has nothing to do with concurrency. Sheesh.

                        I was assuming that, when you already have a connection to a
                        receiving MX, you’d not open another one but reuse the existing
                        one, so it would, of course, have something to do with that.

                        If not… well, that’s the qmail model.

                        > I welcome you to propose a design that is as scalable as the rest

                        I am *not* interested in Postfix. I’m not even primarily
                        working with Postfix. I came here because I was suffering,
                        on the receiving side, from perceived Postfix misbehaviour.
                        I’ve got a configuration item with which I can reduce the
                        possibility of the behaviour becoming an issue, which has
                        helped me. That was it.

                        bye,
                        //mirabilos
                        --
                        tarent solutions GmbH
                        Rochusstraße 2-4, D-53123 Bonn • http://www.tarent.de/
                        Tel: +49 228 54881-393 • Fax: +49 228 54881-314
                        HRB 5168 (AG Bonn) • USt-ID (VAT): DE122264941
                        Geschäftsführer: Boris Esser, Sebastian Mancke
                      • Wietse Venema
                        ... Concurrency is the number of simultaneous connections. It has nothing to do with connection caching, which is using one connection for more than one
                        Message 11 of 14 , Feb 28, 2013
                          Thorsten Glaser:
                          > On Thu, 28 Feb 2013, Wietse Venema wrote:
                          >
                          > > > That?s untrue because it certainly makes a difference whether you
                          > > > send mails in parallel or serial, especially when the receiving
                          > >
                          > > Connection caching has nothing to do with concurrency. Sheesh.
                          >
                          > I was assuming that, when you already have a connection to a
                          > receiving MX, you?d not open another one but reuse the existing

                          Concurrency is the number of simultaneous connections.

                          It has nothing to do with connection caching, which is using one
                          connection for more than one transaction.

                          Both are configurable with Postfix.

                          Wietse
                        • Viktor Dukhovni
                          ... One can t immediately re-use an existing connection when some other process is currently using it! To do that, one would have to always wait for the other
                          Message 12 of 14 , Feb 28, 2013
                            On Thu, Feb 28, 2013 at 01:16:26PM +0100, Thorsten Glaser wrote:

                            > > Connection caching has nothing to do with concurrency. Sheesh.
                            >
                            > I was assuming that, when you already have a connection to a
                            > receiving MX, you'd not open another one but reuse the existing
                            > one, so it would, of course, have something to do with that.
                            >
                            > If not? well, that's the qmail model.

                            One can't immediately re-use an existing connection when some other
                            process is currently using it! To do that, one would have to always
                            wait for the other process to relinquish the connection, which forces
                            concurrency <= 1. This has nothing to do with qmail, which uses a
                            recipient limit of 1 which leads to more deliveries with multi-recipient
                            messages than with Postfix or Sendmail.

                            Postfix reuses *idle* existing connections, but creates new ones
                            in *parallel* up to the concurrency limit to avoid severely limiting
                            throughput to destinations with high capacity in the face of high
                            single message delivery latency.

                            Sendmail has no concurrency limits at all, since each process runs
                            in isolation (no queue-manager), it just hiccups at random
                            when the CPU load is high, which is not a meaningful signal that
                            the MTA should pause deliveries or refuse new connections.

                            You mentioned a Sendmail QueueLA or RefuseLA of 8, this is a meagerly
                            limit on modern multi-cpu server hardware. ~10 years ago, I was
                            tuning multi-CPU Sendmail servers of that era with a QueueLA of 64.

                            > > I welcome you to propose a design that is as scalable as the rest
                            >
                            > I am *not* interested in Postfix. I'm not even primarily
                            > working with Postfix.

                            You're also not interested in any form of critical thinking before
                            you spout non sequiturs, or even a modicum of humility which would
                            suggest asking how to improve latency in your two MTA setup, nor do
                            you care to do any research, rather you barge in here in response to
                            an 11-year old post blaming Postfix for Sendmail design flaws.

                            I smell a troll, please don't come back.

                            --
                            Viktor.
                          Your message has been successfully submitted and would be delivered to recipients shortly.