Loading ...
Sorry, an error occurred while loading the content.

Re: Adjust smtp to limitations of a host

Expand Messages
  • Ultrabug
    ... The result is pretty much that they deny any further email with 4XX codes AND penalize further connections. It just stops accepting any new mail. ... Ok,
    Message 1 of 16 , Apr 1, 2011
    • 0 Attachment
      On 31/03/2011 18:39, Victor Duchovni wrote:
      > On Thu, Mar 31, 2011 at 10:15:55AM +0200, Ultrabug wrote:
      >
      >> Dear list,
      >>
      >> I'm facing a problem where I have to adapt and optimize my smtp servers
      >> to a host's constraints which are as follow :
      >>
      >> - maximum 3 connections to each MX of the host (he has 10 MX so
      >> potentially I should be able to make 30 connections)
      >
      > What happens if you happen to exceed the limit on particular host
      > among the 10? If it just quickly returns a 4XX code, and does not
      > penalize future connections, ignore this limit and let Postfix do
      > what it does by default.
      >

      The result is pretty much that they deny any further email with 4XX
      codes AND penalize further connections. It just stops accepting any new
      mail.

      >> - maximum 1000 connections per MX per hour
      >
      > Connection caching should help if volume is high enough to worry about
      > this. Note this is just less than one connection every 3 seconds, but
      > Postfix caches idle connections for 2 seconds, so if your output rate
      > is 1200 messages spaced perfectly 3 seconds apart, you lose, but this
      > is fairly unlikely.
      >

      Ok, connection caching I do already and I don't think to fall into this
      limitation.

      >> - maximum 100 emails sent per connection
      >
      > Postfix has no such limit, instead a connection is retired if idle
      > for more than 2 seconds, or after 300s (tunable). Again the site
      > should just return a 4XX response to RSET, and Postfix will drop
      > the connection and build a new connection, probably to another host.
      >
      >> My problem is that this setup is far from optimal compared to the
      >> limitations and it slows down a lot my email delivery rate to these domains.
      >>
      >> Anyone have any tips on how I could do this better please ?
      >
      > The receiving sites policies are stupid if they don't implement
      > them sensibly by just returning 4XX responses without penalizing
      > subsequent transactions.
      >
      > Have you in fact observed that default Postfix settings run into trouble
      > with this site?

      Oh yes, it ends up with thousands of mails deferred which in turn would
      try again at a too high rate so the pile remains deferred etc etc

      Have you considered the less aggressive concurrency
      > feedback controls in Postfix 2.5?
      >
      > slow_initial_concurrency = 2
      > slow_destination_concurrency_limit = 15
      > slow_destination_concurrency_failed_cohort_limit = 5
      > slow_destination_concurrency_positive_feedback = 1/5
      > slow_destination_concurrency_negative_feedback = 1/8
      >

      This sounds interesting indeed, I didn't understand fully these
      cohort/feedback options, I'll give them a try !

      > and if absolutely necessary, in master.cf:
      >
      > slow unix - - n - - smtp
      > -o smtp_connection_reuse_time_limit=30s
      >
      > (the remote side starts rejecting traffic consistently instead of
      > sending 421 for the 100th RSET over a given connection).
      >

      Many tanks for your time and help
    • Victor Duchovni
      ... So instead of gently nudging you to the right sending rate, the DoS their service and require the whole planet to hand-tune transports for their domain,
      Message 2 of 16 , Apr 1, 2011
      • 0 Attachment
        On Fri, Apr 01, 2011 at 09:20:39AM +0200, Ultrabug wrote:

        > > What happens if you happen to exceed the limit on particular host
        > > among the 10? If it just quickly returns a 4XX code, and does not
        > > penalize future connections, ignore this limit and let Postfix do
        > > what it does by default.
        > >
        >
        > The result is pretty much that they deny any further email with 4XX
        > codes AND penalize further connections. It just stops accepting any new
        > mail.

        So instead of gently nudging you to the right sending rate, the DoS
        their service and require the whole planet to hand-tune transports for
        their domain, this is insane, and as much as possible, senders should
        not play-along with this insanity. I know that some senders will have
        to jump through hoops for business reasons, but if possible apply some
        spine.

        > > Have you considered the less aggressive concurrency
        > > feedback controls in Postfix 2.5?
        > >
        > > slow_initial_concurrency = 2
        > > slow_destination_concurrency_limit = 15
        > > slow_destination_concurrency_failed_cohort_limit = 5
        > > slow_destination_concurrency_positive_feedback = 1/5
        > > slow_destination_concurrency_negative_feedback = 1/8
        > >
        >
        > This sounds interesting indeed, I didn't understand fully these
        > cohort/feedback options, I'll give them a try !

        This really won't help if the remote sites response to exceeding their
        maximum rate is a sticky refusal to accept further mail.

        If that's the case, you must psychically (i.e. prior hand-tuning) stay
        under their limits. The feedback controls assume that the feedback is in
        the form of transient 421 responses when the connection concurrency or
        re-use limits are reached. No simple feedback algorithm will dynamically
        adjust to a feedback control that goes from open to sticky-closed.

        > > and if absolutely necessary, in master.cf:
        > >
        > > slow unix - - n - - smtp
        > > -o smtp_connection_reuse_time_limit=30s
        > >
        > > (the remote side starts rejecting traffic consistently instead of
        > > sending 421 for the 100th RSET over a given connection).
        >
        > Many tanks for your time and help

        You probably can't do much with my advice, when the receiving system
        is fubared, your options are limited.

        --
        Viktor.
      • Wietse Venema
        ... You ll have to use a combination of /etc/postfix/main.cf: slow_destination_rate_delay=x (this forces concurrency == 1) slow_destination_recipient_limit=y
        Message 3 of 16 , Apr 1, 2011
        • 0 Attachment
          Victor Duchovni:
          > > Many tanks for your time and help
          >
          > You probably can't do much with my advice, when the receiving system
          > is fubared, your options are limited.

          You'll have to use a combination of

          /etc/postfix/main.cf:
          slow_destination_rate_delay=x (this forces concurrency == 1)
          slow_destination_recipient_limit=y

          /etc/postfix/master.cf:
          slow unix - - n - - smtp
          -o smtp_connection_cache_on_demand=no

          that stays within the stated limits (which of course may not bear
          any relationship to reality, but that is a different matter).

          And these clowns will have to get used to abnormal email delays.
          I hope this isn't my tax money at work.

          Wietse
        • Victor Duchovni
          ... Of course, if their policy is that they only accept a trickle of mail from sites that are not their important business partners, the correct place to make
          Message 4 of 16 , Apr 1, 2011
          • 0 Attachment
            On Fri, Apr 01, 2011 at 01:39:41PM -0400, Wietse Venema wrote:

            > Victor Duchovni:
            > > > Many tanks for your time and help
            > >
            > > You probably can't do much with my advice, when the receiving system
            > > is fubared, your options are limited.
            >
            > You'll have to use a combination of
            >
            > /etc/postfix/main.cf:
            > slow_destination_rate_delay=x (this forces concurrency == 1)
            > slow_destination_recipient_limit=y
            >
            > /etc/postfix/master.cf:
            > slow unix - - n - - smtp
            > -o smtp_connection_cache_on_demand=no
            >
            > that stays within the stated limits (which of course may not bear
            > any relationship to reality, but that is a different matter).
            >
            > And these clowns will have to get used to abnormal email delays.
            > I hope this isn't my tax money at work.

            Of course, if their policy is that they only accept a trickle of
            mail from sites that are not their important business partners,
            the correct place to make adjustments is in their white-list.

            If their systems support a whitelist, the OP should find a way to
            get whitelisted.

            --
            Viktor.
          • Ultrabug
            ... I see indeed, I shall try this out on monday at work. ... It s not an american public host thanksfully (but french host, maybe that s why) :) ... Many
            Message 5 of 16 , Apr 1, 2011
            • 0 Attachment
              On 01/04/2011 19:39, Wietse Venema wrote:
              > Victor Duchovni:
              >>> Many tanks for your time and help
              >> You probably can't do much with my advice, when the receiving system
              >> is fubared, your options are limited.
              > You'll have to use a combination of
              >
              > /etc/postfix/main.cf:
              > slow_destination_rate_delay=x (this forces concurrency == 1)
              > slow_destination_recipient_limit=y
              >
              > /etc/postfix/master.cf:
              > slow unix - - n - - smtp
              > -o smtp_connection_cache_on_demand=no
              >
              I see indeed, I shall try this out on monday at work.
              > that stays within the stated limits (which of course may not bear
              > any relationship to reality, but that is a different matter).
              >
              > And these clowns will have to get used to abnormal email delays.
              > I hope this isn't my tax money at work.
              It's not an american public host thanksfully (but french host, maybe
              that's why) :)
              > Wietse
              Many thanks to you both for taking time to look on this matter. I'll
              keep this thread updated if/when I get this working better.

              Regards
            • Mark Alan
              On Thu, 31 Mar 2011 14:53:11 -0400, Victor Duchovni ... Tried the above setup. It does not help. We have much more 421 with this approach than we had with our
              Message 6 of 16 , Apr 2, 2011
              • 0 Attachment
                On Thu, 31 Mar 2011 14:53:11 -0400, Victor Duchovni
                <Victor.Duchovni@...> wrote:

                > > /etc/postfix/master.cf
                > > slow unix - - - - - smtp
                > > -o syslog_name=postfix-slow
                > > -o smtp_connection_reuse_time_limit=30s
                > > EOT
                > >
                > > /etc/postfix/main.cf
                > > slow_initial_destination_concurrency = 2
                > > slow_destination_concurrency_limit = 15
                > > slow_destination_concurrency_failed_cohort_limit = 5
                > > slow_destination_concurrency_positive_feedback = 1/5
                > > slow_destination_concurrency_negative_feedback = 1/8


                > (...) You can certainly try, and report your findings.

                Tried the above setup. It does not help.
                We have much more 421 with this approach than we had with our former
                setup.

                We will now try the following transport settings (based on a
                recent Wietse sugestion):

                slow unix - - - - - smtp
                -o syslog_name=postfix-slow
                -o default_destination_rate_delay=1s
                -o default_destination_recipient_limit=20
                -o smtp_connection_cache_on_demand=no

                As usual we will be reporting here the results.


                Thank you for your time and good will.


                M.
              • Wietse Venema
                ... THAT DOES NOT WORK. Please follow the instructions. As documented, the following parameters: xxx_destination_rate_delay xxx_destination_recipient_limit are
                Message 7 of 16 , Apr 2, 2011
                • 0 Attachment
                  Mark Alan:
                  > We will now try the following transport settings (based on a
                  > recent Wietse sugestion):
                  >
                  > slow unix - - - - - smtp
                  > -o syslog_name=postfix-slow
                  > -o default_destination_rate_delay=1s
                  > -o default_destination_recipient_limit=20
                  > -o smtp_connection_cache_on_demand=no

                  THAT DOES NOT WORK.

                  Please follow the instructions.

                  As documented, the following parameters:

                  xxx_destination_rate_delay
                  xxx_destination_recipient_limit

                  are implemented by the QUEUE MANAGER not SMTP CLIENT.

                  See my earlier email for what parameters go into main.cf and what
                  go into master.cf.

                  Wietse
                • Mark Alan
                  On Sat, 2 Apr 2011 18:03:29 -0400 (EDT), Wietse Venema ... I keep forgetting the inner workings of the multiple inner postfix modules. So here it is the
                  Message 8 of 16 , Apr 3, 2011
                  • 0 Attachment
                    On Sat, 2 Apr 2011 18:03:29 -0400 (EDT), Wietse Venema
                    <wietse@...> wrote:

                    > > slow unix - - - - - smtp
                    > > -o syslog_name=postfix-slow
                    > > -o default_destination_rate_delay=1s
                    > > -o default_destination_recipient_limit=20
                    > > -o smtp_connection_cache_on_demand=no
                    > THAT DOES NOT WORK.
                    > Please follow the instructions.
                    > As documented, the following parameters:
                    > xxx_destination_rate_delay
                    > xxx_destination_recipient_limit
                    > are implemented by the QUEUE MANAGER not SMTP CLIENT.

                    I keep forgetting the inner workings of the multiple inner postfix
                    modules.

                    So here it is the revised setting:

                    /etc/postfix/master.cf
                    slow unix - - - - - smtp
                    -o syslog_name=postfix-slow
                    -o smtp_connection_cache_on_demand=no

                    /etc/postfix/main.cf
                    slow_destination_rate_delay = 1s
                    postconf -e 'slow_destination_recipient_limit = 20


                    I will be reporting the results.

                    Thank you very much for your guidance.

                    M.
                  • Mark Alan
                    On Sat, 2 Apr 2011 18:03:29 -0400 (EDT), Wietse Venema ... I keep forgetting the inner workings of the multiple inner postfix modules. So here it is the
                    Message 9 of 16 , Apr 3, 2011
                    • 0 Attachment
                      On Sat, 2 Apr 2011 18:03:29 -0400 (EDT), Wietse Venema
                      <wietse@...> wrote:
                      > > slow unix - - - - - smtp
                      > > -o syslog_name=postfix-slow
                      > > -o default_destination_rate_delay=1s
                      > > -o default_destination_recipient_limit=20
                      > > -o smtp_connection_cache_on_demand=no
                      > THAT DOES NOT WORK.
                      > Please follow the instructions.
                      > As documented, the following parameters:
                      > xxx_destination_rate_delay
                      > xxx_destination_recipient_limit
                      > are implemented by the QUEUE MANAGER not SMTP CLIENT.

                      I keep forgetting the inner workings of the multiple inner postfix
                      modules.

                      So here it is the revised setting:

                      /etc/postfix/master.cf
                      slow unix - - - - - smtp
                      -o syslog_name=postfix-slow
                      -o smtp_connection_cache_on_demand=no

                      /etc/postfix/main.cf
                      slow_destination_rate_delay = 1s
                      slow_destination_recipient_limit = 20


                      I will be reporting the results.

                      Thank you very much for your guidance.

                      M.
                    • Victor Duchovni
                      ... The number of 421 responses is not a good metric for success. The real metric is the resulting average time in the queue for messages that are finally
                      Message 10 of 16 , Apr 3, 2011
                      • 0 Attachment
                        On Sat, Apr 02, 2011 at 06:59:42PM +0100, Mark Alan wrote:

                        > > > /etc/postfix/master.cf
                        > > > slow unix - - - - - smtp
                        > > > -o syslog_name=postfix-slow
                        > > > -o smtp_connection_reuse_time_limit=30s
                        > > > EOT
                        > > >
                        > > > /etc/postfix/main.cf
                        > > > slow_initial_destination_concurrency = 2
                        > > > slow_destination_concurrency_limit = 15
                        > > > slow_destination_concurrency_failed_cohort_limit = 5
                        > > > slow_destination_concurrency_positive_feedback = 1/5
                        > > > slow_destination_concurrency_negative_feedback = 1/8
                        >
                        > > (...) You can certainly try, and report your findings.
                        >
                        > Tried the above setup. It does not help.
                        > We have much more 421 with this approach than we had with our former
                        > setup.

                        The number of 421 responses is not a good metric for success. The real
                        metric is the resulting average time in the queue for messages that are
                        finally delivered.

                        A dymamic feedback mechanism that is able to go faster, will necessarily
                        elicit 421 responses at a stead rate, to keep it from reaching peak
                        capacity. Naturally, a statically hand-tuned configuration that stays
                        under the remote caps will not elicit 421 responses, but it is likely
                        to stay below the optimal throughput.

                        With sites whose 421 responses are NOT "sticky", the feedback controls
                        will in a variety of cases lead to a nearly optimal throughput with
                        some messages retried at a second MX or deferred, but the overall
                        delay can be smaller than conservative upper-bounds on concurrency
                        and message rates.

                        --
                        Viktor.
                      Your message has been successfully submitted and would be delivered to recipients shortly.