Loading ...
Sorry, an error occurred while loading the content.

dealing with Yahoo slowness

Expand Messages
  • Florin Andrei
    There seems to be a massive difference between the speed of various providers, in terms of accepting messages for delivery. Yahoo stands out as, by far, the
    Message 1 of 26 , Jun 10, 2010
    • 0 Attachment
      There seems to be a massive difference between the speed of various
      providers, in terms of accepting messages for delivery. Yahoo stands out
      as, by far, the slowest of the big ones.

      Because the messages are legitimate, we signed up for the email feedback
      loop with Yahoo, so that messages flagged as spam by Yahoo users are
      reported back to us, so we can silence notifications for those accounts.
      That didn't seem to help.

      Messages @... just accumulate in the deferred queue and stay there
      for a long time. When they do get kicked back into the active queue,
      it's just a trickle.

      Everyone else's emails are delivered very quickly. It might be my
      imagination, but it seems some providers do accept messages more quickly
      if you use the email feedback loop.

      Anyway, after a while, we end up with a bunch of @... messages in
      the queue, some domain typos, and not much else besides.

      One of the tricks some people seem to use is creating a dedicated
      transport for the slow destination. I'm reading the tuning and qshape
      README documents, and there are a lot of good suggestions there, but I
      was wondering what are the solutions that are being used *now*
      specifically for dealing with Yahoo.

      Thanks.

      --
      Florin Andrei
      http://florin.myip.org/
    • mouss
      ... it looks like yahoo throttle you. - if you send few mail, do nothing. just wait and things will hopefully get better. - if on the other hand you send a lot
      Message 2 of 26 , Jun 10, 2010
      • 0 Attachment
        Florin Andrei a écrit :
        > There seems to be a massive difference between the speed of various
        > providers, in terms of accepting messages for delivery. Yahoo stands out
        > as, by far, the slowest of the big ones.
        >
        > Because the messages are legitimate, we signed up for the email feedback
        > loop with Yahoo, so that messages flagged as spam by Yahoo users are
        > reported back to us, so we can silence notifications for those accounts.
        > That didn't seem to help.
        >
        > Messages @... just accumulate in the deferred queue and stay there
        > for a long time. When they do get kicked back into the active queue,
        > it's just a trickle.
        >
        > Everyone else's emails are delivered very quickly. It might be my
        > imagination, but it seems some providers do accept messages more quickly
        > if you use the email feedback loop.
        >
        > Anyway, after a while, we end up with a bunch of @... messages in
        > the queue, some domain typos, and not much else besides.
        >
        > One of the tricks some people seem to use is creating a dedicated
        > transport for the slow destination. I'm reading the tuning and qshape
        > README documents, and there are a lot of good suggestions there, but I
        > was wondering what are the solutions that are being used *now*
        > specifically for dealing with Yahoo.
        >

        it looks like yahoo throttle you.
        - if you send few mail, do nothing. just wait and things will hopefully
        get better.

        - if on the other hand you send a lot of mail, then you need to get in
        touch with yahoo. there's nothing we can do to help you.

        one thing you can do is to ask your recipients to explicitely tag your
        mail as "not spam" if it was ever tagged as spam. or they can reply (of
        course, this doesn't work for silly noreply mail. [sigh, I just got an
        internal one with a URL that doesn't work, and I don't know whom to
        contact...]).
      • Mike Hutchinson
        ... [Michael Hutchinson] Only OK if you re sending one or two E-Mails. ... [Michael Hutchinson] This also does very little. Yahoo don t list a Network
        Message 3 of 26 , Jun 10, 2010
        • 0 Attachment
          > -----Original Message-----
          > Florin Andrei a écrit :
          > > There seems to be a massive difference between the speed of various
          > > providers, in terms of accepting messages for delivery. Yahoo stands out
          > > as, by far, the slowest of the big ones.
          > >
          > > Because the messages are legitimate, we signed up for the email feedback
          > > loop with Yahoo, so that messages flagged as spam by Yahoo users are
          > > reported back to us, so we can silence notifications for those accounts.
          > > That didn't seem to help.
          > >
          > > Messages @... just accumulate in the deferred queue and stay
          > there
          > > for a long time. When they do get kicked back into the active queue,
          > > it's just a trickle.
          > >
          > > Everyone else's emails are delivered very quickly. It might be my
          > > imagination, but it seems some providers do accept messages more
          > quickly
          > > if you use the email feedback loop.
          > >
          > > Anyway, after a while, we end up with a bunch of @... messages
          > in
          > > the queue, some domain typos, and not much else besides.
          > >
          > > One of the tricks some people seem to use is creating a dedicated
          > > transport for the slow destination. I'm reading the tuning and qshape
          > > README documents, and there are a lot of good suggestions there, but I
          > > was wondering what are the solutions that are being used *now*
          > > specifically for dealing with Yahoo.
          > >
          >
          > it looks like yahoo throttle you.
          > - if you send few mail, do nothing. just wait and things will hopefully
          > get better.

          [Michael Hutchinson]
          Only OK if you're sending one or two E-Mails.

          > - if on the other hand you send a lot of mail, then you need to get in
          > touch with yahoo. there's nothing we can do to help you.

          [Michael Hutchinson]
          This also does very little. Yahoo don't list a Network Operations Centre and
          are about the hardest company to deal with when it comes to E-Mail delivery
          resolution. I've filled out dozens of forms to get particular IP's
          whitelisted for specific domains on their Mail systems, with no reply, let
          alone result.

          > one thing you can do is to ask your recipients to explicitely tag your
          > mail as "not spam" if it was ever tagged as spam. or they can reply (of
          > course, this doesn't work for silly noreply mail. [sigh, I just got an
          > internal one with a URL that doesn't work, and I don't know whom to
          > contact...]).

          [Michael Hutchinson]
          Unfortunately it only takes ONE user to click on "This is Spam" and you're
          back to square one. (and being IP banned is Square one).

          [Michael Hutchinson]
          We have had this exact problem, delivering Retail newsletters to people who
          have opted in for it. A lot of them are on Gmail and Yahoo, and this can be
          difficult with Bulk E-Mail. Despite contact with Google themselves and
          signing up for all of their reporting services regarding Spammy Emails and
          Certified Senders, the best result we've had is to use some Postfix
          configuration to resolve the issue. It does this by gently delivering the
          E-Mails at an acceptable rate (discovered with a LOT of testing and a LOT of
          IP bans (good they're not permanent, huh :). In our environment, on our
          servers, this has resolved the issue, and delivers mail to those domains a
          LOT faster than not performing the config on Postfix. In fact, if we don’t
          configure, we get banned straight away against those domains and cannot
          deliver for several hours afterwards.

          We setup the postfix transport file with these entries:
          # destination domains that need to be rate limited
          hotmail.com hotmail:
          msn.com hotmail:
          live.com hotmail:
          windowslive.com hotmail:
          yahoo.com.ar yahoo:
          yahoo.com.au yahoo:
          yahoo.com.br yahoo:
          yahoo.ca yahoo:
          yahoo.com.cn yahoo:
          <snip> - there's more but you get the idea.

          Then we setup master.cf:

          yahoo unix - - - - - smtp
          hotmail unix - - - - - smtp

          Then setup main.cf:

          # Slow these destinations to avoid blacklisting, see /etc/postfix/transport
          for domains configured
          hotmail_destination_concurrency_limit = 2
          hotmail_destination_rate_delay = 2s
          hotmail_destination_recipient_limit = 5
          yahoo_destination_concurrency_limit = 4
          yahoo_destination_rate_delay = 1s
          yahoo_destination_recipient_limit = 5

          These settings can be tweaked depending on what server you're talking to.
          However, these values work for us, after having dealt with not getting
          10,000 mails out per week.

          I hope this helps.

          Cheers,
          Michael Hutchinson
          mhutchinson@...
        • Wietse Venema
          ... Interesting. Really. FYI This should be documented better: Postfix s _rate_delay feature forces a per-destination delivery concurrency of 1, so you could
          Message 4 of 26 , Jun 10, 2010
          • 0 Attachment
            Mike Hutchinson:
            > We have had this exact problem, delivering Retail newsletters to people who
            > have opted in for it. A lot of them are on Gmail and Yahoo, and this can be
            > difficult with Bulk E-Mail. Despite contact with Google themselves and
            > signing up for all of their reporting services regarding Spammy Emails and
            > Certified Senders, the best result we've had is to use some Postfix
            > configuration to resolve the issue. It does this by gently delivering the
            > E-Mails at an acceptable rate (discovered with a LOT of testing and a LOT of
            > IP bans (good they're not permanent, huh :). In our environment, on our
            > servers, this has resolved the issue, and delivers mail to those domains a
            > LOT faster than not performing the config on Postfix. In fact, if we don?t
            > configure, we get banned straight away against those domains and cannot
            > deliver for several hours afterwards.
            >
            > We setup the postfix transport file with these entries:
            > # destination domains that need to be rate limited
            > hotmail.com hotmail:
            > msn.com hotmail:
            > live.com hotmail:
            > windowslive.com hotmail:
            > yahoo.com.ar yahoo:
            > yahoo.com.au yahoo:
            > yahoo.com.br yahoo:
            > yahoo.ca yahoo:
            > yahoo.com.cn yahoo:
            > <snip> - there's more but you get the idea.
            >
            > Then we setup master.cf:
            >
            > yahoo unix - - - - - smtp
            > hotmail unix - - - - - smtp
            >
            > Then setup main.cf:
            >
            > # Slow these destinations to avoid blacklisting, see /etc/postfix/transport
            > for domains configured
            > hotmail_destination_concurrency_limit = 2
            > hotmail_destination_rate_delay = 2s
            > hotmail_destination_recipient_limit = 5
            > yahoo_destination_concurrency_limit = 4
            > yahoo_destination_rate_delay = 1s
            > yahoo_destination_recipient_limit = 5
            >
            > These settings can be tweaked depending on what server you're talking to.
            > However, these values work for us, after having dealt with not getting
            > 10,000 mails out per week.
            >
            > I hope this helps.

            Interesting. Really.

            FYI This should be documented better: Postfix's _rate_delay feature
            forces a per-destination delivery concurrency of 1, so you could
            drop the _destination_concurrency_limit settings. The Postfix
            implementation is utterly simple: schedule one delivery, then
            suspend delivery for N second, then schedule the next delivery.

            Wietse
          • Mike Hutchinson
            ... /etc/postfix/transport ... to. ... [Michael Hutchinson] It is. I don t know of any other software that even comes close to supporting rate limiting for
            Message 5 of 26 , Jun 10, 2010
            • 0 Attachment
              > > # Slow these destinations to avoid blacklisting, see
              /etc/postfix/transport
              > > for domains configured
              > > hotmail_destination_concurrency_limit = 2
              > > hotmail_destination_rate_delay = 2s
              > > hotmail_destination_recipient_limit = 5
              > > yahoo_destination_concurrency_limit = 4
              > > yahoo_destination_rate_delay = 1s
              > > yahoo_destination_recipient_limit = 5
              > >
              > > These settings can be tweaked depending on what server you're talking
              to.
              > > However, these values work for us, after having dealt with not getting
              > > 10,000 mails out per week.
              > >
              > > I hope this helps.
              >
              > Interesting. Really.

              [Michael Hutchinson]
              It is. I don't know of any other software that even comes close to
              supporting rate limiting for outgoing E-Mail. As I understand it, even
              companies that package Postfix in their NetApps are taking advantage of
              these features now.

              > FYI This should be documented better: Postfix's _rate_delay feature
              > forces a per-destination delivery concurrency of 1, so you could
              > drop the _destination_concurrency_limit settings. The Postfix
              > implementation is utterly simple: schedule one delivery, then
              > suspend delivery for N second, then schedule the next delivery.

              [Michael Hutchinson]
              I had thought, whilst I was writing the E-Mail, that this could deserve a
              howto or manual section, perhaps briefly describing a general situation that
              would reflect the real world problem of delivery of E-Mail to servers like
              Yahoo/Google, and how postfix can be configured to react in a more
              Yahoo/Google restricted manner. What assistance can I provide ?

              Cheers,
              Michael Hutchinson
              mhutchinson@...
            • Simon Waters
              ... We don t treat Yahoo! any differently here, so essentially we delivered using Postfix defaults. We have a fragile queue for difficult providers but only
              Message 6 of 26 , Jun 11, 2010
              • 0 Attachment
                On Thursday 10 June 2010 19:51:51 Florin Andrei wrote:
                >
                > One of the tricks some people seem to use is creating a dedicated
                > transport for the slow destination. I'm reading the tuning and qshape
                > README documents, and there are a lot of good suggestions there, but I
                > was wondering what are the solutions that are being used *now*
                > specifically for dealing with Yahoo.

                We don't treat Yahoo! any differently here, so essentially we delivered using
                Postfix defaults.

                We have a "fragile" queue for difficult providers but only Microsoft domains
                are listed.

                Whilst I've seen comment that Yahoo! throttle, I had some logs that suggest
                Yahoo! also can be overloaded at times (hardly surprising given some of the
                botnets out there).

                As such afraid I tend to the view that if Yahoo! email is delayed it is
                largely a Yahoo! problem. Although we've not had any complaints of such in
                recent months (years?), and BT Internet use Yahoo so we ship them quite a
                significant proportion of our outbound email. Most of the feedback we got was
                when BT switched to using Yahoo, so I assume teething problems.
              • M. Fioretti
                On Fri, Jun 11, 2010 13:48:24 PM +1200, Mike Hutchinson ... I would be quite interested to read such a howto. I also happen to publish FOSS related tips and
                Message 7 of 26 , Jun 11, 2010
                • 0 Attachment
                  On Fri, Jun 11, 2010 13:48:24 PM +1200, Mike Hutchinson
                  (packetloss@...) wrote:

                  > I had thought, whilst I was writing the E-Mail, that this could
                  > deserve a howto or manual section...

                  I would be quite interested to read such a howto. I also happen to
                  publish FOSS related tips and tricks, all with CC/GPL licenses, at
                  http://freesoftware.zona-m.net I'd like to properly reformat all this
                  discussion in one short tutorial to publish there, so if anybody has
                  other tips and advice, thanks in advance for sending them here or to
                  me directly, as you prefer.

                  HOWEVER, I surely can't work on this in the next two weeks, so if
                  anybody else feels like doing the same on their own blog, by all means
                  go ahead and post the link.

                  Thanks!
                  Marco F.
                • Wietse Venema
                  ... I ll encourage you and will link to, but won t publish or update myself. The reason is that I am not in the email sending business and therefore I cannot
                  Message 8 of 26 , Jun 11, 2010
                  • 0 Attachment
                    Mike Hutchinson:
                    > I had thought, whilst I was writing the E-Mail, that this could deserve a
                    > howto or manual section, perhaps briefly describing a general situation that
                    > would reflect the real world problem of delivery of E-Mail to servers like
                    > Yahoo/Google, and how postfix can be configured to react in a more
                    > Yahoo/Google restricted manner. What assistance can I provide ?

                    I'll encourage you and will link to, but won't publish or update
                    myself. The reason is that I am not in the email sending business
                    and therefore I cannot judge whether a given strategy needs to be
                    updated or whether it can be generalized.

                    Wietse
                  • Florin Andrei
                    ... Well, we do that already (concurrency = 2, rate_delay = 2s). It s still slow. Do you use multiple outbound email gateways? Maybe I should try to increase
                    Message 9 of 26 , Jun 14, 2010
                    • 0 Attachment
                      On 06/10/2010 05:09 PM, Mike Hutchinson wrote:
                      > yahoo_destination_concurrency_limit = 4
                      > yahoo_destination_rate_delay = 1s

                      Well, we do that already (concurrency = 2, rate_delay = 2s). It's still
                      slow. Do you use multiple outbound email gateways?

                      Maybe I should try to increase our existing parameters, it looks like
                      we're using half your values.

                      Looking at the Postfix queue graphs in Munin, one thing I noticed is
                      that when the scheduled emails go out (it's not a continuous trickle,
                      it's in batches, that's just how the software works), a fraction, maybe
                      25%, go into the active queue right away, the rest seem to be dropped
                      into deferred either immediately or very quickly. Then they stay in
                      deferred a long time. Then they move to active for a short while, being
                      delivered at a slow rate, then they fall back into deferred.

                      So not only is delivery slow, but the messages spend a very long time in
                      deferred, therefore amplifying the issue. They seem to spend more time
                      in deferred than in active.

                      Again, these are 95% @... messages which are legitimate, go to
                      valid addresses, but are just being throttled by Yahoo. By my lights,
                      these should spend no time in deferred.

                      > yahoo_destination_recipient_limit = 5

                      We send exactly one message per destination email address, so that
                      setting would be redundant for us.

                      > These settings can be tweaked depending on what server you're talking to.
                      > However, these values work for us, after having dealt with not getting
                      > 10,000 mails out per week.

                      That's pretty much our order of magnitude.

                      So, roughly, what's the delivery rate you managed to obtain? How many
                      messages / second delivered to Yahoo? It seems like our rate is much
                      lower, for some reason.

                      P.S.: We're using postfix-2.3.3-2.1.el5_2 that comes with Red Hat 5. I'm
                      playing with 2.7 packages and I'm contemplating an upgrade. Would there
                      be any inherent benefits in my case by simply doing the upgrade? I
                      noticed there are new control parameters such as
                      destination_concurrency_failed_cohort_limit but I'm not sure how big of
                      a difference they would make.

                      --
                      Florin Andrei
                      http://florin.myip.org/
                    • Wietse Venema
                      ... That is two Postfix versions before _rate_delay was introduced. You may want to upgrade to Postfix 2.5 or later. If you can throttle down Postfix so it
                      Message 10 of 26 , Jun 14, 2010
                      • 0 Attachment
                        Florin Andrei:
                        > P.S.: We're using postfix-2.3.3-2.1.el5_2 that comes with Red Hat 5. I'm

                        That is two Postfix versions before _rate_delay was introduced.
                        You may want to upgrade to Postfix 2.5 or later.

                        If you can throttle down Postfix so it never hits Yahoo's limits,
                        then it will always be better than having Yahoo throttle your mail
                        and having mail end up in the deferred queue repeatedly.

                        Wietse
                      • Florin Andrei
                        ... Aw great. :( Sometimes Red Hat s conservative approach is just too much. It s good in other areas, but obviously they jumped the gun with Postfix in this
                        Message 11 of 26 , Jun 14, 2010
                        • 0 Attachment
                          On 06/14/2010 11:13 AM, Wietse Venema wrote:
                          > Florin Andrei:
                          >> P.S.: We're using postfix-2.3.3-2.1.el5_2 that comes with Red Hat 5. I'm
                          >
                          > That is two Postfix versions before _rate_delay was introduced.
                          > You may want to upgrade to Postfix 2.5 or later.

                          Aw great. :( Sometimes Red Hat's conservative approach is just too much.
                          It's good in other areas, but obviously they jumped the gun with Postfix
                          in this case.

                          Well, that does it. I got RPM packages with 2.7 from two different
                          sources. Time for testing, then upgrade, and I'll keep y'all posted with
                          the results.

                          Thanks for pointing it out.

                          --
                          Florin Andrei
                          http://florin.myip.org/
                        • Florin Andrei
                          ... And here it is, the status update. I got the 2.7.0 src.rpm packages made by Simon J Mudd http://ftp.wl0.org/official/ and rebuilt them on my devel
                          Message 12 of 26 , Jun 18, 2010
                          • 0 Attachment
                            On 06/14/2010 11:54 AM, Florin Andrei wrote:
                            >
                            > Well, that does it. I got RPM packages with 2.7 from two different
                            > sources. Time for testing, then upgrade, and I'll keep y'all posted with
                            > the results.

                            And here it is, the status update.

                            I got the 2.7.0 src.rpm packages made by Simon J Mudd
                            http://ftp.wl0.org/official/ and rebuilt them on my devel instance.
                            Installed them on the gateways and migrated the old custom settings.

                            Just this upgrade, alone, from 2.3.3 (the default version that comes
                            with Red Hat 5) to 2.7.0 increased our delivery success to Yahoo by, I
                            guess, 50%. Now a lot more messages get delivered before we have to
                            purge the queue because it's too late. Still not perfect, but better.

                            Related and relevant non-default settings:

                            master.cf:
                            fragile unix - - n - 5 smtp
                            -o smtp_helo_timeout=5 -o smtp_connect_timeout=5

                            main.cf:
                            transport_maps = hash:/etc/postfix/transport
                            fragile_destination_concurrency_limit = 2
                            fragile_destination_concurrency_failed_cohort_limit = 1
                            fragile_destination_rate_delay = 2s

                            transport:
                            yahoo.com fragile:
                            # bunch of other Yahoo domains here
                            hotmail.com fragile:
                            comcast.net fragile:

                            Before the upgrade, delivery to Yahoo would get interrupted for two
                            reasons: Yahoo getting annoyed by our delivery rate and stopped
                            accepting messages, and users clicking the Spam button (unjustified,
                            IMO, but that's another story).
                            After the upgrade, it appears that the only reason why Yahoo stops
                            accepting messages (it still does, even now) is users clicking Spam.

                            Things I'm planning to try from now on:

                            1. Use slightly more aggressive *_destination_* settings, as indicated
                            by Mike Hutchinson earlier in this thread.

                            2. Separate the various finicky destinations into their own pathways in
                            master.cf, instead of lumping them together under "fragile". Perhaps
                            even dump some of them back into general delivery, since only Yahoo
                            seems to really cause trouble.

                            3. Use two outbound email gateways instead of one. This might double the
                            delivery rate "for free".

                            4. Upgrade to 2.7.1, try to customize other parameters, etc.

                            Suggestions are welcome.

                            I'll keep you posted if I find anything new and notable.

                            --
                            Florin Andrei
                            http://florin.myip.org/
                          • Victor Duchovni
                            ... Try: # Change from 1 above fragile_destination_concurrency_failed_cohort_limit = 5 # New, more stable feedback controls from 2.5
                            Message 13 of 26 , Jun 18, 2010
                            • 0 Attachment
                              On Fri, Jun 18, 2010 at 02:05:36PM -0700, Florin Andrei wrote:

                              > main.cf:
                              > transport_maps = hash:/etc/postfix/transport
                              > fragile_destination_concurrency_limit = 2
                              > fragile_destination_concurrency_failed_cohort_limit = 1
                              > fragile_destination_rate_delay = 2s

                              Try:

                              # Change from 1 above
                              fragile_destination_concurrency_failed_cohort_limit = 5
                              # New, more stable feedback controls from 2.5
                              fragile_destination_concurrency_positive_feedback = 1/3
                              fragile_destination_concurrency_negative_feedback = 1/8

                              I think that will significantly reduce the rate at which delivery is
                              unnecessarily throttled.

                              --
                              Viktor.
                            • Stefan Foerster
                              ... I might be wrong here, but what you describe - the rest seem to be dropped into deferred either immediately or very quickly - sounds a little like the
                              Message 14 of 26 , Jun 19, 2010
                              • 0 Attachment
                                * Florin Andrei <florin@...>:
                                > Looking at the Postfix queue graphs in Munin, one thing I noticed is
                                > that when the scheduled emails go out (it's not a continuous
                                > trickle, it's in batches, that's just how the software works), a
                                > fraction, maybe 25%, go into the active queue right away, the rest
                                > seem to be dropped into deferred either immediately or very quickly.
                                > Then they stay in deferred a long time. Then they move to active for
                                > a short while, being delivered at a slow rate, then they fall back
                                > into deferred.

                                I might be wrong here, but what you describe - "the rest seem to be
                                dropped into deferred either immediately or very quickly" - sounds a
                                little like the "use case" described in

                                http://www.postfix.org/QSHAPE_README.html#backlog

                                The deferrals on Yahoo's part might cause qmgr(8) to mark that
                                destinations as "dead" (so we can't know that for sure, since you
                                didn't post the rejection logs, and the
                                _destination_concurrency_failed_cohort_limit only applies to
                                connection and handshake failures).

                                I don't send any large volumes to Yahoo, but I had to use a dedicated
                                transport which ignored much more errors for a popular German freemail
                                provider. Since you are using rate delays, your concurrency limit will
                                basically be one, and this might very well be related to what you see.

                                I don't know if you need to reload postfix and/or requeue the messages
                                with "postsuper -r" after changing
                                transport_destination_concurrency_failed_cohort_limit.


                                Stefan
                              • Wietse Venema
                                ... This is a good point. To compensate for this unwanted side effect of reduced concurrency INCREASE the fragile_destination_concurrency_failed_cohort_limit
                                Message 15 of 26 , Jun 19, 2010
                                • 0 Attachment
                                  Stefan Foerster:
                                  > I don't send any large volumes to Yahoo, but I had to use a dedicated
                                  > transport which ignored much more errors for a popular German freemail
                                  > provider. Since you are using rate delays, your concurrency limit will
                                  > basically be one, and this might very well be related to what you see.

                                  This is a good point.

                                  To compensate for this unwanted side effect of reduced concurrency
                                  INCREASE the fragile_destination_concurrency_failed_cohort_limit
                                  to 10-20 or so (or REDUCE fragile_destination_concurrency_negative_feedback
                                  to 1/10 or 1/20).

                                  > I don't know if you need to reload postfix and/or requeue the messages
                                  > with "postsuper -r" after changing
                                  > transport_destination_concurrency_failed_cohort_limit.

                                  No. Use "postfix reload" to update the queue manager.

                                  Wietse
                                • Wietse Venema
                                  ... I just did a few experiments to confirm this. With the default fragile_destination_concurrency_failed_cohort_limit, the scheduler defers all mail after one
                                  Message 16 of 26 , Jun 19, 2010
                                  • 0 Attachment
                                    Wietse Venema:
                                    > Stefan Foerster:
                                    > > I don't send any large volumes to Yahoo, but I had to use a dedicated
                                    > > transport which ignored much more errors for a popular German freemail
                                    > > provider. Since you are using rate delays, your concurrency limit will
                                    > > basically be one, and this might very well be related to what you see.
                                    >
                                    > This is a good point.
                                    >
                                    > To compensate for this unwanted side effect of reduced concurrency
                                    > INCREASE the fragile_destination_concurrency_failed_cohort_limit
                                    > to 10-20 or so (or REDUCE fragile_destination_concurrency_negative_feedback
                                    > to 1/10 or 1/20).

                                    I just did a few experiments to confirm this.

                                    With the default fragile_destination_concurrency_failed_cohort_limit,
                                    the scheduler defers all mail after one connection/handshake failure.

                                    With fragile_destination_concurrency_failed_cohort_limit > 1 the
                                    scheduler defers all mail after multiple connection/handshake
                                    failures in a row, which may be more desirable in the Yahoo scenario.

                                    For now, I'll add a note to the documentation. A clever software
                                    solution is not obvious - when a home office user needs to limit
                                    their output rate, it may not be a good idea to keep sending mail
                                    after the ISP starts pushing back.

                                    > > I don't know if you need to reload postfix and/or requeue the messages
                                    > > with "postsuper -r" after changing
                                    > > transport_destination_concurrency_failed_cohort_limit.
                                    >
                                    > No. Use "postfix reload" to update the queue manager.
                                    >
                                    > Wietse
                                    >
                                    >
                                  • Florin Andrei
                                    ... What happens if both are used with more moderate values? E.g., I separated a few big destinations into their own delivery paths and now this is what I
                                    Message 17 of 26 , Jun 21, 2010
                                    • 0 Attachment
                                      On 06/19/2010 05:37 AM, Wietse Venema wrote:
                                      > Stefan Foerster:
                                      >> I don't send any large volumes to Yahoo, but I had to use a dedicated
                                      >> transport which ignored much more errors for a popular German freemail
                                      >> provider. Since you are using rate delays, your concurrency limit will
                                      >> basically be one, and this might very well be related to what you see.
                                      >
                                      > This is a good point.
                                      >
                                      > To compensate for this unwanted side effect of reduced concurrency
                                      > INCREASE the fragile_destination_concurrency_failed_cohort_limit
                                      > to 10-20 or so (or REDUCE fragile_destination_concurrency_negative_feedback
                                      > to 1/10 or 1/20).

                                      What happens if both are used with more moderate values?
                                      E.g., I separated a few big destinations into their own delivery paths
                                      and now this is what I have:

                                      yahoo_destination_concurrency_limit = 4
                                      yahoo_destination_concurrency_failed_cohort_limit = 5
                                      yahoo_destination_rate_delay = 1s
                                      yahoo_destination_concurrency_positive_feedback = 1/3
                                      yahoo_destination_concurrency_negative_feedback = 1/8

                                      I'll have another batch of emails to send pretty soon, I'm still
                                      tweaking parameters until then.

                                      --
                                      Florin Andrei
                                      http://florin.myip.org/
                                    • Victor Duchovni
                                      ... Negative feedback never decreates the concurrency below 1, so with a non-zero rate-delay, there is no negative feedback, only dead-site detection. The
                                      Message 18 of 26 , Jun 21, 2010
                                      • 0 Attachment
                                        On Mon, Jun 21, 2010 at 11:08:04AM -0700, Florin Andrei wrote:

                                        >> To compensate for this unwanted side effect of reduced concurrency
                                        >> INCREASE the fragile_destination_concurrency_failed_cohort_limit
                                        >> to 10-20 or so (or REDUCE
                                        >> fragile_destination_concurrency_negative_feedback
                                        >> to 1/10 or 1/20).
                                        >
                                        > What happens if both are used with more moderate values?
                                        > E.g., I separated a few big destinations into their own delivery paths and
                                        > now this is what I have:

                                        Negative feedback never decreates the concurrency below 1, so with
                                        a non-zero rate-delay, there is no negative feedback, only dead-site
                                        detection. The feedback tuning is appropriate when the rate delay is zero,
                                        and either the concurrency limit is low, or the destination triggers
                                        bursts of connection failures, delivery timeouts, disconnects, ...

                                        > yahoo_destination_concurrency_limit = 4
                                        > yahoo_destination_concurrency_failed_cohort_limit = 5
                                        > yahoo_destination_rate_delay = 1s
                                        > yahoo_destination_concurrency_positive_feedback = 1/3
                                        > yahoo_destination_concurrency_negative_feedback = 1/8

                                        With a rate delay of 1s, the positive/negative feedback parameters
                                        have no effect. Also the concurrency limit has no effect. Only the
                                        failed_cohort_limit is relevant. If you remove the rate delay, and your
                                        mail stream to Yahoo is not too bursty (not jumbo list mails), then
                                        the feedback controls should suffice to keep mail flowing smoothly. If
                                        you need to spread out bursts from large lists that arrive all at once,
                                        rate delay plus higher cohort limits are the only tools at hand.

                                        --
                                        Viktor.
                                      • Florin Andrei
                                        ... My email is very bursty - event updates and changes sent to many / most / all subscribers. So then I should do this, I guess:
                                        Message 19 of 26 , Jun 21, 2010
                                        • 0 Attachment
                                          On 06/21/2010 11:31 AM, Victor Duchovni wrote:
                                          > On Mon, Jun 21, 2010 at 11:08:04AM -0700, Florin Andrei wrote:
                                          >
                                          >> yahoo_destination_concurrency_limit = 4
                                          >> yahoo_destination_concurrency_failed_cohort_limit = 5
                                          >> yahoo_destination_rate_delay = 1s
                                          >> yahoo_destination_concurrency_positive_feedback = 1/3
                                          >> yahoo_destination_concurrency_negative_feedback = 1/8
                                          >
                                          > With a rate delay of 1s, the positive/negative feedback parameters
                                          > have no effect. Also the concurrency limit has no effect. Only the
                                          > failed_cohort_limit is relevant. If you remove the rate delay, and your
                                          > mail stream to Yahoo is not too bursty (not jumbo list mails), then
                                          > the feedback controls should suffice to keep mail flowing smoothly. If
                                          > you need to spread out bursts from large lists that arrive all at once,
                                          > rate delay plus higher cohort limits are the only tools at hand.

                                          My email is very "bursty" - event updates and changes sent to many /
                                          most / all subscribers. So then I should do this, I guess:

                                          yahoo_destination_concurrency_failed_cohort_limit = 20
                                          yahoo_destination_rate_delay = 1s

                                          I think that's exactly what you are suggesting. Hopefully Yahoo doesn't
                                          get annoyed by the high cohort limit, but we'll see.

                                          I can't say I understand *why* the 1s rate delay makes the feedback and
                                          the concurrency limit parameters irrelevant, so I guess it's time for me
                                          to dig deeper into the documentation. :)

                                          --
                                          Florin Andrei
                                          http://florin.myip.org/
                                        • Victor Duchovni
                                          ... Yahoo won t notice your cohort limit . You may not need 20, if deliveries always fail even 20 won t help. The idea is to ride out a short burst of errors,
                                          Message 20 of 26 , Jun 21, 2010
                                          • 0 Attachment
                                            On Mon, Jun 21, 2010 at 12:21:45PM -0700, Florin Andrei wrote:

                                            > My email is very "bursty" - event updates and changes sent to many / most /
                                            > all subscribers. So then I should do this, I guess:
                                            >
                                            > yahoo_destination_concurrency_failed_cohort_limit = 20
                                            > yahoo_destination_rate_delay = 1s
                                            >
                                            > I think that's exactly what you are suggesting. Hopefully Yahoo doesn't get
                                            > annoyed by the high cohort limit, but we'll see.

                                            Yahoo won't notice your "cohort limit". You may not need 20, if deliveries
                                            always fail even 20 won't help. The idea is to ride out a short burst
                                            of errors, when *some* deliveries are still succeeding. The size of
                                            the longest error burst that does not indicate that further attempts
                                            are futile is situation dependent, at a fixed non-zero success rate,
                                            increases much beyond the typical burst size have no value (geometrically
                                            decreasing gain after the odds of 1+ successes in N deliveries start to
                                            dominate the odds of no successes).

                                            > I can't say I understand *why* the 1s rate delay makes the feedback and the
                                            > concurrency limit parameters irrelevant, so I guess it's time for me to dig
                                            > deeper into the documentation. :)

                                            With a rate limit, there is no dynamic concurrency tuning. The concurrency
                                            is always equal to 1 (or zero once the destination is throttled, after
                                            appropriately many consecutive failures).

                                            --
                                            Viktor.
                                          • Wietse Venema
                                            ... This takes some courage: http://www.postfix.org/SCHEDULER_README.html The concurrency of 1 is not documented, but it is the only practical way to implement
                                            Message 21 of 26 , Jun 21, 2010
                                            • 0 Attachment
                                              Florin Andrei:
                                              > I can't say I understand *why* the 1s rate delay makes the feedback and
                                              > the concurrency limit parameters irrelevant, so I guess it's time for me
                                              > to dig deeper into the documentation. :)

                                              This takes some courage:
                                              http://www.postfix.org/SCHEDULER_README.html

                                              The concurrency of 1 is not documented, but it is the only practical
                                              way to implement this. A rate delay of 1s doesn't have much effect
                                              when you delay on one connection while 99 other connections are in
                                              the middle delivering an email message to the same site.

                                              Wietse
                                            • Florin Andrei
                                              ... Would this analysis be influenced in any way if we take into account the fact that we deliver exactly 1 message / user address? (no multiple recipients at
                                              Message 22 of 26 , Jun 21, 2010
                                              • 0 Attachment
                                                On 06/21/2010 12:42 PM, Victor Duchovni wrote:
                                                > On Mon, Jun 21, 2010 at 12:21:45PM -0700, Florin Andrei wrote:
                                                >
                                                >> yahoo_destination_concurrency_failed_cohort_limit = 20
                                                >> yahoo_destination_rate_delay = 1s
                                                >>
                                                >> I can't say I understand *why* the 1s rate delay makes the feedback and the
                                                >> concurrency limit parameters irrelevant, so I guess it's time for me to dig
                                                >> deeper into the documentation. :)
                                                >
                                                > With a rate limit, there is no dynamic concurrency tuning. The concurrency
                                                > is always equal to 1 (or zero once the destination is throttled, after
                                                > appropriately many consecutive failures).

                                                Would this analysis be influenced in any way if we take into account the
                                                fact that we deliver exactly 1 message / user address? (no multiple
                                                recipients at all)

                                                --
                                                Florin Andrei
                                                http://florin.myip.org/
                                              • Wietse Venema
                                                ... You are asking questions that are answered in the documentation. http://www.postfix.org/postconf.5.html#default_destination_rate_delay Please read this
                                                Message 23 of 26 , Jun 21, 2010
                                                • 0 Attachment
                                                  Florin Andrei:
                                                  > On 06/21/2010 12:42 PM, Victor Duchovni wrote:
                                                  > > On Mon, Jun 21, 2010 at 12:21:45PM -0700, Florin Andrei wrote:
                                                  > >
                                                  > >> yahoo_destination_concurrency_failed_cohort_limit = 20
                                                  > >> yahoo_destination_rate_delay = 1s
                                                  > >>
                                                  > >> I can't say I understand *why* the 1s rate delay makes the feedback and the
                                                  > >> concurrency limit parameters irrelevant, so I guess it's time for me to dig
                                                  > >> deeper into the documentation. :)
                                                  > >
                                                  > > With a rate limit, there is no dynamic concurrency tuning. The concurrency
                                                  > > is always equal to 1 (or zero once the destination is throttled, after
                                                  > > appropriately many consecutive failures).
                                                  >
                                                  > Would this analysis be influenced in any way if we take into account the
                                                  > fact that we deliver exactly 1 message / user address? (no multiple
                                                  > recipients at all)

                                                  You are asking questions that are answered in the documentation.

                                                  http://www.postfix.org/postconf.5.html#default_destination_rate_delay

                                                  Please read this text first. Pay particular attention to the
                                                  _destination_recipient_limit discussion. Then come back if
                                                  the text is not clear, and indicate *what* text is not clear.

                                                  Wietse
                                                • Mike Hutchinson
                                                  ... [Michael Hutchinson] made a late reply: Sounds like you ve run into the version problem I had some time ago, where the rate controls were present, but were
                                                  Message 24 of 26 , Jun 28, 2010
                                                  • 0 Attachment
                                                    > -----Original Message-----
                                                    > From: owner-postfix-users@... [mailto:owner-postfix-
                                                    > users@...] On Behalf Of Florin Andrei
                                                    > Sent: Tuesday, 15 June 2010 6:00 a.m.
                                                    > To: postfix-users@...
                                                    > Subject: Re: dealing with Yahoo slowness
                                                    >
                                                    > On 06/10/2010 05:09 PM, Mike Hutchinson wrote:
                                                    > > yahoo_destination_concurrency_limit = 4
                                                    > > yahoo_destination_rate_delay = 1s
                                                    >
                                                    > Well, we do that already (concurrency = 2, rate_delay = 2s). It's still
                                                    > slow. Do you use multiple outbound email gateways?
                                                    >
                                                    > Maybe I should try to increase our existing parameters, it looks like
                                                    > we're using half your values.

                                                    [Michael Hutchinson] made a late reply:
                                                    Sounds like you've run into the version problem I had some time ago, where
                                                    the rate controls were present, but were a bit buggy. See Wietse's post if
                                                    you haven't already.

                                                    Once we'd performed the upgrade, and applied the rate limiting configuration
                                                    everything went smoothly - perhaps try the same values from the original
                                                    post and work from there.

                                                    Cheers,
                                                    Michael.
                                                  • Florin Andrei
                                                    ... More info. This is how the queues always look, it s a very typical batch: http://i.imgur.com/7MPIx.png The batch is sent out at midnight (not always at
                                                    Message 25 of 26 , Jun 30, 2010
                                                    • 0 Attachment
                                                      On 06/28/2010 03:20 PM, Mike Hutchinson wrote:
                                                      >
                                                      > Once we'd performed the upgrade, and applied the rate limiting configuration
                                                      > everything went smoothly - perhaps try the same values from the original
                                                      > post and work from there.

                                                      More info. This is how the queues always look, it's a very typical batch:

                                                      http://i.imgur.com/7MPIx.png

                                                      The batch is sent out at midnight (not always at midnight, just in this
                                                      case). The active queue is dropping at a pretty good slope - that's
                                                      everyone except Yahoo.

                                                      Then there's a green bump at 3am and the active queue is loaded with
                                                      more mails, but this time it's decreasing more slowly. This is when
                                                      basically there's only @... email in the queue.

                                                      Then it gets thrown in deferred at 4:30am, then briefly in active, then
                                                      deferred again, repeat. It's over 99% @... at that point.

                                                      This is with minimal_backoff_time = 1000 and maximal_backoff_time =
                                                      2000. I'll try 500 and 1000 instead, maybe that makes the blue bumps
                                                      more narrow.

                                                      --
                                                      Florin Andrei
                                                      http://florin.myip.org/
                                                    • Victor Duchovni
                                                      ... This graph has no scale, and would not be very interesting in any case. Have you made attempt to sign-up for Yahoo s feedback loop, get whitelisted, etc.?
                                                      Message 26 of 26 , Jul 1 2:34 AM
                                                      • 0 Attachment
                                                        On Wed, Jun 30, 2010 at 10:15:10AM -0700, Florin Andrei wrote:

                                                        > More info. This is how the queues always look, it's a very typical batch:
                                                        >
                                                        > http://i.imgur.com/7MPIx.png

                                                        This graph has no scale, and would not be very interesting in any case.

                                                        Have you made attempt to sign-up for Yahoo's feedback loop, get
                                                        whitelisted, etc.?

                                                        During each deferred queue run:

                                                        - How many messages are delivered by smtp(8)

                                                        - How many are deferred by smtp(8) and is the connection dropped
                                                        by Yahoo, or are the recipients deferred gracefully?

                                                        - Are you subjected to grey-listing? Is your sender address used
                                                        to reach a given recipient stable, or does it change with every
                                                        mailing?

                                                        - How may messages are deferred by qmgr(8), because the transport
                                                        is throttled after too many consecutive errors?

                                                        - What is your "rate delay"?

                                                        - What is your "failure cohort limit"?

                                                        For performance tuning you need relevant metrics. In this case you need
                                                        to understand the statistics of Yahoo's negative feedback. This said,
                                                        the best solution is to work with Yahoo to whitelist you or to stop
                                                        sending bulk email to a provider that does not wish to receive your email.

                                                        --
                                                        Viktor.
                                                      Your message has been successfully submitted and would be delivered to recipients shortly.