Loading ...
Sorry, an error occurred while loading the content.

Re: Balancing destination concurrency + rate delay

Expand Messages
  • Wietse Venema
    ... As for what settings work better with high-volume receivers, I suggest a search query for aol postmaster , yahoo postmaster etc. With per-destination
    Message 1 of 11 , Jan 18, 2013
    • 0 Attachment
      > I read that warning, but wasn't sure I understood it properly. I
      > know that setting destination recipient limit to anything above
      > one defines a "destination" as a domain, so by setting it to 2
      > we're saying "wait 1s between every delivery to a Yahoo domain,"

      As for what settings work better with high-volume receivers, I
      suggest a search query for "aol postmaster", "yahoo postmaster" etc.

      With per-destination recipient limit > 1, destination = domain,
      and different destinations(=domains) are delivered in parallal
      subject to concurrency limits. This enforces fairness between
      different destinations(=domains).
      With rate delay > 0, per-destination(=domain) concurrency is 1.

      With per-destination recipient limit = 0, destination = recipient,
      and different destinations(=recipients) are delivered in parallal
      subject to concurrency limits. This enforces fairness between
      different destinations(=recipients).
      This setting is used by the local delivery agent.

      Wietse
    • Noel Jones
      ... that was intended to read: With per-destination recipient limit = 1, destination = recipient,
      Message 2 of 11 , Jan 18, 2013
      • 0 Attachment
        On 1/18/2013 7:06 AM, Wietse Venema wrote:
        >> I read that warning, but wasn't sure I understood it properly. I
        >> know that setting destination recipient limit to anything above
        >> one defines a "destination" as a domain, so by setting it to 2
        >> we're saying "wait 1s between every delivery to a Yahoo domain,"
        >
        > As for what settings work better with high-volume receivers, I
        > suggest a search query for "aol postmaster", "yahoo postmaster" etc.
        >
        > With per-destination recipient limit > 1, destination = domain,
        > and different destinations(=domains) are delivered in parallal
        > subject to concurrency limits. This enforces fairness between
        > different destinations(=domains).
        > With rate delay > 0, per-destination(=domain) concurrency is 1.
        >
        > With per-destination recipient limit = 0, destination = recipient,

        that was intended to read:
        With per-destination recipient limit = 1, destination = recipient,

        > and different destinations(=recipients) are delivered in parallal
        > subject to concurrency limits. This enforces fairness between
        > different destinations(=recipients).
        > This setting is used by the local delivery agent.
        >
        > Wietse
        >
      • Steve Jenkins
        ... Agreed - but Yahoo is really the only one we re having issues with (even after complying with all their guidelines here):
        Message 3 of 11 , Jan 18, 2013
        • 0 Attachment
          On Fri, Jan 18, 2013 at 5:06 AM, Wietse Venema <wietse@...> wrote:
          As for what settings work better with high-volume receivers, I
          suggest a search query for "aol postmaster", "yahoo postmaster" etc.

          Agreed - but Yahoo is really the only one we're having issues with (even after complying with all their guidelines here):


          Yes, we do confirmed opt-in, and DKIM, and we unsubscribe bounces within 24 hours, and have FBLs set up, and dedicated IPs, and have good sender reputation scores, and have completed Yahoo's Bulk Sender Application Form, etc, etc. The fact that 2 days after sending nearly 400K newsletters to our subscribers, we still have 70K in our fallback relay's deferred queue, and that 100% of them are @yahoo.com addresses, means we still need to work on our Yahoo-specific Postfix settings.

          Unfortunately, Yahoo's official word on outbound mailer settings is too general to be of much use:

          "Use common-sense settings. While we have not published guidelines for numbers of connections you can concurrently use, we ask that you treat our resources with respect. The more you take, the fewer there are for others, which may force us to defer your connections."

          So that's why I'm asking other Postfix users who run high-volume mailers if they would be so kind as to share their experiences with Postfix concurrency limits and/or rate delays when dealing with Yahoo in particular. Is that a taboo subject here? Or perhaps just not the right forum in which to discuss it?

          Thx,

          SteveJ
        • Robert Schetterer
          ... last time i had to do this , yahoo needs 3 weeks for whitelisting the new ip, used for a mail list server at the end if you already followed recommands for
          Message 4 of 11 , Jan 18, 2013
          • 0 Attachment
            Am 18.01.2013 18:49, schrieb Steve Jenkins:
            > Agreed - but Yahoo is really the only one we're having issues with (even
            > after complying with all their guidelines here)

            last time i had to do this , yahoo needs 3 weeks for whitelisting
            the new ip, used for a mail list server

            at the end if you already followed recommands for bulk mails on this
            list ( and archive ) with your mail server setup , there is less what
            you can do more, you may try more cascading (virtual)mailservers on
            other ips and/or more instances etc as well as other tricks
            but there is no magic ( beyond paying money ) to press someone accept
            your mail in timelimits you prefer, at last its not a question of
            postfix, its yahoos ( in this case ) decision


            Best Regards
            MfG Robert Schetterer

            --
            [*] sys4 AG

            http://sys4.de, +49 (89) 30 90 46 64
            Franziskanerstraße 15, 81669 München

            Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
            Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer
            Aufsichtsratsvorsitzender: Joerg Heidrich
          • Viktor Dukhovni
            ... Yes, they are willing to cripple SMTP and expect everyone to cope, because they are too big to ignore. :-) Rumour has it that established proffesional
            Message 5 of 11 , Jan 18, 2013
            • 0 Attachment
              On Fri, Jan 18, 2013 at 09:49:34AM -0800, Steve Jenkins wrote:

              > Agreed - but Yahoo is really the only one we're having issues with (even
              > after complying with all their guidelines here):
              >
              > http://help.yahoo.com/kb/index?page=content&y=PROD_MAIL_ML&locale=en_US&id=SLN3435
              >

              Yes, they are willing to cripple SMTP and expect everyone to cope,
              because they are too big to ignore. :-)

              Rumour has it that established proffesional bulk-senders typically
              don't turn up new IP addresses (no existing good reputation) to
              full capacity overnight. Rather, their sending software gradually
              ramps-up load to a new address allowing its reputation to build-up
              over time (the first few weeks may be at a low load, then add more,
              ...).

              Once your IP reputation is good, you should be able to use reasonable
              concurrency... You may want disable the demand connection cache
              for Yahoo. This cache is primarily useful for destinations that
              sporadically have some of their MX records unreachable at the TCP
              layer.

              Yahoo's MX SMTP pool essentially never has an IP address that is
              completely down. Instead, you can set a low "smtp_connect_timeout",
              just in case.

              master.cf:
              yahoo unix - - n - - smtp
              -o smtp_connect_timeout=$yahoo_connect_timeout
              -o smtp_connection_cache_on_demand=$yahoo_connection_cache_on_demand

              main.cf:
              # Good enough for light RTT to the moon
              yahoo_connect_timeout=3s

              # Don't annoy them with connection reuse.
              yahoo_connection_cache_on_demand=no

              At that point you may not even need rate delays, just set a modest
              concurrency, and typical SMTP transaction latency of 0.2-0.5s (
              with spam checks, RBL lookups, ...) will give you at most 2-5
              messages per unit concurrency per second.

              --
              Viktor.
            • Steve Jenkins
              On Fri, Jan 18, 2013 at 11:36 AM, Viktor Dukhovni
              Message 6 of 11 , Jan 18, 2013
              • 0 Attachment
                On Fri, Jan 18, 2013 at 11:36 AM, Viktor Dukhovni <postfix-users@...> wrote:
                Yes, they are willing to cripple SMTP and expect everyone to cope,
                because they are too big to ignore. :-)

                Sad, but true.
                 
                At that point you may not even need rate delays, just set a modest
                concurrency, and typical SMTP transaction latency of 0.2-0.5s (
                with spam checks, RBL lookups, ...) will give you at most 2-5
                messages per unit concurrency per second.

                Would this be considered "modest?"

                yahoo_initial_destination_concurrency = 1
                yahoo_destination_concurrency_limit = 4
                yahoo_destination_recipient_limit = 2
                #yahoo_destination_rate_delay = 2s
                yahoo_connect_timeout=3s
                yahoo_connection_cache_on_demand=no

                Thx,

                SteveJ
                 
              • Viktor Dukhovni
                ... You ll have to figure that out for yourself mostly, my gut reaction is that the recipient limit is too low, if you ever did have a multi-recipient message,
                Message 7 of 11 , Jan 18, 2013
                • 0 Attachment
                  On Fri, Jan 18, 2013 at 07:46:45PM -0800, Steve Jenkins wrote:

                  > > At that point you may not even need rate delays, just set a modest
                  > > concurrency, and typical SMTP transaction latency of 0.2-0.5s (
                  > > with spam checks, RBL lookups, ...) will give you at most 2-5
                  > > messages per unit concurrency per second.
                  > >
                  >
                  > Would this be considered "modest?"
                  >
                  > yahoo_initial_destination_concurrency = 1
                  > yahoo_destination_concurrency_limit = 4
                  > yahoo_destination_recipient_limit = 2
                  > #yahoo_destination_rate_delay = 2s
                  > yahoo_connect_timeout=3s
                  > yahoo_connection_cache_on_demand=no

                  You'll have to figure that out for yourself mostly, my gut reaction
                  is that the recipient limit is too low, if you ever did have a
                  multi-recipient message, you'd split it into too many pieces.
                  Rather, set something in the 10-20 range (the RFC requires
                  servers to support at least 100, and Postfix by default sends
                  up to 50 and takes in up to 1000).

                  After burning in your IP at low volume, with mail typically not
                  delayed by Yahoo at all, I would try to grow the (destination aka
                  connection) concurrency limit to something in the 5-10 range.

                  The main thing that will matter is "burning in" a new IP address,
                  once they've learned you're not evil, I would hope their rate limits
                  won't get in your way at all.

                  --
                  Viktor.
                Your message has been successfully submitted and would be delivered to recipients shortly.