Loading ...
Sorry, an error occurred while loading the content.

Re: slow delivery to yahoo

Expand Messages
  • Victor Duchovni
    ... In that case, you should post summary statistics from the delays=a/b/c/d log entries for yahoo. The a and b numbers don t really matter in this
    Message 1 of 7 , Oct 15, 2008
    View Source
    • 0 Attachment
      On Wed, Oct 15, 2008 at 03:48:18PM -0400, Ofer Inbar wrote:

      > Victor Duchovni <Victor.Duchovni@...> wrote:
      > > > So I wonder if any of you administer high volume postfix sites and
      > > > have run into the same problem, and if you've found any workarounds.
      > >
      > > The work-around is to get on the Yahoo whitelist.
      >
      > I should've mentioned that all servers in question are whitelisted.
      >
      > Obviously if they weren't they'd be having problems with the other
      > large ISPs too, so I thought it was implicit (large volume sites get
      > on the relevant whitelists or they just don't work). Sorry.

      In that case, you should post summary statistics from the

      delays=a/b/c/d

      log entries for yahoo. The "a" and "b" numbers don't really matter in
      this context, but "c" and "d" are critical. Simple averages distort the
      picture because one slow message can hide a lot of fast messages. So I
      tend to compute "moving averages" with slow exponential decay, yielding
      output similar to the below:

      $ perl -lne '
      m{\A\S+T(\S+).{5} \S+ \S+: \w+: to=<\S+\@yahoo\.com>, (?:orig_to=\S+, )?relay=\S+, (?:conn_use=\d+, )?delay=\S+, delays=[\d.]+/[\d.]+/([\d.]+)/([\d.])+,} or next;
      $c = 0.95 * $c + 0.05 * $2;
      $d = 0.95 * $d + 0.05 * $3;
      if (++$i % 100 == 0) {
      printf "%s %5.2f %5.2f\n", $1, $c, $d;
      }' maillog
      13:01:39 6.30 5.07
      13:05:59 7.24 4.70
      13:12:06 4.58 5.39
      13:16:15 1.28 4.73
      13:21:56 1.71 5.20
      13:27:33 0.50 4.88
      13:33:35 9.90 4.78
      13:37:57 3.82 4.80
      13:41:40 0.70 4.47
      13:48:30 4.72 4.56
      13:54:10 5.50 5.22
      13:58:01 6.15 5.25

      So, it looks a single connection to Yahoo! takes approximately 5 seconds
      to setup, 5 seconds to deliver each message, and (if your message rate
      is high enough) capable of delivering up to 5 messages. So for me,
      maximal throughput per unit of concurrency is:

      msgs/sec = 5 / (5 + 5 * 5) = 1/6

      If my destination concurrency to Yahoo! is the default 20, (not sure
      they let you make that many connections from the same IP) I can only
      deliver 20/6 ~= 3 msgs/sec per sending host.

      To send 100 msgs/sec, I would need ~30 hosts. Fortunately, I don't have
      nearly that much demand for mail to Yahoo!

      --
      Viktor.

      Disclaimer: off-list followups get on-list replies or get ignored.
      Please do not ignore the "Reply-To" header.

      To unsubscribe from the postfix-users list, visit
      http://www.postfix.org/lists.html or click the link below:
      <mailto:majordomo@...?body=unsubscribe%20postfix-users>

      If my response solves your problem, the best way to thank me is to not
      send an "it worked, thanks" follow-up. If you must respond, please put
      "It worked, thanks" in the "Subject" so I can delete these quickly.
    • Ofer Inbar
      ... I don t have delays= in my logs, only delay=. I think this is because the setup I m administering now is running postfix-2.2 (yes, I want to upgrade them,
      Message 2 of 7 , Oct 16, 2008
      View Source
      • 0 Attachment
        > In that case, you should post summary statistics from the
        >
        > delays=a/b/c/d

        I don't have delays= in my logs, only delay=. I think this is because
        the setup I'm administering now is running postfix-2.2 (yes, I want to
        upgrade them, but this won't happen for at least a month).

        However, independently I came to the same conclusion that concurrency
        per-site is the limiting factor. By setting up a separate transport
        for yahoo messages only, we were able to increase our delivery rate
        from a couple of hundred messages per minute, to a few thousand / min.

        Yahoo told us that we were not anywhere close to sending them messages
        too quickly, and we should be able to send more quickly, but it's just
        a slower process to deliver to yahoo than to the other large ISPs, it
        seems, so sending more messages at a time is apparently the solution.

        Settings for this yahoo-specific transport:
        yahoo_destination_concurrency_limit = 500
        yahoo_connect_timeout = 5s
        yahoo_data_done_timeout = 120s
        yahoo_data_init_timeout = 60s
        yahoo_helo_timeout = 120s
        yahoo_mail_timeout = 120s
        yahoo_quit_timeout = 120s
        yahoo_rcpt_timeout = 120s
        yahoo_rset_timeout = 120s

        I had previously tried testing by changing the concurrency limit from
        20 to 40 but that had a small effect, if any. A large change has a
        dramatic effect.

        -- Cos
      • Barney Desmond
        ... I find this quite interesting. Based on the vague guidelines that Yahoo publishes, it sounds like they consider lots of concurrent connections to also be
        Message 3 of 7 , Oct 16, 2008
        View Source
        • 0 Attachment
          Ofer Inbar wrote:
          > Yahoo told us that we were not anywhere close to sending them messages
          > too quickly, and we should be able to send more quickly, but it's just
          > a slower process to deliver to yahoo than to the other large ISPs, it
          > seems, so sending more messages at a time is apparently the solution.
          >
          > Settings for this yahoo-specific transport:
          > yahoo_destination_concurrency_limit = 500
          > yahoo_connect_timeout = 5s
          > yahoo_data_done_timeout = 120s
          > yahoo_data_init_timeout = 60s
          > yahoo_helo_timeout = 120s
          > yahoo_mail_timeout = 120s
          > yahoo_quit_timeout = 120s
          > yahoo_rcpt_timeout = 120s
          > yahoo_rset_timeout = 120s
          >
          > I had previously tried testing by changing the concurrency limit from
          > 20 to 40 but that had a small effect, if any. A large change has a
          > dramatic effect.

          I find this quite interesting. Based on the vague guidelines that Yahoo
          publishes, it sounds like they consider lots of concurrent connections
          to also be abusive. If you can really get away with such high
          concurrency it may well be worth trying out.

          You mentioned going from a couple of hundred messages per minute, to a
          few thousand per minute; are these mostly Yahoo addresses, or was mail
          to Yahoo just holding up the rest of the queue? I don't suppose you have
          any messages-per-minute figures for that yahoo-specific transport?
        • Victor Duchovni
          ... These are fine (provided your queue manager does not run out of file descriptors, you really should move to Postfix 2.5 on a system with epoll, devpoll or
          Message 4 of 7 , Oct 16, 2008
          View Source
          • 0 Attachment
            On Thu, Oct 16, 2008 at 06:54:45PM -0400, Ofer Inbar wrote:

            > yahoo_destination_concurrency_limit = 500
            > yahoo_connect_timeout = 5s

            These are fine (provided your queue manager does not run out of file
            descriptors, you really should move to Postfix 2.5 on a system with
            epoll, devpoll or kqueeu).

            > yahoo_helo_timeout = 120s

            I would consider setting this one even shorter.

            > yahoo_data_done_timeout = 120s
            > yahoo_data_init_timeout = 60s
            > yahoo_mail_timeout = 120s
            > yahoo_quit_timeout = 120s
            > yahoo_rcpt_timeout = 120s
            > yahoo_rset_timeout = 120s

            But leave these at the default values, it is not a good time to time out
            an active transaction aggressively.

            --
            Viktor.

            Disclaimer: off-list followups get on-list replies or get ignored.
            Please do not ignore the "Reply-To" header.

            To unsubscribe from the postfix-users list, visit
            http://www.postfix.org/lists.html or click the link below:
            <mailto:majordomo@...?body=unsubscribe%20postfix-users>

            If my response solves your problem, the best way to thank me is to not
            send an "it worked, thanks" follow-up. If you must respond, please put
            "It worked, thanks" in the "Subject" so I can delete these quickly.
          Your message has been successfully submitted and would be delivered to recipients shortly.