Loading ...
Sorry, an error occurred while loading the content.
 

rate limit issue

Expand Messages
  • Gary Smith
    Lately I have found that my outgoing queues are getting a little clogged for yahoo and sbcglobal.net. This usually coincides with a bulk set of news letters
    Message 1 of 11 , May 1, 2010
      Lately I have found that my outgoing queues are getting a little clogged for yahoo and sbcglobal.net. This usually coincides with a bulk set of news letters sent out from a couples clients. Typically we are seeing that they dump about 2000msg/per batch, with no more than one batch per week (usually on Friday nights). The problem is they do it in one fatal swoop. When this happens, we seem to get rate limited from yahoo/sbcglobal for a few hours, which directly affects our other users.

      Some time ago I put a rate limiter in place for AOL/yahoo/sbcglobal/gmail so we wouldn't be bombarding but I don't think that it's being honored, probably because I missed something. When I do a postconf I don't see my rate limiter, which is in main.cf, listed there. I'm not sure if that's by design or part of a misconfiguration on my part. Mail goes out from different sets of servers, with the same public IP, so rate limiting will only affect the bulk mail queue.

      Is there something obvious that I missed?

      master.cf:
      ratelimit unix - - n - 3 smtp

      rate_limit_transport:
      aol.com ratelimit:
      yahoo.com ratelimit:
      sbcglobal.net ratelimit:
      gmail.com ratelimit:

      main.cf:
      ratelimit_destination_concurrency_failed_cohort_limit = 100
      ratelimit_destination_concurrency_limit = 10

      postconf -n:

      alias_maps = hash:/etc/aliases
      broken_sasl_auth_clients = yes
      command_directory = /usr/sbin
      config_directory = /etc/postfix
      daemon_directory = /usr/libexec/postfix
      data_directory = /var/lib/postfix
      debug_peer_level = 2
      html_directory = no
      mail_owner = postfix
      mailq_path = /usr/bin/mailq
      manpage_directory = /usr/share/man
      myhostname = ...
      mynetworks = /etc/postfix/custom/mynetworks
      newaliases_path = /usr/bin/newaliases
      queue_directory = /var/spool/postfix
      readme_directory = /usr/share/doc/postfix-2.6.3/README_FILES
      sample_directory = /usr/share/doc/postfix-2.6.3/samples
      sendmail_path = /usr/sbin/sendmail
      setgid_group = postdrop
      smtp_tls_note_starttls_offer = yes
      smtp_use_tls = yes
      smtpd_client_restrictions = permit_mynetworks,permit_sasl_authenticated,reject
      smtpd_recipient_restrictions = permit_mynetworks,permit_sasl_authenticated,reject
      smtpd_sasl_auth_enable = yes
      smtpd_sasl_authenticated_header = yes
      smtpd_sasl_security_options = noanonymous
      smtpd_tls_cert_file = ...
      smtpd_tls_key_file = ...
      smtpd_tls_loglevel = 1
      smtpd_tls_received_header = yes
      smtpd_tls_session_cache_timeout = 3600s
      smtpd_use_tls = yes
      tls_random_source = dev:/dev/urandom
      transport_maps = hash:/etc/postfix/custom/rate_limit_transport, hash:/etc/postfix/maps/transport
      unknown_local_recipient_reject_code = 550
    • Noel Jones
      ... custom parameter names don t display in postconf output by design. ... This looks reasonable to me; no more than 3 connections should be made at a time
      Message 2 of 11 , May 1, 2010
        On 5/1/2010 12:58 PM, Gary Smith wrote:
        > Lately I have found that my outgoing queues are getting a little clogged for yahoo and sbcglobal.net. This usually coincides with a bulk set of news letters sent out from a couples clients. Typically we are seeing that they dump about 2000msg/per batch, with no more than one batch per week (usually on Friday nights). The problem is they do it in one fatal swoop. When this happens, we seem to get rate limited from yahoo/sbcglobal for a few hours, which directly affects our other users.
        >
        > Some time ago I put a rate limiter in place for AOL/yahoo/sbcglobal/gmail so we wouldn't be bombarding but I don't think that it's being honored, probably because I missed something. When I do a postconf I don't see my rate limiter, which is in main.cf, listed there. I'm not sure if that's by design or part of a misconfiguration on my part. Mail goes out from different sets of servers, with the same public IP, so rate limiting will only affect the bulk mail queue.
        >

        "custom" parameter names don't display in postconf output by
        design.

        > Is there something obvious that I missed?
        >
        > master.cf:
        > ratelimit unix - - n - 3 smtp
        >
        > rate_limit_transport:
        > aol.com ratelimit:
        > yahoo.com ratelimit:
        > sbcglobal.net ratelimit:
        > gmail.com ratelimit:
        >
        > main.cf:
        > ratelimit_destination_concurrency_failed_cohort_limit = 100
        > ratelimit_destination_concurrency_limit = 10
        >

        This looks reasonable to me; no more than 3 connections should
        be made at a time to any combination of those destinations.
        Why don't you think it's working?

        Hint: it's helpful to add a custom log name to a master.cf
        transport entry to make sure it's being used.
        -o syslog_name=postfix-ratelimit
        http://www.postfix.org/postconf.5.html#syslog_name

        > postconf -n:
        [not examined]

        -- Noel Jones
      • Gary Smith
        ... I m not sure why I think it s not working. Skimming the log file, shortly after the back was launched, we saw several of these messages: connect to
        Message 3 of 11 , May 1, 2010
          > > rate_limit_transport:
          > > aol.com ratelimit:
          > > yahoo.com ratelimit:
          > > sbcglobal.net ratelimit:
          > > gmail.com ratelimit:
          > >
          > This looks reasonable to me; no more than 3 connections should
          > be made at a time to any combination of those destinations.
          > Why don't you think it's working?

          I'm not sure why I think it's not working. Skimming the log file, shortly after the back was launched, we saw several of these messages:

          connect to sbcgloabal.net[208.73.210.27]:25: Connection refused
          connect to comcst.net[216.240.187.144]:25: Connection refused
          connect to eathlink.net[216.65.41.185]:25: Connection refused

          (obviously the last two aren't on the list, but will be added). Anyway, I will start logging see if it's working. I also just noticed that the rate limiting file was touched this week, so I need to find out what was touched (which it hasn't been touched in a year since we set it up).

          >
          > Hint: it's helpful to add a custom log name to a master.cf
          > transport entry to make sure it's being used.
          > -o syslog_name=postfix-ratelimit
          > http://www.postfix.org/postconf.5.html#syslog_name
          >
          > > postconf -n:
          > [not examined]
          >
          > -- Noel Jones
        • Mike Morris
          ... Do you realize the entries you just posted are misspelled domains? They are not sbcglobal.net, comcast.net, or earthlink.net. -Mike
          Message 4 of 11 , May 1, 2010
            On 05/01/2010 11:31 AM, Gary Smith wrote:
            >>> rate_limit_transport:
            >>> aol.com ratelimit:
            >>> yahoo.com ratelimit:
            >>> sbcglobal.net ratelimit:
            >>> gmail.com ratelimit:
            >>>
            >> This looks reasonable to me; no more than 3 connections should
            >> be made at a time to any combination of those destinations.
            >> Why don't you think it's working?
            >
            > I'm not sure why I think it's not working. Skimming the log file, shortly after the back was launched, we saw several of these messages:
            >
            > connect to sbcgloabal.net[208.73.210.27]:25: Connection refused
            > connect to comcst.net[216.240.187.144]:25: Connection refused
            > connect to eathlink.net[216.65.41.185]:25: Connection refused
            >
            > (obviously the last two aren't on the list, but will be added). Anyway, I will start logging see if it's working. I also just noticed that the rate limiting file was touched this week, so I need to find out what was touched (which it hasn't been touched in a year since we set it up).
            >

            Do you realize the entries you just posted are misspelled domains? They
            are not sbcglobal.net, comcast.net, or earthlink.net.

            -Mike
          • Noel Jones
            ... These are dead domains, either owned by squatters or web redirectors owned by the real domain. The domains exist but will never accept mail, so postfix
            Message 5 of 11 , May 1, 2010
              On 5/1/2010 1:31 PM, Gary Smith wrote:
              >>> rate_limit_transport:
              >>> aol.com ratelimit:
              >>> yahoo.com ratelimit:
              >>> sbcglobal.net ratelimit:
              >>> gmail.com ratelimit:
              >>>
              >> This looks reasonable to me; no more than 3 connections should
              >> be made at a time to any combination of those destinations.
              >> Why don't you think it's working?
              >
              > I'm not sure why I think it's not working. Skimming the log file, shortly after the back was launched, we saw several of these messages:
              >
              > connect to sbcgloabal.net[208.73.210.27]:25: Connection refused
              > connect to comcst.net[216.240.187.144]:25: Connection refused
              > connect to eathlink.net[216.65.41.185]:25: Connection refused
              >

              These are dead domains, either owned by squatters or web
              redirectors owned by the "real" domain. The domains exist but
              will never accept mail, so postfix will retry until the mail
              expires. You can instruct postfix to reject common
              misspellings like this by adding transport map entries such as:

              # transport
              eathlink.net error:5.1.2 no such domain
              sbcgloabal.net error:5.1.2 no such domain
              comcst.net error:5.1.2 no such domain


              > (obviously the last two aren't on the list, but will be added). Anyway, I will start logging see if it's working. I also just noticed that the rate limiting file was touched this week, so I need to find out what was touched (which it hasn't been touched in a year since we set it up).

              For misspelled domains, either ignore them or add a transport
              entry to reject them right away. Obviously you can't add
              every possible bad domain, but it's helpful to add the top 10
              or so for your users.

              -- Noel Jones
            • Gary Smith
              ... Mike, um, belay my last... My eyes are tired from clearing out queue s. Yes, those are wrong, looks like they have some typos, but they queues that were
              Message 6 of 11 , May 1, 2010
                >
                > Do you realize the entries you just posted are misspelled domains? They
                > are not sbcglobal.net, comcast.net, or earthlink.net.
                >
                > -Mike

                Mike, um, belay my last... My eyes are tired from clearing out queue's.

                Yes, those are wrong, looks like they have some typos, but they queues that were effected were the real ones. The log files are about 50mb so I'm just eyeballing them.

                I have added Noel's suggestion for syslog on the rate limiter and it is indeed being hit, which is a good thing. I might create a few separate limiters and limit them to one's and two's. This only affects any bulk mail we have going out, which is an authenticated separate server, so it should be fine.

                I will also look at the dead domain reject issue Noel mentioned.

                Thanks guys.
              • Wietse Venema
                ... This DOES NOT limit your delivery RATE!! This limits only the delivery CONCURRENCY. To limit the delivery RATE, see
                Message 7 of 11 , May 1, 2010
                  Gary Smith:
                  > Lately I have found that my outgoing queues are getting a little clogged for yahoo and sbcglobal.net. This usually coincides with a bulk set of news letters sent out from a couples clients. Typically we are seeing that they dump about 2000msg/per batch, with no more than one batch per week (usually on Friday nights). The problem is they do it in one fatal swoop. When this happens, we seem to get rate limited from yahoo/sbcglobal for a few hours, which directly affects our other users.
                  >
                  > Some time ago I put a rate limiter in place for AOL/yahoo/sbcglobal/gmail so we wouldn't be bombarding but I don't think that it's being honored, probably because I missed something. When I do a postconf I don't see my rate limiter, which is in main.cf, listed there. I'm not sure if that's by design or part of a misconfiguration on my part. Mail goes out from different sets of servers, with the same public IP, so rate limiting will only affect the bulk mail queue.
                  >
                  > Is there something obvious that I missed?
                  >
                  > master.cf:
                  > ratelimit unix - - n - 3 smtp
                  >
                  > rate_limit_transport:
                  > aol.com ratelimit:
                  > yahoo.com ratelimit:
                  > sbcglobal.net ratelimit:
                  > gmail.com ratelimit:
                  >
                  > main.cf:
                  > ratelimit_destination_concurrency_failed_cohort_limit = 100
                  > ratelimit_destination_concurrency_limit = 10

                  This DOES NOT limit your delivery RATE!!

                  This limits only the delivery CONCURRENCY.

                  To limit the delivery RATE, see
                  http://www.postfix.org/postconf.5.html#transport_destination_rate_delay.


                  Wietse
                • Gary Smith
                  ... Looking into it now. Thanks for the pointer Wietse. If I m running multiple outgoing relays, does it make sense to have some type of concurrency rate for
                  Message 8 of 11 , May 1, 2010
                    > This DOES NOT limit your delivery RATE!!
                    >
                    > This limits only the delivery CONCURRENCY.
                    >
                    > To limit the delivery RATE, see
                    > http://www.postfix.org/postconf.5.html#transport_destination_rate_delay.
                    >

                    Looking into it now. Thanks for the pointer Wietse. If I'm running multiple outgoing relays, does it make sense to have some type of concurrency rate for the outgoing messages in this case?
                  • Noel Jones
                    ... As a general rule, reducing concurrency to the usual problem domains is enough to keep mail flowing at a reasonable rate (which is one reason I didn t
                    Message 9 of 11 , May 1, 2010
                      On 5/1/2010 9:43 PM, Gary Smith wrote:
                      >> This DOES NOT limit your delivery RATE!!
                      >>
                      >> This limits only the delivery CONCURRENCY.
                      >>
                      >> To limit the delivery RATE, see
                      >> http://www.postfix.org/postconf.5.html#transport_destination_rate_delay.
                      >>
                      >
                      > Looking into it now. Thanks for the pointer Wietse. If I'm running multiple outgoing relays, does it make sense to have some type of concurrency rate for the outgoing messages in this case?
                      >
                      >

                      As a general rule, reducing concurrency to the usual problem
                      domains is enough to keep mail flowing at a reasonable rate
                      (which is one reason I didn't address the concurrency vs. rate
                      delay issue). It's common for them to temporarily 4xx some of
                      the mail but eventually it all should go through, even with
                      default postfix settings.

                      You should sign up for "feedback loops" at each mail provider
                      if you haven't already. This may or may not give preference
                      to your mail, but is a necessary first step if you're doing
                      any kind of bulk mail.

                      If that still doesn't help, then implement the rate delay as
                      explained in the link Wietse gave you above. Read the link
                      carefully; the intent of the rate delay feature is to
                      significantly slow down mail delivery to the target domain.
                      And some domains will *still* 4xx some of the mail.

                      As far as multiple outgoing relays is concerned, the answer is
                      "maybe". The only way to know what works for *your* situation
                      is to try; first with the default settings, then reduce
                      concurrency to see if long-term delivery is any better. If the
                      average throughput of a mail run is significantly less than 1
                      delivery per second, implementing rate delays *may* allow you
                      to send more mail by not triggering evasive action at the
                      target mail server.

                      -- Noel Jones
                    • Gary Smith
                      ... Just to make sure I m reading the docs correctly, in my case where I use the transport of ratelimit (as per the former email) I would be adding
                      Message 10 of 11 , May 1, 2010
                        > >> This DOES NOT limit your delivery RATE!!
                        > >>
                        > >> This limits only the delivery CONCURRENCY.
                        > >>
                        > >> To limit the delivery RATE, see
                        > >> http://www.postfix.org/postconf.5.html#transport_destination_rate_delay.
                        > >>
                        ...
                        > If that still doesn't help, then implement the rate delay as
                        > explained in the link Wietse gave you above. Read the link
                        > carefully; the intent of the rate delay feature is to
                        > significantly slow down mail delivery to the target domain.
                        > And some domains will *still* 4xx some of the mail.
                        >

                        Just to make sure I'm reading the docs correctly, in my case where I use the transport of ratelimit (as per the former email) I would be adding ratelimit_destination_rate_delay to main.cf. And if my understanding is correct, these are global settings so I shouldn't set them on the transport in master.cf.

                        Can you guys confirm that this assumption is correct?
                      • Noel Jones
                        yes. -- Noel Jones
                        Message 11 of 11 , May 1, 2010
                          yes.
                          -- Noel Jones

                          "Gary Smith" <gary.smith@...> wrote:

                          >> >> This DOES NOT limit your delivery RATE!!
                          >> >>
                          >> >> This limits only the delivery CONCURRENCY.
                          >> >>
                          >> >> To limit the delivery RATE, see
                          >> >> http://www.postfix.org/postconf.5.html#transport_destination_rate_delay.
                          >> >>
                          >...
                          >> If that still doesn't help, then implement the rate delay as
                          >> explained in the link Wietse gave you above. Read the link
                          >> carefully; the intent of the rate delay feature is to
                          >> significantly slow down mail delivery to the target domain.
                          >> And some domains will *still* 4xx some of the mail.
                          >>
                          >
                          >Just to make sure I'm reading the docs correctly, in my case where I use the transport of ratelimit (as per the former email) I would be adding ratelimit_destination_rate_delay to main.cf. And if my understanding is correct, these are global settings so I shouldn't set them on the transport in master.cf.
                          >
                          >Can you guys confirm that this assumption is correct?
                        Your message has been successfully submitted and would be delivered to recipients shortly.