Loading ...
Sorry, an error occurred while loading the content.
 

Re: Problem with slow transport map

Expand Messages
  • Noel Jones
    ... When you set the recipient limit to 1, the delay is applied to deliveries to the SAME RECIPIENT rather than deliveries to the SAME DOMAIN. This is
    Message 1 of 7 , Sep 23, 2013
      On 9/23/2013 7:17 AM, Matt - Opem Solutions wrote:
      > Hi,
      >
      >
      >
      > Ive a postfix server which is used to relay emails to an external
      > smtp server, this was done to prevent the receiving smtp server from
      > being flooeded by to many messages per hour which i did by using the
      > debug command and a sleep 6, in addition to this i added a transport
      > map to slow down delivery even more for certain hosts/recipient domains:
      >
      >
      >
      > [MAIN.CF FILE]
      >
      > transport_maps = hash:/etc/postfix/transport
      >
      > smtpslow_destination_rate_delay = 310s
      >
      > smtpslow_destination_concurrency_limit = 1
      >
      > smtpslow_destination_recipient_limit = 1
      >

      When you set the recipient limit to 1, the delay is applied to
      deliveries to the SAME RECIPIENT rather than deliveries to the SAME
      DOMAIN. This is probably not what you intend. See:
      http://www.postfix.org/postconf.5.html#default_destination_rate_delay

      >
      >
      > debugger_command = /bin/sleep 6

      This shouldn't be necessary after you increase the recipient limit.

      > -o max_use=1

      This shouldn't be necessary either.



      -- Noel Jones



      >
      >
      >
      > [TRANSPORT FILE]
      >
      > yahoo.com smtpslow:[external.server.com]:25
      >
      > yahoo.co.uk smtpslow:[external.server.com]:25
      >
      > googlemail.com smtpslow:[external.server.com]:25
      >
      > gmail.com smtpslow:[external.server.com]:25
      >
      > hotmail.com smtpslow:[external.server.com]:25
      >
      > hotmail.co.uk smtpslow:[external.server.com]:25
      >
      > outlook.com smtpslow:[external.server.com]:25
      >
      > outlook.co.uk smtpslow:[external.server.com]:25
      >
      > aol.co.uk smtpslow:[external.server.com]:25
      >
      > aol.com smtpslow:[external.server.com]:25
      >
      > comcast.com smtpslow:[external.server.com]:25
      >
      > live.com smtpslow:[external.server.com]:25
      >
      > live.co.uk smtpslow:[external.server.com]:25
      >
      > msn.com smtpslow:[external.server.com]:25
      >
      > msn.co.uk smtpslow:[external.server.com]:25
      >
      > sbcglobal.net smtpslow:[external.server.com]:25
      >
      > verizon.net smtpslow:[external.server.com]:25
      >
      > bellsouth.net smtpslow:[external.server.com]:25
      >
      > yahoo.ca smtpslow:[external.server.com]:25
      >
      > cox.net smtpslow:[external.server.com]:25
      >
      > ymail.com smtpslow:[external.server.com]:25
      >
      > btinternet.com smtpslow:[external.server.com]:25
      >
      > btinternet.co.uk smtpslow:[external.server.com]:25
      >
      > btopenworld.co.uk smtpslow:[external.server.com]:25
      >
      > btopenworld.com smtpslow:[external.server.com]:25
      >
      > mail.com smtpslow:[external.server.com]:25
      >
      >
      >
      > The smtpslow is set to use one thread only in master.cf
      >
      >
      >
      > [MASTER.CF FILE]
      >
      > smtpslow unix - - n - 1 smtp -D
      >
      > -o syslog_name=postfix-smtp-slow
      >
      > -o max_use=1
      >
      >
      >
      > The –D (debug) is a bit of a dirty hack as it basically calls a
      > sleep for 6 seconds between messages to ensure it doesn’t hit the
      > external hosts message per hour limit.
      >
      >
      >
      > This all works fine, however ive noticed a big problem
      >
      >
      >
      > If i send 3 emails to my hotmail.com address they all rightly have a
      > 5 minute (310s) delay between them, however if i send an email to 2
      > different hotmail.com address and send 3 messages per recipient it
      > does correctly insert the 310s delay between the recipients
      > messages. However the issue is its trying to deliver to both
      > recipients at the same time, ie
      >
      >
      >
      > Emails spooled
      >
      > Email 1 to both reclipents sent (with the 6 second debug gap)
      >
      > Wait 310s
      >
      > Email 2 to both recipients sent (with the 6 second debug gap)
      >
      > Wait 310s
      >
      > Email 3 to both recipients sent (with the 6 second debug gap)
      >
      >
      >
      > Where as i expected it to instead work like:
      >
      >
      >
      > Emails spooled
      >
      > Email 1 to recipient 1 sent
      >
      > Wait 310s
      >
      > Email 1 to recipient 2 sent
      >
      > Wait 310s
      >
      > Email 2 to recipient 1 sent
      >
      > Etc
      >
      >
      >
      > How can i adjust the config so that instead of matching the
      > recipients its matching the domains as if 50 emails go out to
      > hotmail.com at the same time to different recipients it doesn’t try
      > and delvier them all at the same time.
      >
      >
      >
      > Best Regards
      >
      >
      >
      > Matt.
      >
    • Wietse Venema
      ... As documented -D attempts to start a debugger process when the SMTP client process is created. This process will handle subsequent delivery requests
      Message 2 of 7 , Sep 23, 2013
        Matt - Opem Solutions:
        > [MASTER.CF FILE]
        >
        > smtpslow unix - - n - 1 smtp -D
        >
        > -o syslog_name=postfix-smtp-slow
        >
        > -o max_use=1
        >
        >
        >
        > The -D (debug) is a bit of a dirty hack as it basically calls a sleep for 6
        > seconds between messages to ensure it doesn't hit the external hosts message
        > per hour limit.

        As documented -D attempts to start a debugger process when the SMTP
        client process is created. This process will handle subsequent
        delivery requests without attempting to start another debugger
        process. Therefore -D is not a good way to force delays between
        deliveries.

        To force delays between deliveries to the same destination, see
        http://www.postfix.org/postconf.5.html#transport_destination_rate_delay
        (requires Postfix 2.5 or later).

        Wietse
      • Matt - Opem Solutions
        Hi Noel/ Wietse, I was just about to reply i had worked it out when your email came in changed it to 2 and it all worked: smtpslow_destination_recipient_limit
        Message 3 of 7 , Sep 23, 2013
          Hi Noel/ Wietse,

          I was just about to reply i had worked it out when your email came in
          changed it to 2 and it all worked:

          smtpslow_destination_recipient_limit = 2

          # set this to 2 will take care of the domain. If you set it to 1, throttling
          will work only if you send mails to the same recipient and not for the same
          recipient domain.

          The debugger sleep 6 is overall for both normal smtp relay and smtpslow
          relay to ensure the server never hits the messages per hour limit of the
          external relay server.

          Ive removed the max use (i think i had stuck it in there when testing the
          server previously)

          As per Wietse

          smtp_destination_rate_delay = 6 as an alternative to using the debug
          command.

          Am i right in thinking as the smtpslow then uses smtp for delivery it will
          effect both, as in ensure all emails get a 6 second delay.

          Thank you both for your quick reply.

          Best Regards

          Matt.

          -----Original Message-----
          From: owner-postfix-users@...
          [mailto:owner-postfix-users@...] On Behalf Of Noel Jones
          Sent: 23 September 2013 2:26 PM
          To: postfix-users@...
          Subject: Re: Problem with slow transport map

          On 9/23/2013 7:17 AM, Matt - Opem Solutions wrote:
          > Hi,
          >
          >
          >
          > Ive a postfix server which is used to relay emails to an external smtp
          > server, this was done to prevent the receiving smtp server from being
          > flooeded by to many messages per hour which i did by using the debug
          > command and a sleep 6, in addition to this i added a transport map to
          > slow down delivery even more for certain hosts/recipient domains:
          >
          >
          >
          > [MAIN.CF FILE]
          >
          > transport_maps = hash:/etc/postfix/transport
          >
          > smtpslow_destination_rate_delay = 310s
          >
          > smtpslow_destination_concurrency_limit = 1
          >
          > smtpslow_destination_recipient_limit = 1
          >

          When you set the recipient limit to 1, the delay is applied to deliveries to
          the SAME RECIPIENT rather than deliveries to the SAME DOMAIN. This is
          probably not what you intend. See:
          http://www.postfix.org/postconf.5.html#default_destination_rate_delay

          >
          >
          > debugger_command = /bin/sleep 6

          This shouldn't be necessary after you increase the recipient limit.

          > -o max_use=1

          This shouldn't be necessary either.



          -- Noel Jones



          >
          >
          >
          > [TRANSPORT FILE]
          >
          > yahoo.com smtpslow:[external.server.com]:25
          >
          > yahoo.co.uk smtpslow:[external.server.com]:25
          >
          > googlemail.com smtpslow:[external.server.com]:25
          >
          > gmail.com smtpslow:[external.server.com]:25
          >
          > hotmail.com smtpslow:[external.server.com]:25
          >
          > hotmail.co.uk smtpslow:[external.server.com]:25
          >
          > outlook.com smtpslow:[external.server.com]:25
          >
          > outlook.co.uk smtpslow:[external.server.com]:25
          >
          > aol.co.uk smtpslow:[external.server.com]:25
          >
          > aol.com smtpslow:[external.server.com]:25
          >
          > comcast.com smtpslow:[external.server.com]:25
          >
          > live.com smtpslow:[external.server.com]:25
          >
          > live.co.uk smtpslow:[external.server.com]:25
          >
          > msn.com smtpslow:[external.server.com]:25
          >
          > msn.co.uk smtpslow:[external.server.com]:25
          >
          > sbcglobal.net smtpslow:[external.server.com]:25
          >
          > verizon.net smtpslow:[external.server.com]:25
          >
          > bellsouth.net smtpslow:[external.server.com]:25
          >
          > yahoo.ca smtpslow:[external.server.com]:25
          >
          > cox.net smtpslow:[external.server.com]:25
          >
          > ymail.com smtpslow:[external.server.com]:25
          >
          > btinternet.com smtpslow:[external.server.com]:25
          >
          > btinternet.co.uk smtpslow:[external.server.com]:25
          >
          > btopenworld.co.uk smtpslow:[external.server.com]:25
          >
          > btopenworld.com smtpslow:[external.server.com]:25
          >
          > mail.com smtpslow:[external.server.com]:25
          >
          >
          >
          > The smtpslow is set to use one thread only in master.cf
          >
          >
          >
          > [MASTER.CF FILE]
          >
          > smtpslow unix - - n - 1 smtp -D
          >
          > -o syslog_name=postfix-smtp-slow
          >
          > -o max_use=1
          >
          >
          >
          > The -D (debug) is a bit of a dirty hack as it basically calls a sleep
          > for 6 seconds between messages to ensure it doesn't hit the external
          > hosts message per hour limit.
          >
          >
          >
          > This all works fine, however ive noticed a big problem
          >
          >
          >
          > If i send 3 emails to my hotmail.com address they all rightly have a
          > 5 minute (310s) delay between them, however if i send an email to 2
          > different hotmail.com address and send 3 messages per recipient it
          > does correctly insert the 310s delay between the recipients messages.
          > However the issue is its trying to deliver to both recipients at the
          > same time, ie
          >
          >
          >
          > Emails spooled
          >
          > Email 1 to both reclipents sent (with the 6 second debug gap)
          >
          > Wait 310s
          >
          > Email 2 to both recipients sent (with the 6 second debug gap)
          >
          > Wait 310s
          >
          > Email 3 to both recipients sent (with the 6 second debug gap)
          >
          >
          >
          > Where as i expected it to instead work like:
          >
          >
          >
          > Emails spooled
          >
          > Email 1 to recipient 1 sent
          >
          > Wait 310s
          >
          > Email 1 to recipient 2 sent
          >
          > Wait 310s
          >
          > Email 2 to recipient 1 sent
          >
          > Etc
          >
          >
          >
          > How can i adjust the config so that instead of matching the recipients
          > its matching the domains as if 50 emails go out to hotmail.com at the
          > same time to different recipients it doesn't try and delvier them all
          > at the same time.
          >
          >
          >
          > Best Regards
          >
          >
          >
          > Matt.
          >
        • Wietse Venema
          ... Use smtpslow_destination_rate_delay = 6 , and get rid of the debugger hack. Wietse
          Message 4 of 7 , Sep 23, 2013
            Matt - Opem Solutions:
            > As per Wietse
            >
            > smtp_destination_rate_delay = 6 as an alternative to using the debug
            > command.

            Use "smtpslow_destination_rate_delay = 6", and get rid of the
            debugger hack.

            Wietse
          • Matt - Opem Solutions
            The problem is i need to 6 second slow down for both smtp and smtpslow, whilst smtpslow would slow down using this it wouldn t slow down smtp as well. The
            Message 5 of 7 , Sep 23, 2013
              The problem is i need to 6 second slow down for both smtp and smtpslow,
              whilst smtpslow would slow down using this it wouldn't slow down smtp as
              well.

              The debug hack although dirty was the only way i could to enforce it on both
              transits and all domains, i set:

              smtp_destination_rate_delay = 60s
              smtp_destination_concurrency_limit = 1
              smtp_destination_recipient_limit = 1

              which is for the standard (non slow delivery) and spooled in two messages
              (that are not in the smtpslow transport map) and both got delivered
              instantly.

              Regards

              Matt.


              -----Original Message-----
              From: owner-postfix-users@...
              [mailto:owner-postfix-users@...] On Behalf Of Wietse Venema
              Sent: 23 September 2013 2:48 PM
              To: Postfix users
              Subject: Re: Problem with slow transport map

              Matt - Opem Solutions:
              > As per Wietse
              >
              > smtp_destination_rate_delay = 6 as an alternative to using the debug
              > command.

              Use "smtpslow_destination_rate_delay = 6", and get rid of the debugger hack.

              Wietse
            • Wietse Venema
              ... Then, use both, and set each delay to an appropriate value. Note that setting _rate_delay on one transport WILL NOT affect deliveries made with the other
              Message 6 of 7 , Sep 23, 2013
                Matt - Opem Solutions:
                > The problem is i need to 6 second slow down for both smtp and smtpslow,
                > whilst smtpslow would slow down using this it wouldn't slow down smtp as
                > well.

                Then, use both, and set each delay to an appropriate value.
                Note that setting _rate_delay on one transport WILL NOT affect
                deliveries made with the other transport).

                > smtp_destination_recipient_limit = 1

                That is a mistake. As documented that controls the delay
                for deliveries to the same recipient only.

                Wietse
              Your message has been successfully submitted and would be delivered to recipients shortly.