Loading ...
Sorry, an error occurred while loading the content.

Prioritising outgoing mail

Expand Messages
  • Wouter van Marle
    Hi list, From me a question that seems to be asked now and then here, but I could not find any answers even on whether this is possible in the first place. I
    Message 1 of 8 , Mar 2, 2009
    • 0 Attachment
      Hi list,

      From me a question that seems to be asked now and then here, but I
      could not find any answers even on whether this is possible in the
      first place.

      I would like to be able to prioritise outgoing e-mail so they do not
      get stuck in the queue. This as I now and then send out a large number
      of e-mails with attachments, and that saturates my connection for a
      prolonged time. It doesn't matter that those mails get out slower, as
      long as they get out eventually I'm happy.

      However it can clog up other e-mails that I do want to get tackled
      first.

      Any way to do this?

      Any solution will do as long as it can run on a single server, and an
      upgrade of my uplink is also not an option (financially, and it is too
      infrequent to require more bandwidth but frequent enough to irritate me
      that my other e-mails do not get out quickly). Separate mailqueues,
      whatever: just a way to get normal priority mails first in line.

      Thanks.

      Wouter.
    • Victor Duchovni
      ... Use a custom transport for these messages with a low concurrency limit, or use traffic shaping in the TCP stack to limit the bandwidth per SMTP connection.
      Message 2 of 8 , Mar 2, 2009
      • 0 Attachment
        On Mon, Mar 02, 2009 at 10:44:21PM +0800, Wouter van Marle wrote:

        > Hi list,
        >
        > From me a question that seems to be asked now and then here, but I could
        > not find any answers even on whether this is possible in the first place.
        >
        > I would like to be able to prioritise outgoing e-mail so they do not get
        > stuck in the queue. This as I now and then send out a large number of
        > e-mails with attachments, and that saturates my connection for a prolonged
        > time. It doesn't matter that those mails get out slower, as long as they
        > get out eventually I'm happy.

        Use a custom transport for these messages with a low concurrency limit,
        or use traffic shaping in the TCP stack to limit the bandwidth per
        SMTP connection.

        --
        Viktor.

        Disclaimer: off-list followups get on-list replies or get ignored.
        Please do not ignore the "Reply-To" header.

        To unsubscribe from the postfix-users list, visit
        http://www.postfix.org/lists.html or click the link below:
        <mailto:majordomo@...?body=unsubscribe%20postfix-users>

        If my response solves your problem, the best way to thank me is to not
        send an "it worked, thanks" follow-up. If you must respond, please put
        "It worked, thanks" in the "Subject" so I can delete these quickly.
      • Wouter van Marle
        ... You mean like installing sendmail or so in parallel to postfix and then have sendmail send out the lower-priority mails? ... And how would that get certain
        Message 3 of 8 , Mar 2, 2009
        • 0 Attachment
          On 2 Mar 09, at 23:09, Victor Duchovni wrote:

          > On Mon, Mar 02, 2009 at 10:44:21PM +0800, Wouter van Marle wrote:
          >
          >> Hi list,
          >>
          >> From me a question that seems to be asked now and then here, but I
          >> could
          >> not find any answers even on whether this is possible in the first
          >> place.
          >>
          >> I would like to be able to prioritise outgoing e-mail so they do not
          >> get
          >> stuck in the queue. This as I now and then send out a large number of
          >> e-mails with attachments, and that saturates my connection for a
          >> prolonged
          >> time. It doesn't matter that those mails get out slower, as long as
          >> they
          >> get out eventually I'm happy.
          >
          > Use a custom transport for these messages with a low concurrency limit,

          You mean like installing sendmail or so in parallel to postfix and then
          have sendmail send out the lower-priority mails?

          > or use traffic shaping in the TCP stack to limit the bandwidth per
          > SMTP connection.

          And how would that get certain mails out with priority? It sounds to me
          like this would slow down the overall process. I have up to 100 smtp
          processes running at a time, but as long as new mails end up at the
          back of the queue still no progress there. They have to come first.

          Wouter.

          >
          > --
          > Viktor.
          >
          > Disclaimer: off-list followups get on-list replies or get ignored.
          > Please do not ignore the "Reply-To" header.
          >
          > To unsubscribe from the postfix-users list, visit
          > http://www.postfix.org/lists.html or click the link below:
          > <mailto:majordomo@...?body=unsubscribe%20postfix-users>
          >
          > If my response solves your problem, the best way to thank me is to not
          > send an "it worked, thanks" follow-up. If you must respond, please put
          > "It worked, thanks" in the "Subject" so I can delete these quickly.
          >
          >
        • Wietse Venema
          ... No, use a POSTFIX transport map. ... With the concurrency limit (see above), low priority mail can use up only a limited portion of the bandwidth. Wietse
          Message 4 of 8 , Mar 2, 2009
          • 0 Attachment
            Wouter van Marle:
            >
            > On 2 Mar 09, at 23:09, Victor Duchovni wrote:
            >
            > > On Mon, Mar 02, 2009 at 10:44:21PM +0800, Wouter van Marle wrote:
            > >
            > >> Hi list,
            > >>
            > >> From me a question that seems to be asked now and then here, but I
            > >> could
            > >> not find any answers even on whether this is possible in the first
            > >> place.
            > >>
            > >> I would like to be able to prioritise outgoing e-mail so they do not
            > >> get
            > >> stuck in the queue. This as I now and then send out a large number of
            > >> e-mails with attachments, and that saturates my connection for a
            > >> prolonged
            > >> time. It doesn't matter that those mails get out slower, as long as
            > >> they
            > >> get out eventually I'm happy.
            > >
            > > Use a custom transport for these messages with a low concurrency limit,
            >
            > You mean like installing sendmail or so in parallel to postfix and then
            > have sendmail send out the lower-priority mails?

            No, use a POSTFIX transport map.

            > > or use traffic shaping in the TCP stack to limit the bandwidth per
            > > SMTP connection.
            >
            > And how would that get certain mails out with priority? It sounds to me

            With the concurrency limit (see above), low priority mail can use
            up only a limited portion of the bandwidth.

            Wietse
          • Victor Duchovni
            ... No I mean a Postfix transport , as in transport(5) and master(5). ... It would not, but you won t saturate the entire link with any given email, leaving
            Message 5 of 8 , Mar 2, 2009
            • 0 Attachment
              On Mon, Mar 02, 2009 at 11:59:31PM +0800, Wouter van Marle wrote:

              >> Use a custom transport for these messages with a low concurrency limit,
              >
              > You mean like installing sendmail or so in parallel to postfix and then
              > have sendmail send out the lower-priority mails?
              >

              No I mean a Postfix "transport", as in transport(5) and master(5).

              >> or use traffic shaping in the TCP stack to limit the bandwidth per
              >> SMTP connection.
              >
              > And how would that get certain mails out with priority? It sounds to me
              > like this would slow down the overall process. I have up to 100 smtp
              > processes running at a time, but as long as new mails end up at the back of
              > the queue still no progress there. They have to come first.

              It would not, but you won't saturate the entire link with any given email,
              leaving enough room for other traffic. If you can limit the concurrency
              of this particular message, then you'll have some bandwidth left over for
              other messages.

              --
              Viktor.

              Disclaimer: off-list followups get on-list replies or get ignored.
              Please do not ignore the "Reply-To" header.

              To unsubscribe from the postfix-users list, visit
              http://www.postfix.org/lists.html or click the link below:
              <mailto:majordomo@...?body=unsubscribe%20postfix-users>

              If my response solves your problem, the best way to thank me is to not
              send an "it worked, thanks" follow-up. If you must respond, please put
              "It worked, thanks" in the "Subject" so I can delete these quickly.
            • Wouter van Marle
              ... The problem of a transport map (I have just read the man page, which as usual is highly technical so I am not sure whether I fully understand the purpose
              Message 6 of 8 , Mar 2, 2009
              • 0 Attachment
                On Mon, 2009-03-02 at 11:18 -0500, Victor Duchovni wrote:
                > On Mon, Mar 02, 2009 at 11:59:31PM +0800, Wouter van Marle wrote:
                >
                > >> Use a custom transport for these messages with a low concurrency limit,
                > >
                > > You mean like installing sendmail or so in parallel to postfix and then
                > > have sendmail send out the lower-priority mails?
                >
                > No I mean a Postfix "transport", as in transport(5) and master(5).

                The problem of a transport map (I have just read the man page, which as
                usual is highly technical so I am not sure whether I fully understand
                the purpose and working of transport maps) is that there is a huge
                overlap between receivers of the low-priority mail list and regular
                e-mail receivers. Most of the regular e-mail receivers also receive this
                mail list.

                Many of my mail list receivers are on common domains like gmail.com and
                yahoo.com, thus this would slow down all other mails to those domains as
                well, even if they are not part of the mail list.

                Setting it per recipient is not a good idea because of maintenance
                issues (keeping mail list and transport map in sync), and because of the
                regular mails that may be sent to those recipients while a mail list run
                is in progress.

                The idea of using a "slow:" transport as suggested in the transport(5)
                man page (without elaborating unfortunately...) to limit the number of
                smtp deamons that sounds like the way to go to me. Then I can reserve 80
                deamons for the mail list, or maybe 50, enough to saturate my uplink -
                still allowing regular mails to have an smtp available to be handled
                immediately. This appears to me a way to let the other mails "jump the
                queue" and be processed with priority. That there is little bandwidth
                available is not too much of an issue, then it might take a minute
                instead of seconds to send out, still a major improvement of the hours
                it may take in the current situation.

                > >> or use traffic shaping in the TCP stack to limit the bandwidth per
                > >> SMTP connection.
                > >
                > > And how would that get certain mails out with priority? It sounds to me
                > > like this would slow down the overall process. I have up to 100 smtp
                > > processes running at a time, but as long as new mails end up at the back of
                > > the queue still no progress there. They have to come first.
                >
                > It would not, but you won't saturate the entire link with any given email,
                > leaving enough room for other traffic. If you can limit the concurrency
                > of this particular message, then you'll have some bandwidth left over for
                > other messages.

                I don't like that idea very much: when I have only a few mails to send
                out, I want them to go with the maximum speed possible. I have 2 Mbit
                available, so with 100 smtp connections could limit it to say 20 kbit
                per smtp process. But that would leave the rest of my bandwidth idle
                when there are less than 100 active smtp connections, which is the case
                like 90% of the time.

                Wouter.


                >
              • Wouter van Marle
                Hi all, Would it be possible to add an extension to the user s address, e.g. user+slow@example.com, that would be mapped through a separate transport (e.g. the
                Message 7 of 8 , Mar 2, 2009
                • 0 Attachment
                  Hi all,

                  Would it be possible to add an extension to the user's address, e.g.
                  user+slow@..., that would be mapped through a separate transport
                  (e.g. the slow: as suggested in the man page), and be rewritten by
                  trivial-rewrite to user@... before being sent out.

                  An option like this would do the job for me. It would allow me to easily
                  maintain my maillist, rewriting addresses on the fly when creating the
                  mails, and allowing regular mails to be handled with priority.

                  And any ideas on how/where such a slow: transport (with a limited number
                  of smtp daemons to be started) would be configured? I can't find
                  anything about this in the man pages. Except that it is possible.

                  Wouter.

                  On Tue, 2009-03-03 at 11:25 +0800, Wouter van Marle wrote:
                  > On Mon, 2009-03-02 at 11:18 -0500, Victor Duchovni wrote:
                  > > On Mon, Mar 02, 2009 at 11:59:31PM +0800, Wouter van Marle wrote:
                  > >
                  > > >> Use a custom transport for these messages with a low concurrency limit,
                  > > >
                  > > > You mean like installing sendmail or so in parallel to postfix and then
                  > > > have sendmail send out the lower-priority mails?
                  > >
                  > > No I mean a Postfix "transport", as in transport(5) and master(5).
                  >
                  > The problem of a transport map (I have just read the man page, which as
                  > usual is highly technical so I am not sure whether I fully understand
                  > the purpose and working of transport maps) is that there is a huge
                  > overlap between receivers of the low-priority mail list and regular
                  > e-mail receivers. Most of the regular e-mail receivers also receive this
                  > mail list.
                  >
                  > Many of my mail list receivers are on common domains like gmail.com and
                  > yahoo.com, thus this would slow down all other mails to those domains as
                  > well, even if they are not part of the mail list.
                  >
                  > Setting it per recipient is not a good idea because of maintenance
                  > issues (keeping mail list and transport map in sync), and because of the
                  > regular mails that may be sent to those recipients while a mail list run
                  > is in progress.
                  >
                  > The idea of using a "slow:" transport as suggested in the transport(5)
                  > man page (without elaborating unfortunately...) to limit the number of
                  > smtp deamons that sounds like the way to go to me. Then I can reserve 80
                  > deamons for the mail list, or maybe 50, enough to saturate my uplink -
                  > still allowing regular mails to have an smtp available to be handled
                  > immediately. This appears to me a way to let the other mails "jump the
                  > queue" and be processed with priority. That there is little bandwidth
                  > available is not too much of an issue, then it might take a minute
                  > instead of seconds to send out, still a major improvement of the hours
                  > it may take in the current situation.
                  >
                  > > >> or use traffic shaping in the TCP stack to limit the bandwidth per
                  > > >> SMTP connection.
                  > > >
                  > > > And how would that get certain mails out with priority? It sounds to me
                  > > > like this would slow down the overall process. I have up to 100 smtp
                  > > > processes running at a time, but as long as new mails end up at the back of
                  > > > the queue still no progress there. They have to come first.
                  > >
                  > > It would not, but you won't saturate the entire link with any given email,
                  > > leaving enough room for other traffic. If you can limit the concurrency
                  > > of this particular message, then you'll have some bandwidth left over for
                  > > other messages.
                  >
                  > I don't like that idea very much: when I have only a few mails to send
                  > out, I want them to go with the maximum speed possible. I have 2 Mbit
                  > available, so with 100 smtp connections could limit it to say 20 kbit
                  > per smtp process. But that would leave the rest of my bandwidth idle
                  > when there are less than 100 active smtp connections, which is the case
                  > like 90% of the time.
                  >
                  > Wouter.
                  >
                  >
                  > >
                  >
                  >
                • Victor Duchovni
                  ... You may need to do sender-dependent routing for this sender, and inject the mail into a different queue, or get the originating system to do this directly.
                  Message 8 of 8 , Mar 2, 2009
                  • 0 Attachment
                    On Tue, Mar 03, 2009 at 11:25:55AM +0800, Wouter van Marle wrote:

                    > On Mon, 2009-03-02 at 11:18 -0500, Victor Duchovni wrote:
                    > > On Mon, Mar 02, 2009 at 11:59:31PM +0800, Wouter van Marle wrote:
                    > >
                    > > >> Use a custom transport for these messages with a low concurrency limit,
                    > > >
                    > > > You mean like installing sendmail or so in parallel to postfix and then
                    > > > have sendmail send out the lower-priority mails?
                    > >
                    > > No I mean a Postfix "transport", as in transport(5) and master(5).
                    >
                    > The problem of a transport map (I have just read the man page, which as
                    > usual is highly technical so I am not sure whether I fully understand
                    > the purpose and working of transport maps) is that there is a huge
                    > overlap between receivers of the low-priority mail list and regular
                    > e-mail receivers. Most of the regular e-mail receivers also receive this
                    > mail list.

                    You may need to do sender-dependent routing for this sender, and inject
                    the mail into a different queue, or get the originating system to do
                    this directly.

                    > > It would not, but you won't saturate the entire link with any given email,
                    > > leaving enough room for other traffic. If you can limit the concurrency
                    > > of this particular message, then you'll have some bandwidth left over for
                    > > other messages.
                    >
                    > I don't like that idea very much: when I have only a few mails to send
                    > out, I want them to go with the maximum speed possible. I have 2 Mbit
                    > available, so with 100 smtp connections could limit it to say 20 kbit
                    > per smtp process. But that would leave the rest of my bandwidth idle
                    > when there are less than 100 active smtp connections, which is the case
                    > like 90% of the time.

                    Does limiting bandwidth for small messages signicantly impact delivery
                    latency? Also who said you should divide the bandwidth 100-fold? You
                    give the slow transport 5 parallel threads, and up to half the bandwidth,
                    so each channel gets 10% of the bandwidth.

                    --
                    Viktor.

                    Disclaimer: off-list followups get on-list replies or get ignored.
                    Please do not ignore the "Reply-To" header.

                    To unsubscribe from the postfix-users list, visit
                    http://www.postfix.org/lists.html or click the link below:
                    <mailto:majordomo@...?body=unsubscribe%20postfix-users>

                    If my response solves your problem, the best way to thank me is to not
                    send an "it worked, thanks" follow-up. If you must respond, please put
                    "It worked, thanks" in the "Subject" so I can delete these quickly.
                  Your message has been successfully submitted and would be delivered to recipients shortly.