Loading ...
Sorry, an error occurred while loading the content.

Re: Prioritising outgoing mail

Expand Messages
  • Wietse Venema
    ... No, use a POSTFIX transport map. ... With the concurrency limit (see above), low priority mail can use up only a limited portion of the bandwidth. Wietse
    Message 1 of 8 , Mar 2, 2009
    • 0 Attachment
      Wouter van Marle:
      >
      > On 2 Mar 09, at 23:09, Victor Duchovni wrote:
      >
      > > On Mon, Mar 02, 2009 at 10:44:21PM +0800, Wouter van Marle wrote:
      > >
      > >> Hi list,
      > >>
      > >> From me a question that seems to be asked now and then here, but I
      > >> could
      > >> not find any answers even on whether this is possible in the first
      > >> place.
      > >>
      > >> I would like to be able to prioritise outgoing e-mail so they do not
      > >> get
      > >> stuck in the queue. This as I now and then send out a large number of
      > >> e-mails with attachments, and that saturates my connection for a
      > >> prolonged
      > >> time. It doesn't matter that those mails get out slower, as long as
      > >> they
      > >> get out eventually I'm happy.
      > >
      > > Use a custom transport for these messages with a low concurrency limit,
      >
      > You mean like installing sendmail or so in parallel to postfix and then
      > have sendmail send out the lower-priority mails?

      No, use a POSTFIX transport map.

      > > or use traffic shaping in the TCP stack to limit the bandwidth per
      > > SMTP connection.
      >
      > And how would that get certain mails out with priority? It sounds to me

      With the concurrency limit (see above), low priority mail can use
      up only a limited portion of the bandwidth.

      Wietse
    • Victor Duchovni
      ... No I mean a Postfix transport , as in transport(5) and master(5). ... It would not, but you won t saturate the entire link with any given email, leaving
      Message 2 of 8 , Mar 2, 2009
      • 0 Attachment
        On Mon, Mar 02, 2009 at 11:59:31PM +0800, Wouter van Marle wrote:

        >> Use a custom transport for these messages with a low concurrency limit,
        >
        > You mean like installing sendmail or so in parallel to postfix and then
        > have sendmail send out the lower-priority mails?
        >

        No I mean a Postfix "transport", as in transport(5) and master(5).

        >> or use traffic shaping in the TCP stack to limit the bandwidth per
        >> SMTP connection.
        >
        > And how would that get certain mails out with priority? It sounds to me
        > like this would slow down the overall process. I have up to 100 smtp
        > processes running at a time, but as long as new mails end up at the back of
        > the queue still no progress there. They have to come first.

        It would not, but you won't saturate the entire link with any given email,
        leaving enough room for other traffic. If you can limit the concurrency
        of this particular message, then you'll have some bandwidth left over for
        other messages.

        --
        Viktor.

        Disclaimer: off-list followups get on-list replies or get ignored.
        Please do not ignore the "Reply-To" header.

        To unsubscribe from the postfix-users list, visit
        http://www.postfix.org/lists.html or click the link below:
        <mailto:majordomo@...?body=unsubscribe%20postfix-users>

        If my response solves your problem, the best way to thank me is to not
        send an "it worked, thanks" follow-up. If you must respond, please put
        "It worked, thanks" in the "Subject" so I can delete these quickly.
      • Wouter van Marle
        ... The problem of a transport map (I have just read the man page, which as usual is highly technical so I am not sure whether I fully understand the purpose
        Message 3 of 8 , Mar 2, 2009
        • 0 Attachment
          On Mon, 2009-03-02 at 11:18 -0500, Victor Duchovni wrote:
          > On Mon, Mar 02, 2009 at 11:59:31PM +0800, Wouter van Marle wrote:
          >
          > >> Use a custom transport for these messages with a low concurrency limit,
          > >
          > > You mean like installing sendmail or so in parallel to postfix and then
          > > have sendmail send out the lower-priority mails?
          >
          > No I mean a Postfix "transport", as in transport(5) and master(5).

          The problem of a transport map (I have just read the man page, which as
          usual is highly technical so I am not sure whether I fully understand
          the purpose and working of transport maps) is that there is a huge
          overlap between receivers of the low-priority mail list and regular
          e-mail receivers. Most of the regular e-mail receivers also receive this
          mail list.

          Many of my mail list receivers are on common domains like gmail.com and
          yahoo.com, thus this would slow down all other mails to those domains as
          well, even if they are not part of the mail list.

          Setting it per recipient is not a good idea because of maintenance
          issues (keeping mail list and transport map in sync), and because of the
          regular mails that may be sent to those recipients while a mail list run
          is in progress.

          The idea of using a "slow:" transport as suggested in the transport(5)
          man page (without elaborating unfortunately...) to limit the number of
          smtp deamons that sounds like the way to go to me. Then I can reserve 80
          deamons for the mail list, or maybe 50, enough to saturate my uplink -
          still allowing regular mails to have an smtp available to be handled
          immediately. This appears to me a way to let the other mails "jump the
          queue" and be processed with priority. That there is little bandwidth
          available is not too much of an issue, then it might take a minute
          instead of seconds to send out, still a major improvement of the hours
          it may take in the current situation.

          > >> or use traffic shaping in the TCP stack to limit the bandwidth per
          > >> SMTP connection.
          > >
          > > And how would that get certain mails out with priority? It sounds to me
          > > like this would slow down the overall process. I have up to 100 smtp
          > > processes running at a time, but as long as new mails end up at the back of
          > > the queue still no progress there. They have to come first.
          >
          > It would not, but you won't saturate the entire link with any given email,
          > leaving enough room for other traffic. If you can limit the concurrency
          > of this particular message, then you'll have some bandwidth left over for
          > other messages.

          I don't like that idea very much: when I have only a few mails to send
          out, I want them to go with the maximum speed possible. I have 2 Mbit
          available, so with 100 smtp connections could limit it to say 20 kbit
          per smtp process. But that would leave the rest of my bandwidth idle
          when there are less than 100 active smtp connections, which is the case
          like 90% of the time.

          Wouter.


          >
        • Wouter van Marle
          Hi all, Would it be possible to add an extension to the user s address, e.g. user+slow@example.com, that would be mapped through a separate transport (e.g. the
          Message 4 of 8 , Mar 2, 2009
          • 0 Attachment
            Hi all,

            Would it be possible to add an extension to the user's address, e.g.
            user+slow@..., that would be mapped through a separate transport
            (e.g. the slow: as suggested in the man page), and be rewritten by
            trivial-rewrite to user@... before being sent out.

            An option like this would do the job for me. It would allow me to easily
            maintain my maillist, rewriting addresses on the fly when creating the
            mails, and allowing regular mails to be handled with priority.

            And any ideas on how/where such a slow: transport (with a limited number
            of smtp daemons to be started) would be configured? I can't find
            anything about this in the man pages. Except that it is possible.

            Wouter.

            On Tue, 2009-03-03 at 11:25 +0800, Wouter van Marle wrote:
            > On Mon, 2009-03-02 at 11:18 -0500, Victor Duchovni wrote:
            > > On Mon, Mar 02, 2009 at 11:59:31PM +0800, Wouter van Marle wrote:
            > >
            > > >> Use a custom transport for these messages with a low concurrency limit,
            > > >
            > > > You mean like installing sendmail or so in parallel to postfix and then
            > > > have sendmail send out the lower-priority mails?
            > >
            > > No I mean a Postfix "transport", as in transport(5) and master(5).
            >
            > The problem of a transport map (I have just read the man page, which as
            > usual is highly technical so I am not sure whether I fully understand
            > the purpose and working of transport maps) is that there is a huge
            > overlap between receivers of the low-priority mail list and regular
            > e-mail receivers. Most of the regular e-mail receivers also receive this
            > mail list.
            >
            > Many of my mail list receivers are on common domains like gmail.com and
            > yahoo.com, thus this would slow down all other mails to those domains as
            > well, even if they are not part of the mail list.
            >
            > Setting it per recipient is not a good idea because of maintenance
            > issues (keeping mail list and transport map in sync), and because of the
            > regular mails that may be sent to those recipients while a mail list run
            > is in progress.
            >
            > The idea of using a "slow:" transport as suggested in the transport(5)
            > man page (without elaborating unfortunately...) to limit the number of
            > smtp deamons that sounds like the way to go to me. Then I can reserve 80
            > deamons for the mail list, or maybe 50, enough to saturate my uplink -
            > still allowing regular mails to have an smtp available to be handled
            > immediately. This appears to me a way to let the other mails "jump the
            > queue" and be processed with priority. That there is little bandwidth
            > available is not too much of an issue, then it might take a minute
            > instead of seconds to send out, still a major improvement of the hours
            > it may take in the current situation.
            >
            > > >> or use traffic shaping in the TCP stack to limit the bandwidth per
            > > >> SMTP connection.
            > > >
            > > > And how would that get certain mails out with priority? It sounds to me
            > > > like this would slow down the overall process. I have up to 100 smtp
            > > > processes running at a time, but as long as new mails end up at the back of
            > > > the queue still no progress there. They have to come first.
            > >
            > > It would not, but you won't saturate the entire link with any given email,
            > > leaving enough room for other traffic. If you can limit the concurrency
            > > of this particular message, then you'll have some bandwidth left over for
            > > other messages.
            >
            > I don't like that idea very much: when I have only a few mails to send
            > out, I want them to go with the maximum speed possible. I have 2 Mbit
            > available, so with 100 smtp connections could limit it to say 20 kbit
            > per smtp process. But that would leave the rest of my bandwidth idle
            > when there are less than 100 active smtp connections, which is the case
            > like 90% of the time.
            >
            > Wouter.
            >
            >
            > >
            >
            >
          • Victor Duchovni
            ... You may need to do sender-dependent routing for this sender, and inject the mail into a different queue, or get the originating system to do this directly.
            Message 5 of 8 , Mar 2, 2009
            • 0 Attachment
              On Tue, Mar 03, 2009 at 11:25:55AM +0800, Wouter van Marle wrote:

              > On Mon, 2009-03-02 at 11:18 -0500, Victor Duchovni wrote:
              > > On Mon, Mar 02, 2009 at 11:59:31PM +0800, Wouter van Marle wrote:
              > >
              > > >> Use a custom transport for these messages with a low concurrency limit,
              > > >
              > > > You mean like installing sendmail or so in parallel to postfix and then
              > > > have sendmail send out the lower-priority mails?
              > >
              > > No I mean a Postfix "transport", as in transport(5) and master(5).
              >
              > The problem of a transport map (I have just read the man page, which as
              > usual is highly technical so I am not sure whether I fully understand
              > the purpose and working of transport maps) is that there is a huge
              > overlap between receivers of the low-priority mail list and regular
              > e-mail receivers. Most of the regular e-mail receivers also receive this
              > mail list.

              You may need to do sender-dependent routing for this sender, and inject
              the mail into a different queue, or get the originating system to do
              this directly.

              > > It would not, but you won't saturate the entire link with any given email,
              > > leaving enough room for other traffic. If you can limit the concurrency
              > > of this particular message, then you'll have some bandwidth left over for
              > > other messages.
              >
              > I don't like that idea very much: when I have only a few mails to send
              > out, I want them to go with the maximum speed possible. I have 2 Mbit
              > available, so with 100 smtp connections could limit it to say 20 kbit
              > per smtp process. But that would leave the rest of my bandwidth idle
              > when there are less than 100 active smtp connections, which is the case
              > like 90% of the time.

              Does limiting bandwidth for small messages signicantly impact delivery
              latency? Also who said you should divide the bandwidth 100-fold? You
              give the slow transport 5 parallel threads, and up to half the bandwidth,
              so each channel gets 10% of the bandwidth.

              --
              Viktor.

              Disclaimer: off-list followups get on-list replies or get ignored.
              Please do not ignore the "Reply-To" header.

              To unsubscribe from the postfix-users list, visit
              http://www.postfix.org/lists.html or click the link below:
              <mailto:majordomo@...?body=unsubscribe%20postfix-users>

              If my response solves your problem, the best way to thank me is to not
              send an "it worked, thanks" follow-up. If you must respond, please put
              "It worked, thanks" in the "Subject" so I can delete these quickly.
            Your message has been successfully submitted and would be delivered to recipients shortly.