Loading ...
Sorry, an error occurred while loading the content.

Re: Prioritising outgoing mail

Expand Messages
  • Wouter van Marle
    ... You mean like installing sendmail or so in parallel to postfix and then have sendmail send out the lower-priority mails? ... And how would that get certain
    Message 1 of 8 , Mar 2, 2009
    • 0 Attachment
      On 2 Mar 09, at 23:09, Victor Duchovni wrote:

      > On Mon, Mar 02, 2009 at 10:44:21PM +0800, Wouter van Marle wrote:
      >
      >> Hi list,
      >>
      >> From me a question that seems to be asked now and then here, but I
      >> could
      >> not find any answers even on whether this is possible in the first
      >> place.
      >>
      >> I would like to be able to prioritise outgoing e-mail so they do not
      >> get
      >> stuck in the queue. This as I now and then send out a large number of
      >> e-mails with attachments, and that saturates my connection for a
      >> prolonged
      >> time. It doesn't matter that those mails get out slower, as long as
      >> they
      >> get out eventually I'm happy.
      >
      > Use a custom transport for these messages with a low concurrency limit,

      You mean like installing sendmail or so in parallel to postfix and then
      have sendmail send out the lower-priority mails?

      > or use traffic shaping in the TCP stack to limit the bandwidth per
      > SMTP connection.

      And how would that get certain mails out with priority? It sounds to me
      like this would slow down the overall process. I have up to 100 smtp
      processes running at a time, but as long as new mails end up at the
      back of the queue still no progress there. They have to come first.

      Wouter.

      >
      > --
      > Viktor.
      >
      > Disclaimer: off-list followups get on-list replies or get ignored.
      > Please do not ignore the "Reply-To" header.
      >
      > To unsubscribe from the postfix-users list, visit
      > http://www.postfix.org/lists.html or click the link below:
      > <mailto:majordomo@...?body=unsubscribe%20postfix-users>
      >
      > If my response solves your problem, the best way to thank me is to not
      > send an "it worked, thanks" follow-up. If you must respond, please put
      > "It worked, thanks" in the "Subject" so I can delete these quickly.
      >
      >
    • Wietse Venema
      ... No, use a POSTFIX transport map. ... With the concurrency limit (see above), low priority mail can use up only a limited portion of the bandwidth. Wietse
      Message 2 of 8 , Mar 2, 2009
      • 0 Attachment
        Wouter van Marle:
        >
        > On 2 Mar 09, at 23:09, Victor Duchovni wrote:
        >
        > > On Mon, Mar 02, 2009 at 10:44:21PM +0800, Wouter van Marle wrote:
        > >
        > >> Hi list,
        > >>
        > >> From me a question that seems to be asked now and then here, but I
        > >> could
        > >> not find any answers even on whether this is possible in the first
        > >> place.
        > >>
        > >> I would like to be able to prioritise outgoing e-mail so they do not
        > >> get
        > >> stuck in the queue. This as I now and then send out a large number of
        > >> e-mails with attachments, and that saturates my connection for a
        > >> prolonged
        > >> time. It doesn't matter that those mails get out slower, as long as
        > >> they
        > >> get out eventually I'm happy.
        > >
        > > Use a custom transport for these messages with a low concurrency limit,
        >
        > You mean like installing sendmail or so in parallel to postfix and then
        > have sendmail send out the lower-priority mails?

        No, use a POSTFIX transport map.

        > > or use traffic shaping in the TCP stack to limit the bandwidth per
        > > SMTP connection.
        >
        > And how would that get certain mails out with priority? It sounds to me

        With the concurrency limit (see above), low priority mail can use
        up only a limited portion of the bandwidth.

        Wietse
      • Victor Duchovni
        ... No I mean a Postfix transport , as in transport(5) and master(5). ... It would not, but you won t saturate the entire link with any given email, leaving
        Message 3 of 8 , Mar 2, 2009
        • 0 Attachment
          On Mon, Mar 02, 2009 at 11:59:31PM +0800, Wouter van Marle wrote:

          >> Use a custom transport for these messages with a low concurrency limit,
          >
          > You mean like installing sendmail or so in parallel to postfix and then
          > have sendmail send out the lower-priority mails?
          >

          No I mean a Postfix "transport", as in transport(5) and master(5).

          >> or use traffic shaping in the TCP stack to limit the bandwidth per
          >> SMTP connection.
          >
          > And how would that get certain mails out with priority? It sounds to me
          > like this would slow down the overall process. I have up to 100 smtp
          > processes running at a time, but as long as new mails end up at the back of
          > the queue still no progress there. They have to come first.

          It would not, but you won't saturate the entire link with any given email,
          leaving enough room for other traffic. If you can limit the concurrency
          of this particular message, then you'll have some bandwidth left over for
          other messages.

          --
          Viktor.

          Disclaimer: off-list followups get on-list replies or get ignored.
          Please do not ignore the "Reply-To" header.

          To unsubscribe from the postfix-users list, visit
          http://www.postfix.org/lists.html or click the link below:
          <mailto:majordomo@...?body=unsubscribe%20postfix-users>

          If my response solves your problem, the best way to thank me is to not
          send an "it worked, thanks" follow-up. If you must respond, please put
          "It worked, thanks" in the "Subject" so I can delete these quickly.
        • Wouter van Marle
          ... The problem of a transport map (I have just read the man page, which as usual is highly technical so I am not sure whether I fully understand the purpose
          Message 4 of 8 , Mar 2, 2009
          • 0 Attachment
            On Mon, 2009-03-02 at 11:18 -0500, Victor Duchovni wrote:
            > On Mon, Mar 02, 2009 at 11:59:31PM +0800, Wouter van Marle wrote:
            >
            > >> Use a custom transport for these messages with a low concurrency limit,
            > >
            > > You mean like installing sendmail or so in parallel to postfix and then
            > > have sendmail send out the lower-priority mails?
            >
            > No I mean a Postfix "transport", as in transport(5) and master(5).

            The problem of a transport map (I have just read the man page, which as
            usual is highly technical so I am not sure whether I fully understand
            the purpose and working of transport maps) is that there is a huge
            overlap between receivers of the low-priority mail list and regular
            e-mail receivers. Most of the regular e-mail receivers also receive this
            mail list.

            Many of my mail list receivers are on common domains like gmail.com and
            yahoo.com, thus this would slow down all other mails to those domains as
            well, even if they are not part of the mail list.

            Setting it per recipient is not a good idea because of maintenance
            issues (keeping mail list and transport map in sync), and because of the
            regular mails that may be sent to those recipients while a mail list run
            is in progress.

            The idea of using a "slow:" transport as suggested in the transport(5)
            man page (without elaborating unfortunately...) to limit the number of
            smtp deamons that sounds like the way to go to me. Then I can reserve 80
            deamons for the mail list, or maybe 50, enough to saturate my uplink -
            still allowing regular mails to have an smtp available to be handled
            immediately. This appears to me a way to let the other mails "jump the
            queue" and be processed with priority. That there is little bandwidth
            available is not too much of an issue, then it might take a minute
            instead of seconds to send out, still a major improvement of the hours
            it may take in the current situation.

            > >> or use traffic shaping in the TCP stack to limit the bandwidth per
            > >> SMTP connection.
            > >
            > > And how would that get certain mails out with priority? It sounds to me
            > > like this would slow down the overall process. I have up to 100 smtp
            > > processes running at a time, but as long as new mails end up at the back of
            > > the queue still no progress there. They have to come first.
            >
            > It would not, but you won't saturate the entire link with any given email,
            > leaving enough room for other traffic. If you can limit the concurrency
            > of this particular message, then you'll have some bandwidth left over for
            > other messages.

            I don't like that idea very much: when I have only a few mails to send
            out, I want them to go with the maximum speed possible. I have 2 Mbit
            available, so with 100 smtp connections could limit it to say 20 kbit
            per smtp process. But that would leave the rest of my bandwidth idle
            when there are less than 100 active smtp connections, which is the case
            like 90% of the time.

            Wouter.


            >
          • Wouter van Marle
            Hi all, Would it be possible to add an extension to the user s address, e.g. user+slow@example.com, that would be mapped through a separate transport (e.g. the
            Message 5 of 8 , Mar 2, 2009
            • 0 Attachment
              Hi all,

              Would it be possible to add an extension to the user's address, e.g.
              user+slow@..., that would be mapped through a separate transport
              (e.g. the slow: as suggested in the man page), and be rewritten by
              trivial-rewrite to user@... before being sent out.

              An option like this would do the job for me. It would allow me to easily
              maintain my maillist, rewriting addresses on the fly when creating the
              mails, and allowing regular mails to be handled with priority.

              And any ideas on how/where such a slow: transport (with a limited number
              of smtp daemons to be started) would be configured? I can't find
              anything about this in the man pages. Except that it is possible.

              Wouter.

              On Tue, 2009-03-03 at 11:25 +0800, Wouter van Marle wrote:
              > On Mon, 2009-03-02 at 11:18 -0500, Victor Duchovni wrote:
              > > On Mon, Mar 02, 2009 at 11:59:31PM +0800, Wouter van Marle wrote:
              > >
              > > >> Use a custom transport for these messages with a low concurrency limit,
              > > >
              > > > You mean like installing sendmail or so in parallel to postfix and then
              > > > have sendmail send out the lower-priority mails?
              > >
              > > No I mean a Postfix "transport", as in transport(5) and master(5).
              >
              > The problem of a transport map (I have just read the man page, which as
              > usual is highly technical so I am not sure whether I fully understand
              > the purpose and working of transport maps) is that there is a huge
              > overlap between receivers of the low-priority mail list and regular
              > e-mail receivers. Most of the regular e-mail receivers also receive this
              > mail list.
              >
              > Many of my mail list receivers are on common domains like gmail.com and
              > yahoo.com, thus this would slow down all other mails to those domains as
              > well, even if they are not part of the mail list.
              >
              > Setting it per recipient is not a good idea because of maintenance
              > issues (keeping mail list and transport map in sync), and because of the
              > regular mails that may be sent to those recipients while a mail list run
              > is in progress.
              >
              > The idea of using a "slow:" transport as suggested in the transport(5)
              > man page (without elaborating unfortunately...) to limit the number of
              > smtp deamons that sounds like the way to go to me. Then I can reserve 80
              > deamons for the mail list, or maybe 50, enough to saturate my uplink -
              > still allowing regular mails to have an smtp available to be handled
              > immediately. This appears to me a way to let the other mails "jump the
              > queue" and be processed with priority. That there is little bandwidth
              > available is not too much of an issue, then it might take a minute
              > instead of seconds to send out, still a major improvement of the hours
              > it may take in the current situation.
              >
              > > >> or use traffic shaping in the TCP stack to limit the bandwidth per
              > > >> SMTP connection.
              > > >
              > > > And how would that get certain mails out with priority? It sounds to me
              > > > like this would slow down the overall process. I have up to 100 smtp
              > > > processes running at a time, but as long as new mails end up at the back of
              > > > the queue still no progress there. They have to come first.
              > >
              > > It would not, but you won't saturate the entire link with any given email,
              > > leaving enough room for other traffic. If you can limit the concurrency
              > > of this particular message, then you'll have some bandwidth left over for
              > > other messages.
              >
              > I don't like that idea very much: when I have only a few mails to send
              > out, I want them to go with the maximum speed possible. I have 2 Mbit
              > available, so with 100 smtp connections could limit it to say 20 kbit
              > per smtp process. But that would leave the rest of my bandwidth idle
              > when there are less than 100 active smtp connections, which is the case
              > like 90% of the time.
              >
              > Wouter.
              >
              >
              > >
              >
              >
            • Victor Duchovni
              ... You may need to do sender-dependent routing for this sender, and inject the mail into a different queue, or get the originating system to do this directly.
              Message 6 of 8 , Mar 2, 2009
              • 0 Attachment
                On Tue, Mar 03, 2009 at 11:25:55AM +0800, Wouter van Marle wrote:

                > On Mon, 2009-03-02 at 11:18 -0500, Victor Duchovni wrote:
                > > On Mon, Mar 02, 2009 at 11:59:31PM +0800, Wouter van Marle wrote:
                > >
                > > >> Use a custom transport for these messages with a low concurrency limit,
                > > >
                > > > You mean like installing sendmail or so in parallel to postfix and then
                > > > have sendmail send out the lower-priority mails?
                > >
                > > No I mean a Postfix "transport", as in transport(5) and master(5).
                >
                > The problem of a transport map (I have just read the man page, which as
                > usual is highly technical so I am not sure whether I fully understand
                > the purpose and working of transport maps) is that there is a huge
                > overlap between receivers of the low-priority mail list and regular
                > e-mail receivers. Most of the regular e-mail receivers also receive this
                > mail list.

                You may need to do sender-dependent routing for this sender, and inject
                the mail into a different queue, or get the originating system to do
                this directly.

                > > It would not, but you won't saturate the entire link with any given email,
                > > leaving enough room for other traffic. If you can limit the concurrency
                > > of this particular message, then you'll have some bandwidth left over for
                > > other messages.
                >
                > I don't like that idea very much: when I have only a few mails to send
                > out, I want them to go with the maximum speed possible. I have 2 Mbit
                > available, so with 100 smtp connections could limit it to say 20 kbit
                > per smtp process. But that would leave the rest of my bandwidth idle
                > when there are less than 100 active smtp connections, which is the case
                > like 90% of the time.

                Does limiting bandwidth for small messages signicantly impact delivery
                latency? Also who said you should divide the bandwidth 100-fold? You
                give the slow transport 5 parallel threads, and up to half the bandwidth,
                so each channel gets 10% of the bandwidth.

                --
                Viktor.

                Disclaimer: off-list followups get on-list replies or get ignored.
                Please do not ignore the "Reply-To" header.

                To unsubscribe from the postfix-users list, visit
                http://www.postfix.org/lists.html or click the link below:
                <mailto:majordomo@...?body=unsubscribe%20postfix-users>

                If my response solves your problem, the best way to thank me is to not
                send an "it worked, thanks" follow-up. If you must respond, please put
                "It worked, thanks" in the "Subject" so I can delete these quickly.
              Your message has been successfully submitted and would be delivered to recipients shortly.