Loading ...
Sorry, an error occurred while loading the content.

292646Re: R: Scheduling policies for outgoing smtp server

Expand Messages
  • Viktor Dukhovni
    Apr 15, 2013
    • 0 Attachment
      On Mon, Apr 15, 2013 at 01:20:58PM +0200, Giorgio Luchi wrote:

      > How can we implement round-robin by sender ip/authenticated user
      > and to preserve the memory constraint too?
      >
      > - "sender" is the sender's ip address or the authenticated user
      > name (i.e. "80.93.143.50" or "giorgio.luchi")
      > - "rrsender_message_limit" is the max number of messages in the
      > sender queue (i.e. "3")
      > - "rrsender_queue_limit" is the max number of senders actually
      > loaded in RAM (i.e. "6000")
      >
      > [...]
      >
      > I hope this can help in understanding and in finding a solution
      > to what we need.

      Your specific proposal is not viable. The scheduler (queue manager
      processing the active queue) works with a bounded subset of messages
      and message recipients. Think of these as people who are in the
      airport terminal whose ticket class determines which line they may
      join, with some lines getting better service than others.

      The capacity of the capacity of the roads leading to the terminal
      is finite. A first-class ticket holder cannot take advantage of
      preferrential treatment at the terminal if he is stuck behind
      thousands of economy ticket holders on a congested highway.

      The highway is analogous to the Postfix incoming and deferred queues
      and finite capacity of smtpd(8) processes to take in new mail.

      You must abandon all hope of trying to make the highway more "fair"
      than FIFO, this is not possible.

      What is possible is tweaking the algorithm at the terminal. It is
      in principle possible to adjust the scheduler algorithm for selecting
      the next message to send (out of those already in the active queue).

      Right now the scheduler is able to interleave delivery of later
      arriving small messages with progressive delivery of larger (mailing
      list) messages, so that a single newsletter delivery does not
      substantially delay the delivery of all subsequent mail.

      If the same newsletter were to be injected into the queue as as a
      burst of individual messages the scheduler would not be able to
      apply Patrick Raq's nqmgr FIFO preemption logic.

      One could conceivable enhance the algorithm to support a notion
      of job groups, where the pre-emption logic works at two different
      levels.

      - A job group can be preempted by a later job group after the
      first job group consumes enough delivery slots.

      - A job within a group can be preempted by a later job within that
      group after the first job consumes enough delivery slots.

      It would then be up to the input stage to tag queue files with job
      group identifiers. This could be done a policy service that returns
      a new access(5) action to set the job group id.

      The effect is to logically "re-assemble" a multi-message bulk
      mailing from a single source (be it by IP, SASL username, sender
      address, ...) into a single logical message which is subject to
      the scheduler preemption algorithm.

      This is perhaps a fruitful direction, though an ISP would likely
      get more "bang for the buck" by rate limiting input with a policy
      service. The outlined design would be of greater utility in large
      corporate and some hosted email marketing scenarios.

      If anyone were to design an build such a thing on contract, it
      would probably be Patrick Raq (if he's available). Otherwise,
      nobody else comes to mind, you'd have to hire someone who's capable
      of understanding Patrick's design in detail, and extending it
      correctly.

      The result would be unlikely to be incorporated into the mainstream
      Postfix release unless it were exceedingly well documented and
      implemented with care. Most likely you'd be stuck maintaining it
      as a private patch indefinitely.

      So the best approach is perhaps to find some other MTA that does
      what you want if one exists. Otherwise, try to leverage the existing
      feature set to approximate what you want, rate limiting input is
      my best suggestion.

      --
      Viktor.
    • Show all 17 messages in this topic