Loading ...
Sorry, an error occurred while loading the content.

262331Re: smtpd processes congregating at the pub

Expand Messages
  • Stan Hoeppner
    Jan 31, 2010
    • 0 Attachment
      Wietse Venema put forth on 1/31/2010 10:38 AM:
      > Stan Hoeppner:
      >> This is making good progress. Seeing the smtpd's memory footprint
      >> drop so dramatically is fantastic. However, I'm still curious as
      >> to why proxymap doesn't appear to be honoring $max_idle or $max_use.
      >> Maybe my understanding of $max_use is not correct? It's currently
      >> set to 100, the default. Watching top while sending a test message
      >> through, I see proxymap launch but then exit within 5 seconds,
      >> while smtpd honors max_idle. Is there some other setting I need
      >> to change to keep proxymap around longer?
      >
      > Short answer (workaround for low-traffic sites): set ipc_idle=$max_idle
      > to approximate the expected behavior. This keeps the smtpd-to-proxymap
      > connection open for as long as smtpd runs. Then, proxymap won't
      > terminate before its clients terminate.

      Wietse, thank you for the very thorough and thoughtful response. For a few
      reasons, including the fact I don't trust myself working with source in this
      case, and that I'd rather not throw monkey wrenches into my distro's package
      management, I'm going to go with the short answer workaround above. All factors
      being taken into account, I think it best fits my needs, skills, and usage profile.

      > Better: apply the long-term solution, in the form of the patch below.
      > This undoes the max_idle override (a workaround that I introduced
      > with Postfix 2.3). I already introduced the better solution with
      > Postfix 2.4 while solving a different problem.

      I'm not sure if I fully understand this. I'm using 2.5.5, so shouldn't I
      already have the 2.4 solution mentioned above? I must not be reading this
      correctly.

      > Long answer: in ancient times, all Postfix daemons except qmgr
      > implemented the well-known max_idle=100s and max_use=100, as well
      > as the lesser-known ipc_idle=100s (see "short answer" for the effect
      > of that parameter).
      >
      > While this worked fine for single-client servers such as smtpd, it
      > was not so great for multi-client servers such as proxymap or
      > trivial-rewrite. This problem was known, and the idea was that it
      > would be solved over time.
      >
      > Theoretically, smtpd could run for up to $max_idle * $max_use = 3
      > hours, while proxymap and trivial-rewrite could run for up to
      > $max_idle * $max_use * $max_use = 12 days on low-traffic systems
      > (one SMTP client every 100s, or a little under 900 SMTP clients a
      > day), and it would run forever on systems with a steady mail flow.
      >
      > This was a problem. The point of max_use is to limit the impact of
      > bugs such as memory or file handle leaks, by retiring a process
      > after doing a limited amount of work. I can test Postfix itself
      > with tools such as Purify and Valgrind, but I can't do those tests
      > with every version of everyone's system libraries.

      This is a very smart design philosophy. Just one more reason I feel privileged
      to use Postfix.

      > If a proxymap or trivial-rewrite server can run for 11 days even
      > on systems with a minuscule load, then max_use isn't working as
      > intended.
      >
      > The main cause is that the proxymap etc. clients reuse a connection
      > to improve efficiency. Therefore, the proxymap etc. server politely
      > waits until all its clients have disconnected before checking the
      > max_use counter. While this politeness thing can't be changed
      > easily, it is relatively easy to play with the proxymap etc. server's
      > max_idle value, and with the smtpd etc. ipc_ttl value.
      >
      > Postfix 2.3 reduced the proxymap etc. max_idle to a fixed 1s value
      > to make those processes go away sooner when idle. I think that
      > this was a mistake, because it makes processes terminate too soon,
      > and thereby worsens the low-traffic behavior. Instead, we should
      > speed up the proxymap etc. server's max_use counter.

      > Postfix 2.4 reduced ipc_ttl to 5s. This was done for a different
      > purpose: to allow proxymap etc. clients to switch to the least-loaded
      > proxymap etc. server. But, I think that this was also the right way
      > to deal with long-lived proxymap etc. processes, because it speeds
      > up the proxymap etc. max_use counter.

      Absolutely fascinating background information Wietse. Thank you for sharing
      this. It's always nice to learn how/why some things work "under the hood";
      things that often can't easily be found in any official documentation.

      --
      Stan
    • Show all 16 messages in this topic