Loading ...
Sorry, an error occurred while loading the content.

Running on idle systems

Expand Messages
  • Michael Tokarev
    Hello. I already mentioned this topic several years ago, and described a technique I used to compensate the problem at this time (and it is still usable and in
    Message 1 of 23 , May 1, 2012
    • 0 Attachment
      Hello.

      I already mentioned this topic several years ago, and
      described a technique I used to compensate the problem
      at this time (and it is still usable and in use today).

      The problem is that on typical workstation or any other
      non-mail-heavy-load machine, postfix in its default
      configuration constantly triggeres filesystem writes,
      which results in either disks spinning up or SSDs doing
      writes where they can just idle.

      This happens due to master constantly (every 5 minutes
      by default) poking qmgr to re-scan its queues. This is
      done by writing a byte into public/qmgr pipe, which
      triggers mtime update on filesystem level, which in turn
      results in a write - single write every 5 minutes, and
      this happens all the time even if the system does not
      deliver any mail at all.


      All major Unixes these days have one or another sort of
      tmpfs filesystem - a filesystem residing in RAM entirely
      (or it might be backed up by swap storage) but acting as
      a regular filesystem. Files on such filesystem aren't
      preserved during reboot obviously.

      The "trick" I use with postfix for a long time locally
      to address this issue is to mount a tmpfs on linux on
      /var/spool/postfix/run, create subdirs (pid, public,
      private) there and replace the corresponding subdirs
      in /var/spool/postfix to symlinks to these directories.
      Separate /run is to avoid the need to create several
      filesystems (one for each of pid/, public/ and private/).

      This works, but postfix complains that it finds dirs with
      "wrong" permissions (the symlinks have 0777 mode).

      So, the question is: can postfix change the paths so
      that all these "runtime" dirs are in a single subdir
      (as run/pid, run/public, run/private) - that is changing
      3 strings in postfix code (and postfix-files) and adding
      one extra line into postfix-files, -- in order to simplify
      such a setup ?

      I'm fine to change postfix-files locally so it looks at
      run/pid etc instead of pid, but that appears to be a bit
      ugly...

      (One can argue the same writes will happen due to atime
      updates on the directories qmgr scans, but again, many
      modern OSes addressed this problem already, by -
      optionally - omitting atime updates if atime already
      greather than mtime (on linux, this mount option is
      named relatime, and is on by default on modern kernels))

      Thanks!

      /mjt
    • Wietse Venema
      ... Sounds reasonable. However, this is not the only option. 1) Mounting tmpfs over a mail queue subdirectory introduces additional dependency (something
      Message 2 of 23 , May 1, 2012
      • 0 Attachment
        Michael Tokarev:
        > The "trick" I use with postfix for a long time locally
        > to address this issue is to mount a tmpfs on linux on
        > /var/spool/postfix/run, create subdirs (pid, public,
        > private) there [...]
        >
        > So, the question is: can postfix change the paths so
        > that all these "runtime" dirs are in a single subdir
        > (as run/pid, run/public, run/private) - that is changing
        > 3 strings in postfix code (and postfix-files) and adding
        > one extra line into postfix-files, -- in order to simplify
        > such a setup ?

        Sounds reasonable. However, this is not the only option.

        1) Mounting tmpfs over a mail queue subdirectory introduces additional
        dependency (something needs to mount and populate spool/run
        before any Postfix program is allowed to execute) otherwise there
        will be surprises when a Postfix program needs to be executed
        before everything is up and running (for example, when some boot
        script wants to submit mail about saved editor buffers, this
        will result in "postdrop: warning: unable to look up public/pickup:
        No such file or directory"). Normally, a missing "public/pickup"
        directory is a sign of major trouble (no mail will ever be picked
        up). This would violate one basic Postfix principle, which is
        that it must be able to submit mail at any time even before all
        daemons are up and running.

        2) We could wait until all commodity boxes have solid-state drives.
        SSDs have many desirable properties and prices are coming down.

        3) Instead of changing the file system layout, we could change
        master.cf to use "unix" instead of "fifo" endpoints for the queue
        manager and pickup services. The reason for using FIFOs is that
        Solaris 2.4 UNIX-domain sockets locked up the kernel during
        Postfix tests (the change to FIFOs then exposed bugs in several
        4.4BSD implementations, but I disgress).

        To this date, both the trigger client (used in postdrop and cleanup)
        and trigger server (used in qmgr and pickup) contain code for
        UNIX-domain sockets. This code has not been tested in 12 years,
        so minor updates are likely needed. But the benefit of this is
        that there are no visible changes except four bytes in master.cf.

        I would prefer the third solution, as it introduces no external
        dependencies on boot scripts to mount and populate spool/postfix/run,
        and it introduces no visible changes to the file system layout.

        > I'm fine to change postfix-files locally so it looks at
        > run/pid etc instead of pid, but that appears to be a bit
        > ugly...

        I don't understand why this is needed at all. On my own systems,
        "ls -lat /var/spool/postfix/pid" shows that there are no writes to
        the directory itself or to any files in that directory after the
        mail system has started up.

        Wietse
      • Wietse Venema
        ... I just checked: this code works fine on FreeBSD, and I expect no surprises on Linux. With this, there are no write operations under the public
        Message 3 of 23 , May 1, 2012
        • 0 Attachment
          Wietse Venema:
          > 3) Instead of changing the file system layout, we could change
          > master.cf to use "unix" instead of "fifo" endpoints for the queue
          > manager and pickup services. The reason for using FIFOs is that
          > Solaris 2.4 UNIX-domain sockets locked up the kernel during
          > Postfix tests (the change to FIFOs then exposed bugs in several
          > 4.4BSD implementations, but I disgress).
          >
          > To this date, both the trigger client (used in postdrop and cleanup)
          > and trigger server (used in qmgr and pickup) contain code for
          > UNIX-domain sockets. This code has not been tested in 12 years,
          > so minor updates are likely needed. But the benefit of this is
          > that there are no visible changes except four bytes in master.cf.

          I just checked: this code works fine on FreeBSD, and I expect no
          surprises on Linux.

          With this, there are no "write" operations under the "public"
          directory once Postfix has been started, so the disk should
          not spin up due to mtime changes.

          However, this can't be used on Solaris until it can be demonstrated
          that their UNIX-domain sockets are up to the job (and even then,
          one has to allow for the existence of Solaris systems that do
          not run the latest and greatest. So I will be stuck supporting
          multiple IPC models.

          Wietse

          > I would prefer the third solution, as it introduces no external
          > dependencies on boot scripts to mount and populate spool/postfix/run,
          > and it introduces no visible changes to the file system layout.
          >
          > > I'm fine to change postfix-files locally so it looks at
          > > run/pid etc instead of pid, but that appears to be a bit
          > > ugly...
          >
          > I don't understand why this is needed at all. On my own systems,
          > "ls -lat /var/spool/postfix/pid" shows that there are no writes to
          > the directory itself or to any files in that directory after the
          > mail system has started up.
          >
          > Wietse
          >
        • Michael Tokarev
          ... By default these dirs -- /var/spool/postfix/run/{pid,private,public} - should be created at install time just like now these dirs are create in
          Message 4 of 23 , May 1, 2012
          • 0 Attachment
            On 01.05.2012 17:09, Wietse Venema wrote:
            > Michael Tokarev:
            >> The "trick" I use with postfix for a long time locally
            >> to address this issue is to mount a tmpfs on linux on
            >> /var/spool/postfix/run, create subdirs (pid, public,
            >> private) there [...]
            >>
            >> So, the question is: can postfix change the paths so
            >> that all these "runtime" dirs are in a single subdir
            >> (as run/pid, run/public, run/private) - that is changing
            >> 3 strings in postfix code (and postfix-files) and adding
            >> one extra line into postfix-files, -- in order to simplify
            >> such a setup ?
            >
            > Sounds reasonable. However, this is not the only option.
            >
            > 1) Mounting tmpfs over a mail queue subdirectory introduces additional
            > dependency (something needs to mount and populate spool/run
            > before any Postfix program is allowed to execute) otherwise there
            > will be surprises when a Postfix program needs to be executed
            > before everything is up and running (for example, when some boot
            > script wants to submit mail about saved editor buffers, this
            > will result in "postdrop: warning: unable to look up public/pickup:
            > No such file or directory"). Normally, a missing "public/pickup"
            > directory is a sign of major trouble (no mail will ever be picked
            > up). This would violate one basic Postfix principle, which is
            > that it must be able to submit mail at any time even before all
            > daemons are up and running.

            By default these dirs -- /var/spool/postfix/run/{pid,private,public} -
            should be created at install time just like now these dirs are create
            in /var/spool/postfix. The only change I'm asking about is to move
            the "runtime" directories into a subdir (suggesting to name it "run"),
            I don't ask to mount anything in there or to create these on boot.

            With this scheme, all default installs will be functional just like
            now, but the option to mount some tmpfs over /var/spool/postfix/run
            _and_ create {pid,private,public} subdirs in there will be easier.

            That's whole difference: just moving these 3 into a subdir, to
            group them together, all the rest remains the same.

            This change does not add any additional external dependency. Only
            when some separate filesystem gets mounted over there, only in that
            case something should create the subdirs inside, and this "something"
            can be the same which did the mount.

            > 2) We could wait until all commodity boxes have solid-state drives.
            > SSDs have many desirable properties and prices are coming down.

            This question has been asked in context of SSDs too, like, it is a
            good idea to avoid extra writes to SSD. I've no idea how reliable
            these beasts are, actually, - reports varies greatly, probably as
            with everything else, there are cheaper less reliable models and
            more expensive, but more reliable. I dunno, I never used any SSD
            myself so far.

            > 3) Instead of changing the file system layout, we could change
            > master.cf to use "unix" instead of "fifo" endpoints for the queue
            > manager and pickup services. The reason for using FIFOs is that
            > Solaris 2.4 UNIX-domain sockets locked up the kernel during
            > Postfix tests (the change to FIFOs then exposed bugs in several
            > 4.4BSD implementations, but I disgress).

            Yes, I remember that change to work around an issue on solaris.

            After all, this can be a compile-time option in sys_defs.h. I
            think I suggested just that when the original bug has been
            identified, but I don't remember why FIFO has been choosen
            unconditionally -- most likely because it just works everywhere.

            But I know you dislike partial solutions - me dislikes them too :)

            > I would prefer the third solution, as it introduces no external
            > dependencies on boot scripts to mount and populate spool/postfix/run,
            > and it introduces no visible changes to the file system layout.
            >
            >> I'm fine to change postfix-files locally so it looks at
            >> run/pid etc instead of pid, but that appears to be a bit
            >> ugly...
            >
            > I don't understand why this is needed at all. On my own systems,
            > "ls -lat /var/spool/postfix/pid" shows that there are no writes to
            > the directory itself or to any files in that directory after the
            > mail system has started up.

            Note the word "etc" after run/pid. I added all 3 "runtime" dirs into
            the mix -- pid, private, public -- the directories where the files
            makes any sense at all till postfix is running _only_. Files in
            there does not exist, so to say, when it is not running - even if
            formally corresponding inodes do exist.

            This is exactly the case for /var/run directory, and nowadays it
            become quite common to use some tmpfs for this directory - one of
            the reasons is a problem of pid boot files after fresh boot (or,
            you may say it is a problem of stupid startup scripts who does not
            verify that the pid listed in a pidfile actually correspond to our
            daemon and not some other random process - but tmpfs on /var/run
            makes this problem go away completely).

            That's why I grouped all 3 "runtime" dirs together in my setup and
            moved them into a tmpfs - there's just no reason for them to be in
            a regular filesystem which preserves content over reboots. Yes
            it is obviously not necessary to do this just to stop the filesystem
            writes and spinning up disks.

            Besides, the same effect can be achieved by using subdirs of "pid"
            directory for private and public dirs, -- ie, instead of

            run/pid, run/private, run/public

            it is equally easy to have

            pid, pid/private, pid/public

            since files in pid/ does not clash with names "private" and "public".
            This way, filesystem layout changes less, but that's unimportant
            details.

            And yes, I verified the socket code (instead of pipe code) on linux
            a few days ago and it appears to work fine there too. So indeed, this
            is a very good possibility too, but it does not cover solaris well.

            Thanks,

            /mjt
          • Postfix Support Mail
            I m currently running postgrey, but a recent thread here got me thinking about postscreen, which I hadn t considered before. What are the pros and cons of one
            Message 5 of 23 , May 1, 2012
            • 0 Attachment
              I'm currently running postgrey, but a recent thread here got me thinking
              about postscreen, which I hadn't considered before.

              What are the pros and cons of one versus the other? Are there advantages of
              one over the other for a given application?

              --Mac
            • karavelov@mail.bg
              ... They use different, complementing strategies to fight spam. We currently run both of them (postscreen, greylisting) with great success. Another greylisting
              Message 6 of 23 , May 1, 2012
              • 0 Attachment
                ----- Цитат от Postfix Support Mail (postfixsm@...), на 01.05.2012 в 22:14 -----

                > I'm currently running postgrey, but a recent thread here got me thinking
                > about postscreen, which I hadn't considered before.
                >
                > What are the pros and cons of one versus the other? Are there advantages of
                > one over the other for a given application?
                >
                > --Mac


                They use different, complementing strategies to fight spam. We currently run
                both of them (postscreen, greylisting) with great success.

                Another greylisting server that uses memcached as a backend and is written for
                postfix could be found here:
                https://github.com/luben/postfix-utils


                --
                Luben Karavelov
              • Noel Jones
                ... Please don t hijack threads. Start a new message instead of changing the subject of an existing message. Postgrey and postscreen do different things.
                Message 7 of 23 , May 1, 2012
                • 0 Attachment
                  postfixsm@...:
                  > I'm currently running postgrey, but a recent thread here got me thinking
                  > about postscreen, which I hadn't considered before.
                  >
                  > What are the pros and cons of one versus the other? Are there advantages of
                  > one over the other for a given application?
                  >
                  > --Mac

                  Please don't hijack threads. Start a new message instead of
                  changing the subject of an existing message.

                  Postgrey and postscreen do different things. After you read up on
                  them, you may want both.

                  http://www.postfix.org/POSTSCREEN_README.html
                  http://postgrey.schweikert.ch/



                  -- Noel Jones
                • Postfix Support Mail
                  Sorry about that. Reading the postscreen readme is what spawned the question. ## -----Original Message----- ## From: Noel Jones
                  Message 8 of 23 , May 1, 2012
                  • 0 Attachment
                    Sorry about that.

                    Reading the postscreen readme is what spawned the question.

                    ## >> -----Original Message-----
                    ## >> From: Noel Jones [mailto:njones@...]
                    ## >> Sent: Tuesday, 01 May, 2012 12:29
                    ## >> To: postfix users; postfixsm@...
                    ## >> Subject: Re: postgrey vs postscreen
                    ## >>
                    ## >> postfixsm@...:
                    ## >> > I'm currently running postgrey, but a recent thread here got me
                    ## >> > thinking about postscreen, which I hadn't considered before.
                    ## >> >
                    ## >> > What are the pros and cons of one versus the other? Are there
                    ## >> > advantages of one over the other for a given application?
                    ## >> >
                    ## >> > --Mac
                    ## >>
                    ## >> Please don't hijack threads. Start a new message instead
                    ## >> of changing the subject of an existing message.
                    ## >>
                    ## >> Postgrey and postscreen do different things. After you
                    ## >> read up on them, you may want both.
                    ## >>
                    ## >> http://www.postfix.org/POSTSCREEN_README.html
                    ## >> http://postgrey.schweikert.ch/
                    ## >>
                    ## >>
                    ## >>
                    ## >> -- Noel Jones
                  • Wietse Venema
                    Michael Tokarev: [using unix instead of fifo ] ... The preferred pickup/qmgr IPC type (fifo or unix) can be a main.cf parameter setting (with an
                    Message 9 of 23 , May 1, 2012
                    • 0 Attachment
                      Michael Tokarev:
                      [using "unix" instead of "fifo"]
                      > And yes, I verified the socket code (instead of pipe code) on linux
                      > a few days ago and it appears to work fine there too. So indeed, this
                      > is a very good possibility too, but it does not cover solaris well.

                      The preferred pickup/qmgr IPC type (fifo or unix) can be a main.cf
                      parameter setting (with an OS-dependent default value, e.g., fifo
                      for Solaris and unix for everything else), and post-install can be
                      updated to edit master.cf accordingly.

                      This solves the "idle" mtime update problem a manner that is 100%
                      maintainable. It involves adding one configuration parameter to
                      main.cf, and adding a few lines to post-install that edit master.cf
                      accordingly.

                      > By default these dirs -- /var/spool/postfix/run/{pid,private,public} -
                      > should be created at install time just like now these dirs are create
                      > in /var/spool/postfix. The only change I'm asking about is to move
                      > the "runtime" directories into a subdir (suggesting to name it "run"),
                      > I don't ask to mount anything in there or to create these on boot.

                      Sorry, that is not how Postfix is maintained.

                      All aspects of Postfix are meant to be for general use, so that
                      they are widely used, so that they can be routinely tested during
                      development, so that they can be properly supported, and so that
                      they don't break as Postfix evolves.

                      Also, mounting stuff over Postfix directories breaks the basic model
                      of how Postfix manages its files, and I definitely don't want some
                      (Linux) distro maintainer to pick up on the idea. I have enough
                      work on my hands to deal with damage control of (Linux) distro
                      maintainers.

                      [empty run directory]
                      You seem to believe that Postfix IPC endpoints don't need to exist
                      while Postfix is not running.

                      > the mix -- pid, private, public -- the directories where the files
                      > makes any sense at all till postfix is running _only_. Files in
                      > there does not exist, so to say, when it is not running - even if
                      > formally corresponding inodes do exist.

                      That is incorrect.

                      For example, the postdrop program will produce a warning message
                      if the public/pickup node does not exist. By design this program
                      MUST be able to run without errors while NO Postfix daemons are
                      running. Normally, a missing public/pickup node is a sign of great
                      trouble (mail would be queued forever) and it would therefore be
                      wrong to shut up the error message.

                      > [waiting for SSD to become a commodity product]

                      An SSD has a write cycle limit of a million times or so. It should
                      last forever with one mtime update per 5 minutes.

                      Wietse
                    • Michael Orlitzky
                      ... If you enable the deep protocol tests, postscreen works pretty much like greylisting since it will 4xx any client that passes. When they reconnect, they
                      Message 10 of 23 , May 1, 2012
                      • 0 Attachment
                        On 05/01/2012 03:42 PM, Postfix Support Mail wrote:
                        > Sorry about that.
                        >
                        > Reading the postscreen readme is what spawned the question.
                        >

                        If you enable the deep protocol tests, postscreen works pretty much like
                        greylisting since it will 4xx any client that passes. When they
                        reconnect, they skip postscreen entirely; in effect, they've passed
                        greylisting.

                        Postgrey is a little more configurable (re: greylisting), and its
                        introduction is less intrusive than switching will be. If you have to
                        support older versions of postfix (sans postscreen), it also allows you
                        to consolidate your configurations.

                        On the other hand, I trust the postscreen code to be of the same
                        quality as the rest of postfix. Postscreen does provide some additional
                        anti-spam measures and reduces the load on your system. Finally, when
                        all's said and done, you've got one less[1] moving part to worry about.

                        Perhaps my favorite benefit is that with postscreen, we can afford the
                        resources to use a pre-queue content filter. Now senders get a rejection
                        if we e.g. detect a virus in their message. This greatly reduces the
                        amount of time I have to spend on the phone.

                        I would recommend switching eventually.


                        [1] Technically there are probably more, but let's say "logical" moving
                        parts.
                      • Terry Barnum
                        ... Mac, I made the switch to postscreen about week ago. I used http://www.postfix.org/postscreen.8.html and /dev/rob0 s guide at
                        Message 11 of 23 , May 1, 2012
                        • 0 Attachment
                          On May 1, 2012, at 12:14 PM, Postfix Support Mail wrote:

                          > I'm currently running postgrey, but a recent thread here got me thinking
                          > about postscreen, which I hadn't considered before.
                          >
                          > What are the pros and cons of one versus the other? Are there advantages of
                          > one over the other for a given application?
                          >
                          > --Mac

                          Mac,

                          I made the switch to postscreen about week ago. I used http://www.postfix.org/postscreen.8.html and /dev/rob0's guide at http://permalink.gmane.org/gmane.mail.postfix.user/218114 and also turned on deep protocol tests. I have not done any empirical testing here but my impression is that it works at least as well as postgrey. I like that it's seamlessly integrated into postfix.

                          -Terry
                        • Robert Schetterer
                          ... depending on your setup, ( read postscreen readme.... ) you can combine them, specially if you use postgrey only selective which you should allready do yet
                          Message 12 of 23 , May 1, 2012
                          • 0 Attachment
                            Am 01.05.2012 21:14, schrieb Postfix Support Mail:
                            > I'm currently running postgrey, but a recent thread here got me thinking
                            > about postscreen, which I hadn't considered before.
                            >
                            > What are the pros and cons of one versus the other? Are there advantages of
                            > one over the other for a given application?
                            >
                            > --Mac
                            >
                            >

                            depending on your setup, ( read postscreen readme.... )
                            you can combine them, specially if you use postgrey only selective
                            which you should allready do yet

                            --
                            Best Regards

                            MfG Robert Schetterer

                            Germany/Munich/Bavaria
                          • Michael Tokarev
                            ... Maybe this is something which don t really need any configuration parameters, just pick up whatever is best on current OS, from prior knowlege. But
                            Message 13 of 23 , May 2, 2012
                            • 0 Attachment
                              02.05.2012 00:14, Wietse Venema wrote:
                              > Michael Tokarev:
                              > [using "unix" instead of "fifo"]
                              >> And yes, I verified the socket code (instead of pipe code) on linux
                              >> a few days ago and it appears to work fine there too. So indeed, this
                              >> is a very good possibility too, but it does not cover solaris well.
                              >
                              > The preferred pickup/qmgr IPC type (fifo or unix) can be a main.cf
                              > parameter setting (with an OS-dependent default value, e.g., fifo
                              > for Solaris and unix for everything else), and post-install can be
                              > updated to edit master.cf accordingly.

                              Maybe this is something which don't really need any configuration
                              parameters, just pick up whatever is best on current OS, from prior
                              knowlege. But indeed, config parameter there will do the job, and
                              will even let us test easily if things have changed on solaris,
                              without a need to recompile postfix.

                              I'm a bit afraid of possible questions this change may generate
                              in the future, when people will start asking "what it is for,
                              why can't it be hardcoded" etc. :)

                              > This solves the "idle" mtime update problem a manner that is 100%
                              > maintainable. It involves adding one configuration parameter to
                              > main.cf, and adding a few lines to post-install that edit master.cf
                              > accordingly.

                              Yes, it should solve the idle mtime update issue. Hopefully unix
                              sockets does not have this problem -- at least on linux, mtime on
                              these isn't updated on write.

                              >> By default these dirs -- /var/spool/postfix/run/{pid,private,public} -
                              >> should be created at install time just like now these dirs are create
                              >> in /var/spool/postfix. The only change I'm asking about is to move
                              >> the "runtime" directories into a subdir (suggesting to name it "run"),
                              >> I don't ask to mount anything in there or to create these on boot.
                              >
                              > Sorry, that is not how Postfix is maintained.
                              >
                              > All aspects of Postfix are meant to be for general use, so that
                              > they are widely used, so that they can be routinely tested during
                              > development, so that they can be properly supported, and so that
                              > they don't break as Postfix evolves.

                              Renaming directories does not break this. But see below...

                              > Also, mounting stuff over Postfix directories breaks the basic model
                              > of how Postfix manages its files, and I definitely don't want some
                              > (Linux) distro maintainer to pick up on the idea. I have enough
                              > work on my hands to deal with damage control of (Linux) distro
                              > maintainers.

                              Well, I already suggested this to Debian - to mount /var/spool/postfix/run
                              and change pid, private and public dirs to be symlinks into subdirs
                              in run/ -- exactly how I manage it locally for many years.

                              Switching to unix sockets should eliminate the need for this.
                              But yet again, see below...

                              > [empty run directory]
                              > You seem to believe that Postfix IPC endpoints don't need to exist
                              > while Postfix is not running.
                              >
                              >> the mix -- pid, private, public -- the directories where the files
                              >> makes any sense at all till postfix is running _only_. Files in
                              >> there does not exist, so to say, when it is not running - even if
                              >> formally corresponding inodes do exist.
                              >
                              > That is incorrect.
                              >
                              > For example, the postdrop program will produce a warning message
                              > if the public/pickup node does not exist. By design this program
                              > MUST be able to run without errors while NO Postfix daemons are
                              > running. Normally, a missing public/pickup node is a sign of great
                              > trouble (mail would be queued forever) and it would therefore be
                              > wrong to shut up the error message.

                              Aha. And that's exactly what I overlooked. I didn't knw that in
                              case public/pickup is not present, postdrop will complain loudly.
                              Indeed, this is a serious enough reason to rethink the whole idea.
                              So I'll hold on with the mentioned changes for debian, I definitely
                              don't want to break postfix in any way.

                              Besides, the same issue (missing pickup node) exists when postfix
                              has been installed but hasn't been run even once. That one appears
                              to be of much less severity.

                              So, to sum it all up, it looks like the best solution will be to
                              switch to unix sockets, either with a config parameter or without.
                              In any way, I'd pick saner per-OS defaults for this, with sockets
                              for everything but solaris which will continue using fifos.

                              Thanks,

                              /mjt
                            • Wietse Venema
                              ... $ postconf -d|wc -l 920 No-one has complained sofar. ... Thank you for making my worst nightmares come true. I will do my best to prevent this from
                              Message 14 of 23 , May 2, 2012
                              • 0 Attachment
                                Michael Tokarev:
                                > > The preferred pickup/qmgr IPC type (fifo or unix) can be a main.cf
                                > > parameter setting (with an OS-dependent default value, e.g., fifo
                                > > for Solaris and unix for everything else), and post-install can be
                                > > updated to edit master.cf accordingly.
                                >
                                > Maybe this is something which don't really need any configuration
                                > parameters, just pick up whatever is best on current OS, from prior
                                > knowlege. But indeed, config parameter there will do the job, and
                                > will even let us test easily if things have changed on solaris,
                                > without a need to recompile postfix.
                                >
                                > I'm a bit afraid of possible questions this change may generate
                                > in the future, when people will start asking "what it is for,
                                > why can't it be hardcoded" etc. :)

                                $ postconf -d|wc -l
                                920

                                No-one has complained sofar.

                                > > Also, mounting stuff over Postfix directories breaks the basic model
                                > > of how Postfix manages its files, and I definitely don't want some
                                > > (Linux) distro maintainer to pick up on the idea. I have enough
                                > > work on my hands to deal with damage control of (Linux) distro
                                > > maintainers.
                                >
                                > Well, I already suggested this to Debian - to mount /var/spool/postfix/run

                                Thank you for making my worst nightmares come true. I will do
                                my best to prevent this from happening, and if I find out that
                                they do it anyway, then I will raise hell and it won't be pretty.

                                Wietse
                              • Stan Hoeppner
                                ... To who at Debian? Lamont Jones? Has he replied to your idiotic idea yet? ... All of this nonsense because one guy on the planet feels he can t simply use
                                Message 15 of 23 , May 3, 2012
                                • 0 Attachment
                                  On 5/2/2012 6:02 AM, Wietse Venema wrote:
                                  > Michael Tokarev:
                                  >>> The preferred pickup/qmgr IPC type (fifo or unix) can be a main.cf
                                  >>> parameter setting (with an OS-dependent default value, e.g., fifo
                                  >>> for Solaris and unix for everything else), and post-install can be
                                  >>> updated to edit master.cf accordingly.
                                  >>
                                  >> Maybe this is something which don't really need any configuration
                                  >> parameters, just pick up whatever is best on current OS, from prior
                                  >> knowlege. But indeed, config parameter there will do the job, and
                                  >> will even let us test easily if things have changed on solaris,
                                  >> without a need to recompile postfix.
                                  >>
                                  >> I'm a bit afraid of possible questions this change may generate
                                  >> in the future, when people will start asking "what it is for,
                                  >> why can't it be hardcoded" etc. :)
                                  >
                                  > $ postconf -d|wc -l
                                  > 920
                                  >
                                  > No-one has complained sofar.
                                  >
                                  >>> Also, mounting stuff over Postfix directories breaks the basic model
                                  >>> of how Postfix manages its files, and I definitely don't want some
                                  >>> (Linux) distro maintainer to pick up on the idea. I have enough
                                  >>> work on my hands to deal with damage control of (Linux) distro
                                  >>> maintainers.
                                  >>
                                  >> Well, I already suggested this to Debian - to mount /var/spool/postfix/run

                                  To who at Debian? Lamont Jones? Has he replied to your idiotic idea yet?

                                  > Thank you for making my worst nightmares come true. I will do
                                  > my best to prevent this from happening, and if I find out that
                                  > they do it anyway, then I will raise hell and it won't be pretty.

                                  All of this nonsense because one guy on the planet feels he can't simply
                                  use an MUA with submission like everyone else does, but demands he be
                                  able to run an MTA on his damn desktop/laptop, and demands the default
                                  MTA config allows him to do what he wants seamlessly, possibly to the
                                  detriment of others, mainly the guy who wrote this MTA for your use in
                                  the first place. At least that's my read of this thread.

                                  --
                                  Stan
                                • Viktor Dukhovni
                                  ... Stan, please have the decency to not reflexively abuse others on this list, or otherwise leave. Michael, thanks for all your contributions over the years.
                                  Message 16 of 23 , May 3, 2012
                                  • 0 Attachment
                                    On Thu, May 03, 2012 at 08:16:33AM -0500, Stan Hoeppner wrote:

                                    > Has he replied to your idiotic idea yet?

                                    Stan, please have the decency to not reflexively abuse others on
                                    this list, or otherwise leave.

                                    Michael, thanks for all your contributions over the years.

                                    The issue you raised here is a valid concern. It is however best
                                    to avoid needless distribution-specific tweaks to Postfix, so it
                                    would be best if issues are resolved on this list, rather than by
                                    downstream maintainers, who may not always take all the consequences
                                    into account. To pick on Debian for example, we have the Debian
                                    OpenSSL random number fiasco...

                                    --
                                    Viktor.
                                  • Michael Tokarev
                                    On 03.05.2012 17:16, Stan Hoeppner wrote: [] ... Please refrain from using such words in public forum. Such usage makes you to be of that kind. ... Your read
                                    Message 17 of 23 , May 3, 2012
                                    • 0 Attachment
                                      On 03.05.2012 17:16, Stan Hoeppner wrote:
                                      []
                                      > To who at Debian? Lamont Jones? Has he replied to your idiotic idea yet?

                                      Please refrain from using such words in public forum.
                                      Such usage makes you to be of that kind.

                                      >> Thank you for making my worst nightmares come true. I will do
                                      >> my best to prevent this from happening, and if I find out that
                                      >> they do it anyway, then I will raise hell and it won't be pretty.
                                      >
                                      > All of this nonsense because one guy on the planet feels he can't simply
                                      > use an MUA with submission like everyone else does, but demands he be
                                      > able to run an MTA on his damn desktop/laptop, and demands the default
                                      > MTA config allows him to do what he wants seamlessly, possibly to the
                                      > detriment of others, mainly the guy who wrote this MTA for your use in
                                      > the first place. At least that's my read of this thread.

                                      Your read is incorrect. World is much larger than your imagination.

                                      Thanks,

                                      /mjt
                                    • Wietse Venema
                                      I already thanked Michael for his contributions in private email. Michael, does editing master.cf and s/fifo/unix/ solve the mtime file system updates problem?
                                      Message 18 of 23 , May 3, 2012
                                      • 0 Attachment
                                        I already thanked Michael for his contributions in private email.

                                        Michael, does editing master.cf and s/fifo/unix/ solve the mtime
                                        file system updates problem? This is already supported by existing
                                        code, works on Linux and *BSD, and I can make a config parameter
                                        that makes this configurable with system-dependent defaults.

                                        If so, then we can avoid controversial changes, such as support to
                                        mount over a critical Postfix directory without Postfix's knowledge
                                        of such things happening.

                                        Wietse
                                      • Stan Hoeppner
                                        ... My apologies for allowing my passion to transform into abrasiveness. ... Please (re)explain the use case you have in mind. It seemed to me the changes
                                        Message 19 of 23 , May 3, 2012
                                        • 0 Attachment
                                          On 5/3/2012 8:48 AM, Michael Tokarev wrote:
                                          > On 03.05.2012 17:16, Stan Hoeppner wrote:
                                          > []
                                          >> To who at Debian? Lamont Jones? Has he replied to your idiotic idea yet?
                                          >
                                          > Please refrain from using such words in public forum.
                                          > Such usage makes you to be of that kind.

                                          My apologies for allowing my passion to transform into abrasiveness.

                                          >>> Thank you for making my worst nightmares come true. I will do
                                          >>> my best to prevent this from happening, and if I find out that
                                          >>> they do it anyway, then I will raise hell and it won't be pretty.
                                          >>
                                          >> All of this nonsense because one guy on the planet feels he can't simply
                                          >> use an MUA with submission like everyone else does, but demands he be
                                          >> able to run an MTA on his damn desktop/laptop, and demands the default
                                          >> MTA config allows him to do what he wants seamlessly, possibly to the
                                          >> detriment of others, mainly the guy who wrote this MTA for your use in
                                          >> the first place. At least that's my read of this thread.
                                          >
                                          > Your read is incorrect. World is much larger than your imagination.

                                          Please (re)explain the use case you have in mind. It seemed to me the
                                          changes you're proposing will have a positive effect, immediately
                                          anyway, for only a very small subset of Postfix users, for a niche
                                          configuration.

                                          This request seems very similar to one made on the XFS list not all that
                                          long ago. A user with a home theater PC and a single large WD Green
                                          drive was irked that the drive wouldn't stay asleep for more than 30
                                          seconds. He debugged it himself, and found a long standing XFS behavior
                                          of accessing the journal or filesystem superblock every 30s IIRC. He
                                          said this wasn't necessary and pleaded with the devs to change this
                                          behavior, just so his HTPC drive could sleep. XFS was never intended
                                          for such a setup, this behavior existing since ~1994/95. The average
                                          XFS setup is a server with a dozen to a few hundred or more drives in
                                          hardware RAID running 24x7--no sleeping. An SGI employee mentioned just
                                          a couple of weeks ago working with a single XFS filesystem spaning 600
                                          drives in an IS16000 array. Not your average XFS drive count, but it is
                                          a typical large XFS configuration, and quite a contrast from a single
                                          drive HTPC server in a living room.

                                          IIRC a patch was eventually developed after many months, when it was
                                          determined there was likely no downside, and mainlined after much
                                          regression testing and tweaking. All for the benefit of very very few
                                          non-typical XFS users.

                                          Anyway, I see this as a similar case, and a similar waste of resources
                                          expended for the benefit of very few users, when there is nothing
                                          inherently "wrong" with the current Postfix implementation, as far as I
                                          understand the request. Maybe I simply don't fully understand the issue
                                          and the potential benefits yet.

                                          --
                                          Stan
                                        • john
                                          I do not see where Stan was abusive. Abrasive maybe, but then sometimes bumps on logs need sanding down this would appear to be one of those occasions.
                                          Message 20 of 23 , May 3, 2012
                                          • 0 Attachment
                                            I do not see where Stan was abusive.
                                            Abrasive maybe, but then sometimes bumps on logs need sanding down this
                                            would appear to be one of those occasions.

                                            On 03/05/2012 11:29 AM, Stan Hoeppner wrote:
                                            > On 5/3/2012 8:48 AM, Michael Tokarev wrote:
                                            >> On 03.05.2012 17:16, Stan Hoeppner wrote:
                                            >> []
                                            >>> To who at Debian? Lamont Jones? Has he replied to your idiotic idea yet?
                                            >> Please refrain from using such words in public forum.
                                            >> Such usage makes you to be of that kind.
                                            > My apologies for allowing my passion to transform into abrasiveness.
                                            >
                                            >>>> Thank you for making my worst nightmares come true. I will do
                                            >>>> my best to prevent this from happening, and if I find out that
                                            >>>> they do it anyway, then I will raise hell and it won't be pretty.
                                            >>> All of this nonsense because one guy on the planet feels he can't simply
                                            >>> use an MUA with submission like everyone else does, but demands he be
                                            >>> able to run an MTA on his damn desktop/laptop, and demands the default
                                            >>> MTA config allows him to do what he wants seamlessly, possibly to the
                                            >>> detriment of others, mainly the guy who wrote this MTA for your use in
                                            >>> the first place. At least that's my read of this thread.
                                            >> Your read is incorrect. World is much larger than your imagination.
                                            > Please (re)explain the use case you have in mind. It seemed to me the
                                            > changes you're proposing will have a positive effect, immediately
                                            > anyway, for only a very small subset of Postfix users, for a niche
                                            > configuration.
                                            >
                                            > This request seems very similar to one made on the XFS list not all that
                                            > long ago. A user with a home theater PC and a single large WD Green
                                            > drive was irked that the drive wouldn't stay asleep for more than 30
                                            > seconds. He debugged it himself, and found a long standing XFS behavior
                                            > of accessing the journal or filesystem superblock every 30s IIRC. He
                                            > said this wasn't necessary and pleaded with the devs to change this
                                            > behavior, just so his HTPC drive could sleep. XFS was never intended
                                            > for such a setup, this behavior existing since ~1994/95. The average
                                            > XFS setup is a server with a dozen to a few hundred or more drives in
                                            > hardware RAID running 24x7--no sleeping. An SGI employee mentioned just
                                            > a couple of weeks ago working with a single XFS filesystem spaning 600
                                            > drives in an IS16000 array. Not your average XFS drive count, but it is
                                            > a typical large XFS configuration, and quite a contrast from a single
                                            > drive HTPC server in a living room.
                                            >
                                            > IIRC a patch was eventually developed after many months, when it was
                                            > determined there was likely no downside, and mainlined after much
                                            > regression testing and tweaking. All for the benefit of very very few
                                            > non-typical XFS users.
                                            >
                                            > Anyway, I see this as a similar case, and a similar waste of resources
                                            > expended for the benefit of very few users, when there is nothing
                                            > inherently "wrong" with the current Postfix implementation, as far as I
                                            > understand the request. Maybe I simply don't fully understand the issue
                                            > and the potential benefits yet.
                                            >
                                          • Stan Hoeppner
                                            On 5/3/2012 6:54 PM, Bill Cole wrote: ... This could be completely resolved by PXE/bootp and NFS mounted root filesystems, and save you $200-500/node in disk
                                            Message 21 of 23 , May 4, 2012
                                            • 0 Attachment
                                              On 5/3/2012 6:54 PM, Bill Cole wrote:
                                              ...
                                              > For many of these systems,
                                              > the OS resides on a mirrored pair of local disks which see very
                                              > infrequent writes because every filesystem with significant flux is
                                              > physically resident across the SAN. Spinning disks draw power. Anything
                                              > drawing power generates heat. Heat requires cooling. Cooling typically
                                              > requires more power than the devices it is compensating for. Cooling
                                              > also requires careful attention to the details of physical server
                                              > density and rack design and so on...

                                              This could be completely resolved by PXE/bootp and NFS mounted root
                                              filesystems, and save you $200-500/node in disk drive costs after
                                              spending $1000-2000 for the NFS server hardware, or nothing using a VM
                                              server. It would also save you substantial admin time by using
                                              templates for new node deployments. This diskless node methodology has
                                              been around for ~30 years.

                                              > A local mail submission and trivial outbound transport subsystem is a
                                              > normal feature of any Unix-like machine. To operate robustly, it needs a
                                              > queueing and retry mechanism. It is helpful for environments with power
                                              > and cooling concerns if a mechanical disk (or worse: a mirrored pair of
                                              > disks) isn't forced to spin up every time that mechanism activates.
                                              > Every little wattage savings is useful, and avoiding truly pointless
                                              > disk writes is never a bad thing.

                                              SSD is a perfect solution here, in cases of non netboot machines. And
                                              right now small SSDs are less expensive than their rusty disk
                                              counterparts. If one is truly concerned about spurious spin ups eating
                                              power and generating heat, I would think one would not go after the
                                              software stack in a piecemeal fashion to solve the problem. The MTA
                                              isn't the only software waking the disk. The kernel will write logs far
                                              more often in many/most situations.

                                              > Well, beyond the data center environment there is also a very widespread
                                              > deployment of Postfix as the legacy mail subsystem on MacOS personal
                                              > machines, where the mail flow is typically extremely low.
                                              ...
                                              > Ultimately the result is having to choose
                                              > between power management and timely delivery. If the periodic wakeups
                                              > didn't force a disk write, it would be less onerous to let master run in
                                              > its normal persistent mode for a lot of Postfix users (many of whom may
                                              > not even be aware that they are Postfix users.)

                                              This is only true if two things persist into the future:

                                              1. Postfix isn't modified in order to perform a power management role
                                              2. Laptops will forever have spinning rust storage

                                              Addressing the first point, should it be the responsibility of
                                              application software to directly address power management concerns? Or
                                              should this be left to the OS and hardware platform/BIOS?

                                              Addressing the 2nd, within a few years all new laptops will ship with
                                              SSD instead of SRD specifically to address battery run time
                                              issues. Many are shipping now with SSDs. All netbooks already do,
                                              smart phones use other flash types.

                                              > Whether it is actually worthwhile to make a change that is only
                                              > significant for people who are barely using Postfix isn't a judgment I
                                              > can make. It's obvious that Dr. Venema takes significantly more care
                                              > with his code than I can really relate to, so I don't really know what
                                              > effort a conceptually small change in Postfix really entails.

                                              Wietse will make his own decisions as he always has.

                                              I'm simply making the point that issues such as power/cooling,
                                              wake/sleep, etc should be addressed at the hardware platform/OS level,
                                              or system or network architecture level, at the application level,
                                              especially if the effort to implement it is more than trivial.

                                              This is especially true when any such coding effort may only produce
                                              very short term gains, as these issues are already being addressed and
                                              will be completely resolved by other means (SSD) in the near
                                              future, or have already been resolved by 30 year old
                                              technology/architecture methods (netboot/NFS), depending on the platform
                                              scenario.

                                              --
                                              Stan
                                            • Bill Cole
                                              ... Yes, it is possible to fundamentally re-architect working environments that have been organically developed over years by adding significant new
                                              Message 22 of 23 , May 4, 2012
                                              • 0 Attachment
                                                On 4 May 2012, at 17:00, Stan Hoeppner wrote:

                                                > On 5/3/2012 6:54 PM, Bill Cole wrote:
                                                > ...
                                                >> For many of these systems,
                                                >> the OS resides on a mirrored pair of local disks which see very
                                                >> infrequent writes because every filesystem with significant flux is
                                                >> physically resident across the SAN. Spinning disks draw power.
                                                >> Anything
                                                >> drawing power generates heat. Heat requires cooling. Cooling
                                                >> typically
                                                >> requires more power than the devices it is compensating for. Cooling
                                                >> also requires careful attention to the details of physical server
                                                >> density and rack design and so on...
                                                >
                                                > This could be completely resolved by PXE/bootp and NFS mounted root
                                                > filesystems, and save you $200-500/node in disk drive costs after
                                                > spending $1000-2000 for the NFS server hardware, or nothing using a VM
                                                > server. It would also save you substantial admin time by using
                                                > templates for new node deployments. This diskless node methodology
                                                > has
                                                > been around for ~30 years.

                                                Yes, it is possible to fundamentally re-architect working environments
                                                that have been "organically" developed over years by adding significant
                                                new infrastructure to save on capital costs of hypothetical growth and
                                                maybe on future admin time. The idea that a server in the $1000-$2000
                                                range would be part of a global conversion to diskless servers or even
                                                the largest capital cost of such a project reveals that I failed to
                                                communicate an accurate understanding of the environment, but that's not
                                                terribly important. There's no shortage of well-informed well-developed
                                                specific proposals for comprehensive infrastructure overhaul, and in the
                                                interim between now and the distant never when one of those meets up
                                                with a winning lottery ticket and an unutilized skilled head or three, I
                                                have sufficient workarounds in place.

                                                I didn't mention that environment seeking a solution, but rather to
                                                point out that there are real-world systems that take advantage of the
                                                power management capabilities of modern disks and have nothing else in
                                                common with the average personal system. I think that was responsive to
                                                the paragraph of yours that I originally quoted. It's easy to come up
                                                with flippant advice for others to spend time and money to replace
                                                stable working systems, but it is also irrelevant and a bit rude.

                                                [...]
                                                >> Ultimately the result is having to choose
                                                >> between power management and timely delivery. If the periodic wakeups
                                                >> didn't force a disk write, it would be less onerous to let master run
                                                >> in
                                                >> its normal persistent mode for a lot of Postfix users (many of whom
                                                >> may
                                                >> not even be aware that they are Postfix users.)
                                                >
                                                > This is only true if two things persist into the future:
                                                >
                                                > 1. Postfix isn't modified in order to perform a power management role

                                                No reason for it to "perform" but it would be nice for it to "stop
                                                thwarting."

                                                > 2. Laptops will forever have spinning rust storage

                                                Who said anything about laptops?

                                                > Addressing the first point, should it be the responsibility of
                                                > application software to directly address power management concerns?
                                                > Or
                                                > should this be left to the OS and hardware platform/BIOS?

                                                Applications should not do things that are actively hostile to
                                                housekeeping functions of lower-level software (in this case: drive
                                                firmware) without a functional justification. It's not wrong for a
                                                filesystem to change the mtime on a pipe with every write to it, nor is
                                                it wrong for a filesystem to commit every change in a timely manner.
                                                This is not really fixable at a lower level without eliminating the
                                                hardware in question or making changes to filesystem software that could
                                                cause wide-ranging problems with other software.

                                                > Addressing the 2nd, within a few years all new laptops will ship with
                                                > SSD instead of SRD specifically to address battery run time
                                                > issues. Many are shipping now with SSDs. All netbooks already do,
                                                > smart phones use other flash types.

                                                This is not about laptops. Really.

                                                Systems can live a long time without drive replacements. Spinning rust
                                                with power management firmware is not going to be rare in running
                                                systems until at least 5 years after dependable & fast SSD's hit $1/GB
                                                for devices larger than 100GB. Of course, those drives may die out a lot
                                                faster where applications do periodic pointless writes that keep them
                                                running continuously.

                                                Note that the reason this issue exists *AT ALL* is to work around a bug
                                                in Solaris 2.4. I spent most of the last 14 years working mostly on
                                                Solaris systems in change-averse places and the last time I saw Solaris
                                                2.4 was 1999. I don't have the details of the bug or the free time to
                                                rig up a test system to prove it gone in whatever version Postfix needs
                                                to work on today, but I have no gripe with that relatively ancient and
                                                *likely* inoperative history being the blocking issue. I hope someone
                                                else can settle the issue. An argument that time will soon make this fix
                                                pointless is a bit ironic.


                                                >> Whether it is actually worthwhile to make a change that is only
                                                >> significant for people who are barely using Postfix isn't a judgment
                                                >> I
                                                >> can make. It's obvious that Dr. Venema takes significantly more care
                                                >> with his code than I can really relate to, so I don't really know
                                                >> what
                                                >> effort a conceptually small change in Postfix really entails.
                                                >
                                                > Wietse will make his own decisions as he always has.
                                                >
                                                > I'm simply making the point that issues such as power/cooling,
                                                > wake/sleep, etc should be addressed at the hardware platform/OS level,
                                                > or system or network architecture level, at the application level,
                                                > especially if the effort to implement it is more than trivial.

                                                See his discussion of the details. The code exists, what remains is the
                                                harder work of testing and getting all the defaults right.


                                                P.S.: Note that I have respected with your Reply-To header. Please
                                                return that courtesy.
                                              • Reindl Harald
                                                ... but only if you do not permanently spin them up and down power managment is the dead of a drive i have here disks with 35.000 uptime you can be sure with
                                                Message 23 of 23 , May 4, 2012
                                                • 0 Attachment
                                                  Am 05.05.2012 03:05, schrieb Bill Cole:
                                                  > Systems can live a long time without drive replacements.

                                                  but only if you do not permanently spin them up and down
                                                  power managment is the dead of a drive

                                                  i have here disks with > 35.000 uptime
                                                  you can be sure with "power-managment" they
                                                  would still be dead

                                                  try it out: spin down a drive running some years
                                                  and you have a real good change that the next
                                                  spin up is the final one

                                                  > Spinning rust with power management firmware is not going
                                                  > to be rare in running systems until at least 5 years

                                                  and should be the first to get disabled in the real world

                                                  the power you save in spin down a disk is meaningless
                                                  in any way, the power for produce a new drive because yours
                                                  died by permanently spin up/down is much higher and the cost
                                                  of the new drive also compared with let the drive run
                                                Your message has been successfully submitted and would be delivered to recipients shortly.