Loading ...
Sorry, an error occurred while loading the content.

Re: Large incoming que

Expand Messages
  • Adrian Ulrich
    ... Are you using Linux or *BSD? Running something like iostat (Solaris /BSD?) or sysstat (Linux: http://perso.orange.fr/sebastien.godard/) could give you a
    Message 1 of 10 , Oct 1, 2006
    • 0 Attachment
      > Looking at vmstat we see that when this happens the 'b' column seems high,
      > 30+ values..

      Are you using Linux or *BSD?

      Running something like iostat (Solaris /BSD?) or sysstat (Linux: http://perso.orange.fr/sebastien.godard/)
      could give you a clue about how busy your Raid-system is:

      Example output, showing a *very* busy 'sda' (svctm / %util)
      # iostat -x 5
      Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
      md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
      sda 4.99 0.00 153.49 1.00 6125.35 7.98 39.70 1.17 7.54 6.34 97.88


      > Are we not having enough disk speed to process the load?

      If sysstat/iostat shows high svctm/%util values: yes.
      Otherwise your SQL-Server may be the bottleneck.



      > Any way to decrease from the mailstore the amount of mail accepted to
      > incoming and speedup delivery?

      You could try to decrase the number of smtpd processes (master.cf) or
      play with *_destination_concurrency_limit on your frontend servers.


      > The two mailstores in question have the heaviest domains in terms of how
      > much incoming traffic they get.

      How about spreading the heavy-domains across multiple backends?


      > Any recommendationgs greatly appreciated.

      You are using Maildir, correct?

      Regards,
      Adrian


      --
      A. Top posters
      Q. What's the most annoying thing on Usenet?
    • Sandy Drobic
      ... In that case no fiddling with Postfix settings will alleviate the problem. ... Increasing the cache size will help a bit, also adding one or two more hdds
      Message 2 of 10 , Oct 1, 2006
      • 0 Attachment
        Francisco Reyes wrote:
        > Sandy Drobic writes:
        >
        >> Try to find out which process consumes most of the I/O. Is it really
        >> the local delivery from Postfix or is it an imap process?
        >
        > Imap processes.

        In that case no fiddling with Postfix settings will alleviate the problem.

        >> Are the raid controllers equipped with battery backup units? It helps
        >> a lot, if the write cache is enabled
        >
        > It has and the write cache is enabled.
        >
        >> has lots of fast cache RAM.
        >
        > About 128MB of cache. It is a 3ware controller. 9500SX

        Increasing the cache size will help a bit, also adding one or two more
        hdds to the raid to spread the I/O over more hdds. How many hdds are there
        in your raid? Which raid level?

        Though I am afraid your system is just overtaxed. In that case you can
        probably only decrease the load or increase the system performance with
        better hardware.

        You need some hard figures to find out if it is sufficient to tweak your
        system or you need new hardware. Try "sar" on linux to get continuous
        measure of your system load.

        Sandy

        --
        List replies only please!
        Please address PMs to: news-reply2 (@) japantest (.) homelinux (.) com
      • Francisco Reyes
        ... FreeBSD 6.1 ... Have used iostat and I see lots of small transactions. ... What do you mean by svctm/%util ? ... Eliminated 2 of the 3 SQL lookups and
        Message 3 of 10 , Oct 2, 2006
        • 0 Attachment
          Adrian Ulrich writes:

          > Are you using Linux or *BSD?

          FreeBSD 6.1

          > Running something like iostat (Solaris /BSD?) or sysstat (Linux: http://perso.orange.fr/sebastien.godard/)
          > could give you a clue about how busy your Raid-system is:

          Have used iostat and I see lots of small transactions.

          > If sysstat/iostat shows high svctm/%util

          What do you mean by "svctm/%util"?


          > Otherwise your SQL-Server may be the bottleneck.

          Eliminated 2 of the 3 SQL lookups and plan to remove the 3rd today.

          > You could try to decrase the number of smtpd processes (master.cf) or
          > play with *_destination_concurrency_limit on your frontend servers.

          Ideally I would like to decrease incoming mails only to these two machines.
          Other machines are fine.


          > How about spreading the heavy-domains across multiple backends?

          We have 4 mailstores. These two have the better hardware so we moved the
          heaviest smtp traffic domains to these two machines. I am still trying to
          convince the powers that be.. to get a SCSI setup for these domains.

          > You are using Maildir, correct?

          Correct. Maildir run off Courier IMAP, serving imap and pop3.
        • Victor Duchovni
          ... Is syslogd writing synchronously to disk? If so, disable that behaviour. -- Viktor. Disclaimer: off-list followups get on-list replies or get ignored.
          Message 4 of 10 , Oct 2, 2006
          • 0 Attachment
            On Mon, Oct 02, 2006 at 10:55:10AM -0400, Francisco Reyes wrote:

            > Adrian Ulrich writes:
            >
            > >Are you using Linux or *BSD?
            >
            > FreeBSD 6.1
            >
            > >Running something like iostat (Solaris /BSD?) or sysstat (Linux:
            > >http://perso.orange.fr/sebastien.godard/)
            > >could give you a clue about how busy your Raid-system is:
            >
            > Have used iostat and I see lots of small transactions.

            Is syslogd writing synchronously to disk? If so, disable that behaviour.

            --
            Viktor.

            Disclaimer: off-list followups get on-list replies or get ignored.
            Please do not ignore the "Reply-To" header.

            To unsubscribe from the postfix-users list, visit
            http://www.postfix.org/lists.html or click the link below:
            <mailto:majordomo@...?body=unsubscribe%20postfix-users>

            If my response solves your problem, the best way to thank me is to not
            send an "it worked, thanks" follow-up. If you must respond, please put
            "It worked, thanks" in the "Subject" so I can delete these quickly.
          • Francisco Reyes
            ... So far eliminating two of the 3 SQL lookups helped. Also moved IMAP connections to a gigabit switch... so hopefully IMAP connections will get served
            Message 5 of 10 , Oct 2, 2006
            • 0 Attachment
              Sandy Drobic writes:

              >> Imap processes.
              >
              > In that case no fiddling with Postfix settings will alleviate the problem.

              So far eliminating two of the 3 SQL lookups helped.
              Also moved IMAP connections to a gigabit switch... so hopefully IMAP
              connections will get served faster.

              > Increasing the cache size will help a bit

              Don't think these controllers can take any more memory.

              >, also adding one or two more
              > hdds to the raid to spread the I/O over more hdds. How many hdds are there
              > in your raid? Which raid level?

              8 disk totals. 6 in RAID 10 and 2 as hot spares. I wanted to get all 8 in
              the RAID, but the owner preferred to have them as hot spares.

              > Though I am afraid your system is just overtaxed. In that case you can
              > probably only decrease the load or increase the system performance with
              > better hardware.

              Thanks for the help. After a couple of weeks of looking into this I was
              starting to get that impression. Waiting to see how the last changes help
              the system. Also plan to remove the last SQL lookup today.

              > You need some hard figures to find out if it is sufficient to tweak your
              > system or you need new hardware.

              Started to work on that too. Basically all the heavy SMTP domains were moved
              to these machines and I think we need to move those domains to a new machine
              with a SCSI setup.

              We are also considering to move to Dovecot. As customers let email pile up
              on directories Courier takes longer and longer to open large folders.

              Our early testing with Dovecot looks very promissing. The first time a
              directory is read it takes long, but still quicker than Courier, and after
              that access is through index files which seems noticeable faster than
              Courier. Hopefully a change in IMAP will decrease I/O tso here will me more
              resources for Postfix.
            • Wietse Venema
              ... On FreeBSD, this gives cpu, memory, disk utilization and more: systat -vmstat ... If none of cpu/memory/disk are saturated, then it s likely waiting for
              Message 6 of 10 , Oct 2, 2006
              • 0 Attachment
                Francisco Reyes:
                > Adrian Ulrich writes:
                >
                > > Are you using Linux or *BSD?
                >
                > FreeBSD 6.1
                >
                > > Running something like iostat (Solaris /BSD?) or sysstat (Linux: http://perso.orange.fr/sebastien.godard/)
                > > could give you a clue about how busy your Raid-system is:
                >
                > Have used iostat and I see lots of small transactions.

                On FreeBSD, this gives cpu, memory, disk utilization and more:

                systat -vmstat

                > > Otherwise your SQL-Server may be the bottleneck.
                >
                > Eliminated 2 of the 3 SQL lookups and plan to remove the 3rd today.
                >
                > > You could try to decrase the number of smtpd processes (master.cf) or
                > > play with *_destination_concurrency_limit on your frontend servers.
                >
                > Ideally I would like to decrease incoming mails only to these two machines.
                > Other machines are fine.

                If none of cpu/memory/disk are saturated, then it's likely waiting
                for things going across the network.

                "netstat -I" can identify problems with collision or bad packets;
                "netstat -s" can help to identity trouble higher up the network
                stack.

                Wietse
                >
                > > How about spreading the heavy-domains across multiple backends?
                >
                > We have 4 mailstores. These two have the better hardware so we moved the
                > heaviest smtp traffic domains to these two machines. I am still trying to
                > convince the powers that be.. to get a SCSI setup for these domains.
                >
                > > You are using Maildir, correct?
                >
                > Correct. Maildir run off Courier IMAP, serving imap and pop3.
                >
                >
              • Adrian Ulrich
                Hi, ... It s a Mailserver afterall ;-) ... Service-Time + Percent-Busy I don t have access to a FreeBSD-Host ATM, but the output of solaris iostat -xnz 5 looks
                Message 7 of 10 , Oct 2, 2006
                • 0 Attachment
                  Hi,

                  > Have used iostat and I see lots of small transactions.

                  It's a Mailserver afterall ;-)


                  > What do you mean by "svctm/%util"?

                  Service-Time + Percent-Busy

                  I don't have access to a FreeBSD-Host ATM, but the output of solaris
                  iostat -xnz 5 looks like this: (Output on FreeBSD should be similar):

                  r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
                  20.6 8.9 1281.1 53.1 1.8 0.3 62.1 11.2 9 21 c0d0

                  If %b(usy) is above ~>70% you are having speed-issues with your
                  storage system.


                  > Ideally I would like to decrease incoming mails only to these two machines.
                  > Other machines are fine.

                  Limit the number of smtpd processes (= Incoming mail) on this two hosts


                  > I am still trying to
                  > convince the powers that be.. to get a SCSI setup for these domains.

                  How about a 'shared storage' for all 4 Backends?

                  <ADV>
                  I'm a very happy NetApp(.com) customer: We are using a bunch
                  of FAS270 Boxes for Mailqueues and never had any (Storage-Related)
                  speed issues. They can do NFS, FCP, iSCSI (urgs) provide Raid4 and
                  Raid4-DP (Think 'raid6' -> Double Parity) you can create
                  Snapshots (without any performance issues unlike
                  ${everyone_else's_solution}) .. etc..
                  </ADV>



                  > > You are using Maildir, correct?
                  > Correct. Maildir run off Courier IMAP, serving imap and pop3.

                  So switching to Dovecot might help: IMAP with dovecot seems to be
                  pretty fast.

                  Regards,
                  Adrian
                Your message has been successfully submitted and would be delivered to recipients shortly.