Loading ...
Sorry, an error occurred while loading the content.

Re: Large incoming que

Expand Messages
  • Francisco Reyes
    ... FreeBSD 6.1 ... Have used iostat and I see lots of small transactions. ... What do you mean by svctm/%util ? ... Eliminated 2 of the 3 SQL lookups and
    Message 1 of 10 , Oct 2, 2006
    • 0 Attachment
      Adrian Ulrich writes:

      > Are you using Linux or *BSD?

      FreeBSD 6.1

      > Running something like iostat (Solaris /BSD?) or sysstat (Linux: http://perso.orange.fr/sebastien.godard/)
      > could give you a clue about how busy your Raid-system is:

      Have used iostat and I see lots of small transactions.

      > If sysstat/iostat shows high svctm/%util

      What do you mean by "svctm/%util"?


      > Otherwise your SQL-Server may be the bottleneck.

      Eliminated 2 of the 3 SQL lookups and plan to remove the 3rd today.

      > You could try to decrase the number of smtpd processes (master.cf) or
      > play with *_destination_concurrency_limit on your frontend servers.

      Ideally I would like to decrease incoming mails only to these two machines.
      Other machines are fine.


      > How about spreading the heavy-domains across multiple backends?

      We have 4 mailstores. These two have the better hardware so we moved the
      heaviest smtp traffic domains to these two machines. I am still trying to
      convince the powers that be.. to get a SCSI setup for these domains.

      > You are using Maildir, correct?

      Correct. Maildir run off Courier IMAP, serving imap and pop3.
    • Victor Duchovni
      ... Is syslogd writing synchronously to disk? If so, disable that behaviour. -- Viktor. Disclaimer: off-list followups get on-list replies or get ignored.
      Message 2 of 10 , Oct 2, 2006
      • 0 Attachment
        On Mon, Oct 02, 2006 at 10:55:10AM -0400, Francisco Reyes wrote:

        > Adrian Ulrich writes:
        >
        > >Are you using Linux or *BSD?
        >
        > FreeBSD 6.1
        >
        > >Running something like iostat (Solaris /BSD?) or sysstat (Linux:
        > >http://perso.orange.fr/sebastien.godard/)
        > >could give you a clue about how busy your Raid-system is:
        >
        > Have used iostat and I see lots of small transactions.

        Is syslogd writing synchronously to disk? If so, disable that behaviour.

        --
        Viktor.

        Disclaimer: off-list followups get on-list replies or get ignored.
        Please do not ignore the "Reply-To" header.

        To unsubscribe from the postfix-users list, visit
        http://www.postfix.org/lists.html or click the link below:
        <mailto:majordomo@...?body=unsubscribe%20postfix-users>

        If my response solves your problem, the best way to thank me is to not
        send an "it worked, thanks" follow-up. If you must respond, please put
        "It worked, thanks" in the "Subject" so I can delete these quickly.
      • Francisco Reyes
        ... So far eliminating two of the 3 SQL lookups helped. Also moved IMAP connections to a gigabit switch... so hopefully IMAP connections will get served
        Message 3 of 10 , Oct 2, 2006
        • 0 Attachment
          Sandy Drobic writes:

          >> Imap processes.
          >
          > In that case no fiddling with Postfix settings will alleviate the problem.

          So far eliminating two of the 3 SQL lookups helped.
          Also moved IMAP connections to a gigabit switch... so hopefully IMAP
          connections will get served faster.

          > Increasing the cache size will help a bit

          Don't think these controllers can take any more memory.

          >, also adding one or two more
          > hdds to the raid to spread the I/O over more hdds. How many hdds are there
          > in your raid? Which raid level?

          8 disk totals. 6 in RAID 10 and 2 as hot spares. I wanted to get all 8 in
          the RAID, but the owner preferred to have them as hot spares.

          > Though I am afraid your system is just overtaxed. In that case you can
          > probably only decrease the load or increase the system performance with
          > better hardware.

          Thanks for the help. After a couple of weeks of looking into this I was
          starting to get that impression. Waiting to see how the last changes help
          the system. Also plan to remove the last SQL lookup today.

          > You need some hard figures to find out if it is sufficient to tweak your
          > system or you need new hardware.

          Started to work on that too. Basically all the heavy SMTP domains were moved
          to these machines and I think we need to move those domains to a new machine
          with a SCSI setup.

          We are also considering to move to Dovecot. As customers let email pile up
          on directories Courier takes longer and longer to open large folders.

          Our early testing with Dovecot looks very promissing. The first time a
          directory is read it takes long, but still quicker than Courier, and after
          that access is through index files which seems noticeable faster than
          Courier. Hopefully a change in IMAP will decrease I/O tso here will me more
          resources for Postfix.
        • Wietse Venema
          ... On FreeBSD, this gives cpu, memory, disk utilization and more: systat -vmstat ... If none of cpu/memory/disk are saturated, then it s likely waiting for
          Message 4 of 10 , Oct 2, 2006
          • 0 Attachment
            Francisco Reyes:
            > Adrian Ulrich writes:
            >
            > > Are you using Linux or *BSD?
            >
            > FreeBSD 6.1
            >
            > > Running something like iostat (Solaris /BSD?) or sysstat (Linux: http://perso.orange.fr/sebastien.godard/)
            > > could give you a clue about how busy your Raid-system is:
            >
            > Have used iostat and I see lots of small transactions.

            On FreeBSD, this gives cpu, memory, disk utilization and more:

            systat -vmstat

            > > Otherwise your SQL-Server may be the bottleneck.
            >
            > Eliminated 2 of the 3 SQL lookups and plan to remove the 3rd today.
            >
            > > You could try to decrase the number of smtpd processes (master.cf) or
            > > play with *_destination_concurrency_limit on your frontend servers.
            >
            > Ideally I would like to decrease incoming mails only to these two machines.
            > Other machines are fine.

            If none of cpu/memory/disk are saturated, then it's likely waiting
            for things going across the network.

            "netstat -I" can identify problems with collision or bad packets;
            "netstat -s" can help to identity trouble higher up the network
            stack.

            Wietse
            >
            > > How about spreading the heavy-domains across multiple backends?
            >
            > We have 4 mailstores. These two have the better hardware so we moved the
            > heaviest smtp traffic domains to these two machines. I am still trying to
            > convince the powers that be.. to get a SCSI setup for these domains.
            >
            > > You are using Maildir, correct?
            >
            > Correct. Maildir run off Courier IMAP, serving imap and pop3.
            >
            >
          • Adrian Ulrich
            Hi, ... It s a Mailserver afterall ;-) ... Service-Time + Percent-Busy I don t have access to a FreeBSD-Host ATM, but the output of solaris iostat -xnz 5 looks
            Message 5 of 10 , Oct 2, 2006
            • 0 Attachment
              Hi,

              > Have used iostat and I see lots of small transactions.

              It's a Mailserver afterall ;-)


              > What do you mean by "svctm/%util"?

              Service-Time + Percent-Busy

              I don't have access to a FreeBSD-Host ATM, but the output of solaris
              iostat -xnz 5 looks like this: (Output on FreeBSD should be similar):

              r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
              20.6 8.9 1281.1 53.1 1.8 0.3 62.1 11.2 9 21 c0d0

              If %b(usy) is above ~>70% you are having speed-issues with your
              storage system.


              > Ideally I would like to decrease incoming mails only to these two machines.
              > Other machines are fine.

              Limit the number of smtpd processes (= Incoming mail) on this two hosts


              > I am still trying to
              > convince the powers that be.. to get a SCSI setup for these domains.

              How about a 'shared storage' for all 4 Backends?

              <ADV>
              I'm a very happy NetApp(.com) customer: We are using a bunch
              of FAS270 Boxes for Mailqueues and never had any (Storage-Related)
              speed issues. They can do NFS, FCP, iSCSI (urgs) provide Raid4 and
              Raid4-DP (Think 'raid6' -> Double Parity) you can create
              Snapshots (without any performance issues unlike
              ${everyone_else's_solution}) .. etc..
              </ADV>



              > > You are using Maildir, correct?
              > Correct. Maildir run off Courier IMAP, serving imap and pop3.

              So switching to Dovecot might help: IMAP with dovecot seems to be
              pretty fast.

              Regards,
              Adrian
            Your message has been successfully submitted and would be delivered to recipients shortly.