Loading ...
Sorry, an error occurred while loading the content.

Long time sending

Expand Messages
  • Jan Steinman
    I know the drill, but before you start in with show us the logs, consider that this problem is generating some 60MB of logs EACH DAY! I have changed
    Message 1 of 8 , Jul 1, 2006
    • 0 Attachment
      I know the drill, but before you start in with "show us the logs," consider that this problem is generating some 60MB of logs EACH DAY!

      I have changed locations and servers, going from a 350MHz G4 to a 2.5GHz quad G5, from 128kB IDSL to 4MB ADSL. (Both are supposed to be "commercial quality" connections, which in this case, simply means more expensive.) "Should be a lot faster," I thought.

      But sending from other machines on our network takes increasingly long times after starting the server. It has been three days since a restart, and it now takes about 20 seconds pause between hitting "send" and having the message leave the queue.

      There are over a hundred process entries like the following:
        postfix   2272   0.0 -0.0    27524    924  ??  S     4:33PM   0:00.03 smtpd -n smtp -t inet -u

      Checking logs, I am getting 10-20 rejects PER MINUTE! All of them appear to be legit rejects -- generally "User unknown in local recipient table". Although I have had one or two people tell me their legit email was bounced, we seem to be getting most (if not all) our legit email. I suspect the bounces are a result of Inadvertent Denial of Service from the heavy reject traffic.

      I am using "virtual_maps = hash:/etc/postfix/virtual". I have not set "relay_recipient_maps" nor "local_recipient_maps".

      I suspect that spammers are hitting me particularly hard because my network segment is known to be a residential high-speed subnet, whereas I rarely got more than a few per minute on my old, slow IDSL subnet -- they didn't bother.

      I am not doing any particular postfix spam prevention -- but neither was I before, on my IDSL connection.

      I suspect that the volume of spam rejected is what is causing the extremely slow legit relaying through my SMTP server from machines on my subnet. I have also enabled a few discrete addresses for relaying (via "mynetworks") for places where I frequent wireless networks.

      So I'm thinking of doing one or more of the following:

      1) aggressive firewalling to block it BEFORE postfix sees it: CONS: a lot of work, may block legit email, the spammers keep moving

      2) start a second postfix instance on a different port, and use it exclusively for outbound email: CONS: does nothing to reduce the huge load on my incoming service

      3) switch IAPs: CONS: a pain in the neck, and they're all resellers, anyway, so I'd probably still end up on the same residential ADSL subnet

      4) Tweak main.cf somehow that I don't understand: CONS: if postfix still has to look at it, it still slows it down, no?

      5) Throttle the number of smtpd process instances: CONS: may increase Inadvertent Denial of Service to legit SMTP traffic.

      6) Your Idea Here! Feel free to steer me to a URL, FAQ, or book.

      Thanks in advance for any advice offered!

      :::: Jan Steinman, Communication Steward, EcoReality: http://www.EcoReality.org ::::
      :::: 160 Sharp Road, Salt Spring Island, BC V8K 2P6, Canada, 250.537.2024 ::::


    • Wietse Venema
      A problem report without confguration and actual evidence? Do some home work. http://www.postfix.org/QSHAPE_README.html
      Message 2 of 8 , Jul 1, 2006
      • 0 Attachment
        A problem report without confguration and actual evidence?

        Do some home work.

        http://www.postfix.org/QSHAPE_README.html
      • Jan Steinman
        ... Thanks for the starting point. Exactly what I was looking for. I even expected the attitude. # no qshape in /var/root/bin /Developer/Tools /usr/local/bin
        Message 3 of 8 , Jul 1, 2006
        • 0 Attachment
          On 1 Jul 06, at 19:06 , Wietse Venema wrote:

          > A problem report without confguration and actual evidence?
          >
          > Do some home work.
          >
          > http://www.postfix.org/QSHAPE_README.html

          Thanks for the starting point. Exactly what I was looking for. I even
          expected the attitude.

          # no qshape in /var/root/bin /Developer/Tools /usr/local/bin /usr/
          bin /bin /usr/sbin /sbin /sw/bin /sw/sbin /opt/local/bin

          Ah, never mind. I forgot that I'm running an "unsupported" version --
          the one that comes with the latest MacOS X updates. Last time I tried
          to install the supported version, I and my customers were without
          mail for a week.

          :::: Sell your cleverness, and purchase bewilderment -- Rumi
          :::: Jan Steinman <http://www.Bytesmiths.com/Item/99-6313-15>
        • Brian Collins
          ... You should expect the attitude when you thrust a complaint onto this list with essentially, I know you re going to tell me to present evidence of what I m
          Message 4 of 8 , Jul 1, 2006
          • 0 Attachment
            > Thanks for the starting point. Exactly what I was looking for. I even
            > expected the attitude.

            You should expect the attitude when you thrust a complaint onto this
            list with essentially, "I know you're going to tell me to present
            evidence of what I'm complaining about, but my case is special so you're
            not getting it."

            --Brian
          • Brian Collins
            Now if I can encourage you a bit as to why logs and config are helpful to us... ... One of my mail filters generates 750MB of logs each day. grep is your
            Message 5 of 8 , Jul 1, 2006
            • 0 Attachment
              Now if I can encourage you a bit as to why logs and config are helpful
              to us...

              > ...this problem is generating some 60MB of logs EACH DAY!

              One of my mail filters generates 750MB of logs each day. grep is your
              friend, and even a minimal knowledge of regular expressions can help you
              get exactly what you need.

              > I have changed locations and servers, going from a 350MHz G4 to a
              > 2.5GHz quad G5, from 128kB IDSL to 4MB ADSL. (Both are supposed to be
              > "commercial quality" connections, which in this case, simply means
              > more expensive.) "Should be a lot faster," I thought.

              Same mail filter I mentioned above is dual Athlon 900Mhz, 1GB RAM.
              100Mbps NIC, lots-o-bandwidth available to it. Protecting 23,000+ email
              accounts, 150+ email domains. Based on what I've learned from this
              list, that falls into the category of small-to-medium (more small than
              medium).

              > But sending from other machines on our network takes increasingly long
              > times after starting the server. It has been three days since a
              > restart, and it now takes about 20 seconds pause between hitting
              > "send" and having the message leave the queue.

              This is why config is helpful. If we know what your Postfix is doing,
              we (others more than myself, but I'm getting more experience) can help
              point to potential reasons. But your hardware seems fine.

              > Checking logs, I am getting 10-20 rejects PER MINUTE! All of them
              > appear to be legit rejects -- generally "User unknown in local
              > recipient table". Although I have had one or two people tell me their
              > legit email was bounced, we seem to be getting most (if not all) our
              > legit email. I suspect the bounces are a result of Inadvertent Denial
              > of Service from the heavy reject traffic.

              Same mail filter I mentioned above averages 120+ rejects per minute.
              During spam storms it has reached over 1,000 per minute. Keeps ticking
              right along.

              > I suspect that spammers are hitting me particularly hard because my
              > network segment is known to be a residential high-speed subnet,

              Makes sense. I see similar.

              > I am not doing any particular postfix spam prevention -- but neither
              > was I before, on my IDSL connection.

              Again, this is where configs come in handy.

              > I suspect that the volume of spam rejected is what is causing the
              > extremely slow legit relaying through my SMTP server from machines on
              > my subnet. I have also enabled a few discrete addresses for relaying
              > (via "mynetworks") for places where I frequent wireless networks.

              Eek.

              > So I'm thinking of doing one or more of the following:
              >
              >
              > 1) aggressive firewalling to block it BEFORE postfix sees it: CONS: a
              > lot of work, may block legit email, the spammers keep moving

              How would you determine what to block? What criteria will you use?

              > 2) start a second postfix instance on a different port, and use it
              > exclusively for outbound email: CONS: does nothing to reduce the huge
              > load on my incoming service

              Should not be necessary. Your hardware should be able to handle your
              mail load. Another Postfix instance wouldn't help this. Setting up a
              separate outbound server might; but again, you shouldn't have to do
              this.

              > 3) switch IAPs: CONS: a pain in the neck, and they're all resellers,
              > anyway, so I'd probably still end up on the same residential ADSL
              > subnet

              Again, not necessary. Get Postfix working right and you shouldn't need
              to do this

              > 4) Tweak main.cf somehow that I don't understand: CONS: if postfix
              > still has to look at it, it still slows it down, no?

              Not necessarily. It could be (and likely is) that your current
              configuration is exactly the problem. Which is, again, why it's good to
              see it.

              > 5) Throttle the number of smtpd process instances: CONS: may increase
              > Inadvertent Denial of Service to legit SMTP traffic.

              If we could see your configuration we could help you determine whether
              that's a good idea. I think I'm still running the defaults on the
              filter I mentioned above. :)

              > 6) Your Idea Here! Feel free to steer me to a URL, FAQ, or book.

              The Postfix doc is excellent. As well, there are numerous good howtos
              on the net (obviously Google is your friend). If you want a book, I
              recommend Ralf & Patrick's The Postfix Book.

              --Brian


              >
            • Stephan Budach
              Hi Jan, I do not know, if the answer form Vietse affended you in any way, but in genral is is of course right. Your problem is, like mine, that you are running
              Message 6 of 8 , Jul 2, 2006
              • 0 Attachment
                Hi Jan,

                I do not know, if the answer form Vietse affended you in any way, but
                in genral is is of course right. Your problem is, like mine, that you
                are running Postfix on a Mac OS X Server, and that Postfix does not
                build out of the box on Mac OS X.

                We are running Postfix as a Mail-Gateway in front of or Oracle Mail
                Server because it was so easy to equip Postfix with all that Anti-
                Spam/Anti-Virus softwzare that comes loaded with Mac OS X Server. So,
                our setup might differ from yours, since we do only the MTA part
                using Postfix only as a relay to and from our network. We are running
                Postfix 2.15 on a Xserve G5 CL 2.5 GHz with 4 GB of RAM on a 100Mbit
                leased line.

                In general the time a message needs to completely pass our Postfix
                system ranges from 3 to 50 seconds, depending on size and
                destination. Have you ever had a close look at the logs. Especially
                at the "delay" value?

                Unfortanetly qshape is not part of the Postfix installation coming
                with Mac OS X, but if you really need the complete Postfix stuff youc
                ould go with Fink and install Postfix from scratch. It will take some
                effort to do so, but you would get a complete Postfix installation.

                Cheers,
                Stephan

                Am 02.07.2006 um 04:33 schrieb Jan Steinman:

                > On 1 Jul 06, at 19:06 , Wietse Venema wrote:
                >
                >> A problem report without confguration and actual evidence?
                >>
                >> Do some home work.
                >>
                >> http://www.postfix.org/QSHAPE_README.html
                >
                > Thanks for the starting point. Exactly what I was looking for. I
                > even expected the attitude.
                >
                > # no qshape in /var/root/bin /Developer/Tools /usr/local/bin /usr/
                > bin /bin /usr/sbin /sbin /sw/bin /sw/sbin /opt/local/bin
                >
                > Ah, never mind. I forgot that I'm running an "unsupported" version
                > -- the one that comes with the latest MacOS X updates. Last time I
                > tried to install the supported version, I and my customers were
                > without mail for a week.
                >
                > :::: Sell your cleverness, and purchase bewilderment -- Rumi
                > :::: Jan Steinman <http://www.Bytesmiths.com/Item/99-6313-15>
                >
                >

                --
                Stephan Budach

                Jung von Matt/it-services GmbH
                Glashüttenstrasse 79
                20357 Hamburg

                Tel: +49 (40) 4321-1353
                Fax: +49 (40) 4321-1114

                http://www.jvm.de
              • Ralf Hildebrandt
                ... So your box has a lot of incoming connections. How many smtpd processes are configured (check master.cf to see)? ... OK, that s not much. What is
                Message 7 of 8 , Jul 2, 2006
                • 0 Attachment
                  * Jan Steinman <Jan@...>:

                  > But sending from other machines on our network takes increasingly long
                  > times after starting the server. It has been three days since a
                  > restart, and it now takes about 20 seconds pause between hitting
                  > "send" and having the message leave the queue.
                  >
                  > There are over a hundred process entries like the following:
                  > postfix 2272 0.0 -0.0 27524 924 ?? S 4:33PM 0:00.03 smtpd -n smtp -t inet -u

                  So your box has a lot of incoming connections. How many smtpd processes
                  are configured (check master.cf to see)?

                  > Checking logs, I am getting 10-20 rejects PER MINUTE! All of them
                  > appear to be legit rejects -- generally "User unknown in local
                  > recipient table".

                  OK, that's not much. What is smtpd_error_sleep_time set to?
                  % postconf smtpd_error_sleep_time
                  tells you.

                  > I suspect that spammers are hitting me particularly hard
                  Hard? 10-20 rejects per minute is not a lot.

                  > 5) Throttle the number of smtpd process instances: CONS: may increase
                  > Inadvertent Denial of Service to legit SMTP traffic.

                  Indeed. So why not up it?

                  Show postconf -n :)

                  --
                  Ralf Hildebrandt (Ralf.Hildebrandt@...) spamtrap@...
                  Postfix - Einrichtung, Betrieb und Wartung Tel. +49 (0)30-450 570-155
                  http://www.postfix-buch.com
                  Why you can't find your system administrators:
                  They are hiding under the stairs
                • Wietse Venema
                  ... Citing the first paragraph of Purpose of this document of QSHAPE_README.html: This document describes the qshape(1) program which helps the administrator
                  Message 8 of 8 , Jul 2, 2006
                  • 0 Attachment
                    Jan Steinman:
                    > On 1 Jul 06, at 19:06 , Wietse Venema wrote:
                    >
                    > > A problem report without confguration and actual evidence?
                    > >
                    > > Do some home work.
                    > >
                    > > http://www.postfix.org/QSHAPE_README.html
                    >
                    > Thanks for the starting point. Exactly what I was looking for. I even
                    > expected the attitude.
                    >
                    > # no qshape in /var/root/bin /Developer/Tools /usr/local/bin /usr/
                    > bin /bin /usr/sbin /sbin /sw/bin /sw/sbin /opt/local/bin

                    Citing the first paragraph of

                    Purpose of this document of QSHAPE_README.html:

                    This document describes the qshape(1) program which helps the
                    administrator understand the Postfix queue message distribution
                    sorted by time and by sender or recipient domain. QSHAPE(1) IS
                    BUNDLED WITH THE POSTFIX 2.1 SOURCE UNDER THE "AUXILIARY"
                    DIRECTORY.

                    I can point you to the docs, but I can't teach you to read.

                    Wietse
                  Your message has been successfully submitted and would be delivered to recipients shortly.