Loading ...
Sorry, an error occurred while loading the content.

Our postfix works fine, but it is very slow when we send newsletter

Expand Messages
  • Vince Wang
    Hello, We have a configured postfix email server worked well when we had it on the public IP. After we moved it behind our firewall on a intranet with ip
    Message 1 of 6 , Feb 20, 2013
    • 0 Attachment

      Hello,

       

      We have a configured postfix email server worked well when we had it on the public IP. After we moved  it behind our firewall on a intranet with ip 192.168.xxx.xxx, we found it is very slow when we send newsletter.

       

      Server info:     Ubuntu 10.4 32 bit running on 4cpus + 8GB memory VM ( VMware host )

      postfix: 2.70

       

      As I just start learning about  postfix so  I tried to figure how it works.  I sent a newsletter to 1100 members last week and monitored  the queue in the webmin and mailq, and the postfix log.  After I clicked the "send" button on our web page, I  found that the messages are added into the queue for 15 minutes and then I saw messages are sent out  from the log file for around 15 minutes.   

       

      Is there anything wrong in our configure?

       

      Thanks a lot,

      Vince

       

      Here is  the main.cf:

      # amavis loop

      content_filter = smtp-amavis:[127.0.0.1]:10024

      smtpd_banner = $myhostname ESMTP $mail_name (Ubuntu)

      biff = no

      mailbox_size_limit = 2147483648

      smtpd_sender_restrictions = permit_mynetworks, check_sender_access hash:/etc/postfix/users, check_sender_access pcre:/etc/postfix/user_catchall, reject

      smtpd_recipient_restrictions = check_recipient_access pcre:/etc/postfix/recipients, reject

      header_checks = pcre:/etc/postfix/header_checks

      # ALIAS DATABASE

      alias_maps = hash:/etc/postfix/aliases

      alias_database = hash:/etc/postfix/aliases

       

      # Virtual Maps

      virtual_alias_maps = hash:/etc/postfix/virtual

      readme_directory = no

       

      # TLS parameters

      smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem

      smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key

      smtpd_use_tls=yes

      smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache

      smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache

       

      # See /usr/share/doc/postfix/TLS_README.gz in the postfix-doc package for

      # information on enabling SSL in the smtp client.

       

      myhostname = www.xxx.com

      myorigin = /etc/mailname

       

      mydestination = $myhostname, localhost.$mydomain, localhost

      mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128

      mailbox_command = procmail -a "$EXTENSION"

       

      recipient_delimiter = +

      smtpd_recipient_limit = 100000

      error_notice_recipient = error@...

       

      proxy_interfaces = 123.123.1223.158 ( NAT IP address )

      # (the proxy/NAT external network address)

      qmgr_message_active_limit = 50000

      line_length_limit = 204800

      maximal_queue_lifetime = 2d

      queue_run_delay = 4000s

      minimal_backoff_time = 4000s

       

      here is the master.cf

      smtp      inet  n       -       -       -       -       smtpd

      #submission inet n       -       -       -       -       smtpd

      #  -o smtpd_tls_security_level=encrypt

      #  -o smtpd_sasl_auth_enable=yes

      #  -o smtpd_client_restrictions=permit_sasl_authenticated,reject

      #  -o milter_macro_daemon_name=ORIGINATING

      #smtps     inet  n       -       -       -       -       smtpd

      #  -o smtpd_tls_wrappermode=yes

      #  -o smtpd_sasl_auth_enable=yes

      #  -o smtpd_client_restrictions=permit_sasl_authenticated,reject

      #  -o milter_macro_daemon_name=ORIGINATING

      #628       inet  n       -       -       -       -       qmqpd

      pickup    fifo  n       -       -       60      1       pickup

      cleanup   unix  n       -       -       -       0       cleanup

      #qmgr      fifo  n       -       n       300     1       qmgr

      qmgr     fifo  n       -       -       300     1       oqmgr

      tlsmgr    unix  -       -       -       1000?   1       tlsmgr

      rewrite   unix  -       -       -       -       -       trivial-rewrite

      bounce    unix  -       -       -       -       0       bounce

      defer     unix  -       -       -       -       0       bounce

      trace     unix  -       -       -       -       0       bounce

      verify    unix  -       -       -       -       1       verify

      flush     unix  n       -       -       1000?   0       flush

      proxymap  unix  -       -       n       -       -       proxymap

      proxywrite unix -       -       n       -       1       proxymap

      smtp      unix  -       -       -       -       -       smtp

      # When relaying mail as backup MX, disable fallback_relay to avoid MX loops

      relay     unix  -       -       -       -       -       smtp

                      -o smtp_fallback_relay=

      #       -o smtp_helo_timeout=5 -o smtp_connect_timeout=5

      showq     unix  n       -       -       -       -       showq

      error     unix  -       -       -       -       -       error

      retry     unix  -       -       -       -       -       error

      discard   unix  -       -       -       -       -       discard

      local     unix  -       n       n       -       -       local

      virtual   unix  -       n       n       -       -       virtual

      lmtp      unix  -       -       -       -       -       lmtp

      anvil     unix  -       -       -       -       1       anvil

      scache    unix  -       -       -       -       1       scache

      #

      # ====================================================================

      # Interfaces to non-Postfix software. Be sure to examine the manual

      # pages of the non-Postfix software to find out what options it wants.

      #

      # Many of the following services use the Postfix pipe(8) delivery

      # agent.  See the pipe(8) man page for information about ${recipient}

      # and other message envelope options.

      # ====================================================================

      #

      # maildrop. See the Postfix MAILDROP_README file for details.

      # Also specify in main.cf: maildrop_destination_recipient_limit=1

      #

      maildrop  unix  -       n       n       -       -       pipe

        flags=DRhu user=vmail argv=/usr/bin/maildrop -d ${recipient}

      #

      # ====================================================================

      #

      # Recent Cyrus versions can use the existing "lmtp" master.cf entry.

      #

      # Specify in cyrus.conf:

      #   lmtp    cmd="lmtpd -a" listen="localhost:lmtp" proto=tcp4

      #

      # Specify in main.cf one or more of the following:

      #  mailbox_transport = lmtp:inet:localhost

      #  virtual_transport = lmtp:inet:localhost

      #

      # ====================================================================

      #

      # Cyrus 2.1.5 (Amos Gouaux)

      # Also specify in main.cf: cyrus_destination_recipient_limit=1

      #

      #cyrus     unix  -       n       n       -       -       pipe

      #  user=cyrus argv=/cyrus/bin/deliver -e -r ${sender} -m ${extension} ${user}

      #

      # ====================================================================

      # Old example of delivery via Cyrus.

      #

      #old-cyrus unix  -       n       n       -       -       pipe

      #  flags=R user=cyrus argv=/cyrus/bin/deliver -e -m ${extension} ${user}

      #

      # ====================================================================

      #

      # See the Postfix UUCP_README file for configuration details.

      #

      uucp      unix  -       n       n       -       -       pipe

        flags=Fqhu user=uucp argv=uux -r -n -z -a$sender - $nexthop!rmail ($recipient)

      #

      # Other external delivery methods.

      #

      ifmail    unix  -       n       n       -       -       pipe

        flags=F user=ftn argv=/usr/lib/ifmail/ifmail -r $nexthop ($recipient)

      bsmtp     unix  -       n       n       -       -       pipe

        flags=Fq. user=bsmtp argv=/usr/lib/bsmtp/bsmtp -t$nexthop -f$sender $recipient

      scalemail-backend unix -              n             n             -              2              pipe

        flags=R user=scalemail argv=/usr/lib/scalemail/bin/scalemail-store ${nexthop} ${user} ${extension}

      mailman   unix  -       n       n       -       -       pipe

        flags=FR user=list argv=/usr/lib/mailman/bin/postfix-to-mailman.py

        ${nexthop} ${user}

       

      #

      # copid from https://help.ubuntu.com/community/PostfixAmavisNew

      # to enable the amavis service in the postfix

      # sudo /etc/init.d/postfix reload

      #

      smtp-amavis     unix    -       -       -       -       10       smtp

              -o smtp_data_done_timeout=1200

              -o smtp_send_xforward_command=yes

              -o disable_dns_lookups=yes

              -o max_use=20

       

      127.0.0.1:10025 inet    n       -       -       -       -       smtpd

              -o content_filter=

              -o local_recipient_maps=

              -o relay_recipient_maps=

              -o smtpd_restriction_classes=

              -o smtpd_delay_reject=no

              -o smtpd_client_restrictions=permit_mynetworks,reject

              -o smtpd_helo_restrictions=

              -o smtpd_sender_restrictions=

              -o smtpd_recipient_restrictions=permit_mynetworks,reject

              -o smtpd_data_restrictions=reject_unauth_pipelining

              -o smtpd_end_of_data_restrictions=

              -o mynetworks=127.0.0.0/8

              -o smtpd_error_sleep_time=0

              -o smtpd_soft_error_limit=1001

              -o smtpd_hard_error_limit=1000

              -o smtpd_client_connection_count_limit=0

              -o smtpd_client_connection_rate_limit=0

              -o receive_override_options=no_header_body_checks,no_unknown_recipient_checks

    • Ansgar Wiechers
      ... Since the message gets delivered there might be nothing wrong with your configuration. Im wondering, though, why you chose so large values for ... Anyway,
      Message 2 of 6 , Feb 20, 2013
      • 0 Attachment
        On 2013-02-20 Vince Wang wrote:
        > We have a configured postfix email server worked well when we had it
        > on the public IP. After we moved it behind our firewall on a intranet
        > with ip 192.168.xxx.xxx, we found it is very slow when we send
        > newsletter.
        >
        > Server info: Ubuntu 10.4 32 bit running on 4cpus + 8GB memory VM (
        > VMware host )
        > postfix: 2.70
        >
        > As I just start learning about postfix so I tried to figure how it
        > works. I sent a newsletter to 1100 members last week and monitored
        > the queue in the webmin and mailq, and the postfix log. After I
        > clicked the "send" button on our web page, I found that the messages
        > are added into the queue for 15 minutes and then I saw messages are
        > sent out from the log file for around 15 minutes.
        >
        > Is there anything wrong in our configure?

        Since the message gets delivered there might be nothing wrong with your
        configuration. Im wondering, though, why you chose so large values for
        $queue_run_delay and $minimal_backoff_time:

        > queue_run_delay = 4000s
        > minimal_backoff_time = 4000s

        Anyway, there isn't much we could tell without seeing the logs. Please
        post a log excerpt showing a full transaction of a delayed mail (from
        the point where the mail enters postfix to the point where it's
        delivered). You can get that data by greping for the queue ID of such a
        message.

        Also, always post the output of "postconf -n", not the content of
        main.cf. The latter isn't guaranteed to be the active configuration. The
        former is.

        Regards
        Ansgar Wiechers
        --
        "Abstractions save us time working, but they don't save us time learning."
        --Joel Spolsky
      • Wietse Venema
        ... Please define slow , and don t forget to include your outbound speed and message size. If you are sending mail to Yahoo and other large mailers, they will
        Message 3 of 6 , Feb 20, 2013
        • 0 Attachment
          Vince Wang:
          > Hello,
          >
          > We have a configured postfix email server worked well when we had
          > it on the public IP. After we moved it behind our firewall on a
          > intranet with ip 192.168.xxx.xxx, we found it is very slow when
          > we send newsletter.

          Please define "slow", and don't forget to include your outbound
          speed and message size. If you are sending mail to Yahoo and other
          large mailers, they will slow you down intentionally.

          Wietse
        • Ralf Hildebrandt
          ... Logs? ... It could be that the process injecting the mails into the queue is stalling the queuemanager, thus sending out can only begin AFTER the injection
          Message 4 of 6 , Feb 21, 2013
          • 0 Attachment
            * Vince Wang <vwang@...>:
            > Hello,
            >
            > We have a configured postfix email server worked well when we had it on the public IP. After we moved it behind our firewall on a intranet with ip 192.168.xxx.xxx, we found it is very slow when we send newsletter.

            Logs?

            > As I just start learning about postfix so I tried to figure how it
            > works. I sent a newsletter to 1100 members last week and monitored
            > the queue in the webmin and mailq, and the postfix log. After I
            > clicked the "send" button on our web page, I found that the messages
            > are added into the queue for 15 minutes and then I saw messages are
            > sent out from the log file for around 15 minutes.

            It could be that the process injecting the mails into the queue is
            stalling the queuemanager, thus sending out can only begin AFTER the
            injection period.

            > Here is the main.cf:
            > # amavis loop
            > content_filter = smtp-amavis:[127.0.0.1]:10024

            You're filtering the mail? I hope not.
            --
            [*] sys4 AG

            http://sys4.de, +49 (89) 30 90 46 64
            Franziskanerstraße 15, 81669 München

            Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
            Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer
            Aufsichtsratsvorsitzender: Joerg Heidrich
          • Jeroen Geilman
            ... How is DNS set up in comparison with the previous server ? Badly configured DNS can certainly slow things down, especially on outgoing mail. Any even
            Message 5 of 6 , Feb 21, 2013
            • 0 Attachment
              On 02/20/2013 07:16 PM, Vince Wang wrote:

              Hello,

               

              We have a configured postfix email server worked well when we had it on the public IP.
              After we moved  it behind our firewall on a intranet with ip 192.168.xxx.xxx, we found it is very slow when we send newsletter.


              How is DNS set up in comparison with the previous server ?
              Badly configured DNS can certainly slow things down, especially on outgoing mail.
              Any even moderately busy mailserver should have a local DNS cache.

                Server info:     Ubuntu 10.4 32 bit running on 4cpus + 8GB memory VM ( VMware host )


              A 32-bit OS with 8GB of memory ? only 3.5GB of that will be used, ever.
              Regardless, postfix hardly uses any memory, unless you are receiving hundreds of 10MB messages concurrently.
              That is much more relevant for mail performance is storage I/O - and you don't mention anything related to storage.

              As I just start learning about  postfix so  I tried to figure how it works.  I sent a newsletter to 1100 members last week

              How many *messages* did you send ?

              and monitored  the queue in the webmin and mailq, and the postfix log.  After I clicked the "send" button on our web page, I  found that the messages are added into the queue for 15 minutes and then I saw messages are sent out  from the log file for around 15 minutes.   


              So you are seeing an average processing speed of 1.2 messages per second before queue, and another average 1.2 messages per second during delivery ?
              Show logs that exhibit these delays; postfix logs detailed delay statistics for each message delivered.

                content_filter = smtp-amavis:[127.0.0.1]:10024


              If you're submitting via smtpd(8) then all locally submitted mail will be scanned, which is patently useless in this case.

              smtpd_recipient_limit = 100000

              That is insane.

              qmgr_message_active_limit = 50000

              line_length_limit = 204800

              maximal_queue_lifetime = 2d

              queue_run_delay = 4000s

              minimal_backoff_time = 4000s


              Do not mess with these values unless you know exactly what they do.

              No logs, so how do you expect us to deduce what is happening here ?


              -- 
              J.
              
            • Jeroen Geilman
              ... ... how ? Either pickup(8) or smtpd(8) do the queueing; the qmgr only SENDS mail. There could be disk I/O contention, sure, but that would never translate
              Message 6 of 6 , Feb 21, 2013
              • 0 Attachment
                On 02/21/2013 03:34 PM, Ralf Hildebrandt wrote:
                > It could be that the process injecting the mails into the queue is
                > stalling the queuemanager, thus sending out can only begin AFTER the
                > injection period.

                ... how ?

                Either pickup(8) or smtpd(8) do the queueing; the qmgr only SENDS mail.
                There could be disk I/O contention, sure, but that would never translate
                into a scenario where no mail could be de-queued before all mail was
                finished queueing.
                These are wholly separate processes after all, and the only point of
                contact is the mail queue, which is concurrent read-write by design.
                By default, there may be many simultaneous processes accessing the queue
                (100 each of smtpd and smtp, for starters.)

                Of course, it could be that he really is sending every single submitted
                message through amavisd and then re-injecting into postfix, thus
                effectively forcing every single message through the pipeline twice.

                This would be inane no matter what kind of IP address it has, but the
                cause of the delays would be the content_filter, nothing else.

                There are settings in amavisd-new that govern what to do when a message
                originates from a trusted or untrusted IP range, offering the option to
                pass it through without scanning.
                If this was impacted by the IP change, that could easily explain the
                delays - but they would still never be sequential.

                Of course, you did ask for logs as well :)

                --
                J.
              Your message has been successfully submitted and would be delivered to recipients shortly.