Loading ...
Sorry, an error occurred while loading the content.

Persistant LDAP connections

Expand Messages
  • Geoff Shang
    Hi, First, thanks to everyone for your help so far. We ve got all our customer information in LDAP and we ve set up our Postfix and Dovecot instances to talk
    Message 1 of 15 , Mar 8, 2013
    • 0 Attachment
      Hi,

      First, thanks to everyone for your help so far.

      We've got all our customer information in LDAP and we've set up our
      Postfix and Dovecot instances to talk to it.

      Given the high focus on secrity at our company, we've determined that
      password verification in LDAP is a costly operation. Therefore, we need
      to try to limit LDAP lookups, specifically ones that depend on either
      verifying a customer's password or logging in (binding) with an account
      (which obviously needs to verify a password).

      I have two queries that I'll put in separate threads.

      I've configured our MX to lookup customer domains and addresses in LDAP
      before accepting mail. This is, of course, going to be our most common
      lookup, as it will be done for every Email received from the outside
      world.

      Right now, Postfix is connecting to LDAP every time it needs to do one of
      these lookups, then disconnects again. I thought that specifying "proxy:"
      in the entry might deal with this, but it doesn't appear to have done so.

      My question is, is it possible to get proxymap to open a persistant
      connection for LDAP to do relay_domain and relay_recipient lookups?

      Thanks for any advice.

      Cheers,
      Geoff.

      Configuration info follows:

      postconf -n output:

      alias_database = hash:/etc/aliases
      alias_maps = hash:/etc/aliases
      append_dot_mydomain = no
      biff = no
      config_directory = /etc/postfix
      html_directory = /usr/share/doc/postfix/html
      inet_interfaces = all
      inet_protocols = ipv6,ipv4
      mailbox_size_limit = 0
      mydestination = mx.ourdomain.com, localhost
      myhostname = mx.ourdomain.com
      mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
      <our IPv6 allocation> <our IPv4 allocation>
      myorigin = /etc/mailname
      readme_directory = /usr/share/doc/postfix
      recipient_delimiter = +
      relay_domains = proxy:ldap:/etc/postfix/ldap-domains.cf ourdomain.com
      relay_recipient_maps =
      proxy:pgsql:/etc/postfix/pgsql_corporate_recipients.cf
      proxy:ldap:/etc/postfix/ldap-users.cf
      relay_transport = relay:[scanner.ourdomain.com]
      smtp_tls_ciphers = high
      smtp_tls_mandatory_ciphers = high
      smtp_tls_mandatory_exclude_ciphers = RC4,MD5
      smtp_tls_note_starttls_offer = yes
      smtp_tls_protocols = !SSLv2,!SSLv3
      smtp_tls_security_level = may
      smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
      smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU)
      smtpd_error_sleep_time = 2s
      smtpd_hard_error_limit = 10
      smtpd_helo_required = yes
      smtpd_helo_restrictions = permit_mynetworks
      reject_invalid_helo_hostname
      reject_non_fqdn_helo_hostname
      smtpd_recipient_restrictions = permit_mynetworks
      reject_unauth_pipelining reject_non_fqdn_sender
      reject_invalid_hostname reject_non_fqdn_hostname
      reject_unknown_sender_domain reject_unlisted_recipient
      reject_non_fqdn_recipient reject_unknown_recipient_domain
      reject_unauth_destination reject_multi_recipient_bounce
      smtpd_soft_error_limit = 5
      smtpd_tls_cert_file = /etc/ssl/certs/ssl-cert-snakeoil.pem
      smtpd_tls_key_file = /etc/ssl/private/ssl-cert-snakeoil.key
      smtpd_tls_loglevel = 1
      smtpd_tls_received_header = yes
      smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
      smtpd_use_tls = yes

      /etc/postfix/ldap-domains.cf:

      version = 3
      timeout = 20
      size_limit = 1
      expansion_limit = 1
      start_tls = no
      tls_require_cert = no
      scope = one
      query_filter = o=%s
      result_attribute = o
      server_host = ldap://ldap-server.ourdomain.com
      search_base =ou=mail,dc=ourdomain,dc=com

      /etc/postfix/ldap-users.cf:

      version = 3
      timeout = 20
      size_limit = 1
      expansion_limit = 1
      start_tls = no
      tls_require_cert = no
      scope = sub
      query_filter = mail=%s
      result_attribute = mail
      server_host = ldap://ldap-server.ourdomain.com
      search_base =o=%d,ou=mail,dc=ourdomain,dc=com

      # The return value is only used in a transport map (i.e. on our scanner)
      result_format = lmtp:[imap.ourdomain.com]:24
    • Bastian Blank
      ... Why is it costly? And how does costly fit into security? And password verification is not necessary for looking up stuff. ... Add a LDAP replica on each
      Message 2 of 15 , Mar 8, 2013
      • 0 Attachment
        On Fri, Mar 08, 2013 at 03:45:57PM +0200, Geoff Shang wrote:
        > Given the high focus on secrity at our company, we've determined
        > that password verification in LDAP is a costly operation.

        Why is it costly? And how does "costly" fit into security? And password
        verification is not necessary for looking up stuff.

        > Therefore, we need to try to limit LDAP lookups, specifically ones
        > that depend on either verifying a customer's password or logging in
        > (binding) with an account (which obviously needs to verify a
        > password).

        Add a LDAP replica on each postfix and dovecot server. This is a good
        idea for scallability and rudandancy anyway.

        > My question is, is it possible to get proxymap to open a persistant
        > connection for LDAP to do relay_domain and relay_recipient lookups?

        It does this in all of my setups. They use Postfix 2.9.

        > mydestination = mx.ourdomain.com, localhost
        > myhostname = mx.ourdomain.com

        I don't think this is correct. Maybe mx.example.com.

        Bastian

        --
        History tends to exaggerate.
        -- Col. Green, "The Savage Curtain", stardate 5906.4
      • Geoff Shang
        ... Because the passwords are stored in some highly encrypted form (not my area) and comparing passwords means likewise encrypting the password to be checked.
        Message 3 of 15 , Mar 8, 2013
        • 0 Attachment
          On Fri, 8 Mar 2013, Bastian Blank wrote:

          > On Fri, Mar 08, 2013 at 03:45:57PM +0200, Geoff Shang wrote:
          >> Given the high focus on secrity at our company, we've determined
          >> that password verification in LDAP is a costly operation.
          >
          > Why is it costly? And how does "costly" fit into security?

          Because the passwords are stored in some highly encrypted form (not my
          area) and comparing passwords means likewise encrypting the password to be
          checked.

          > And password verification is not necessary for looking up stuff.

          Not if you bind anonymously. But if you bind with a specific account
          (i.e. log in with a username and password), this will need to be verified.
          This is no big deal if it happens once but can be a performance drain if
          it has to happen for every single lookup.

          The other issue is TLS negociation. If it can be set up once, this is
          fine. Frequent TLS negotiations will likewise be a performance hit.

          We could do anonymous binds in the clear, but we're taking this as a last
          resort position.

          > Add a LDAP replica on each postfix and dovecot server. This is a good
          > idea for scallability and rudandancy anyway.

          Not sure how wild people will be about this idea.

          >> My question is, is it possible to get proxymap to open a persistant
          >> connection for LDAP to do relay_domain and relay_recipient lookups?
          >
          > It does this in all of my setups. They use Postfix 2.9.

          Good point. I'm using 2.7.1 (Debian stable).

          >> mydestination = mx.ourdomain.com, localhost
          >> myhostname = mx.ourdomain.com
          >
          > I don't think this is correct. Maybe mx.example.com.

          It's correct. All hosted domains will be relay_domains.

          Geoff.
        • Bastian Blank
          ... Then just don t do it. You IPSEC or so if you want network security. ... Don t do that. ... As always: don t shoot the messenger. ... And why? You need the
          Message 4 of 15 , Mar 8, 2013
          • 0 Attachment
            On Fri, Mar 08, 2013 at 05:23:27PM +0200, Geoff Shang wrote:
            > On Fri, 8 Mar 2013, Bastian Blank wrote:
            > >On Fri, Mar 08, 2013 at 03:45:57PM +0200, Geoff Shang wrote:
            > >And password verification is not necessary for looking up stuff.
            > Not if you bind anonymously. But if you bind with a specific
            > account (i.e. log in with a username and password), this will need
            > to be verified. This is no big deal if it happens once but can be a
            > performance drain if it has to happen for every single lookup.

            Then just don't do it. You IPSEC or so if you want network security.

            > The other issue is TLS negociation. If it can be set up once, this
            > is fine. Frequent TLS negotiations will likewise be a performance
            > hit.

            Don't do that.

            > We could do anonymous binds in the clear, but we're taking this as a
            > last resort position.

            As always: don't shoot the messenger.

            > >Add a LDAP replica on each postfix and dovecot server. This is a good
            > >idea for scallability and rudandancy anyway.
            > Not sure how wild people will be about this idea.

            And why? You need the information at this location and already have
            access.

            > >>mydestination = mx.ourdomain.com, localhost
            > >>myhostname = mx.ourdomain.com
            > >I don't think this is correct. Maybe mx.example.com.
            > It's correct. All hosted domains will be relay_domains.

            No, it is not correct:

            | $ drill mx.ourdomain.com any
            | ;; ->>HEADER<<- opcode: QUERY, rcode: NXDOMAIN, id: 40228

            There is no mx.ourdomain.com in the public DNS.

            Bastian

            --
            War isn't a good life, but it's life.
            -- Kirk, "A Private Little War", stardate 4211.8
          • Quanah Gibson-Mount
            --On Friday, March 08, 2013 3:45 PM +0200 Geoff Shang ... This is exactly what postfix does for our setup, using postfix 2.10.0. Example connection from this
            Message 5 of 15 , Mar 8, 2013
            • 0 Attachment
              --On Friday, March 08, 2013 3:45 PM +0200 Geoff Shang
              <geoff@...> wrote:

              > Hi,

              > Right now, Postfix is connecting to LDAP every time it needs to do one of
              > these lookups, then disconnects again. I thought that specifying
              > "proxy:" in the entry might deal with this, but it doesn't appear to have
              > done so.
              >
              > My question is, is it possible to get proxymap to open a persistant
              > connection for LDAP to do relay_domain and relay_recipient lookups?

              This is exactly what postfix does for our setup, using postfix 2.10.0.
              Example connection from this morning:

              Mar 8 08:01:49 ldap01-zcs slapd[7644]: conn=29791 fd=84 ACCEPT from
              IP=X.X.X.X:53128 (IP=X.X.X.X:389)
              Mar 8 08:01:49 ldap01-zcs slapd[7644]: conn=29791 op=0 EXT
              oid=1.3.6.1.4.1.1466.20037
              Mar 8 08:01:49 ldap01-zcs slapd[7644]: conn=29791 op=0 STARTTLS
              Mar 8 08:01:49 ldap01-zcs slapd[7644]: conn=29791 op=0 RESULT oid= err=0
              text=
              Mar 8 08:01:49 ldap01-zcs slapd[7644]: conn=29791 fd=84 TLS established
              tls_ssf=256 ssf=256
              Mar 8 08:01:49 ldap01-zcs slapd[7644]: conn=29791 op=1 BIND
              dn="uid=zmpostfix,cn=appaccts,cn=zimbra" method=128
              Mar 8 08:01:49 ldap01-zcs slapd[7644]: conn=29791 op=1 BIND
              dn="uid=zmpostfix,cn=appaccts,cn=zimbra" mech=SIMPLE ssf=0
              Mar 8 08:01:49 ldap01-zcs slapd[7644]: conn=29791 op=1 RESULT tag=97 err=0
              text=


              [... lots of queries ...]

              Mar 8 08:12:27 ldap01-zcs slapd[7644]: conn=29791 op=7801 SRCH
              attr=zimbraMailCanonicalAddress zimbraMailCatchAllCanonicalAddress
              Mar 8 08:12:27 ldap01-zcs slapd[7644]: conn=29791 op=7801 SEARCH RESULT
              tag=101 err=0 nentries=0 text=
              Mar 8 08:12:32 ldap01-zcs slapd[7644]: conn=29791 fd=84 closed (connection
              lost)


              Or as you can see, it stayed connected from 08:01:49 to 08:12:32, during
              which time it performed 7,800 operations (all mail delivery lookups)
              outside of the initial bind before disconnecting.

              --Quanah

              --

              Quanah Gibson-Mount
              Sr. Member of Technical Staff
              Zimbra, Inc
              A Division of VMware, Inc.
              --------------------
              Zimbra :: the leader in open source messaging and collaboration
            • Viktor Dukhovni
              ... No Postfix release has ever done that. LDAP connections have always been cached by the process that makes the queries. The first query triggers a
              Message 6 of 15 , Mar 8, 2013
              • 0 Attachment
                On Fri, Mar 08, 2013 at 03:45:57PM +0200, Geoff Shang wrote:

                > Right now, Postfix is connecting to LDAP every time it needs to do
                > one of these lookups, then disconnects again.

                No Postfix release has ever done that. LDAP connections have always
                been cached by the process that makes the queries. The first query
                triggers a connection, subsequent queries re-use the connection.

                > I thought that specifying "proxy:" in the entry might deal with
                > this, but it doesn't appear to have done so.

                The effect of the "proxy:" prefix is to reduce the number of
                processes that make LDAP connections, by pooling all the connections
                via a small number of proxy processes.

                On a sufficiently idle system (say a test system which only receives
                intermittent email messages) the processes that are connected to LDAP
                may exit when idle for long enough, and then new connections will be
                made later. The same happens on systems where some misguidedly runs
                "postfix reload" frequently.

                > My question is, is it possible to get proxymap to open a persistant
                > connection for LDAP to do relay_domain and relay_recipient lookups?

                It is not possible to open non-persisten connections. If the LDAP
                server closes connections that the LDAP client did not actively
                close, that could be a reason for connections to not stay open.

                > /etc/postfix/ldap-domains.cf:
                >
                > version = 3
                > start_tls = no
                > tls_require_cert = no
                > server_host = ldap://ldap-server.ourdomain.com
                >
                > /etc/postfix/ldap-users.cf:
                >
                > version = 3
                > start_tls = no
                > tls_require_cert = no
                > server_host = ldap://ldap-server.ourdomain.com

                Furthermore, both tables have the same connection-related
                Parameters, and so as of postfix-2.0.16-2003091 both tables
                use the same LDAP connection.

                20030917

                Multiple LDAP lookup tables in the one Postfix process now
                share one LDAP connection. ...

                This snapshot eventually evolved into Postfix 2.1. So Postfix 2.1
                or newer supports both connection caching and connection consolidation
                for multiple tables that differ only in the query paramers (search base,
                scope, query, returned attributes).

                The proxymap service was introduced in postfix-2.0.0-20030103.

                --
                Viktor.
              • Quanah Gibson-Mount
                --On Friday, March 08, 2013 5:29 PM +0000 Viktor Dukhovni ... This is not really the behavior I see from proxy:. For example, the connection I pasted from
                Message 7 of 15 , Mar 8, 2013
                • 0 Attachment
                  --On Friday, March 08, 2013 5:29 PM +0000 Viktor Dukhovni
                  <postfix-users@...> wrote:

                  > The effect of the "proxy:" prefix is to reduce the number of
                  > processes that make LDAP connections, by pooling all the connections
                  > via a small number of proxy processes.
                  >
                  > On a sufficiently idle system (say a test system which only receives
                  > intermittent email messages) the processes that are connected to LDAP
                  > may exit when idle for long enough, and then new connections will be
                  > made later. The same happens on systems where some misguidedly runs
                  > "postfix reload" frequently.

                  This is not really the behavior I see from proxy:. For example, the
                  connection I pasted from this morning:

                  Mar 8 08:12:27 ldap01-zcs slapd[7644]: conn=29791 op=7801 SRCH
                  attr=zimbraMailCanonicalAddress zimbraMailCatchAllCanonicalAddress
                  Mar 8 08:12:27 ldap01-zcs slapd[7644]: conn=29791 op=7801 SEARCH RESULT
                  tag=101 err=0 nentries=0 text=
                  Mar 8 08:12:32 ldap01-zcs slapd[7644]: conn=29791 fd=84 closed (connection
                  lost)


                  You can see that postfix closed the connection (without sending an unbind)
                  5 seconds after operation 7801. Postfix has established 258 connections
                  since 4 am this morning, all of which it initiates a close on after some
                  amount of time (i.e., the LDAP server is not the one closing the
                  connection).

                  --Quanah


                  --

                  Quanah Gibson-Mount
                  Sr. Member of Technical Staff
                  Zimbra, Inc
                  A Division of VMware, Inc.
                  --------------------
                  Zimbra :: the leader in open source messaging and collaboration
                • Quanah Gibson-Mount
                  --On Friday, March 08, 2013 9:41 AM -0800 Quanah Gibson-Mount ... Vs, for example, our OpenDKIM setup, which has used a persistent connection to our ldap
                  Message 8 of 15 , Mar 8, 2013
                  • 0 Attachment
                    --On Friday, March 08, 2013 9:41 AM -0800 Quanah Gibson-Mount
                    <quanah@...> wrote:

                    > --On Friday, March 08, 2013 5:29 PM +0000 Viktor Dukhovni
                    > <postfix-users@...> wrote:
                    >
                    >> The effect of the "proxy:" prefix is to reduce the number of
                    >> processes that make LDAP connections, by pooling all the connections
                    >> via a small number of proxy processes.
                    >>
                    >> On a sufficiently idle system (say a test system which only receives
                    >> intermittent email messages) the processes that are connected to LDAP
                    >> may exit when idle for long enough, and then new connections will be
                    >> made later. The same happens on systems where some misguidedly runs
                    >> "postfix reload" frequently.
                    >
                    > This is not really the behavior I see from proxy:. For example, the
                    > connection I pasted from this morning:
                    >
                    > Mar 8 08:12:27 ldap01-zcs slapd[7644]: conn=29791 op=7801 SRCH
                    > attr=zimbraMailCanonicalAddress zimbraMailCatchAllCanonicalAddress
                    > Mar 8 08:12:27 ldap01-zcs slapd[7644]: conn=29791 op=7801 SEARCH RESULT
                    > tag=101 err=0 nentries=0 text=
                    > Mar 8 08:12:32 ldap01-zcs slapd[7644]: conn=29791 fd=84 closed
                    > (connection lost)
                    >
                    >
                    > You can see that postfix closed the connection (without sending an
                    > unbind) 5 seconds after operation 7801. Postfix has established 258
                    > connections since 4 am this morning, all of which it initiates a close on
                    > after some amount of time (i.e., the LDAP server is not the one closing
                    > the connection).

                    Vs, for example, our OpenDKIM setup, which has used a persistent connection
                    to our ldap server for a few days, and is now on op=137251.

                    --Quanah

                    --

                    Quanah Gibson-Mount
                    Sr. Member of Technical Staff
                    Zimbra, Inc
                    A Division of VMware, Inc.
                    --------------------
                    Zimbra :: the leader in open source messaging and collaboration
                  • Viktor Dukhovni
                    ... Not all connections are necessarily from proxy processes, and not all proxy processes are eternal. ... Postfix does not do graceful shutdown of database
                    Message 9 of 15 , Mar 8, 2013
                    • 0 Attachment
                      On Fri, Mar 08, 2013 at 09:41:10AM -0800, Quanah Gibson-Mount wrote:

                      > >On a sufficiently idle system (say a test system which only receives
                      > >intermittent email messages) the processes that are connected to LDAP
                      > >may exit when idle for long enough, and then new connections will be
                      > >made later. The same happens on systems where some misguidedly runs
                      > >"postfix reload" frequently.
                      >
                      > This is not really the behavior I see from proxy:. For example, the
                      > connection I pasted from this morning:

                      Not all connections are necessarily from proxy processes, and not
                      all proxy processes are eternal.

                      > Mar 8 08:12:27 ldap01-zcs slapd[7644]: conn=29791 op=7801 SRCH
                      > attr=zimbraMailCanonicalAddress zimbraMailCatchAllCanonicalAddress
                      > Mar 8 08:12:27 ldap01-zcs slapd[7644]: conn=29791 op=7801 SEARCH
                      > RESULT tag=101 err=0 nentries=0 text=
                      > Mar 8 08:12:32 ldap01-zcs slapd[7644]: conn=29791 fd=84 closed
                      > (connection lost)
                      >
                      > You can see that postfix closed the connection (without sending an
                      > unbind) 5 seconds after operation 7801. Postfix has established 258
                      > connections since 4 am this morning, all of which it initiates a
                      > close on after some amount of time (i.e., the LDAP server is not the
                      > one closing the connection).

                      Postfix does not do "graceful shutdown" of database tables on exit.
                      When a process is done, it exits. What is your max_idle set to?

                      --
                      Viktor.
                    • Quanah Gibson-Mount
                      --On Friday, March 08, 2013 5:48 PM +0000 Viktor Dukhovni ... Ok. And I m not complaining, really. ;) The proxy: functionality is definitely much more robust
                      Message 10 of 15 , Mar 8, 2013
                      • 0 Attachment
                        --On Friday, March 08, 2013 5:48 PM +0000 Viktor Dukhovni
                        <postfix-users@...> wrote:

                        > On Fri, Mar 08, 2013 at 09:41:10AM -0800, Quanah Gibson-Mount wrote:
                        >
                        >> > On a sufficiently idle system (say a test system which only receives
                        >> > intermittent email messages) the processes that are connected to LDAP
                        >> > may exit when idle for long enough, and then new connections will be
                        >> > made later. The same happens on systems where some misguidedly runs
                        >> > "postfix reload" frequently.
                        >>
                        >> This is not really the behavior I see from proxy:. For example, the
                        >> connection I pasted from this morning:
                        >
                        > Not all connections are necessarily from proxy processes, and not
                        > all proxy processes are eternal.

                        Ok. And I'm not complaining, really. ;) The proxy: functionality is
                        definitely much more robust than without it.

                        > Postfix does not do "graceful shutdown" of database tables on exit.
                        > When a process is done, it exits. What is your max_idle set to?

                        [zimbra@edge01-zcs ~]$ postconf | grep max_idle
                        max_idle = 100s
                        smtpd_policy_service_max_idle = 300s


                        --Quanah

                        --

                        Quanah Gibson-Mount
                        Sr. Member of Technical Staff
                        Zimbra, Inc
                        A Division of VMware, Inc.
                        --------------------
                        Zimbra :: the leader in open source messaging and collaboration
                      • Viktor Dukhovni
                        ... Postfix is robust without proxy: , but LDAP servers sometimes don t like hundreds of connections from Postfix when each smtpd(8) and each cleanup(8) makes
                        Message 11 of 15 , Mar 8, 2013
                        • 0 Attachment
                          On Fri, Mar 08, 2013 at 09:57:49AM -0800, Quanah Gibson-Mount wrote:

                          > >Not all connections are necessarily from proxy processes, and not
                          > >all proxy processes are eternal.
                          >
                          > Ok. And I'm not complaining, really. ;) The proxy: functionality
                          > is definitely much more robust than without it.

                          Postfix is robust without "proxy:", but LDAP servers sometimes
                          don't like hundreds of connections from Postfix when each smtpd(8)
                          and each cleanup(8) makes its own connection. Proxymap is kinder
                          and gentler to LDAP, it makes little difference to Postfix.

                          > >Postfix does not do "graceful shutdown" of database tables on exit.
                          > >When a process is done, it exits. What is your max_idle set to?
                          >
                          > [zimbra@edge01-zcs ~]$ postconf | grep max_idle
                          > max_idle = 100s
                          > smtpd_policy_service_max_idle = 300s

                          Processes that accept multiple client connections at the same time
                          like proxymap and trivial-rewrite will also exit after handling at
                          least 100 ($max_use) client connections if at some point their
                          client connection count drops to zero. You can also check whether
                          there are any max_idle overrides in master.cf.

                          Anyway your observations in no way contradict what I said, I don't
                          know why you thought they did.

                          --
                          Viktor.
                        • Quanah Gibson-Mount
                          --On Friday, March 08, 2013 6:13 PM +0000 Viktor Dukhovni ... I ve bumped max_use to 5000 to see what happens. My point is that the connections are not as
                          Message 12 of 15 , Mar 8, 2013
                          • 0 Attachment
                            --On Friday, March 08, 2013 6:13 PM +0000 Viktor Dukhovni
                            <postfix-users@...> wrote:


                            > Processes that accept multiple client connections at the same time
                            > like proxymap and trivial-rewrite will also exit after handling at
                            > least 100 ($max_use) client connections if at some point their
                            > client connection count drops to zero. You can also check whether
                            > there are any max_idle overrides in master.cf.
                            >
                            > Anyway your observations in no way contradict what I said, I don't
                            > know why you thought they did.

                            I've bumped max_use to 5000 to see what happens. My point is that the
                            connections are not as persistent as one may desire. ;) I.e., OpenDKIM
                            stays connected forever until the server closes. Postfix is not
                            (currently) doing that for me, but as you note, may well be related to the
                            max_use setting.

                            --Quanah

                            --

                            Quanah Gibson-Mount
                            Sr. Member of Technical Staff
                            Zimbra, Inc
                            A Division of VMware, Inc.
                            --------------------
                            Zimbra :: the leader in open source messaging and collaboration
                          • Viktor Dukhovni
                            ... This is not a feature, it is a bug. OpenDKIM is a multi-threaded process that does not periodically exit to be replaced by a fresh process. As such it
                            Message 13 of 15 , Mar 8, 2013
                            • 0 Attachment
                              On Fri, Mar 08, 2013 at 10:20:20AM -0800, Quanah Gibson-Mount wrote:

                              > My point is that
                              > the connections are not as persistent as one may desire. ;) I.e.,
                              > OpenDKIM stays connected forever until the server closes.

                              This is not a feature, it is a bug. OpenDKIM is a multi-threaded
                              process that does not periodically exit to be replaced by a fresh
                              process. As such it does not tolerate memory leaks in its own code
                              or in the libraries it uses.

                              Postfix avoids this design pattern as much as possible. Other than
                              the tiny master server, only the queue manager (which does no table
                              lookups directly, and does not use SSL, GSSAPI, LDAP, ...), the
                              pickup server and tlsmgr run indefinitely. All three are simple and
                              have minimal interactions with non-Postfix resources.

                              > Postfix
                              > is not (currently) doing that for me, but as you note, may well be
                              > related to the max_use setting.

                              This is a feature. Also this keeps the load on your LDAP servers
                              more balanced, connections don't stick to one server forever.

                              --
                              Viktor.
                            • Quanah Gibson-Mount
                              --On Friday, March 08, 2013 7:05 PM +0000 Viktor Dukhovni ... OpenDKIM does what I ask. It makes a persistent connection and cuts out the overhead of
                              Message 14 of 15 , Mar 8, 2013
                              • 0 Attachment
                                --On Friday, March 08, 2013 7:05 PM +0000 Viktor Dukhovni
                                <postfix-users@...> wrote:

                                > On Fri, Mar 08, 2013 at 10:20:20AM -0800, Quanah Gibson-Mount wrote:
                                >
                                >> My point is that
                                >> the connections are not as persistent as one may desire. ;) I.e.,
                                >> OpenDKIM stays connected forever until the server closes.
                                >
                                > This is not a feature, it is a bug. OpenDKIM is a multi-threaded
                                > process that does not periodically exit to be replaced by a fresh
                                > process. As such it does not tolerate memory leaks in its own code
                                > or in the libraries it uses.

                                OpenDKIM does what I ask. It makes a persistent connection and cuts out
                                the overhead of persistent rebinding.

                                > Postfix avoids this design pattern as much as possible. Other than
                                > the tiny master server, only the queue manager (which does no table
                                > lookups directly, and does not use SSL, GSSAPI, LDAP, ...), the
                                > pickup server and tlsmgr run indefinitely. All three are simple and
                                > have minimal interactions with non-Postfix resources.
                                >
                                >> Postfix
                                >> is not (currently) doing that for me, but as you note, may well be
                                >> related to the max_use setting.
                                >
                                > This is a feature. Also this keeps the load on your LDAP servers
                                > more balanced, connections don't stick to one server forever.

                                I don't see an issue with them sticking to the first server in its URL
                                list, which is how postfix behaves. I organize my URLs as necessary on the
                                MTAs to distribute out the load. If I needed something more complicated
                                than that, I'd use a load balancer and load balanced name to return to
                                postfix. In any case, lookups from postfix cause an insignificant amount
                                of load, as long as they are persistent.

                                Thanks for pointing out max_use. Now instead of postfix rebinding every
                                4-5 minutes to the LDAP servers, it is at least every 20 minutes between
                                binds, significantly cutting out startTLS negotiation overhead and
                                improving performance.

                                It is trivial to see what a significant difference it makes in postfix
                                behavior to go from the default of 100 to 5000:
                                <http://www.pastebin.ca/2330089>

                                --Quanah

                                --

                                Quanah Gibson-Mount
                                Sr. Member of Technical Staff
                                Zimbra, Inc
                                A Division of VMware, Inc.
                                --------------------
                                Zimbra :: the leader in open source messaging and collaboration
                              • Viktor Dukhovni
                                ... Just because you want it, does not mean it is better. :-) ... It is surely trivial to see what an insignificant difference this makes. Between all those
                                Message 15 of 15 , Mar 8, 2013
                                • 0 Attachment
                                  On Fri, Mar 08, 2013 at 11:24:25AM -0800, Quanah Gibson-Mount wrote:

                                  > >This is not a feature, it is a bug. OpenDKIM is a multi-threaded
                                  > >process that does not periodically exit to be replaced by a fresh
                                  > >process. As such it does not tolerate memory leaks in its own code
                                  > >or in the libraries it uses.
                                  >
                                  > OpenDKIM does what I ask. It makes a persistent connection and cuts
                                  > out the overhead of persistent rebinding.

                                  Just because you want it, does not mean it is better. :-)

                                  > Thanks for pointing out max_use. Now instead of postfix rebinding
                                  > every 4-5 minutes to the LDAP servers, it is at least every 20
                                  > minutes between binds, significantly cutting out startTLS
                                  > negotiation overhead and improving performance.
                                  >
                                  > It is trivial to see what a significant difference it makes in
                                  > postfix behavior to go from the default of 100 to 5000:
                                  > <http://www.pastebin.ca/2330089>

                                  It is surely trivial to see what an insignificant difference this
                                  makes. Between all those connections thousands of lookups are
                                  made, the connection overhead is negligible.

                                  The difference between a TLS handshake and LDAP bind every 4-5
                                  minutes vs. every 20 minutes (or even infinity as with DKIM) is
                                  negligible. Almost all the payoff from re-use is in the first
                                  O(10) uses, after that it is diminishing returns all the way....

                                  It is similar with max_use, it is of couse reasonably safe to have
                                  it higher than 100, but the benefit is marginal at best.

                                  --
                                  Viktor.
                                Your message has been successfully submitted and would be delivered to recipients shortly.