Loading ...
Sorry, an error occurred while loading the content.
 

transport_maps "loops back to myself"

Expand Messages
  • MV
    I m trying to use multiple smtp IPs without having multiple postfix instances. So I tried to split smtp using transport_maps and that works ok for local mail
    Message 1 of 27 , Mar 26, 2014
      I'm trying to use multiple smtp IPs without having multiple postfix instances.

      So I tried to split smtp using transport_maps and that works ok for
      local mail destined to foreign destinations.
      But it breaks the delivery of foreign mail destined to local
      destinations with the error "mail for example.com loops back to
      myself".

      Where's the place place to "plug" the custom service
      (postfix-smtp-roundrobin.pl)?

      # main.cf
      transport_maps = tcp:127.0.0.1:9999

      # -------%<----------------------------------
      # master.cf

      smtp unix - - n - - smtp
      relay unix - - n - - smtp

      127.0.0.1:smtp inet n - n - - smtpd

      1.1.1.1:smtp inet n - n - - smtpd
      -o myhostname=mx1.example.com
      -o smtp_helo_name=mx1.example.com
      -o smtp_bind_address=1.1.1.1
      -o syslog_name=smtpd-mx1
      -o smtpd_tls_cert_file=/etc/pki/postfix/mx1.example.com.cert
      -o smtpd_tls_key_file=/etc/pki/postfix/mx1.example.com.key
      -o smtp_tls_cert_file=/etc/pki/postfix/mx1.example.com.cert
      -o smtp_tls_key_file=/etc/pki/postfix/mx1.example.com.key
      -o smtpd_sasl_local_domain=mx1.example.com

      2.2.2.2:smtp inet n - n - - smtpd
      -o myhostname=mx2.example.com
      -o smtp_helo_name=mx2.example.com
      -o smtp_bind_address=2.2.2.2
      -o syslog_name=smtpd-mx2
      -o smtpd_tls_cert_file=/etc/pki/postfix/mx2.example.com.cert
      -o smtpd_tls_key_file=/etc/pki/postfix/mx2.example.com.key
      -o smtp_tls_cert_file=/etc/pki/postfix/mx2.example.com.cert
      -o smtp_tls_key_file=/etc/pki/postfix/mx2.example.com.key
      -o smtpd_sasl_local_domain=mx2.example.com


      # Round-robin outgoing smtp
      # And the contents of /usr/local/sbin/postfix-smtp-roundrobin.pl
      # can be found at https://gist.github.com/mvsantos/9786697
      #
      127.0.0.1:9999 inet n n n - 0 spawn
      user=nobody argv=/usr/local/sbin/postfix-smtp-roundrobin.pl

      smtp1 unix - - n - - smtp
      -o myhostname=mx1.example.com
      -o smtp_helo_name=mx1.example.com
      -o smtp_bind_address=1.1.1.1
      -o syslog_name=smtp-mx1

      smtp2 unix - - n - - smtp
      -o myhostname=mx2.example.com
      -o smtp_helo_name=mx2.example.com
      -o smtp_bind_address=2.2.2.2
      -o syslog_name=smtp-mx2

      # ------->%----------------------------------


      --version--
      CentOS release 6.5
      Postfix 2.11.0

      --Mailbox locking methods--
      flock fcntl dotlock

      --Supported Lookup tables--
      btree cidr environ fail hash internal ldap memcache mysql nis pcre
      proxy regexp socketmap static tcp texthash unix


      Thanks
      Marcus
    • Noel Jones
      ... You can t set smtp options within the smtpd service. Use separate postfix instances. -- Noel Jones
      Message 2 of 27 , Mar 26, 2014
        On 3/26/2014 2:42 PM, MV wrote:
        > I'm trying to use multiple smtp IPs without having multiple postfix instances.
        >
        > So I tried to split smtp using transport_maps and that works ok for
        > local mail destined to foreign destinations.
        > But it breaks the delivery of foreign mail destined to local
        > destinations with the error "mail for example.com loops back to
        > myself".
        >
        > Where's the place place to "plug" the custom service
        > (postfix-smtp-roundrobin.pl)?
        >
        > # main.cf
        > transport_maps = tcp:127.0.0.1:9999
        >
        > # -------%<----------------------------------
        > # master.cf
        >
        > smtp unix - - n - - smtp
        > relay unix - - n - - smtp
        >
        > 127.0.0.1:smtp inet n - n - - smtpd
        >
        > 1.1.1.1:smtp inet n - n - - smtpd
        > -o myhostname=mx1.example.com
        > -o smtp_helo_name=mx1.example.com
        > -o smtp_bind_address=1.1.1.1

        You can't set "smtp" options within the "smtpd" service.

        Use separate postfix instances.



        -- Noel Jones
      • Wietse Venema
        ... Don t route LOCAL destinations to the smtp delivery agent. The SMTP client is for REMOTE deliveries. Wietse
        Message 3 of 27 , Mar 26, 2014
          MV:
          > I'm trying to use multiple smtp IPs without having multiple postfix instances.
          >
          > So I tried to split smtp using transport_maps and that works ok for
          > local mail destined to foreign destinations.
          > But it breaks the delivery of foreign mail destined to local
          > destinations with the error "mail for example.com loops back to
          > myself".

          Don't route LOCAL destinations to the "smtp" delivery agent.
          The SMTP client is for REMOTE deliveries.

          Wietse
        • MV
          ... smtp options won t affect the smtpd service (they are harmless), but fair point.. they shouldn t be there. But that doesn t solve the initial problem. ...
          Message 4 of 27 , Mar 26, 2014
            Noel Jones wrote:
            > MV wrote:
            >>
            >> 1.1.1.1:smtp inet n - n - - smtpd
            >> -o myhostname=mx1.example.com
            >> -o smtp_helo_name=mx1.example.com
            >> -o smtp_bind_address=1.1.1.1
            >
            > You can't set "smtp" options within the "smtpd" service.
            smtp options won't affect the smtpd service (they are harmless), but
            fair point.. they shouldn't be there.
            But that doesn't solve the initial problem.

            > Use separate postfix instances.
            That's not an option.
          • Noel Jones
            ... Don t use the smtp transport for local addresses. -- Noel Jones
            Message 5 of 27 , Mar 26, 2014
              On 3/26/2014 3:42 PM, MV wrote:
              > Noel Jones wrote:
              >> MV wrote:
              >>>
              >>> 1.1.1.1:smtp inet n - n - - smtpd
              >>> -o myhostname=mx1.example.com
              >>> -o smtp_helo_name=mx1.example.com
              >>> -o smtp_bind_address=1.1.1.1
              >>
              >> You can't set "smtp" options within the "smtpd" service.
              > smtp options won't affect the smtpd service (they are harmless), but
              > fair point.. they shouldn't be there.
              > But that doesn't solve the initial problem.


              Don't use the smtp transport for local addresses.




              -- Noel Jones
            • MV
              ... I guess this should ve been my initial question: how can I add a transport to REMOTE deliveries only? Marcus
              Message 6 of 27 , Mar 26, 2014
                >Wietse wrote:
                >> MV wrote:
                >> I'm trying to use multiple smtp IPs without having multiple postfix instances.
                >>
                >> So I tried to split smtp using transport_maps and that works ok for
                >> local mail destined to foreign destinations.
                >> But it breaks the delivery of foreign mail destined to local
                >> destinations with the error "mail for example.com loops back to
                >> myself".
                >
                > Don't route LOCAL destinations to the "smtp" delivery agent.
                > The SMTP client is for REMOTE deliveries.

                I guess this should've been my initial question:
                how can I add a transport to REMOTE deliveries only?

                Marcus
              • MV
                ... And how would I do that? I mean how can I have custom smtp for REMOTE deliveries and not breaking the LOCAL deliveries, Marcus
                Message 7 of 27 , Mar 26, 2014
                  > Noel Jones wrote:
                  > Don't use the smtp transport for local addresses.

                  And how would I do that? I mean how can I have custom smtp for REMOTE
                  deliveries and not breaking the LOCAL deliveries,

                  Marcus
                • Wietse Venema
                  ... By having the transport map reply not found for local destinations. Wietse
                  Message 8 of 27 , Mar 26, 2014
                    MV:
                    > > Wietse:
                    > >> I'm trying to use multiple smtp IPs without having multiple postfix instances.
                    > >>
                    > >> So I tried to split smtp using transport_maps and that works ok for
                    > >> local mail destined to foreign destinations.
                    > >> But it breaks the delivery of foreign mail destined to local
                    > >> destinations with the error "mail for example.com loops back to
                    > >> myself".
                    > >
                    > > Don't route LOCAL destinations to the "smtp" delivery agent.
                    > > The SMTP client is for REMOTE deliveries.
                    >
                    > I guess this should've been my initial question: how can I add a
                    > transport to REMOTE deliveries only?

                    By having the transport map reply "not found" for local destinations.

                    Wietse
                  • Noel Jones
                    ... I see. You re using a TCP based table for your transport_maps and it s incorrectly answering for local addresses too. Don t do that. Configure your TCP
                    Message 9 of 27 , Mar 26, 2014
                      On 3/26/2014 3:59 PM, MV wrote:
                      >> Wietse wrote:
                      >>> MV wrote:
                      >>> I'm trying to use multiple smtp IPs without having multiple postfix instances.
                      >>>
                      >>> So I tried to split smtp using transport_maps and that works ok for
                      >>> local mail destined to foreign destinations.
                      >>> But it breaks the delivery of foreign mail destined to local
                      >>> destinations with the error "mail for example.com loops back to
                      >>> myself".
                      >>
                      >> Don't route LOCAL destinations to the "smtp" delivery agent.
                      >> The SMTP client is for REMOTE deliveries.
                      >
                      > I guess this should've been my initial question:
                      > how can I add a transport to REMOTE deliveries only?
                      >
                      > Marcus
                      >


                      I see. You're using a TCP based table for your transport_maps and
                      it's incorrectly answering for local addresses too.

                      Don't do that. Configure your TCP table ignore local addresses. In
                      the context of a TCP table, the response should be something like
                      500 not found



                      -- Noel Jones
                    • Viktor Dukhovni
                      ... To attempt to evade receiving system client controls by spreading load over multiple local IPs use sender_dependent_default_transport_maps. If all you want
                      Message 10 of 27 , Mar 26, 2014
                        On Wed, Mar 26, 2014 at 03:58:16PM -0500, Noel Jones wrote:

                        > Don't use the smtp transport for local addresses.

                        To attempt to evade receiving system client controls by spreading
                        load over multiple local IPs use sender_dependent_default_transport_maps.

                        If all you want is round-robin of the IPs, without regard to the
                        actual sender, you can use the socketmap table support to implement
                        the round-robin policy.

                        http://www.postfix.org/postconf.5.html#sender_dependent_default_transport_maps
                        http://www.postfix.org/socketmap_table.5.html

                        --
                        Viktor.
                      • MV
                        ... So, am I right to assume that by replying to smtp with 500 not found it will fallback to local delivery? Also I ve noticed that different queries are
                        Message 11 of 27 , Mar 26, 2014
                          > Wietse wrote:
                          > By having the transport map reply "not found" for local destinations.


                          > Noel Jones wrote:
                          > ... Configure your TCP table ignore local addresses. In
                          > the context of a TCP table, the response should be something like
                          > 500 not found

                          So, am I right to assume that by replying to smtp with "500 not found"
                          it will fallback to local delivery?


                          Also I've noticed that different queries are sent to the transport
                          map. Looking at the logs I see that early on in the request the
                          transport map is queried as follow
                          get *
                          get *
                          get bar@...
                          get foo@...

                          and I can't figure out what those two initial "get *" queries are and
                          where they come from and why are they there. Any ideas?
                          Here's the complete log crop of an inbound email (from gmail to
                          mydomain) and the subsequent bounce mail back to gmail because of the
                          "look back to myself".
                          For clarity, the queries sent to the transport map are logged under
                          "smtp-roundrobin" and can be found after the string "(...)transport
                          service. Query: "



                          Mar 26 18:58:09 mailer2 smtpd-mx1/smtpd[22438]: connect from
                          mail-ob0-f178.google.com[209.85.214.178]:53853
                          Mar 26 18:58:09 mailer2 smtpd-mx1/smtpd[22438]: Anonymous TLS
                          connection established from
                          mail-ob0-f178.google.com[209.85.214.178]:53853: TLSv1 with cipher
                          ECDHE-RSA-RC4-SHA (128/128 bits)
                          Mar 26 18:58:09 mailer2 smtp-roundrobin[22444]: Using: 'smtp2:'
                          transport service. Query: get *
                          Mar 26 18:58:09 mailer2 smtp-roundrobin[22444]: Using: 'smtp1:'
                          transport service. Query: get *
                          Mar 26 18:58:09 mailer2 smtp-roundrobin[22444]: Using: 'smtp2:'
                          transport service. Query: get XXXXXXXX@...
                          Mar 26 18:58:10 mailer2 smtp-roundrobin[22444]: Using: 'smtp1:'
                          transport service. Query: get foo@...
                          Mar 26 18:58:10 mailer2 smtpd-mx1/smtpd[22438]: 3fvJW60bZtzyg2:
                          client=mail-ob0-f178.google.com[209.85.214.178]:53853
                          Mar 26 18:58:10 mailer2 postfix/cleanup[22445]: 3fvJW60bZtzyg2:
                          message-id=<CAAv8khP0+Ak91B=4=WvwE-vRPv6Bawz9DVgGUbk-mvLzPJpAYg@...>
                          Mar 26 18:58:10 mailer2 postfix/qmgr[22435]: 3fvJW60bZtzyg2:
                          from=<XXXXXXXX@...>, size=1728, nrcpt=1 (queue active)
                          Mar 26 18:58:10 mailer2 postfix/qmgr[22435]: amavisfeed:
                          default_destination_concurrency_positive_feedback feedback type 0
                          value at 5: 1
                          Mar 26 18:58:10 mailer2 postfix/qmgr[22435]: amavisfeed:
                          default_destination_concurrency_negative_feedback feedback type 0
                          value at 5: 1
                          Mar 26 18:58:10 mailer2 smtpd-mx1/smtpd[22438]: disconnect from
                          mail-ob0-f178.google.com[209.85.214.178]:53853
                          Mar 26 18:58:11 mailer2 postfix/smtpd[22449]: connect from
                          localhost.localdomain[127.0.0.1]:49877
                          Mar 26 18:58:11 mailer2 smtp-roundrobin[22444]: Using: 'smtp2:'
                          transport service. Query: get XXXXXXXX@...
                          Mar 26 18:58:11 mailer2 smtp-roundrobin[22444]: Using: 'smtp1:'
                          transport service. Query: get foo@...
                          Mar 26 18:58:11 mailer2 postfix/smtpd[22449]: 3fvJW7374jzyg3:
                          client=localhost.localdomain[127.0.0.1]:49877,
                          orig_queue_id=3fvJW60bZtzyg2,
                          orig_client=mail-ob0-f178.google.com[209.85.214.178]:53853
                          Mar 26 18:58:11 mailer2 postfix/cleanup[22445]: 3fvJW7374jzyg3:
                          message-id=<CAAv8khP0+Ak91B=4=WvwE-vRPv6Bawz9DVgGUbk-mvLzPJpAYg@...>
                          Mar 26 18:58:11 mailer2 postfix/smtpd[22449]: disconnect from
                          localhost.localdomain[127.0.0.1]:49877
                          Mar 26 18:58:11 mailer2 postfix/qmgr[22435]: 3fvJW7374jzyg3:
                          from=<XXXXXXXX@...>, size=2347, nrcpt=1 (queue active)
                          Mar 26 18:58:11 mailer2 smtp-roundrobin[22444]: Using: 'smtp2:'
                          transport service. Query: get foo@...
                          Mar 26 18:58:11 mailer2 postfix/qmgr[22435]: smtp2:
                          default_destination_concurrency_positive_feedback feedback type 0
                          value at 5: 1
                          Mar 26 18:58:11 mailer2 postfix/qmgr[22435]: smtp2:
                          default_destination_concurrency_negative_feedback feedback type 0
                          value at 5: 1
                          Mar 26 18:58:11 mailer2 amavis[13883]: (13883-13) Passed CLEAN
                          {RelayedInbound}, [209.85.214.178]:53853 [209.85.214.178]
                          <XXXXXXXX@...> -> <foo@...>, Queue-ID: 3fvJW60bZtzyg2,
                          Message-ID: <CAAv8khP0+Ak91B=4=WvwE-vRPv6Bawz9DVgGUbk-mvLzPJpAYg@...>,
                          mail_id: 1UI9pBamXvhy, Hits: 0.012, size: 1728, queued_as:
                          3fvJW7374jzyg3, dkim_sd=20120113:gmail.com, 1058 ms
                          Mar 26 18:58:11 mailer2 postfix/lmtp[22446]: 3fvJW60bZtzyg2:
                          to=<foo@...>, relay=127.0.0.1[127.0.0.1]:10111, delay=1.6,
                          delays=0.52/0/0/1.1, dsn=2.0.0, status=sent (250 2.0.0 from
                          MTA(smtp:[127.0.0.1]:10211): 250 2.0.0 Ok: queued as 3fvJW7374jzyg3)
                          Mar 26 18:58:11 mailer2 postfix/qmgr[22435]: qmgr_queue_unthrottle: feedback 1
                          Mar 26 18:58:11 mailer2 postfix/qmgr[22435]: qmgr_queue_unthrottle:
                          queue [127.0.0.1]:10111: limit 20 window 6 success 0 failure 0
                          fail_cohorts 0
                          Mar 26 18:58:11 mailer2 postfix/qmgr[22435]: 3fvJW60bZtzyg2: removed
                          Mar 26 18:58:11 mailer2 smtp-mx2/smtp[22450]: 3fvJW7374jzyg3:
                          to=<foo@...>, relay=none, delay=0.14, delays=0.1/0.01/0.03/0,
                          dsn=5.4.6, status=bounced (mail for MYDOMAIN.com loops back to myself)
                          Mar 26 18:58:11 mailer2 postfix/qmgr[22435]: qmgr_queue_unthrottle: feedback 1
                          Mar 26 18:58:11 mailer2 postfix/qmgr[22435]: qmgr_queue_unthrottle:
                          queue MYDOMAIN.com: limit 20 window 6 success 0 failure 0 fail_cohorts
                          0
                          Mar 26 18:58:11 mailer2 postfix/cleanup[22445]: 3fvJW74mxHzyg5:
                          message-id=<3fvJW74mxHzyg5@...>
                          Mar 26 18:58:11 mailer2 postfix/bounce[22451]: 3fvJW7374jzyg3: sender
                          non-delivery notification: 3fvJW74mxHzyg5
                          Mar 26 18:58:11 mailer2 postfix/qmgr[22435]: 3fvJW74mxHzyg5: from=<>,
                          size=4382, nrcpt=1 (queue active)
                          Mar 26 18:58:11 mailer2 smtp-roundrobin[22444]: Using: 'smtp1:'
                          transport service. Query: get XXXXXXXX@...
                          Mar 26 18:58:11 mailer2 postfix/qmgr[22435]: smtp1:
                          default_destination_concurrency_positive_feedback feedback type 0
                          value at 5: 1
                          Mar 26 18:58:11 mailer2 postfix/qmgr[22435]: smtp1:
                          default_destination_concurrency_negative_feedback feedback type 0
                          value at 5: 1
                          Mar 26 18:58:11 mailer2 postfix/qmgr[22435]: 3fvJW7374jzyg3: removed
                          Mar 26 18:58:12 mailer2 smtp-mx1/smtp[22452]: Trusted TLS connection
                          established to gmail-smtp-in.l.google.com[173.194.78.26]:25: TLSv1.2
                          with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)
                          Mar 26 18:58:12 mailer2 smtp-mx1/smtp[22452]: 3fvJW74mxHzyg5:
                          to=<XXXXXXXX@...>,
                          relay=gmail-smtp-in.l.google.com[173.194.78.26]:25, delay=1.2,
                          delays=0.07/0.01/0.57/0.57, dsn=2.0.0, status=sent (250 2.0.0 OK
                          1395860292 ek7si2054896wib.117 - gsmtp)
                          Mar 26 18:58:12 mailer2 postfix/qmgr[22435]: qmgr_queue_unthrottle: feedback 1
                          Mar 26 18:58:12 mailer2 postfix/qmgr[22435]: qmgr_queue_unthrottle:
                          queue gmail.com: limit 20 window 6 success 0 failure 0 fail_cohorts 0
                          Mar 26 18:58:12 mailer2 postfix/qmgr[22435]: 3fvJW74mxHzyg5: removed
                        • Wietse Venema
                          ... These secrets are revealed in http://www.postfix.org/transport.5.html In other words, RTFM. Wietse
                          Message 12 of 27 , Mar 26, 2014
                            MV:
                            > Also I've noticed that different queries are sent to the transport
                            > map. Looking at the logs I see that early on in the request the
                            > transport map is queried as follow
                            > get *
                            > get *
                            > get bar@...
                            > get foo@...
                            >
                            > and I can't figure out what those two initial "get *" queries are and
                            > where they come from and why are they there. Any ideas?

                            These secrets are revealed in http://www.postfix.org/transport.5.html
                            In other words, RTFM.

                            Wietse
                          • MV
                            ... I d love to say I haven t read the manual and thank you for pointing it out to me, but my OCD is too damn high, so I always read manuals. Unfortunately
                            Message 13 of 27 , Mar 27, 2014
                              > Wietse Venema wrote:
                              > In other words, RTFM.

                              I'd love to say I haven't read the manual and thank you for pointing
                              it out to me, but my OCD is too damn high, so I always read manuals.
                              Unfortunately this time I can't quite get my head around it to figure
                              it out on my own how to correctly and sanely configure postfix to use
                              a different smtp delivery transports for mail destined to foreign
                              destinations.

                              I believe the below is irrelevant to issue I'm trying to solve, but
                              I'd to understand postfix better instead of just copy+paste or get
                              ready-made crafted answer.
                              The documentation says:

                              ...
                              With lookups from indexed files such as DB or DBM, or from
                              networked tables such as NIS, LDAP or SQL, patterns are tried *in the
                              order as listed below*:
                              user+extension@domain transport:nexthop
                              user@domain transport:nexthop
                              domain transport:nexthop
                              .domain transport:nexthop
                              * transport:nexthop
                              ...
                              TCP-BASED TABLES
                              Each lookup operation uses the entire recipient address once.
                              Thus, some.domain.hierarchy is not looked up via its parent domains,
                              nor is user+foo@domain looked up as user@domain.
                              ...

                              As far as I can tell, in my case since I'm using sing tcp-base tables
                              some look ups are not performed and that's fine. But there are no
                              mentions to the change in the order which patterns are checked.
                              So am I wrong to expect to see the logs showing "get bar <at>
                              foreign.tld" and "get foo <at> mydomain.tld" before "get *"?
                              As oppose to what I see now:
                              get *
                              get *
                              get bar <at> foreign.tld
                              get foo <at> mydomain.tld



                              Regards, Marcus
                            • Wietse Venema
                              ... The * is the documented wild-card pattern. The query result is not expected to change, therefore it is cached once during the initialization of a
                              Message 14 of 27 , Mar 27, 2014
                                MV:
                                > As far as I can tell, in my case since I'm using sing tcp-base tables
                                > some look ups are not performed and that's fine. But there are no
                                > mentions to the change in the order which patterns are checked.
                                > So am I wrong to expect to see the logs showing "get bar <at>
                                > foreign.tld" and "get foo <at> mydomain.tld" before "get *"?
                                > As oppose to what I see now:
                                > get *
                                > get *

                                The "*" is the documented wild-card pattern. The query result is
                                not expected to change, therefore it is cached once during the
                                initialization of a trivial-rewrite process instance.

                                Wietse
                              • Viktor Dukhovni
                                ... The documentation describes semantic precedence order in which results are used. With cached results, order of use is not necessarily the same as order of
                                Message 15 of 27 , Mar 27, 2014
                                  On Thu, Mar 27, 2014 at 12:04:34PM +0000, MV wrote:

                                  > As far as I can tell, in my case since I'm using sing tcp-base tables
                                  > some look ups are not performed and that's fine. But there are no
                                  > mentions to the change in the order which patterns are checked.
                                  > So am I wrong to expect to see the logs showing "get bar <at>
                                  > foreign.tld" and "get foo <at> mydomain.tld" before "get *"?
                                  > As oppose to what I see now:
                                  > get *
                                  > get *
                                  > get bar <at> foreign.tld
                                  > get foo <at> mydomain.tld

                                  The documentation describes semantic precedence order in which
                                  results are used. With cached results, order of use is not
                                  necessarily the same as order of lookup. When you're looking at
                                  low level activity logs you encounter implementation details
                                  which are not fixed and hence not documented.

                                  --
                                  Viktor.
                                • MV
                                  ... Thanks for explaining that. Got a much better picture now (and the of the diagrams at http://www.postfix.org/OVERVIEW.html also helped) Going back to my
                                  Message 16 of 27 , Mar 27, 2014
                                    >Wietse:
                                    >> MV:
                                    >> As far as I can tell, in my case since I'm using sing tcp-base tables
                                    >> some look ups are not performed and that's fine. But there are no
                                    >> mentions to the change in the order which patterns are checked.
                                    >> So am I wrong to expect to see the logs showing "get bar <at>
                                    >> foreign.tld" and "get foo <at> mydomain.tld" before "get *"?
                                    >> As oppose to what I see now:
                                    >> get *
                                    >> get *
                                    >
                                    > The "*" is the documented wild-card pattern. The query result is
                                    > not expected to change, therefore it is cached once during the
                                    > initialization of a trivial-rewrite process instance.

                                    Thanks for explaining that.
                                    Got a much better picture now (and the of the diagrams at
                                    http://www.postfix.org/OVERVIEW.html also helped)


                                    Going back to my original question...

                                    Lets say I have "transport_maps = hash:/etc/postfix/transport", and
                                    the contents of /etc/postfix/transport are as follows:
                                    # Begin /etc/postfix/transport
                                    mydomain.ltd :
                                    .mydomain.ltd :
                                    * smtpX
                                    # End /etc/postfix/transport

                                    And in the master.cf file I have
                                    # -------%<------------------------------------
                                    smtpX unix - - n - - smtp
                                    -o myhostname=X.mydomain.tld
                                    -o smtp_helo_name=mx1.mydomain.tld
                                    -o smtp_bind_address=1.1.1.1
                                    -o syslog_name=smtpX

                                    smtpY unix - - n - - smtp
                                    -o myhostname=Y.mydomain.tld
                                    -o smtp_helo_name=mx2.mydomain.tld
                                    -o smtp_bind_address=2.2.2.2
                                    -o syslog_name=smtpY
                                    # ------->%------------------------------------

                                    The config above should work as expected.
                                    Now going a step further, how can I split the "*" (all non-local)
                                    between smtpX and smtpY (without running multiple postfix instances) ?
                                    Something like ...

                                    mydomain.ltd :
                                    .mydomain.ltd :
                                    * smtpX
                                    * smtpY
                                  • Viktor Dukhovni
                                    ... No. To get round-robit results, you need a dynamic transport reply. Thus socketmap or similar, with the replies provided by a program, not a fixed
                                    Message 17 of 27 , Mar 27, 2014
                                      On Thu, Mar 27, 2014 at 04:50:25PM +0000, MV wrote:

                                      > Now going a step further, how can I split the "*" (all non-local)
                                      > between smtpX and smtpY (without running multiple postfix instances) ?
                                      > Something like ...
                                      >
                                      > mydomain.ltd :
                                      > .mydomain.ltd :
                                      > * smtpX
                                      > * smtpY

                                      No. To get round-robit results, you need a dynamic transport reply.
                                      Thus socketmap or similar, with the replies provided by a program, not
                                      a fixed key->value table.

                                      Furthermore, because "*" is cached, you really don't want to use
                                      "*" at all for dynamic transport resolution.

                                      I answered your question upthread, use:

                                      sender_dependent_default_transport_maps

                                      For some reason you're still looking elsewhere...

                                      --
                                      Viktor.
                                    • MV
                                      ... Thanks for your input RE the caching of the special pattern * results. ... I can t use sender_dependent_default_transport_maps because I don t want to
                                      Message 18 of 27 , Mar 27, 2014
                                        >Viktor Dukhovni wrote:
                                        > Furthermore, because "*" is cached, you really don't want to use
                                        > "*" at all for dynamic transport resolution.
                                        Thanks for your input RE the caching of the special pattern "*" results.

                                        > I answered your question upthread, use:
                                        > sender_dependent_default_transport_maps
                                        > For some reason you're still looking elsewhere...
                                        I can't use sender_dependent_default_transport_maps because I don't
                                        want to use sender-based static "routes". I'm looking for a "random"
                                        or round-robin-ish split of smtp that provides consistent "helo ..
                                        hostname .. ip .. reverse-dns-lookup"


                                        Anyways, I've got the round-robin working now, using transport_maps
                                        and this script https://gist.github.com/mvsantos/9813415
                                        But I'm finding the solution very ugly. There must be a more elegant
                                        way to do dynamic smtp deliveries...

                                        $ postconf mail_version
                                        mail_version = 2.11.0

                                        # main.cf
                                        transport_maps = tcp:127.0.0.1:9999

                                        # master.cf
                                        127.0.0.1:9999 inet n n n - 0 spawn
                                        user=nobody argv=/usr/local/sbin/postfix-smtp-roundrobin.pl

                                        smtp1 unix - - n - - smtp
                                        -o myhostname=X.mydomain.tld
                                        -o smtp_helo_name=X.mydomain.tld
                                        -o smtp_bind_address=1.1.1.1
                                        -o syslog_name=smtp-1

                                        smtp2 unix - - n - - smtp
                                        -o myhostname=Y.mydomain.tld
                                        -o smtp_helo_name=Y.mydomain.tld
                                        -o smtp_bind_address=2.2.2.2
                                        -o syslog_name=smtp-2
                                        # End of master.cf


                                        And for every mail sent or received I have 8 extra lines in my logs
                                        (below, not in the correct order) and the time of processing has
                                        increased a bit - not because of the extra logging, but because of the
                                        transport_maps.

                                        Let's see how it performs under load tomorrow, when I make it live...

                                        smtp-roundrobin[1234]: Using: 'smtp2:' transport service. Query: get *
                                        smtp-roundrobin[1234]: Using: 'smtp1:' transport service. Query: get *
                                        smtp-roundrobin[1234]: Using: 'smtp2:' transport service. Query: get
                                        bar@...
                                        smtp-roundrobin[1234]: Using: 'smtp1:' transport service. Query: get
                                        foo@...
                                        postfix/qmgr[4567]: smtp1:
                                        default_destination_concurrency_positive_feedback feedback type 0
                                        value at 5: 1
                                        postfix/qmgr[4567]: smtp1:
                                        default_destination_concurrency_negative_feedback feedback type 0
                                        value at 5: 1
                                        postfix/qmgr[4567]: smtp2:
                                        default_destination_concurrency_positive_feedback feedback type 0
                                        value at 5: 1
                                        postfix/qmgr[4567]: smtp2:
                                        default_destination_concurrency_negative_feedback feedback type 0
                                        value at 5: 1
                                      • Wietse Venema
                                        ... What is the legitimate use case for this kind of policy evasion? Wietse
                                        Message 19 of 27 , Mar 27, 2014
                                          MV:
                                          > I don't want to use sender-based static "routes". I'm looking for
                                          > a "random" or round-robin-ish split of smtp that provides consistent
                                          > "helo .. hostname .. ip .. reverse-dns-lookup"

                                          What is the legitimate use case for this kind of policy evasion?

                                          Wietse
                                        • Viktor Dukhovni
                                          ... Of course you can. You re just not listening carefully. Your sender dependent maps would actually largely ignore the sender, and just provide a
                                          Message 20 of 27 , Mar 27, 2014
                                            On Thu, Mar 27, 2014 at 06:23:39PM +0000, MV wrote:

                                            > >Viktor Dukhovni wrote:
                                            > > Furthermore, because "*" is cached, you really don't want to use
                                            > > "*" at all for dynamic transport resolution.
                                            > Thanks for your input RE the caching of the special pattern "*" results.
                                            >
                                            > > I answered your question upthread, use:
                                            > > sender_dependent_default_transport_maps
                                            > > For some reason you're still looking elsewhere...
                                            >
                                            > I can't use sender_dependent_default_transport_maps because I don't
                                            > want to use sender-based static "routes". I'm looking for a "random"
                                            > or round-robin-ish split of smtp that provides consistent "helo ..
                                            > hostname .. ip .. reverse-dns-lookup"

                                            Of course you can. You're just not listening carefully. Your
                                            sender dependent maps would actually largely ignore the sender,
                                            and just provide a round-robin response. The important part is
                                            that this mechanism returns a "default_transport" which never
                                            overrides local or other more specific transport information.

                                            This mechanism (being a lookup on the sender, which is a message,
                                            not recipient property) can help avoid unnecessarily splitting of
                                            the envelope for multi-recipient mail. You should try to send all
                                            the recipients of a message to the same default transport, therefore
                                            you should have an modestly sized LRU cache within the round-robin
                                            process that returns a cached answer for a cached sender, but
                                            returns and caches a random answer for a new sender.

                                            If many messages come predominantly from a single sender and (since
                                            your traffic is likely single-recipient bulk mail, solicited or
                                            otherwise) multi-recipient messages are rare, you may not need the
                                            cache, since envelope splitting will not be useful, but the cache
                                            might defeat the load-balancing you want.

                                            > Anyways, I've got the round-robin working now, using transport_maps
                                            > and this script https://gist.github.com/mvsantos/9813415
                                            > But I'm finding the solution very ugly. There must be a more elegant
                                            > way to do dynamic smtp deliveries...

                                            Use socketmap (or the obsolete tcp table) with:

                                            sender_dependent_default_transport_maps

                                            Don't bother logging the queries except briefly if you're unsure
                                            it is working.

                                            --
                                            Viktor.
                                          • MV
                                            ... Just to be clear, I m not a spammer, if anything, I couldn t be more far from it. I m in the business of (strictly subscription-only) monitoring stuff . I
                                            Message 21 of 27 , Mar 27, 2014
                                              > Wietse:
                                              > What is the legitimate use case for this kind of policy evasion?

                                              Just to be clear, I'm not a spammer, if anything, I couldn't be more
                                              far from it.
                                              I'm in the business of (strictly subscription-only) "monitoring
                                              stuff". I mean, as soon as an event happens the subscribers who signed
                                              up to that kind of event must be alerted immediately (99.9999% of the
                                              times subscribers get a single email per day). But because we have
                                              grown considerably recently, we needed extra outbound IPs because we
                                              started to hit some ESP's limit of mail sent per hour from a single
                                              source IP.

                                              So I added an extra IP to the existing stack and used iptables to
                                              split the outbound connections using the nat table.
                                              But that created a problem because postfix would picks up the hostname
                                              used in the helo (and other stuff???) from IP 1 and iptables would
                                              route the connection through IP 2, so the client would see the message
                                              headers like this one

                                              Received: from AAAA.mydomain.tld (BBBB.mydomain.tld. [1.1.1.1])
                                              by mta.foreign.tld with ESMTPS id abcdf....
                                              for <foo@...>
                                              (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
                                              Thu, 27 Mar 2014 ....

                                              See how postfix helo was "AAAA.mydomain.tld" but the message actually
                                              came from "BBBB.mydomain.tld".

                                              And that simple header inconsistency bothers me (OCD)

                                              So instead of splitting smtp using iptables I now use that script I
                                              posted upthread, and the headers now look like this:

                                              Received: from AAAA.mydomain.tld (AAAA.mydomain.tld. [1.1.1.1])
                                              by mta.foreign.tld with ESMTPS id abcdf....
                                              for <foo@...>
                                              (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
                                              Thu, 27 Mar 2014 ....


                                              Received: from BBBB.mydomain.tld (BBBB.mydomain.tld. [2.2.2.2])
                                              by mta.foreign.tld with ESMTPS id abcdf....
                                              for <foo@...>
                                              (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
                                              Thu, 27 Mar 2014 ....


                                              Is there a more elegant way achieve this?
                                            • MV
                                              ... I m failing to grasp the concept and can t find any working examples online... Finding this thread http://thread.gmane.org/gmane.mail.postfix.user/203958
                                              Message 22 of 27 , Mar 27, 2014
                                                Viktor Dukhovni:
                                                > Of course you can. You're just not listening carefully. Your
                                                I'm failing to grasp the concept and can't find any working examples online...
                                                Finding this thread
                                                http://thread.gmane.org/gmane.mail.postfix.user/203958 has helped a
                                                bit..

                                                > sender dependent maps would actually largely ignore the sender,
                                                But how to I define the sender (map), if that's not asking too much,
                                                could you please provide me with an example file?

                                                > and just provide a round-robin response. The important part is
                                                > that this mechanism returns a "default_transport" which never
                                                > overrides local or other more specific transport information.
                                                > ...
                                                > ... can help avoid unnecessarily splitting of
                                                > the envelope for multi-recipient mail. You should try to send all
                                                And therefore would cut the overhead I currently have, right?

                                                > the cache might defeat the load-balancing you want.
                                                yep!

                                                > Use socketmap (or the obsolete tcp table) with:
                                                > sender_dependent_default_transport_maps
                                                (mental note: this surely is the right way to do it, not the way I've done it.)

                                                > Don't bother logging the queries except briefly if you're unsure
                                                > it is working.
                                                I'm just logging while debugging..


                                                Marcus
                                              • Viktor Dukhovni
                                                ... It is a program! Not a fixed mapping. It receives a sender address, and replies (with a possibly cached per-sender) answer which is computed on a
                                                Message 23 of 27 , Mar 27, 2014
                                                  On Thu, Mar 27, 2014 at 07:44:46PM +0000, MV wrote:

                                                  > > sender dependent maps would actually largely ignore the sender,
                                                  >
                                                  > But how to I define the sender (map), if that's not asking too much,
                                                  > could you please provide me with an example file?

                                                  It is a program! Not a fixed mapping. It receives a sender address,
                                                  and replies (with a possibly cached per-sender) answer which is
                                                  computed on a mostly-round-robin basis.

                                                  > > and just provide a round-robin response. The important part is
                                                  > > that this mechanism returns a "default_transport" which never
                                                  > > overrides local or other more specific transport information.
                                                  > > ...
                                                  > > ... can help avoid unnecessarily splitting of
                                                  > > the envelope for multi-recipient mail. You should try to send all
                                                  >
                                                  > And therefore would cut the overhead I currently have, right?

                                                  Don't know what overhead you have in mind. Transport lookups happen
                                                  in many places in Postfix. Your transport switch needs to be fast,
                                                  (no verbose logging, ...) the rest is irrelevant.

                                                  > > the cache might defeat the load-balancing you want.
                                                  >
                                                  > yep!

                                                  Or it might not. Do you send a flood of separate messages each
                                                  from the same sender? If not the cache will help with multi-recipient
                                                  mail. If all mail is single-recipient, you don't need a cache.

                                                  --
                                                  Viktor.
                                                • MV
                                                  ... If that s not asking too much, could you please provide me with a practical example or point me to where I could to find one? ... The overhead caused by my
                                                  Message 24 of 27 , Mar 27, 2014
                                                    Viktor Dukhovni wrote:
                                                    > It is a program! Not a fixed mapping. It receives a sender address,
                                                    > and replies (with a possibly cached per-sender) answer which is
                                                    > computed on a mostly-round-robin basis.
                                                    If that's not asking too much, could you please provide me with a
                                                    practical example or point me to where I could to find one?

                                                    > Don't know what overhead you have in mind.
                                                    The overhead caused by my crappy perl script and its IO ops... but
                                                    this is irrelevant right now.

                                                    > Do you send a flood of separate messages each from the same sender?
                                                    By sender, do you mean email address or just $mydomain ?
                                                    If you mean sender=foo@..., then yes, most of the mail sent
                                                    comes from a single email account and is sent to a great number of
                                                    indivual recipients.


                                                    Marcus
                                                  • Wietse Venema
                                                    ... In that case, arrange for whitelisting like ever legitimate sender does. Wietse
                                                    Message 25 of 27 , Mar 27, 2014
                                                      MV:
                                                      > > Wietse:
                                                      > > What is the legitimate use case for this kind of policy evasion?
                                                      >
                                                      > Just to be clear, I'm not a spammer, if anything, I couldn't be more
                                                      > far from it.
                                                      > I'm in the business of (strictly subscription-only) "monitoring
                                                      > stuff". I mean, as soon as an event happens the subscribers who signed
                                                      > up to that kind of event must be alerted immediately (99.9999% of the
                                                      > times subscribers get a single email per day). But because we have
                                                      > grown considerably recently, we needed extra outbound IPs because we
                                                      > started to hit some ESP's limit of mail sent per hour from a single
                                                      > source IP.

                                                      In that case, arrange for whitelisting like ever legitimate sender does.

                                                      Wietse
                                                    • Viktor Dukhovni
                                                      ... A small socketmap or tcp table service that ignores the lookup key and returns the next element of a circular list of values with each lookup is surely
                                                      Message 26 of 27 , Mar 27, 2014
                                                        On Thu, Mar 27, 2014 at 08:48:17PM +0000, MV wrote:

                                                        > > It is a program! Not a fixed mapping. It receives a sender address,
                                                        > > and replies (with a possibly cached per-sender) answer which is
                                                        > > computed on a mostly-round-robin basis.
                                                        >
                                                        > If that's not asking too much, could you please provide me with a
                                                        > practical example or point me to where I could to find one?

                                                        A small socketmap or tcp table service that ignores the lookup key
                                                        and returns the next element of a circular list of values with each
                                                        lookup is surely just a few lines of custom code. As for how to
                                                        write a socketmap or tcp table driver in general, your Google skills
                                                        are likely not too dissimilar from mine.

                                                        In perl the core lookup function boils down to:

                                                        @choices = ( ... );
                                                        my $count = @choices;
                                                        my $next = 0;
                                                        sub lookup { return $choices[$next++ % $count]; }

                                                        which ignores the lookup key and returns the next choice. You have
                                                        many choices of languages to choose from, and ways to run this as
                                                        a network or unix-domain socket service.

                                                        > > Don't know what overhead you have in mind.
                                                        >
                                                        > The overhead caused by my crappy perl script and its IO ops... but
                                                        > this is irrelevant right now.

                                                        A tcp or unix-domain socket request-reply round-trip should be
                                                        quite fast.

                                                        > > Do you send a flood of separate messages each from the same sender?
                                                        >
                                                        > By sender, do you mean email address or just $mydomain ?

                                                        Full envelope sender address.

                                                        > If you mean sender=foo@..., then yes, most of the mail sent
                                                        > comes from a single email account and is sent to a great number of
                                                        > individual recipients.

                                                        If you have the same sender for many concurrent messages (not just
                                                        multiple recipients of a single message), then you would not use
                                                        the proposed cache, making the code simpler.

                                                        --
                                                        Viktor.
                                                      • MV
                                                        ... I do that for Gmail, Yahoo, Microsoft, AOL .. and it works, so much so that we have never been graylisted by any of those folks despite the tens of
                                                        Message 27 of 27 , Mar 27, 2014
                                                          Wietse wrote:
                                                          > In that case, arrange for whitelisting like ever legitimate sender does.

                                                          I do that for Gmail, Yahoo, Microsoft, AOL .. and it works, so much
                                                          so that we have never been graylisted by any of those folks despite
                                                          the tens of thousands emails we send daily.
                                                          But I can't afford do that for the thousands of other ESPs that my clients use.
                                                          We actually stop emailing users after the second failure and let them
                                                          know about it upon login to the web dashboard.
                                                          Users normally contact their ESPs on their own and once sorted they
                                                          normally ask us to restore their alerts. And sometimes they just
                                                          create a new email account in a big free ESP .
                                                        Your message has been successfully submitted and would be delivered to recipients shortly.