Loading ...
Sorry, an error occurred while loading the content.

Re: postfix mysql lookup table has some kind of caching?

Expand Messages
  • Wietse Venema
    ... You did not provide enough context to figure out what Postfix caching feature you re referring to, so I will assume that this is the one-element reply
    Message 1 of 18 , Dec 31, 2012
    • 0 Attachment
      Marcin Owsiany:
      > Wietse Venema <wietse <at> porcupine.org> writes:
      > > It is not a major effort to give those cache entries a finite
      > > time-to-live but this needs a better justification that what appears
      > > to be an attempt to circumvent Yahoo rate limits.
      >
      > I have a need similar to the original poster. In my case the use case
      > is to gradually move outgoing traffic to a new egress IP.
      >
      > I have separate transports set up in master.cf per address like this:
      >
      > smtp unix - - - - - smtp
      > out1 unix - - - - 50 smtp \
      > -o smtp_bind_address=10.150.15.101
      > out2 unix - - - - 50 smtp \
      > -o smtp_bind_address=10.150.15.102
      > [...]
      >
      > and additionally the senders are partitioned across IPs
      > with sender_dependent_default_transport_maps
      >
      > Moving a big chunk of traffic in one go to a new IP that was never
      > sending any traffic before will just cause it to get greylisted
      > or blocked altogether.
      >
      > Given I have a low number of high-volume senders, I cannot find
      > an easy way to start sending (say) 1% of given sender's traffic
      > from a new IP. Any suggestions?

      You did not provide enough context to figure out what Postfix caching
      feature you're referring to, so I will assume that this is the
      one-element reply cache with transport map lookups.

      In that case the obvious solution is not to send 100% of all email
      to the same destination domain. That will flush the one-element
      cache frequently, and force fresh transport map lookups.

      What is the business use case to send 100% of all email to the same
      destination domain?

      Wietse
    • Marcin Owsiany
      ... Gmane nagged me to remove excessive citation, and I guess it got too terse. ... That s not what is happening. Let me explain a bit more: There s a dozen or
      Message 2 of 18 , Dec 31, 2012
      • 0 Attachment
        On Mon, Dec 31, 2012 at 02:05:38PM -0500, Wietse Venema wrote:
        > Marcin Owsiany:
        > > Wietse Venema <wietse <at> porcupine.org> writes:
        > > > It is not a major effort to give those cache entries a finite
        > > > time-to-live but this needs a better justification that what appears
        > > > to be an attempt to circumvent Yahoo rate limits.
        > >
        > > I have a need similar to the original poster. In my case the use case
        > > is to gradually move outgoing traffic to a new egress IP.
        > >
        > > I have separate transports set up in master.cf per address like this:
        > >
        > > smtp unix - - - - - smtp
        > > out1 unix - - - - 50 smtp \
        > > -o smtp_bind_address=10.150.15.101
        > > out2 unix - - - - 50 smtp \
        > > -o smtp_bind_address=10.150.15.102
        > > [...]
        > >
        > > and additionally the senders are partitioned across IPs
        > > with sender_dependent_default_transport_maps
        > >
        > > Moving a big chunk of traffic in one go to a new IP that was never
        > > sending any traffic before will just cause it to get greylisted
        > > or blocked altogether.
        > >
        > > Given I have a low number of high-volume senders, I cannot find
        > > an easy way to start sending (say) 1% of given sender's traffic
        > > from a new IP. Any suggestions?
        >
        > You did not provide enough context to figure out what Postfix caching
        > feature you're referring to, so I will assume that this is the
        > one-element reply cache with transport map lookups.

        Gmane nagged me to remove excessive citation, and I guess it got too
        terse.

        > In that case the obvious solution is not to send 100% of all email
        > to the same destination domain. That will flush the one-element
        > cache frequently, and force fresh transport map lookups.
        >
        > What is the business use case to send 100% of all email to the same
        > destination domain?

        That's not what is happening. Let me explain a bit more:

        There's a dozen or so envelope-senders (distribution lists), each
        periodically sends to a lot of recipients (few thousand to a few dozen
        thousand). The recipients are distributed over what I think is a typical
        country-wide sample of domains, which inevitably means the few big
        webmail services get a large portion.

        In the basic case, sender_dependent_default_transport_maps assigns each
        sender to one outgoing IP address, to try and limit colateral damage if
        one of them screws up and gets blacklisted.

        I imagine this might sound like a spam sending service, but it's not :-)

        Now the problem is when we want to move a sender to a different IP
        address. Because receivers don't like a previously-idle IP to suddenly
        start sending lots of mail, we need to start with a trickle, to build
        reputation.

        Ideally the smtp daemon would be able to bind to a few addresses and
        there would be an option to assign egress traffic split between them.
        However I did not find anything like that, so my next idea was to use a
        custom randomized "tcp" sender_dependent_default_transport_maps lookup
        table that would map "sender1" to, say, "out1" in 99% of cases and
        "out2" in the remaining 1%. However the builtin caching would defeat
        that.

        regards,
        --
        Marcin Owsiany <marcin@...> http://marcin.owsiany.pl/
        GnuPG: 2048R/02F946FC 35E9 1344 9F77 5F43 13DD 6423 DBF4 80C6 02F9 46FC

        "Every program in development at MIT expands until it can read mail."
        -- Unknown
      • Wietse Venema
        ... According to source code, the Postfix 2.5 and later resolver client will reuse the trivial-rewrite daemon result for up to 30 seconds when the request s
        Message 3 of 18 , Dec 31, 2012
        • 0 Attachment
          Marcin Owsiany:
          > Ideally the smtp daemon would be able to bind to a few addresses and
          > there would be an option to assign egress traffic split between them.

          According to source code, the Postfix 2.5 and later resolver client
          will reuse the trivial-rewrite daemon result for up to 30 seconds
          when the request's (address class, sender address, recipient address)
          are the same as the previous request.

          According to the following comment, a similar strategy was adopted
          for address rewriting and transport map wild-card results.

          20070414

          Cleanup: expire cached results from addres rewriting, address
          resolution, and from transport map lookups. Results expire
          after 30 seconds; short enough that it doesn't freak out
          people who run the same test repeatedly, and long enough
          that it doesn't upset other people with continuous streams
          of "*" transport map lookups. Files: global/rewrite_clnt.c,
          global/resolve_clnt.c, trivial-rewrite/transport.c.

          Now the question is are 30 seconds too much or too little.

          Wietse
        • Marcin Owsiany
          ... Much too much in my case. The client app sends messages in batches, and in 30 seconds much more than 1% of a batch will get processed, especially given
          Message 4 of 18 , Dec 31, 2012
          • 0 Attachment
            On Mon, Dec 31, 2012 at 05:00:52PM -0500, Wietse Venema wrote:
            > Marcin Owsiany:
            > > Ideally the smtp daemon would be able to bind to a few addresses and
            > > there would be an option to assign egress traffic split between them.
            >
            > According to source code, the Postfix 2.5 and later resolver client
            > will reuse the trivial-rewrite daemon result for up to 30 seconds
            > when the request's (address class, sender address, recipient address)
            > are the same as the previous request.
            >
            > According to the following comment, a similar strategy was adopted
            > for address rewriting and transport map wild-card results.
            >
            > 20070414
            >
            > Cleanup: expire cached results from addres rewriting, address
            > resolution, and from transport map lookups. Results expire
            > after 30 seconds; short enough that it doesn't freak out
            > people who run the same test repeatedly, and long enough
            > that it doesn't upset other people with continuous streams
            > of "*" transport map lookups. Files: global/rewrite_clnt.c,
            > global/resolve_clnt.c, trivial-rewrite/transport.c.
            >
            > Now the question is are 30 seconds too much or too little.

            Much too much in my case. The client app sends messages in batches, and
            in 30 seconds much more than 1% of a batch will get processed,
            especially given that the canary IP address transport will mostly see
            fast rejections.

            --
            Marcin Owsiany <marcin@...> http://marcin.owsiany.pl/
            GnuPG: 2048R/02F946FC 35E9 1344 9F77 5F43 13DD 6423 DBF4 80C6 02F9 46FC

            "Every program in development at MIT expands until it can read mail."
            -- Unknown
          • Wietse Venema
            ... In that case, a burst of 1s would still be undesirable, so one would have to be able to specify 0 (no caching) while the default would remain at 30s.
            Message 5 of 18 , Dec 31, 2012
            • 0 Attachment
              Wietse:
              > 20070414
              > Cleanup: expire cached results from addres rewriting, address
              > resolution, and from transport map lookups. Results expire
              > after 30 seconds; short enough that it doesn't freak out
              > people who run the same test repeatedly, and long enough
              > that it doesn't upset other people with continuous streams
              > of "*" transport map lookups. [...]
              >
              > Now the question is are 30 seconds too much or too little.

              Marcin Owsiany:
              > Much too much in my case. The client app sends messages in batches,
              > and in 30 seconds much more than 1% of a batch will get processed,
              > especially given that the canary IP address transport will mostly
              > see fast rejections.

              In that case, a burst of 1s would still be undesirable, so one would
              have to be able to specify 0 (no caching) while the default would
              remain at 30s.

              Wietse
            • Marcin Owsiany
              ... Right. I hope that disabling caching altogether would not affect performance too much. Perhaps it would be a per-map option, so that it would only affect
              Message 6 of 18 , Dec 31, 2012
              • 0 Attachment
                On Mon, Dec 31, 2012 at 05:45:57PM -0500, Wietse Venema wrote:
                > In that case, a burst of 1s would still be undesirable, so one would
                > have to be able to specify 0 (no caching) while the default would
                > remain at 30s.

                Right.

                I hope that disabling caching altogether would not affect performance
                too much. Perhaps it would be a per-map option, so that it would only
                affect the maps that it needs to affect?

                --
                Marcin Owsiany <marcin@...> http://marcin.owsiany.pl/
                GnuPG: 2048R/02F946FC 35E9 1344 9F77 5F43 13DD 6423 DBF4 80C6 02F9 46FC

                "Every program in development at MIT expands until it can read mail."
                -- Unknown
              • Wietse Venema
                ... Caching is mostly a client feature. A client doesn t care what tables a server is using. Wietse
                Message 7 of 18 , Dec 31, 2012
                • 0 Attachment
                  Marcin Owsiany:
                  > On Mon, Dec 31, 2012 at 05:45:57PM -0500, Wietse Venema wrote:
                  > > In that case, a burst of 1s would still be undesirable, so one would
                  > > have to be able to specify 0 (no caching) while the default would
                  > > remain at 30s.
                  >
                  > Right.
                  >
                  > I hope that disabling caching altogether would not affect performance
                  > too much. Perhaps it would be a per-map option, so that it would only
                  > affect the maps that it needs to affect?

                  Caching is mostly a client feature. A client doesn't care what
                  tables a server is using.

                  Wietse
                • Stan Hoeppner
                  ... Hi Marcin, Q1: Why is a Google Site Reliability Engineer and long time Debian maintainer, with two Masters degrees, working on an email blasting
                  Message 8 of 18 , Dec 31, 2012
                  • 0 Attachment
                    On 12/31/2012 3:06 PM, Marcin Owsiany wrote:

                    > I imagine this might sound like a spam sending service, but it's not :-)

                    Hi Marcin,

                    Q1: Why is a Google Site Reliability Engineer and long time Debian
                    maintainer, with two Masters degrees, working on an email blasting
                    infrastructure?

                    Q2: Why are you not leveraging the expertise of the Google messaging
                    folks for this project?

                    FYI, my first Linux was Potato and I've been Debian ever since. :)
                    Thank you for your contributions over the years.

                    --
                    Stan
                  • Marcin Owsiany
                    ... This is probably not completely on topic, but just to make sure this does not turn into some kind of conspiracy theory: a) it has nothing to do with my
                    Message 9 of 18 , Jan 1, 2013
                    • 0 Attachment
                      On Mon, Dec 31, 2012 at 08:46:03PM -0600, Stan Hoeppner wrote:
                      > On 12/31/2012 3:06 PM, Marcin Owsiany wrote:
                      >
                      > > I imagine this might sound like a spam sending service, but it's not :-)
                      >
                      > Hi Marcin,
                      >
                      > Q1: Why is a Google Site Reliability Engineer and long time Debian
                      > maintainer, with two Masters degrees, working on an email blasting
                      > infrastructure?
                      >
                      > Q2: Why are you not leveraging the expertise of the Google messaging
                      > folks for this project?

                      This is probably not completely on topic, but just to make sure this
                      does not turn into some kind of conspiracy theory:
                      a) it has nothing to do with my employer whatsoever,
                      b) I'm doing it because someone I know asked me for advice,
                      c) as I said, it's not UCE-related, sending a lot of email fast does
                      have legitimate purposes
                      d) asking a postfix question on postfix-users seems like a reasonable
                      thing to me

                      > FYI, my first Linux was Potato and I've been Debian ever since. :)
                      > Thank you for your contributions over the years.

                      Not that I contributed a whole lot, compared to some other DDs, but
                      you're welcome :-)

                      regards,
                      --
                      Marcin Owsiany <marcin@...> http://marcin.owsiany.pl/
                      GnuPG: 2048R/02F946FC 35E9 1344 9F77 5F43 13DD 6423 DBF4 80C6 02F9 46FC

                      "Every program in development at MIT expands until it can read mail."
                      -- Unknown
                    • Stan Hoeppner
                      ... I didn t intend to imply anything nefarious. By email blasting I simply meant high volume, short duration--not spamming. ... The reason I asked this is
                      Message 10 of 18 , Jan 1, 2013
                      • 0 Attachment
                        On 1/1/2013 7:33 AM, Marcin Owsiany wrote:
                        > On Mon, Dec 31, 2012 at 08:46:03PM -0600, Stan Hoeppner wrote:
                        >> On 12/31/2012 3:06 PM, Marcin Owsiany wrote:
                        >>
                        >>> I imagine this might sound like a spam sending service, but it's not :-)
                        >>
                        >> Hi Marcin,
                        >>
                        >> Q1: Why is a Google Site Reliability Engineer and long time Debian
                        >> maintainer, with two Masters degrees, working on an email blasting
                        >> infrastructure?
                        >>
                        >> Q2: Why are you not leveraging the expertise of the Google messaging
                        >> folks for this project?
                        >
                        > This is probably not completely on topic, but just to make sure this
                        > does not turn into some kind of conspiracy theory:

                        I didn't intend to imply anything nefarious. By "email blasting" I
                        simply meant high volume, short duration--not spamming.

                        > a) it has nothing to do with my employer whatsoever,
                        > b) I'm doing it because someone I know asked me for advice,
                        > c) as I said, it's not UCE-related, sending a lot of email fast does
                        > have legitimate purposes
                        > d) asking a postfix question on postfix-users seems like a reasonable
                        > thing to me

                        The reason I asked this is that Google is one of the "few big
                        webmail services get a large portion" that you mentioned. Your approach
                        here seems to be optimizing Postfix to "circumvent defenses". Normally
                        one would sign up with these services as a bulk mailer to avoid bulk
                        delivery problems. Maybe you are doing that as well but did not mention it.

                        >> FYI, my first Linux was Potato and I've been Debian ever since. :)
                        >> Thank you for your contributions over the years.
                        >
                        > Not that I contributed a whole lot, compared to some other DDs, but
                        > you're welcome :-)

                        Every small contribution helps. :)

                        --
                        Stan
                      Your message has been successfully submitted and would be delivered to recipients shortly.