Loading ...
Sorry, an error occurred while loading the content.

postfix mysql lookup table has some kind of caching?

Expand Messages
  • Bányász Botond
    Hy, I want to setup a system who warmes the sending ip`s, so i made a mysql  transport map where per domain i can add how much % to relay from main ip pool to
    Message 1 of 18 , Feb 24, 2012
    • 0 Attachment
      Hy,

      I want to setup a system who warmes the sending ip`s, so i made a mysql  transport map where per domain i can add how much % to relay from main ip pool to the warmup ip pool. The problem is that if manually I change in the database for example yahoo.com domain  from 0 percent to 100 percent the postfix uses the old settings for around 20 seconds, so it means that i cant control  the system, if i specify let say 20%.

      Thank U.


    • Noel Jones
      ... The transport lookup has a 1-element cache. This is not configurable. -- Noel Jones
      Message 2 of 18 , Feb 24, 2012
      • 0 Attachment
        On 2/24/2012 2:04 AM, Bányász Botond wrote:
        > Hy,
        >
        > I want to setup a system who warmes the sending ip`s, so i made a
        > mysql transport map where per domain i can add how much % to relay
        > from main ip pool to the warmup ip pool. The problem is that if
        > manually I change in the database for example yahoo.com
        > <http://yahoo.com/> domain from 0 percent to 100 percent the
        > postfix uses the old settings for around 20 seconds, so it means
        > that i cant control the system, if i specify let say 20%.
        >
        > Thank U.
        >
        >

        The transport lookup has a 1-element cache. This is not configurable.


        -- Noel Jones
      • Bányász Botond
        What means this 1-element cache? it caches the last lookup?   Banyasz Botond Phone:0740138717 Ymes:banyasz_b ________________________________ From: Noel Jones
        Message 3 of 18 , Feb 24, 2012
        • 0 Attachment
          What means this 1-element cache? it caches the last lookup?
           
          Banyasz Botond
          Phone:0740138717
          Ymes:banyasz_b


          From: Noel Jones <njones@...>
          To: Bányász Botond <banyasz_b@...>; "postfix-users@..." <postfix-users@...>
          Sent: Friday, February 24, 2012 3:41 PM
          Subject: Re: postfix mysql lookup table has some kind of caching?

          On 2/24/2012 2:04 AM, Bányász Botond wrote:
          > Hy,
          >
          > I want to setup a system who warmes the sending ip`s, so i made a
          > mysql  transport map where per domain i can add how much % to relay
          > from main ip pool to the warmup ip pool. The problem is that if
          > manually I change in the database for example yahoo.com
          > <http://yahoo.com/> domain  from 0 percent to 100 percent the
          > postfix uses the old settings for around 20 seconds, so it means
          > that i cant control  the system, if i specify let say 20%.
          >
          > Thank U.
          >
          >

          The transport lookup has a 1-element cache.  This is not configurable.


            -- Noel Jones


        • Noel Jones
          ... Right. The cache is not specific to mysql, but is a feature of the trivial-rewrite transport lookup. This is only likely to be noticed when you use
          Message 4 of 18 , Feb 24, 2012
          • 0 Attachment
            On 2/24/2012 8:04 AM, Bányász Botond wrote:
            > What means this 1-element cache? it caches the last lookup?
            >

            Right. The cache is not specific to mysql, but is a feature of the
            trivial-rewrite transport lookup.

            This is only likely to be noticed when you use mysql-based
            transport_maps and a high percentage of the queue is for a single
            destination. This is not configurable.

            The workaround is to use a hash: or cdb: table, which triggers a
            restart of trivial-rewrite upon changes -- but note that frequent
            restarts of trivial-rewrite may be bad for performance.




            -- Noel Jones
          • Bányász Botond
            I dont think that the problem cames from transport map 1-element caching, because the destination is variable, we have millions of email addresses. I made some
            Message 5 of 18 , Feb 27, 2012
            • 0 Attachment
              I dont think that the problem cames from transport map 1-element caching, because the destination is variable, we have millions of email addresses.
              I made some tests with mysql general logging enabled and i found that when mysql lookup table checks a transport for a destination it does a lookup for email@... domain.tld .tld but the lookup for * is done just once, but for the rest it does all the time. This is how it was designed to be the * lookup? 

              > What means this 1-element cache? it caches the last lookup?


              Right.  The cache is not specific to mysql, but is a feature of the
              trivial-rewrite transport lookup.

              This is only likely to be noticed when you use mysql-based
              transport_maps and a high percentage of the queue is for a single
              destination.  This is not configurable.

              The workaround is to use a hash: or cdb: table, which triggers a
              restart of trivial-rewrite upon changes -- but note that frequent
              restarts of trivial-rewrite may be bad for performance.



            • Wietse Venema
              ... The trivial-rewrite server has a 1-element cache for the wild-card lookup result, and the trivial-rewrite client has a 1-element cache for repeated
              Message 6 of 18 , Feb 27, 2012
              • 0 Attachment
                B?ny?sz Botond:
                > I dont think that the problem cames from transport map 1-element
                > caching, because the destination is variable, we have millions of
                > email addresses.

                The trivial-rewrite server has a 1-element cache for the wild-card
                lookup result, and the trivial-rewrite client has a 1-element cache
                for repeated lookups.

                It is not a major effort to give those cache entries a finite
                time-to-live but this needs a better justification that what appears
                to be an attempt to circumvent Yahoo rate limits.

                Wietse
              • Bányász Botond
                ... The trivial-rewrite server has a 1-element cache for the wild-card lookup result, and the trivial-rewrite client has a 1-element cache for repeated
                Message 7 of 18 , Feb 28, 2012
                • 0 Attachment


                  B?ny?sz Botond:
                  > I dont think that the problem cames from transport map 1-element
                  > caching, because the destination is variable, we have millions of
                  > email addresses.

                  The trivial-rewrite server has a 1-element cache for the wild-card
                  lookup result, and the trivial-rewrite client has a 1-element cache
                  for repeated lookups.

                  It is not a major effort to give those cache entries a finite
                  time-to-live but this needs a better justification that what appears
                  to be an attempt to circumvent Yahoo rate limits.

                      Wietse



                  Thank U for your help, unfortunately i didn`t found this in the documentation.
                • Marcin Owsiany
                  ... I have a need similar to the original poster. In my case the use case is to gradually move outgoing traffic to a new egress IP. I have separate transports
                  Message 8 of 18 , Dec 31, 2012
                  • 0 Attachment
                    Wietse Venema <wietse <at> porcupine.org> writes:
                    > It is not a major effort to give those cache entries a finite
                    > time-to-live but this needs a better justification that what appears
                    > to be an attempt to circumvent Yahoo rate limits.

                    I have a need similar to the original poster. In my case the use case
                    is to gradually move outgoing traffic to a new egress IP.

                    I have separate transports set up in master.cf per address like this:

                    smtp unix - - - - - smtp
                    out1 unix - - - - 50 smtp \
                    -o smtp_bind_address=10.150.15.101
                    out2 unix - - - - 50 smtp \
                    -o smtp_bind_address=10.150.15.102
                    [...]

                    and additionally the senders are partitioned across IPs
                    with sender_dependent_default_transport_maps

                    Moving a big chunk of traffic in one go to a new IP that was never
                    sending any traffic before will just cause it to get greylisted
                    or blocked altogether.

                    Given I have a low number of high-volume senders, I cannot find
                    an easy way to start sending (say) 1% of given sender's traffic
                    from a new IP. Any suggestions?

                    Marcin
                  • Wietse Venema
                    ... You did not provide enough context to figure out what Postfix caching feature you re referring to, so I will assume that this is the one-element reply
                    Message 9 of 18 , Dec 31, 2012
                    • 0 Attachment
                      Marcin Owsiany:
                      > Wietse Venema <wietse <at> porcupine.org> writes:
                      > > It is not a major effort to give those cache entries a finite
                      > > time-to-live but this needs a better justification that what appears
                      > > to be an attempt to circumvent Yahoo rate limits.
                      >
                      > I have a need similar to the original poster. In my case the use case
                      > is to gradually move outgoing traffic to a new egress IP.
                      >
                      > I have separate transports set up in master.cf per address like this:
                      >
                      > smtp unix - - - - - smtp
                      > out1 unix - - - - 50 smtp \
                      > -o smtp_bind_address=10.150.15.101
                      > out2 unix - - - - 50 smtp \
                      > -o smtp_bind_address=10.150.15.102
                      > [...]
                      >
                      > and additionally the senders are partitioned across IPs
                      > with sender_dependent_default_transport_maps
                      >
                      > Moving a big chunk of traffic in one go to a new IP that was never
                      > sending any traffic before will just cause it to get greylisted
                      > or blocked altogether.
                      >
                      > Given I have a low number of high-volume senders, I cannot find
                      > an easy way to start sending (say) 1% of given sender's traffic
                      > from a new IP. Any suggestions?

                      You did not provide enough context to figure out what Postfix caching
                      feature you're referring to, so I will assume that this is the
                      one-element reply cache with transport map lookups.

                      In that case the obvious solution is not to send 100% of all email
                      to the same destination domain. That will flush the one-element
                      cache frequently, and force fresh transport map lookups.

                      What is the business use case to send 100% of all email to the same
                      destination domain?

                      Wietse
                    • Marcin Owsiany
                      ... Gmane nagged me to remove excessive citation, and I guess it got too terse. ... That s not what is happening. Let me explain a bit more: There s a dozen or
                      Message 10 of 18 , Dec 31, 2012
                      • 0 Attachment
                        On Mon, Dec 31, 2012 at 02:05:38PM -0500, Wietse Venema wrote:
                        > Marcin Owsiany:
                        > > Wietse Venema <wietse <at> porcupine.org> writes:
                        > > > It is not a major effort to give those cache entries a finite
                        > > > time-to-live but this needs a better justification that what appears
                        > > > to be an attempt to circumvent Yahoo rate limits.
                        > >
                        > > I have a need similar to the original poster. In my case the use case
                        > > is to gradually move outgoing traffic to a new egress IP.
                        > >
                        > > I have separate transports set up in master.cf per address like this:
                        > >
                        > > smtp unix - - - - - smtp
                        > > out1 unix - - - - 50 smtp \
                        > > -o smtp_bind_address=10.150.15.101
                        > > out2 unix - - - - 50 smtp \
                        > > -o smtp_bind_address=10.150.15.102
                        > > [...]
                        > >
                        > > and additionally the senders are partitioned across IPs
                        > > with sender_dependent_default_transport_maps
                        > >
                        > > Moving a big chunk of traffic in one go to a new IP that was never
                        > > sending any traffic before will just cause it to get greylisted
                        > > or blocked altogether.
                        > >
                        > > Given I have a low number of high-volume senders, I cannot find
                        > > an easy way to start sending (say) 1% of given sender's traffic
                        > > from a new IP. Any suggestions?
                        >
                        > You did not provide enough context to figure out what Postfix caching
                        > feature you're referring to, so I will assume that this is the
                        > one-element reply cache with transport map lookups.

                        Gmane nagged me to remove excessive citation, and I guess it got too
                        terse.

                        > In that case the obvious solution is not to send 100% of all email
                        > to the same destination domain. That will flush the one-element
                        > cache frequently, and force fresh transport map lookups.
                        >
                        > What is the business use case to send 100% of all email to the same
                        > destination domain?

                        That's not what is happening. Let me explain a bit more:

                        There's a dozen or so envelope-senders (distribution lists), each
                        periodically sends to a lot of recipients (few thousand to a few dozen
                        thousand). The recipients are distributed over what I think is a typical
                        country-wide sample of domains, which inevitably means the few big
                        webmail services get a large portion.

                        In the basic case, sender_dependent_default_transport_maps assigns each
                        sender to one outgoing IP address, to try and limit colateral damage if
                        one of them screws up and gets blacklisted.

                        I imagine this might sound like a spam sending service, but it's not :-)

                        Now the problem is when we want to move a sender to a different IP
                        address. Because receivers don't like a previously-idle IP to suddenly
                        start sending lots of mail, we need to start with a trickle, to build
                        reputation.

                        Ideally the smtp daemon would be able to bind to a few addresses and
                        there would be an option to assign egress traffic split between them.
                        However I did not find anything like that, so my next idea was to use a
                        custom randomized "tcp" sender_dependent_default_transport_maps lookup
                        table that would map "sender1" to, say, "out1" in 99% of cases and
                        "out2" in the remaining 1%. However the builtin caching would defeat
                        that.

                        regards,
                        --
                        Marcin Owsiany <marcin@...> http://marcin.owsiany.pl/
                        GnuPG: 2048R/02F946FC 35E9 1344 9F77 5F43 13DD 6423 DBF4 80C6 02F9 46FC

                        "Every program in development at MIT expands until it can read mail."
                        -- Unknown
                      • Wietse Venema
                        ... According to source code, the Postfix 2.5 and later resolver client will reuse the trivial-rewrite daemon result for up to 30 seconds when the request s
                        Message 11 of 18 , Dec 31, 2012
                        • 0 Attachment
                          Marcin Owsiany:
                          > Ideally the smtp daemon would be able to bind to a few addresses and
                          > there would be an option to assign egress traffic split between them.

                          According to source code, the Postfix 2.5 and later resolver client
                          will reuse the trivial-rewrite daemon result for up to 30 seconds
                          when the request's (address class, sender address, recipient address)
                          are the same as the previous request.

                          According to the following comment, a similar strategy was adopted
                          for address rewriting and transport map wild-card results.

                          20070414

                          Cleanup: expire cached results from addres rewriting, address
                          resolution, and from transport map lookups. Results expire
                          after 30 seconds; short enough that it doesn't freak out
                          people who run the same test repeatedly, and long enough
                          that it doesn't upset other people with continuous streams
                          of "*" transport map lookups. Files: global/rewrite_clnt.c,
                          global/resolve_clnt.c, trivial-rewrite/transport.c.

                          Now the question is are 30 seconds too much or too little.

                          Wietse
                        • Marcin Owsiany
                          ... Much too much in my case. The client app sends messages in batches, and in 30 seconds much more than 1% of a batch will get processed, especially given
                          Message 12 of 18 , Dec 31, 2012
                          • 0 Attachment
                            On Mon, Dec 31, 2012 at 05:00:52PM -0500, Wietse Venema wrote:
                            > Marcin Owsiany:
                            > > Ideally the smtp daemon would be able to bind to a few addresses and
                            > > there would be an option to assign egress traffic split between them.
                            >
                            > According to source code, the Postfix 2.5 and later resolver client
                            > will reuse the trivial-rewrite daemon result for up to 30 seconds
                            > when the request's (address class, sender address, recipient address)
                            > are the same as the previous request.
                            >
                            > According to the following comment, a similar strategy was adopted
                            > for address rewriting and transport map wild-card results.
                            >
                            > 20070414
                            >
                            > Cleanup: expire cached results from addres rewriting, address
                            > resolution, and from transport map lookups. Results expire
                            > after 30 seconds; short enough that it doesn't freak out
                            > people who run the same test repeatedly, and long enough
                            > that it doesn't upset other people with continuous streams
                            > of "*" transport map lookups. Files: global/rewrite_clnt.c,
                            > global/resolve_clnt.c, trivial-rewrite/transport.c.
                            >
                            > Now the question is are 30 seconds too much or too little.

                            Much too much in my case. The client app sends messages in batches, and
                            in 30 seconds much more than 1% of a batch will get processed,
                            especially given that the canary IP address transport will mostly see
                            fast rejections.

                            --
                            Marcin Owsiany <marcin@...> http://marcin.owsiany.pl/
                            GnuPG: 2048R/02F946FC 35E9 1344 9F77 5F43 13DD 6423 DBF4 80C6 02F9 46FC

                            "Every program in development at MIT expands until it can read mail."
                            -- Unknown
                          • Wietse Venema
                            ... In that case, a burst of 1s would still be undesirable, so one would have to be able to specify 0 (no caching) while the default would remain at 30s.
                            Message 13 of 18 , Dec 31, 2012
                            • 0 Attachment
                              Wietse:
                              > 20070414
                              > Cleanup: expire cached results from addres rewriting, address
                              > resolution, and from transport map lookups. Results expire
                              > after 30 seconds; short enough that it doesn't freak out
                              > people who run the same test repeatedly, and long enough
                              > that it doesn't upset other people with continuous streams
                              > of "*" transport map lookups. [...]
                              >
                              > Now the question is are 30 seconds too much or too little.

                              Marcin Owsiany:
                              > Much too much in my case. The client app sends messages in batches,
                              > and in 30 seconds much more than 1% of a batch will get processed,
                              > especially given that the canary IP address transport will mostly
                              > see fast rejections.

                              In that case, a burst of 1s would still be undesirable, so one would
                              have to be able to specify 0 (no caching) while the default would
                              remain at 30s.

                              Wietse
                            • Marcin Owsiany
                              ... Right. I hope that disabling caching altogether would not affect performance too much. Perhaps it would be a per-map option, so that it would only affect
                              Message 14 of 18 , Dec 31, 2012
                              • 0 Attachment
                                On Mon, Dec 31, 2012 at 05:45:57PM -0500, Wietse Venema wrote:
                                > In that case, a burst of 1s would still be undesirable, so one would
                                > have to be able to specify 0 (no caching) while the default would
                                > remain at 30s.

                                Right.

                                I hope that disabling caching altogether would not affect performance
                                too much. Perhaps it would be a per-map option, so that it would only
                                affect the maps that it needs to affect?

                                --
                                Marcin Owsiany <marcin@...> http://marcin.owsiany.pl/
                                GnuPG: 2048R/02F946FC 35E9 1344 9F77 5F43 13DD 6423 DBF4 80C6 02F9 46FC

                                "Every program in development at MIT expands until it can read mail."
                                -- Unknown
                              • Wietse Venema
                                ... Caching is mostly a client feature. A client doesn t care what tables a server is using. Wietse
                                Message 15 of 18 , Dec 31, 2012
                                • 0 Attachment
                                  Marcin Owsiany:
                                  > On Mon, Dec 31, 2012 at 05:45:57PM -0500, Wietse Venema wrote:
                                  > > In that case, a burst of 1s would still be undesirable, so one would
                                  > > have to be able to specify 0 (no caching) while the default would
                                  > > remain at 30s.
                                  >
                                  > Right.
                                  >
                                  > I hope that disabling caching altogether would not affect performance
                                  > too much. Perhaps it would be a per-map option, so that it would only
                                  > affect the maps that it needs to affect?

                                  Caching is mostly a client feature. A client doesn't care what
                                  tables a server is using.

                                  Wietse
                                • Stan Hoeppner
                                  ... Hi Marcin, Q1: Why is a Google Site Reliability Engineer and long time Debian maintainer, with two Masters degrees, working on an email blasting
                                  Message 16 of 18 , Dec 31, 2012
                                  • 0 Attachment
                                    On 12/31/2012 3:06 PM, Marcin Owsiany wrote:

                                    > I imagine this might sound like a spam sending service, but it's not :-)

                                    Hi Marcin,

                                    Q1: Why is a Google Site Reliability Engineer and long time Debian
                                    maintainer, with two Masters degrees, working on an email blasting
                                    infrastructure?

                                    Q2: Why are you not leveraging the expertise of the Google messaging
                                    folks for this project?

                                    FYI, my first Linux was Potato and I've been Debian ever since. :)
                                    Thank you for your contributions over the years.

                                    --
                                    Stan
                                  • Marcin Owsiany
                                    ... This is probably not completely on topic, but just to make sure this does not turn into some kind of conspiracy theory: a) it has nothing to do with my
                                    Message 17 of 18 , Jan 1, 2013
                                    • 0 Attachment
                                      On Mon, Dec 31, 2012 at 08:46:03PM -0600, Stan Hoeppner wrote:
                                      > On 12/31/2012 3:06 PM, Marcin Owsiany wrote:
                                      >
                                      > > I imagine this might sound like a spam sending service, but it's not :-)
                                      >
                                      > Hi Marcin,
                                      >
                                      > Q1: Why is a Google Site Reliability Engineer and long time Debian
                                      > maintainer, with two Masters degrees, working on an email blasting
                                      > infrastructure?
                                      >
                                      > Q2: Why are you not leveraging the expertise of the Google messaging
                                      > folks for this project?

                                      This is probably not completely on topic, but just to make sure this
                                      does not turn into some kind of conspiracy theory:
                                      a) it has nothing to do with my employer whatsoever,
                                      b) I'm doing it because someone I know asked me for advice,
                                      c) as I said, it's not UCE-related, sending a lot of email fast does
                                      have legitimate purposes
                                      d) asking a postfix question on postfix-users seems like a reasonable
                                      thing to me

                                      > FYI, my first Linux was Potato and I've been Debian ever since. :)
                                      > Thank you for your contributions over the years.

                                      Not that I contributed a whole lot, compared to some other DDs, but
                                      you're welcome :-)

                                      regards,
                                      --
                                      Marcin Owsiany <marcin@...> http://marcin.owsiany.pl/
                                      GnuPG: 2048R/02F946FC 35E9 1344 9F77 5F43 13DD 6423 DBF4 80C6 02F9 46FC

                                      "Every program in development at MIT expands until it can read mail."
                                      -- Unknown
                                    • Stan Hoeppner
                                      ... I didn t intend to imply anything nefarious. By email blasting I simply meant high volume, short duration--not spamming. ... The reason I asked this is
                                      Message 18 of 18 , Jan 1, 2013
                                      • 0 Attachment
                                        On 1/1/2013 7:33 AM, Marcin Owsiany wrote:
                                        > On Mon, Dec 31, 2012 at 08:46:03PM -0600, Stan Hoeppner wrote:
                                        >> On 12/31/2012 3:06 PM, Marcin Owsiany wrote:
                                        >>
                                        >>> I imagine this might sound like a spam sending service, but it's not :-)
                                        >>
                                        >> Hi Marcin,
                                        >>
                                        >> Q1: Why is a Google Site Reliability Engineer and long time Debian
                                        >> maintainer, with two Masters degrees, working on an email blasting
                                        >> infrastructure?
                                        >>
                                        >> Q2: Why are you not leveraging the expertise of the Google messaging
                                        >> folks for this project?
                                        >
                                        > This is probably not completely on topic, but just to make sure this
                                        > does not turn into some kind of conspiracy theory:

                                        I didn't intend to imply anything nefarious. By "email blasting" I
                                        simply meant high volume, short duration--not spamming.

                                        > a) it has nothing to do with my employer whatsoever,
                                        > b) I'm doing it because someone I know asked me for advice,
                                        > c) as I said, it's not UCE-related, sending a lot of email fast does
                                        > have legitimate purposes
                                        > d) asking a postfix question on postfix-users seems like a reasonable
                                        > thing to me

                                        The reason I asked this is that Google is one of the "few big
                                        webmail services get a large portion" that you mentioned. Your approach
                                        here seems to be optimizing Postfix to "circumvent defenses". Normally
                                        one would sign up with these services as a bulk mailer to avoid bulk
                                        delivery problems. Maybe you are doing that as well but did not mention it.

                                        >> FYI, my first Linux was Potato and I've been Debian ever since. :)
                                        >> Thank you for your contributions over the years.
                                        >
                                        > Not that I contributed a whole lot, compared to some other DDs, but
                                        > you're welcome :-)

                                        Every small contribution helps. :)

                                        --
                                        Stan
                                      Your message has been successfully submitted and would be delivered to recipients shortly.