Loading ...
Sorry, an error occurred while loading the content.

Tweaking DNS timeouts (was: postscreen_dnsbl_whitelist_threshold)

Expand Messages
  • Wietse Venema
    ... All postscreen versions work that way. When the DNSBL score is not final before the pregreet test completes, the DNSBL test remains undecided, and the test
    Message 1 of 25 , May 16, 2013
    • 0 Attachment
      /dev/rob0:
      > In the time since I've been running this, I saw the first thing that
      > might be seen as a problem: dnsblog timing out on one of the DNSBL
      > lookups:
      >
      > May 16 21:51:44 harrier postfix/postscreen[29502]: CONNECT from [208.66.205.36]:53814 to [207.223.116.211]:25
      > May 16 21:51:44 harrier postfix/dnsblog[29507]: addr 208.66.205.36 listed by domain list.dnswl.org as 127.0.15.0
      >
      > This gives it a -2 so far, but when the greet pause is finished,
      > postscreen proceeds anyway:

      All postscreen versions work that way. When the DNSBL score is not
      final before the pregreet test completes, the DNSBL test remains
      undecided, and the test will be repeated the next time the client
      connects.

      Increasing the greet-wait to 10+ seconds could result in legitimate
      clients hanging up, so I would not recommend that.

      You can try to change the DNS resolver timeout/retry behavior:

      /etc/resolv.conf:
      # Typical default settings shown here. See resolver(5).
      options timeout:5 attempts:2 ...

      However, this changes all DNS lookups of every program on the system,
      and that may be undesirable.

      You can instead specify these settings for Postfix only by setting
      the RES_OPTIONS environment variable.

      /etc/postfix/main.cf:
      import_environment = ... RES_OPTIONS=timeout:3 ...

      Unfortunately main.cf does not support RES_OPTIONS values that
      contain spaces (there is no support for quotes) and multiple
      RES_OPTIONS=whatever settings don't add up, so you can override
      only one of "timeout" or "attempts" but not both.

      From here on things only gets worse. The following information is
      only for completeness. I would not recommend anyone to take this
      path. To override RES_OPTIONS with spaces and all you would have
      to set it in $daemon_directory/postfix-script.

      /usr/libexec/postfix/postfix-script:
      export RES_OPTIONS; RES_OPTIONS="xxx yyy zzz"

      etc/postfix/main.cf:
      import_environment = ... RES_OPTIONS ...

      This will import an environment setting literally. But it will break
      the next time Postfix is updated.

      Wietse
    • /dev/rob0
      ... Do we have any testing to validate this? I m pretty sure I recall from a few years back on the old original SPAM-L list that some Sendmail people[1] were
      Message 2 of 25 , May 17, 2013
      • 0 Attachment
        On Thu, May 16, 2013 at 07:48:24PM -0400, Wietse Venema wrote:
        > /dev/rob0:
        > > In the time since I've been running this, I saw the first thing
        > > that might be seen as a problem: dnsblog timing out on one of
        > > the DNSBL lookups:
        > >
        > > May 16 21:51:44 harrier postfix/postscreen[29502]: CONNECT from [208.66.205.36]:53814 to [207.223.116.211]:25
        > > May 16 21:51:44 harrier postfix/dnsblog[29507]: addr 208.66.205.36 listed by domain list.dnswl.org as 127.0.15.0
        > >
        > > This gives it a -2 so far, but when the greet pause is finished,
        > > postscreen proceeds anyway:
        >
        > All postscreen versions work that way. When the DNSBL score is not
        > final before the pregreet test completes, the DNSBL test remains
        > undecided, and the test will be repeated the next time the client
        > connects.
        >
        > Increasing the greet-wait to 10+ seconds could result in
        > legitimate clients hanging up, so I would not recommend that.

        Do we have any testing to validate this? I'm pretty sure I recall
        from a few years back on the old original SPAM-L list that some
        Sendmail people[1] were saying they used greet pauses in excess of 30
        seconds.

        > You can try to change the DNS resolver timeout/retry behavior:

        Thanks for all that. As it happens, I have a quick fix for this:

        $ grep 'dnsblog.*timeout' /var/log/maillog | wc
        35 420 3731
        $ grep 'dnsblog.*timeout' /var/log/maillog | grep -v surriel | wc
        0 0 0

        PSBL seems to be a bit slow for me. I've taken it out of my
        postscreen_dnsbl_sites; I had only recently added it.

        What this shows is that there's no good, risk-free way to test
        potential new DNSBLs. No great harm done: at the most, 35 delayed
        mails. But could a site which is consistently timing out cause
        positive scores to be ignored? Apparently not here:

        May 12 05:05:39 harrier postfix/postscreen[17895]: CONNECT from [24.227.47.42]:1362 to [207.223.116.211]:25
        May 12 05:05:39 harrier postfix/postscreen[17895]: PREGREET 21 after 0.03 from [24.227.47.42]:1362: EHLO [192.168.2.33]\r\n
        May 12 05:05:39 harrier postfix/dnsblog[17901]: addr 24.227.47.42 listed by domain dnsbl.sorbs.net as 127.0.0.7
        May 12 05:05:39 harrier postfix/dnsblog[17897]: addr 24.227.47.42 listed by domain b.barracudacentral.org as 127.0.0.2
        May 12 05:05:40 harrier postfix/dnsblog[17900]: addr 24.227.47.42 listed by domain zen.spamhaus.org as 127.0.0.4
        May 12 05:05:45 harrier postfix/postscreen[17895]: DNSBL rank 6 for [24.227.47.42]:1362
        May 12 05:05:45 harrier postfix/postscreen[17895]: NOQUEUE: reject: RCPT from [24.227.47.42]:1362: 550 5.7.1 Service unavailable; client [24.227.47.42] blocked using zen.spamhaus.org; from=<test@...>, to=<therichsheickc@...>, proto=ESMTP, helo=<[192.168.2.33]>
        May 12 05:05:45 harrier postfix/postscreen[17895]: DISCONNECT [24.227.47.42]:1362
        May 12 05:05:49 harrier postfix/postscreen[17895]: warning: dnsblog reply timeout 10s for psbl.surriel.com
        May 12 05:05:59 harrier postfix/dnsblog[17902]: warning: dnsblog_query: lookup error for DNS query 42.47.227.24.psbl.surriel.com: Host or domain name not found. Name service error for name=42.47.227.24.psbl.surriel.com type=A: Host not found, try again

        I guess this says that postscreen_dnsbl_action fires at the end of
        the greet pause when postscreen_dnsbl_threshold is met, but
        postscreen_dnsbl_whitelist_threshold is not calculated. Here's the
        same botnet from a different zombie, which does not meet the
        threshold, rejected for protocol error:

        May 12 05:43:09 harrier postfix/postscreen[19787]: CONNECT from [80.24.21.133]:23652 to [207.223.116.211]:25
        May 12 05:43:09 harrier postfix/dnsblog[19790]: addr 80.24.21.133 listed by domain bl.spameatingmonkey.net as 127.0.0.2
        May 12 05:43:09 harrier postfix/postscreen[19787]: PREGREET 21 after 0.22 from [80.24.21.133]:23652: EHLO [192.168.2.33]\r\n
        May 12 05:43:19 harrier postfix/postscreen[19787]: warning: dnsblog reply timeout 10s for psbl.surriel.com
        May 12 05:43:20 harrier postfix/postscreen[19787]: NOQUEUE: reject: RCPT from [80.24.21.133]:23652: 550 5.5.1 Protocol error; from=<test@...>, to=<therichsheickc@...>, proto=ESMTP, helo=<[192.168.2.33]>
        May 12 05:43:21 harrier postfix/postscreen[19787]: DISCONNECT [80.24.21.133]:23652

        Here's one without the pregreet:

        May 13 06:21:09 harrier postfix/postscreen[3805]: CONNECT from [89.121.129.184]:43448 to [207.223.116.211]:25
        May 13 06:21:09 harrier postfix/dnsblog[3807]: addr 89.121.129.184 listed by domain b.barracudacentral.org as 127.0.0.2
        May 13 06:21:09 harrier postfix/dnsblog[3813]: addr 89.121.129.184 listed by domain zen.spamhaus.org as 127.0.0.11
        May 13 06:21:09 harrier postfix/dnsblog[3813]: addr 89.121.129.184 listed by domain zen.spamhaus.org as 127.0.0.4
        May 13 06:21:09 harrier postfix/dnsblog[3808]: addr 89.121.129.184 listed by domain bl.mailspike.net as 127.0.0.12
        May 13 06:21:15 harrier postfix/postscreen[3805]: DNSBL rank 6 for [89.121.129.184]:43448
        May 13 06:21:16 harrier postfix/postscreen[3805]: NOQUEUE: reject: RCPT from [89.121.129.184]:43448: 550 5.7.1 Service unavailable; client [89.121.129.184] blocked using zen.spamhaus.org; from=<watcheslz@...>, to=<mungeduser@...>, proto=ESMTP, helo=<89-121-129-184.romtelecom.net>
        May 13 06:21:16 harrier postfix/postscreen[3805]: HANGUP after 0.68 from [89.121.129.184]:43448 in tests after SMTP handshake
        May 13 06:21:16 harrier postfix/postscreen[3805]: DISCONNECT [89.121.129.184]:43448
        May 13 06:21:19 harrier postfix/postscreen[3805]: warning: dnsblog reply timeout 10s for psbl.surriel.com


        [Snip all the good resolver(5) information]


        [1] Specifically I am thinking of the late Bruce Gingery, a true
        master spamfighter. I will ask about this on SDLU[2] also.
        [2] http://spammers.dontlike.us/
        --
        http://rob0.nodns4.us/ -- system administration and consulting
        Offlist GMX mail is seen only if "/dev/rob0" is in the Subject:
      • Viktor Dukhovni
        ... It creates a lot of needless congestion on legitimate sending systems even if they don t hang up. Now every message (from a small MTA that does not visit
        Message 3 of 25 , May 17, 2013
        • 0 Attachment
          On Fri, May 17, 2013 at 12:26:13PM -0500, /dev/rob0 wrote:

          > > Increasing the greet-wait to 10+ seconds could result in
          > > legitimate clients hanging up, so I would not recommend that.
          >
          > Do we have any testing to validate this? I'm pretty sure I recall
          > from a few years back on the old original SPAM-L list that some
          > Sendmail people[1] were saying they used greet pauses in excess of 30
          > seconds.

          It creates a lot of needless congestion on legitimate sending
          systems even if they don't hang up.

          Now every message (from a small MTA that does not visit often)
          starts to take 30s to make a delivery. Queue throughput collapses
          and Patrick Raq's MTA can't deliver new mail in a timely fashion.
          On the plus side, Wietse and Patrick may finally consider my
          "concurrency balooning" suggestion. :-)

          Much of the damage to the SMTP infrastructure is done by well-meaning
          anti-spam measures. Let's not take it too far.

          --
          Viktor.
        • /dev/rob0
          ... snip ... I understand all this and agree. I m not advocating a 30+ second greet pause. My original goal was to reduce delays. Most of those who manage
          Message 4 of 25 , May 17, 2013
          • 0 Attachment
            On Fri, May 17, 2013 at 05:53:47PM +0000, Viktor Dukhovni wrote:
            > On Fri, May 17, 2013 at 12:26:13PM -0500, /dev/rob0 wrote:
            > Wietse:
            > > > Increasing the greet-wait to 10+ seconds could result in
            > > > legitimate clients hanging up, so I would not recommend that.
            > >
            > > Do we have any testing to validate this? I'm pretty sure I
            > > recall from a few years back on the old original SPAM-L list
            > > that some Sendmail people[1] were saying they used greet
            > > pauses in excess of 30 seconds.
            >
            > It creates a lot of needless congestion on legitimate sending
            > systems even if they don't hang up.
            >
            snip
            >
            > Much of the damage to the SMTP infrastructure is done by
            > well-meaning anti-spam measures. Let's not take it too far.

            I understand all this and agree. I'm not advocating a 30+ second
            greet pause. My original goal was to reduce delays.

            Most of those who manage really busy outbounds will have gone to the
            trouble of getting listed on DNS whitelists. And for these outbounds,
            an occasional 10-second greet pause is better than "Service currently
            unavailable" and PASS NEW.

            But I think this is all moot, and my quick fix, to stop querying
            psbl.surriel.com, was the best. The moral of the story being, use
            DNSBL sites with adequate response times and five nines. It's
            probably also moot if the postscreen_dnsbl_threshold score is only
            calculated when in excess thereof in case of DNS timeouts.
            --
            http://rob0.nodns4.us/ -- system administration and consulting
            Offlist GMX mail is seen only if "/dev/rob0" is in the Subject:
          • Wietse Venema
            ... [begin background material] I mis-understood how postscreen works (I do not constantly stare at Postfix source code, having other things to work on that
            Message 5 of 25 , May 17, 2013
            • 0 Attachment
              /dev/rob0:
              >
              > I guess this says that postscreen_dnsbl_action fires at the end of
              > the greet pause when postscreen_dnsbl_threshold is met, but
              > postscreen_dnsbl_whitelist_threshold is not calculated. Here's the

              [begin background material]

              I mis-understood how postscreen works (I do not constantly stare
              at Postfix source code, having other things to work on that pay the
              bills).

              I thought that the whitelist will be applied only when DNS lookups
              complete *before* the pregreet timer expires. That is,

              - When some DNS lookup is taking too long, no DNS score is available.

              This is consistent with how postscreen whitelisting works for non-DNS
              tests. It applies the whitelist threshold only when DNS lookup
              completes before the pregreet timer expires.

              However, the bullet above is incorrect. When soe DNS lookup takes
              too long, a DNS score is available, and the postscreen DNS blocking
              code uses that partial score.

              This is safe when there are only positive scores (if the partial
              client is already over the threshold then the client should be
              blocked even if some DNS results are not yet in).

              This is less safe when there may also be exculpatory evidence (in
              the form of DNSWL lookups). But, sites are usually not listed in
              both white and block lists.

              [end background material]

              I can change postscreen to also use partial scores for whitelisting
              of non-DNS tests, and thereby make whitelisting of non-DNS tests
              consistent with DNS-based blocking (that's one less WTF factor).
              This requires minor code duplication.

              Wietse
            • Wietse Venema
              ... Released as snapshot 20130517. Wietse
              Message 6 of 25 , May 17, 2013
              • 0 Attachment
                Wietse Venema:
                > I can change postscreen to also use partial scores for whitelisting
                > of non-DNS tests, and thereby make whitelisting of non-DNS tests
                > consistent with DNS-based blocking (that's one less WTF factor).
                > This requires minor code duplication.

                Released as snapshot 20130517.

                Wietse
              • /dev/rob0
                ... For testing I reenabled PSBL, and I ll see what comes in overnight. I thought I could make my own pseudo-DNSBL on a random IP address with blocked ports
                Message 7 of 25 , May 17, 2013
                • 0 Attachment
                  On Fri, May 17, 2013 at 10:06:38PM -0400, Wietse Venema wrote:
                  > Wietse Venema:
                  > > I can change postscreen to also use partial scores for
                  > > whitelisting of non-DNS tests, and thereby make whitelisting
                  > > of non-DNS tests consistent with DNS-based blocking (that's one
                  > > less WTF factor). This requires minor code duplication.
                  >
                  > Released as snapshot 20130517.

                  For testing I reenabled PSBL, and I'll see what comes in overnight.
                  I thought I could make my own pseudo-DNSBL on a random IP address
                  with blocked ports 53, but I need to set up an NS record to point to
                  that. I'll do that tomorrow if results tonight are inconclusive.
                  --
                  http://rob0.nodns4.us/ -- system administration and consulting
                  Offlist GMX mail is seen only if "/dev/rob0" is in the Subject:
                • Wietse Venema
                  ... For whitelisting I used a wild-card A record, and for timeout testing I used an NS record that resolves to a firewalled port (a black hole). This
                  Message 8 of 25 , May 18, 2013
                  • 0 Attachment
                    /dev/rob0:
                    > On Fri, May 17, 2013 at 10:06:38PM -0400, Wietse Venema wrote:
                    > > Wietse Venema:
                    > > > I can change postscreen to also use partial scores for
                    > > > whitelisting of non-DNS tests, and thereby make whitelisting
                    > > > of non-DNS tests consistent with DNS-based blocking (that's one
                    > > > less WTF factor). This requires minor code duplication.
                    > >
                    > > Released as snapshot 20130517.
                    >
                    > For testing I reenabled PSBL, and I'll see what comes in overnight.
                    > I thought I could make my own pseudo-DNSBL on a random IP address
                    > with blocked ports 53, but I need to set up an NS record to point to
                    > that. I'll do that tomorrow if results tonight are inconclusive.

                    For whitelisting I used a wild-card "A" record, and for timeout
                    testing I used an NS record that resolves to a firewalled port (a
                    black hole).

                    This confirmed that postscreen will now use partial scores to
                    whitelist pending non-dnbsbl tests.

                    I can make those domain names available for general testing (but
                    not now as I am in the middle of a copper-to-fiber conversion).

                    Wietse
                  • /dev/rob0
                    Still watching logs, this one just passed by. Probably unrelated to the changes in 20130517, but I was curious about it: May 19 13:24:20 harrier
                    Message 9 of 25 , May 19, 2013
                    • 0 Attachment
                      Still watching logs, this one just passed by. Probably unrelated to
                      the changes in 20130517, but I was curious about it:

                      May 19 13:24:20 harrier postfix/postscreen[3533]: CONNECT from [188.42.15.19]:48706 to [207.223.116.211]:25
                      May 19 13:24:26 harrier postfix/postscreen[3533]: NOQUEUE: reject: RCPT from [188.42.15.19]:48706: 450 4.3.2 Service currently unavailable; from=<bounce@...>, to=<munged@...>, proto=ESMTP, helo=<mail18.consumer-news123.com>
                      May 19 13:24:26 harrier postfix/postscreen[3533]: PASS NEW [188.42.15.19]:48706
                      May 19 13:24:26 harrier postfix/postscreen[3533]: DISCONNECT [188.42.15.19]:48706

                      All is well and good for a non-whitelisted host, but apparently it
                      was too quick in coming back to the secondary MX IP address ...

                      May 19 13:24:26 harrier postfix/postscreen[3533]: CONNECT from [188.42.15.9]:33610 to [207.223.116.214]:25
                      May 19 13:24:26 harrier postfix/postscreen[3533]: WHITELIST VETO [188.42.15.9]:33610

                      ... all in the same second, but according to syslog, sequentially
                      after having earned whitelist status.

                      May 19 13:24:32 harrier postfix/postscreen[3533]: NOQUEUE: reject: RCPT from [188.42.15.9]:33610: 450 4.3.2 Service currently unavailable; from=<bounce@...>, to=<munged@...>, proto=ESMTP, helo=<mail8.consumer-news123.com>
                      May 19 13:24:32 harrier postfix/postscreen[3533]: DISCONNECT [188.42.15.9]:33610

                      Another six seconds pass before this one is turned away, which
                      suggests that the greet pause was repeated. Makes sense, because
                      "WHITELIST VETO" means it was not seen before.
                      --
                      http://rob0.nodns4.us/ -- system administration and consulting
                      Offlist GMX mail is seen only if "/dev/rob0" is in the Subject:
                    • Wietse Venema
                      ... postscreen does not find the client IP address in the permanent postscreen_access_list, does not find client the IP address in the temporary
                      Message 10 of 25 , May 19, 2013
                      • 0 Attachment
                        /dev/rob0:
                        > Still watching logs, this one just passed by. Probably unrelated to
                        > the changes in 20130517, but I was curious about it:
                        >
                        > May 19 13:24:20 harrier postfix/postscreen[3533]: CONNECT from [188.42.15.19]:48706 to [207.223.116.211]:25
                        > May 19 13:24:26 harrier postfix/postscreen[3533]: NOQUEUE: reject: RCPT from [188.42.15.19]:48706: 450 4.3.2 Service currently unavailable; from=<bounce@...>, to=<munged@...>, proto=ESMTP, helo=<mail18.consumer-news123.com>
                        > May 19 13:24:26 harrier postfix/postscreen[3533]: PASS NEW [188.42.15.19]:48706
                        > May 19 13:24:26 harrier postfix/postscreen[3533]: DISCONNECT [188.42.15.19]:48706

                        postscreen does not find the client IP address in the permanent
                        postscreen_access_list, does not find client the IP address in the
                        temporary postscreen_cache_map, logs the "all tests passed" status,
                        updates the temporary postscreen_cache_map with the expiration time
                        for each test, and forgets the test results.

                        > All is well and good for a non-whitelisted host, but apparently it
                        > was too quick in coming back to the secondary MX IP address ...
                        >
                        > May 19 13:24:26 harrier postfix/postscreen[3533]: CONNECT from [188.42.15.9]:33610 to [207.223.116.214]:25
                        > May 19 13:24:26 harrier postfix/postscreen[3533]: WHITELIST VETO [188.42.15.9]:33610
                        >
                        > ... all in the same second, but according to syslog, sequentially
                        > after having earned whitelist status.

                        postscreen logs "CONNECT from", does not find the client IP address
                        in the permanent postscreen_access_list, and does not find the
                        client IP address in the temporary postscreen_cache_map. Therefore
                        this is handled as a non-whitelisted client that connects to the
                        "wrong" IP address.

                        Why wasn't the client IP address found in the temporary
                        postscreen_cache_map? Maybe silent corruption of the cache database.

                        Wietse
                      Your message has been successfully submitted and would be delivered to recipients shortly.