Loading ...
Sorry, an error occurred while loading the content.

Re: [feature request] Subzero postscreen/dnsblog score to bypass after-220 tests?

Expand Messages
  • Stan Hoeppner
    On 4/12/2013 12:58 AM, /dev/rob0 wrote: ... ... Or.... Maybe you could bash script this: dig +short txt _netblocks.google.com|sed s/ip4://g ... which yields
    Message 1 of 9 , Apr 12, 2013
    • 0 Attachment
      On 4/12/2013 12:58 AM, /dev/rob0 wrote:
      ...
      > So here's my idea (I think the parameter names are lousy, but it's
      > the best I could come up with this late at night):
      ...

      Or....

      Maybe you could bash script this:

      dig +short txt _netblocks.google.com|sed s/ip4://g \
      |mawk '{for(i=2; i<=(NF-1); i++){print($i)}}'

      which yields this formatted list of Google outbound CIDRs:

      216.239.32.0/19
      64.233.160.0/19
      66.249.80.0/20
      72.14.192.0/18
      209.85.128.0/17
      66.102.0.0/20
      74.125.0.0/16
      64.18.0.0/20
      207.126.144.0/20
      173.194.0.0/16

      then diff this against your postscreen whitelist and append any new
      entries. You'd cron this to a $suitable_interval, say nightly. If/when
      Google adds any new outbound networks you're covered.

      This seems quite a bit less effort than Wietse adding the feature you
      requested. The end result is nearly identical, at least for the Google
      case, and can easily be extended to cover other domains. And with this
      method the Google outbounds skip all Postscreen processing entirely, not
      just the after 220 tests.

      --
      Stan
    • Wietse Venema
      ... Disabling tests based on DNSWL score would make sense (currently they disable DNSBL tests only). Perhaps this needs a disable flag in the postscreen
      Message 2 of 9 , Apr 12, 2013
      • 0 Attachment
        /dev/rob0:
        > I finally got around to my upgrade to 2.11-20130405 and was watching
        > logs. A gmail message fell afoul of the after-220 tests; each time it
        > came from a different host. Each one got a "PASS NEW" and of course
        > the "450 4.3.2 Service currently unavailable" rejection.
        >
        > These gmail outbounds are all listed in list.dnswl.org as 127.0.5.1,
        > and I give that a negative score in my postscreen_dnsbl_sites. So
        > with no offsetting DNSBL scores, these hosts all got a subzero score.
        > It would be nice if we could put those whitelist scores to work, and
        > not have to maintain so big of a postscreen_access_list whitelist.

        Disabling tests based on DNSWL score would make sense (currently
        they "disable" DNSBL tests only). Perhaps this needs a "disable"
        flag in the postscreen cache.

        Wietse
      • /dev/rob0
        On Fri, Apr 12, 2013 at 04:39:29AM -0500, Stan Hoeppner wrote ... I did think of this, and yes, it would save us the pain which seems to hit every 30 days, as
        Message 3 of 9 , Apr 12, 2013
        • 0 Attachment
          On Fri, Apr 12, 2013 at 04:39:29AM -0500, Stan Hoeppner wrote
          Re: scripting a list of Google outbound CIDRs:
          > This seems quite a bit less effort than Wietse adding the feature
          > you requested. The end result is nearly identical, at least for
          > the Google case, and can easily be extended to cover other domains.

          I did think of this, and yes, it would save us the pain which seems
          to hit every 30 days, as the after-220 tests for gmail expire. But
          extending it to cover other domains would not scale well. Which
          domains? What's the structure of their SPF records?

          When you "easily extend" this idea it becomes much more onerous. And
          still sitting out there are those unused DNSWL scores.

          Yes, unused. As it stands I could drop those checks from my config
          without noticing a change. There is very little overlap between the
          DNSWLs (I currently use SWL and dnswl.org) and reasonable, well-run
          DNSBLs. In my experience a few of the spamtrap-driven automated
          DNSBLs occasionally list a dnswl.org whitelisted host, but I don't
          recall having seen an instance where whitelisting prevented a
          rejection. And I have never found a blacklist entry for the (much
          smaller, I think) SWL zone.

          A DNSWL entry says two things:
          1. This is a real MTA, not a zombie
          2. At one point someone trustworthy thought it was not
          spammer-controlled

          Case 1 mostly entitles it to speak to smtpd, unless of course
          offsetting DNSBL scores overcome the whitelist score. By continuing
          on to check DNSBLs, Case 2 is addressed.

          I believe that DNS-based whitelisting will grow in importance,
          especially in the IPv6 world. I expect to move into IPv6 with a
          default-deny policy, where non-whitelisted hosts are rejected.

          > And with this method the Google outbounds skip all Postscreen
          > processing entirely, not just the after 220 tests.

          I wouldn't want that. :) If one of these providers is seriously
          compromised, they'll be blacklisted, and I would want to check for
          that. I don't give Google my absolute trust. I think they may have
          improved, but I know they're not infallible.
          --
          http://rob0.nodns4.us/ -- system administration and consulting
          Offlist GMX mail is seen only if "/dev/rob0" is in the Subject:
        • Reindl Harald
          ... how do you imagine this working? in this case it would be better you stay at ipv4 at all instead answer AAA dns-requests which may be preferred from
          Message 4 of 9 , Apr 12, 2013
          • 0 Attachment
            Am 12.04.2013 16:52, schrieb /dev/rob0:
            > I believe that DNS-based whitelisting will grow in importance,
            > especially in the IPv6 world. I expect to move into IPv6 with a
            > default-deny policy, where non-whitelisted hosts are rejected

            how do you imagine this working?

            in this case it would be better you stay at ipv4 at all instead
            answer AAA dns-requests which may be preferred from dual-stack
            machines try to deliver to your customer

            it does not work that anybody who wants to send you e-mail he
            must prove that he is no spammer, really this does not work
          • Wietse Venema
            ... On second consideration, this can be done as follows: - One parameter with the (negative) postscreen_dnsbl_sites score that is needed to allow the client
            Message 5 of 9 , Apr 23, 2013
            • 0 Attachment
              On Fri, Apr 12, 2013 at 06:34:24AM -0400, Wietse Venema wrote:
              > /dev/rob0:
              > > I finally got around to my upgrade to 2.11-20130405 and was watching
              > > logs. A gmail message fell afoul of the after-220 tests; each time it
              > > came from a different host. Each one got a "PASS NEW" and of course
              > > the "450 4.3.2 Service currently unavailable" rejection.
              > >
              > > These gmail outbounds are all listed in list.dnswl.org as 127.0.5.1,
              > > and I give that a negative score in my postscreen_dnsbl_sites. So
              > > with no offsetting DNSBL scores, these hosts all got a subzero score.
              > > It would be nice if we could put those whitelist scores to work, and
              > > not have to maintain so big of a postscreen_access_list whitelist.
              >
              > Disabling tests based on DNSWL score would make sense (currently
              > they "disable" DNSBL tests only). Perhaps this needs a "disable"
              > flag in the postscreen cache.

              On second consideration, this can be done as follows:

              - One parameter with the (negative) postscreen_dnsbl_sites score
              that is needed to allow the client to skip tests.

              - One parameter with the names of tests that are skipped (using
              !name to exclude a name, and static:all to match everything).
              This may include "greet" to cancel a "greet wait" in progress.

              The procedure is: postscreen does a postscreen_dnsbl_sites query
              for the client IP address. If the score satifies the threshold in
              the first parameter, then all tests with a name that matches the
              second parameter will be skipped until the next postscreen_dnsbl_sites
              query for that client IP address (i.e. after postscreen_dnsbl_ttl).

              Wietse
            • /dev/rob0
              ... postscreen_skip_tests_threshold , or should there be a _dnsbl or _dnswl in there? postscreen_dnsbl_skip_tests_threshold is good because it lumps it in
              Message 6 of 9 , Apr 23, 2013
              • 0 Attachment
                On Tue, Apr 23, 2013 at 08:05:34PM -0400, Wietse Venema wrote:
                > On Fri, Apr 12, 2013 at 06:34:24AM -0400, Wietse Venema wrote:
                > > /dev/rob0:
                > > > I finally got around to my upgrade to 2.11-20130405 and was
                > > > watching logs. A gmail message fell afoul of the after-220
                > > > tests; each time it came from a different host. Each one got
                > > > a "PASS NEW" and of course the "450 4.3.2 Service currently
                > > > unavailable" rejection.
                > > >
                > > > These gmail outbounds are all listed in list.dnswl.org as
                > > > 127.0.5.1, and I give that a negative score in my
                > > > postscreen_dnsbl_sites. So with no offsetting DNSBL scores,
                > > > these hosts all got a subzero score. It would be nice if we
                > > > could put those whitelist scores to work, and not have to
                > > > maintain so big of a postscreen_access_list whitelist.
                > >
                > > Disabling tests based on DNSWL score would make sense (currently
                > > they "disable" DNSBL tests only). Perhaps this needs a "disable"
                > > flag in the postscreen cache.
                >
                > On second consideration, this can be done as follows:
                >
                > - One parameter with the (negative) postscreen_dnsbl_sites score
                > that is needed to allow the client to skip tests.

                "postscreen_skip_tests_threshold", or should there be a _dnsbl or
                _dnswl in there? "postscreen_dnsbl_skip_tests_threshold" is good
                because it lumps it in with the other postscreen_dnsbl_* settings
                and makes it clear that this is associated with the DNSBL lookups.
                OTOH _dnswl almost does this too, and it's more accurate. To see how
                it looks: "postscreen_dnswl_skip_tests_threshold".

                > - One parameter with the names of tests that are skipped (using
                > !name to exclude a name, and static:all to match everything).
                > This may include "greet" to cancel a "greet wait" in progress.

                As for "type:name", like postscreen_access_list, you're going to want
                to discourage lookups which might slow this down. I guess only pcre,
                regexp, static, and texthash would be suitable for this? I did not
                include such a warning in the proposed documentation below.

                I think you might want to consider an "all", because that's used in
                numerous other places and meets the "minimum surprise" ideal.


                * Postscreen Skippable Test Names *

                before-220 after-220
                ========== =========
                bare_newline
                blacklist
                greet
                mx_policy
                non_smtp_command
                pipelining
                static:all


                I don't think there's much point in skipping the blacklist test. It
                would be very strange if the postscreen_access_list lookup came in
                after the DNSBL lookups. Furthermore, if I listed something there
                that I don't want to see, it should never pass. But maybe someone
                would want this? A test can be "skipped" even if already completed.
                (But what if the postscreen_access_list result is reject? Shouldn't
                that be done immediately, before the DNSBL lookups are in?)

                Likewise, I think the mx_policy (postscreen_whitelist_interfaces)
                test should also be absolute. If a client is not connecting on the
                proper IP address, this should be cause for at least having it talk
                to postscreen and try again later. But again, maybe someone would
                trust the DNS whitelists' judgment?

                I like the idea of two umbrella categories, before-220 and after-220,
                in the spirit of inet_interfaces' "all" and "loopback-only". But
                there's only three per category, so this is not major at this point.
                (I suppose in the future more tests could be added.)

                I think the default should be either "after-220" or "greet,
                after-220". Typically the result would come in during a greet pause,
                and even though it's only a few seconds, it can add up in the Big
                Scheme of Things.

                > The procedure is: postscreen does a postscreen_dnsbl_sites query
                > for the client IP address. If the score satifies the threshold
                > in the first parameter, then all tests with a name that matches
                > the second parameter will be skipped until the next
                > postscreen_dnsbl_sites query for that client IP address (i.e.
                > after postscreen_dnsbl_ttl).

                """
                postscreen_skip_tests (default: greet, after-220)

                Allow a remote SMTP client with a score less than or equal to
                postscreen_skip_tests_threshold based on its combined DNSBL
                score as defined with the postscreen_dnsbl_sites parameter,
                to skip the listed tests, if enabled. Specify zero or more of
                blacklist, greet, mx_policy (these three collectively can be
                "before-220"), bare_newline, non_smtp_command, pipelining
                (these three collectively can be "after-220"), or
                "static:all" to skip all postscreen tests except for the
                DNSBL test itself. Specify "!pattern" to exclude a test from
                the list.

                Example:

                /etc/postfix/main.cf:
                postscreen_dnsbl_sites = dnsbl.example.org,
                whitelist.example.com*-1
                postscreen_skip_tests = !blacklist, !mx_policy,
                static:all

                This feature is available in Postfix 2.11.

                postscreen_skip_tests_threshold (default: -1)

                The inclusive upper bound for allowing a remote SMTP client,
                based on its combined DNSBL score as defined with the
                postscreen_dnsbl_sites parameter, to bypass the tests listed
                in the postscreen_skip_tests parameter.

                Note: this typically would be a negative number, and it only
                makes sense when using DNS whitelists with negative weights
                in the postscreen_dnsbl_sites list. See the example at
                postscreen_skip_tests.

                This feature is available in Postfix 2.11.
                """

                I hope this is getting closer? The only point of confusion about it
                in my mind is whether/how to skip the blacklist test. Should
                postscreen, knowing "blacklist" is in the postscreen_skip_tests list,
                await the dnsblog results for a blacklisted client? Why? It's
                certainly not going to hold up a postscreen_access_list "permit"
                client.

                Thanks again for considering this.
                --
                http://rob0.nodns4.us/ -- system administration and consulting
                Offlist GMX mail is seen only if "/dev/rob0" is in the Subject:
              • /dev/rob0
                Here s a proposed diff for the POSTSCREEN_README: rob0@harrier:~/stuff/postscreen.dnswl$ diff -Nru POSTSCREEN_README* ... +++ POSTSCREEN_README.new
                Message 7 of 9 , Apr 24, 2013
                • 0 Attachment
                  Here's a proposed diff for the POSTSCREEN_README:

                  rob0@harrier:~/stuff/postscreen.dnswl$ diff -Nru POSTSCREEN_README*
                  --- POSTSCREEN_README 2013-04-12 03:34:16.000000000 +0000
                  +++ POSTSCREEN_README.new 2013-04-24 21:04:06.155395154 +0000
                  @@ -245,6 +245,7 @@

                  * Pregreet test
                  * DNS White/blacklist test
                  + * Skipping other tests for whitelisted clients
                  * When tests fail before the 220 SMTP server greeting

                  Pregreet test
                  @@ -315,6 +316,17 @@
                  the combined DNSBL score is equal to or greater than the threshold. See "When
                  tests fail before the 220 SMTP server greeting" below.

                  +Skipping other tests for whitelisted clients
                  +
                  +The postscreen_skip_tests parameter lists the short names of tests which will
                  +be skipped if a client's combined DNSBL score is less than or equal to
                  +postscreen_skip_tests_threshold. This only makes sense when using whitelists
                  +with negative weights in the postscreen_dnsbl_sites list.
                  +
                  +The tests which can be skipped are all but the DNSBL test itself. The default
                  +is to perform the blacklist and MX policy tests, but skip the greet test and
                  +all the "deep protocol" tests, described below.
                  +
                  When tests fail before the 220 SMTP server greeting

                  When the client address matches the permanent blacklist, or when the client
                  @@ -612,6 +624,7 @@
                  postscreen_dnsbl_threshold = 2
                  postscreen_dnsbl_sites = zen.spamhaus.org*2
                  bl.spamcop.net*1 b.barracudacentral.org*1
                  + list.dnswl.org*-1 swl.spamhaus.org*-1

                  Note: if your DNSBL queries have a "secret" in the domain name, you must
                  censor this information from the postscreen(8) SMTP replies. For example:
                  --
                  http://rob0.nodns4.us/ -- system administration and consulting
                  Offlist GMX mail is seen only if "/dev/rob0" is in the Subject:
                • Wietse Venema
                  ... This will not skip (or otherwise overrule) the *static* access list that is queried before DNS lookup. Normally a client must pass *dynamic* test X to
                  Message 8 of 9 , Apr 24, 2013
                  • 0 Attachment
                    /dev/rob0:
                    > Here's a proposed diff for the POSTSCREEN_README:
                    >
                    > rob0@harrier:~/stuff/postscreen.dnswl$ diff -Nru POSTSCREEN_README*
                    > --- POSTSCREEN_README 2013-04-12 03:34:16.000000000 +0000
                    > +++ POSTSCREEN_README.new 2013-04-24 21:04:06.155395154 +0000
                    > @@ -245,6 +245,7 @@
                    >
                    > * Pregreet test
                    > * DNS White/blacklist test
                    > + * Skipping other tests for whitelisted clients

                    This will not "skip" (or otherwise overrule) the *static* access
                    list that is queried before DNS lookup.

                    Normally a client must "pass *dynamic* test X" to become temporarily
                    whitelisted for test X. Instead of passing test X, the new feature
                    temporarily whitelists the client for test X based on DNSBL score.

                    In theory there could be different DNSBL thresholds for different
                    tests. But I expect that no sane person would use Postfix that
                    way.

                    Wietse
                  Your message has been successfully submitted and would be delivered to recipients shortly.