Loading ...
Sorry, an error occurred while loading the content.

[feature request] Subzero postscreen/dnsblog score to bypass after-220 tests?

Expand Messages
  • /dev/rob0
    I finally got around to my upgrade to 2.11-20130405 and was watching logs. A gmail message fell afoul of the after-220 tests; each time it came from a
    Message 1 of 9 , Apr 11 10:58 PM
    View Source
    • 0 Attachment
      I finally got around to my upgrade to 2.11-20130405 and was watching
      logs. A gmail message fell afoul of the after-220 tests; each time it
      came from a different host. Each one got a "PASS NEW" and of course
      the "450 4.3.2 Service currently unavailable" rejection.

      These gmail outbounds are all listed in list.dnswl.org as 127.0.5.1,
      and I give that a negative score in my postscreen_dnsbl_sites. So
      with no offsetting DNSBL scores, these hosts all got a subzero score.
      It would be nice if we could put those whitelist scores to work, and
      not have to maintain so big of a postscreen_access_list whitelist.

      This has been a common concern among the new postscreen users I have
      talked to. Gmail in particular is troublesome with after-220 because
      they never try the lower priority MX on the same host. The first
      attempt was at 03:00 UTC tonight, the last one (of 8) was 05:45, just
      a few minutes ago, and I still apparently haven't got all the gmail
      outbounds whitelisted. :(

      So here's my idea (I think the parameter names are lousy, but it's
      the best I could come up with this late at night):


      """
      postscreen_after_220_bypass_enable (default: no)

      Allow a remote SMTP client with a score less than or equal to
      postscreen_after_220_bypass_threshold based on its combined
      DNSBL score as defined with the postscreen_dnsbl_sites
      parameter, to bypass the after-220 tests, if enabled. Those
      tests include postscreen_bare_newline_enable,
      postscreen_non_smtp_command_enable, and
      postscreen_pipelining_enable.

      If enabled, this means that whitelisted hosts would get to
      talk directly to a real Postfix SMTP server, if all other
      pre-220 tests are passed. For examples, see the
      POSTSCREEN_README.

      This feature is available in Postfix 2.11.

      postscreen_after_220_bypass_threshold (default: -1)

      The inclusive upper bound for allowing a remote SMTP client,
      based on its combined DNSBL score as defined with the
      postscreen_dnsbl_sites parameter, to bypass the after-220
      tests, if those tests are enabled and the
      postscreen_after_220_bypass_enable parameter is "yes".

      This feature is available in Postfix 2.11.
      """

      For reference, my postscreen settings are online here:
      http://rob0.nodns4.us/postscreen.html
      (I'm planning to maintain that page as an example configuration.)

      Some questions remain: will the whitelist result give these hosts an
      entry in the after-220 databases? Or would the pre-220 DNSBL test be
      done every time?
      --
      http://rob0.nodns4.us/ -- system administration and consulting
      Offlist GMX mail is seen only if "/dev/rob0" is in the Subject:
    • Stan Hoeppner
      On 4/12/2013 12:58 AM, /dev/rob0 wrote: ... ... Or.... Maybe you could bash script this: dig +short txt _netblocks.google.com|sed s/ip4://g ... which yields
      Message 2 of 9 , Apr 12 2:39 AM
      View Source
      • 0 Attachment
        On 4/12/2013 12:58 AM, /dev/rob0 wrote:
        ...
        > So here's my idea (I think the parameter names are lousy, but it's
        > the best I could come up with this late at night):
        ...

        Or....

        Maybe you could bash script this:

        dig +short txt _netblocks.google.com|sed s/ip4://g \
        |mawk '{for(i=2; i<=(NF-1); i++){print($i)}}'

        which yields this formatted list of Google outbound CIDRs:

        216.239.32.0/19
        64.233.160.0/19
        66.249.80.0/20
        72.14.192.0/18
        209.85.128.0/17
        66.102.0.0/20
        74.125.0.0/16
        64.18.0.0/20
        207.126.144.0/20
        173.194.0.0/16

        then diff this against your postscreen whitelist and append any new
        entries. You'd cron this to a $suitable_interval, say nightly. If/when
        Google adds any new outbound networks you're covered.

        This seems quite a bit less effort than Wietse adding the feature you
        requested. The end result is nearly identical, at least for the Google
        case, and can easily be extended to cover other domains. And with this
        method the Google outbounds skip all Postscreen processing entirely, not
        just the after 220 tests.

        --
        Stan
      • Wietse Venema
        ... Disabling tests based on DNSWL score would make sense (currently they disable DNSBL tests only). Perhaps this needs a disable flag in the postscreen
        Message 3 of 9 , Apr 12 3:34 AM
        View Source
        • 0 Attachment
          /dev/rob0:
          > I finally got around to my upgrade to 2.11-20130405 and was watching
          > logs. A gmail message fell afoul of the after-220 tests; each time it
          > came from a different host. Each one got a "PASS NEW" and of course
          > the "450 4.3.2 Service currently unavailable" rejection.
          >
          > These gmail outbounds are all listed in list.dnswl.org as 127.0.5.1,
          > and I give that a negative score in my postscreen_dnsbl_sites. So
          > with no offsetting DNSBL scores, these hosts all got a subzero score.
          > It would be nice if we could put those whitelist scores to work, and
          > not have to maintain so big of a postscreen_access_list whitelist.

          Disabling tests based on DNSWL score would make sense (currently
          they "disable" DNSBL tests only). Perhaps this needs a "disable"
          flag in the postscreen cache.

          Wietse
        • /dev/rob0
          On Fri, Apr 12, 2013 at 04:39:29AM -0500, Stan Hoeppner wrote ... I did think of this, and yes, it would save us the pain which seems to hit every 30 days, as
          Message 4 of 9 , Apr 12 7:52 AM
          View Source
          • 0 Attachment
            On Fri, Apr 12, 2013 at 04:39:29AM -0500, Stan Hoeppner wrote
            Re: scripting a list of Google outbound CIDRs:
            > This seems quite a bit less effort than Wietse adding the feature
            > you requested. The end result is nearly identical, at least for
            > the Google case, and can easily be extended to cover other domains.

            I did think of this, and yes, it would save us the pain which seems
            to hit every 30 days, as the after-220 tests for gmail expire. But
            extending it to cover other domains would not scale well. Which
            domains? What's the structure of their SPF records?

            When you "easily extend" this idea it becomes much more onerous. And
            still sitting out there are those unused DNSWL scores.

            Yes, unused. As it stands I could drop those checks from my config
            without noticing a change. There is very little overlap between the
            DNSWLs (I currently use SWL and dnswl.org) and reasonable, well-run
            DNSBLs. In my experience a few of the spamtrap-driven automated
            DNSBLs occasionally list a dnswl.org whitelisted host, but I don't
            recall having seen an instance where whitelisting prevented a
            rejection. And I have never found a blacklist entry for the (much
            smaller, I think) SWL zone.

            A DNSWL entry says two things:
            1. This is a real MTA, not a zombie
            2. At one point someone trustworthy thought it was not
            spammer-controlled

            Case 1 mostly entitles it to speak to smtpd, unless of course
            offsetting DNSBL scores overcome the whitelist score. By continuing
            on to check DNSBLs, Case 2 is addressed.

            I believe that DNS-based whitelisting will grow in importance,
            especially in the IPv6 world. I expect to move into IPv6 with a
            default-deny policy, where non-whitelisted hosts are rejected.

            > And with this method the Google outbounds skip all Postscreen
            > processing entirely, not just the after 220 tests.

            I wouldn't want that. :) If one of these providers is seriously
            compromised, they'll be blacklisted, and I would want to check for
            that. I don't give Google my absolute trust. I think they may have
            improved, but I know they're not infallible.
            --
            http://rob0.nodns4.us/ -- system administration and consulting
            Offlist GMX mail is seen only if "/dev/rob0" is in the Subject:
          • Reindl Harald
            ... how do you imagine this working? in this case it would be better you stay at ipv4 at all instead answer AAA dns-requests which may be preferred from
            Message 5 of 9 , Apr 12 8:02 AM
            View Source
            • 0 Attachment
              Am 12.04.2013 16:52, schrieb /dev/rob0:
              > I believe that DNS-based whitelisting will grow in importance,
              > especially in the IPv6 world. I expect to move into IPv6 with a
              > default-deny policy, where non-whitelisted hosts are rejected

              how do you imagine this working?

              in this case it would be better you stay at ipv4 at all instead
              answer AAA dns-requests which may be preferred from dual-stack
              machines try to deliver to your customer

              it does not work that anybody who wants to send you e-mail he
              must prove that he is no spammer, really this does not work
            • Wietse Venema
              ... On second consideration, this can be done as follows: - One parameter with the (negative) postscreen_dnsbl_sites score that is needed to allow the client
              Message 6 of 9 , Apr 23 5:05 PM
              View Source
              • 0 Attachment
                On Fri, Apr 12, 2013 at 06:34:24AM -0400, Wietse Venema wrote:
                > /dev/rob0:
                > > I finally got around to my upgrade to 2.11-20130405 and was watching
                > > logs. A gmail message fell afoul of the after-220 tests; each time it
                > > came from a different host. Each one got a "PASS NEW" and of course
                > > the "450 4.3.2 Service currently unavailable" rejection.
                > >
                > > These gmail outbounds are all listed in list.dnswl.org as 127.0.5.1,
                > > and I give that a negative score in my postscreen_dnsbl_sites. So
                > > with no offsetting DNSBL scores, these hosts all got a subzero score.
                > > It would be nice if we could put those whitelist scores to work, and
                > > not have to maintain so big of a postscreen_access_list whitelist.
                >
                > Disabling tests based on DNSWL score would make sense (currently
                > they "disable" DNSBL tests only). Perhaps this needs a "disable"
                > flag in the postscreen cache.

                On second consideration, this can be done as follows:

                - One parameter with the (negative) postscreen_dnsbl_sites score
                that is needed to allow the client to skip tests.

                - One parameter with the names of tests that are skipped (using
                !name to exclude a name, and static:all to match everything).
                This may include "greet" to cancel a "greet wait" in progress.

                The procedure is: postscreen does a postscreen_dnsbl_sites query
                for the client IP address. If the score satifies the threshold in
                the first parameter, then all tests with a name that matches the
                second parameter will be skipped until the next postscreen_dnsbl_sites
                query for that client IP address (i.e. after postscreen_dnsbl_ttl).

                Wietse
              • /dev/rob0
                ... postscreen_skip_tests_threshold , or should there be a _dnsbl or _dnswl in there? postscreen_dnsbl_skip_tests_threshold is good because it lumps it in
                Message 7 of 9 , Apr 23 9:59 PM
                View Source
                • 0 Attachment
                  On Tue, Apr 23, 2013 at 08:05:34PM -0400, Wietse Venema wrote:
                  > On Fri, Apr 12, 2013 at 06:34:24AM -0400, Wietse Venema wrote:
                  > > /dev/rob0:
                  > > > I finally got around to my upgrade to 2.11-20130405 and was
                  > > > watching logs. A gmail message fell afoul of the after-220
                  > > > tests; each time it came from a different host. Each one got
                  > > > a "PASS NEW" and of course the "450 4.3.2 Service currently
                  > > > unavailable" rejection.
                  > > >
                  > > > These gmail outbounds are all listed in list.dnswl.org as
                  > > > 127.0.5.1, and I give that a negative score in my
                  > > > postscreen_dnsbl_sites. So with no offsetting DNSBL scores,
                  > > > these hosts all got a subzero score. It would be nice if we
                  > > > could put those whitelist scores to work, and not have to
                  > > > maintain so big of a postscreen_access_list whitelist.
                  > >
                  > > Disabling tests based on DNSWL score would make sense (currently
                  > > they "disable" DNSBL tests only). Perhaps this needs a "disable"
                  > > flag in the postscreen cache.
                  >
                  > On second consideration, this can be done as follows:
                  >
                  > - One parameter with the (negative) postscreen_dnsbl_sites score
                  > that is needed to allow the client to skip tests.

                  "postscreen_skip_tests_threshold", or should there be a _dnsbl or
                  _dnswl in there? "postscreen_dnsbl_skip_tests_threshold" is good
                  because it lumps it in with the other postscreen_dnsbl_* settings
                  and makes it clear that this is associated with the DNSBL lookups.
                  OTOH _dnswl almost does this too, and it's more accurate. To see how
                  it looks: "postscreen_dnswl_skip_tests_threshold".

                  > - One parameter with the names of tests that are skipped (using
                  > !name to exclude a name, and static:all to match everything).
                  > This may include "greet" to cancel a "greet wait" in progress.

                  As for "type:name", like postscreen_access_list, you're going to want
                  to discourage lookups which might slow this down. I guess only pcre,
                  regexp, static, and texthash would be suitable for this? I did not
                  include such a warning in the proposed documentation below.

                  I think you might want to consider an "all", because that's used in
                  numerous other places and meets the "minimum surprise" ideal.


                  * Postscreen Skippable Test Names *

                  before-220 after-220
                  ========== =========
                  bare_newline
                  blacklist
                  greet
                  mx_policy
                  non_smtp_command
                  pipelining
                  static:all


                  I don't think there's much point in skipping the blacklist test. It
                  would be very strange if the postscreen_access_list lookup came in
                  after the DNSBL lookups. Furthermore, if I listed something there
                  that I don't want to see, it should never pass. But maybe someone
                  would want this? A test can be "skipped" even if already completed.
                  (But what if the postscreen_access_list result is reject? Shouldn't
                  that be done immediately, before the DNSBL lookups are in?)

                  Likewise, I think the mx_policy (postscreen_whitelist_interfaces)
                  test should also be absolute. If a client is not connecting on the
                  proper IP address, this should be cause for at least having it talk
                  to postscreen and try again later. But again, maybe someone would
                  trust the DNS whitelists' judgment?

                  I like the idea of two umbrella categories, before-220 and after-220,
                  in the spirit of inet_interfaces' "all" and "loopback-only". But
                  there's only three per category, so this is not major at this point.
                  (I suppose in the future more tests could be added.)

                  I think the default should be either "after-220" or "greet,
                  after-220". Typically the result would come in during a greet pause,
                  and even though it's only a few seconds, it can add up in the Big
                  Scheme of Things.

                  > The procedure is: postscreen does a postscreen_dnsbl_sites query
                  > for the client IP address. If the score satifies the threshold
                  > in the first parameter, then all tests with a name that matches
                  > the second parameter will be skipped until the next
                  > postscreen_dnsbl_sites query for that client IP address (i.e.
                  > after postscreen_dnsbl_ttl).

                  """
                  postscreen_skip_tests (default: greet, after-220)

                  Allow a remote SMTP client with a score less than or equal to
                  postscreen_skip_tests_threshold based on its combined DNSBL
                  score as defined with the postscreen_dnsbl_sites parameter,
                  to skip the listed tests, if enabled. Specify zero or more of
                  blacklist, greet, mx_policy (these three collectively can be
                  "before-220"), bare_newline, non_smtp_command, pipelining
                  (these three collectively can be "after-220"), or
                  "static:all" to skip all postscreen tests except for the
                  DNSBL test itself. Specify "!pattern" to exclude a test from
                  the list.

                  Example:

                  /etc/postfix/main.cf:
                  postscreen_dnsbl_sites = dnsbl.example.org,
                  whitelist.example.com*-1
                  postscreen_skip_tests = !blacklist, !mx_policy,
                  static:all

                  This feature is available in Postfix 2.11.

                  postscreen_skip_tests_threshold (default: -1)

                  The inclusive upper bound for allowing a remote SMTP client,
                  based on its combined DNSBL score as defined with the
                  postscreen_dnsbl_sites parameter, to bypass the tests listed
                  in the postscreen_skip_tests parameter.

                  Note: this typically would be a negative number, and it only
                  makes sense when using DNS whitelists with negative weights
                  in the postscreen_dnsbl_sites list. See the example at
                  postscreen_skip_tests.

                  This feature is available in Postfix 2.11.
                  """

                  I hope this is getting closer? The only point of confusion about it
                  in my mind is whether/how to skip the blacklist test. Should
                  postscreen, knowing "blacklist" is in the postscreen_skip_tests list,
                  await the dnsblog results for a blacklisted client? Why? It's
                  certainly not going to hold up a postscreen_access_list "permit"
                  client.

                  Thanks again for considering this.
                  --
                  http://rob0.nodns4.us/ -- system administration and consulting
                  Offlist GMX mail is seen only if "/dev/rob0" is in the Subject:
                • /dev/rob0
                  Here s a proposed diff for the POSTSCREEN_README: rob0@harrier:~/stuff/postscreen.dnswl$ diff -Nru POSTSCREEN_README* ... +++ POSTSCREEN_README.new
                  Message 8 of 9 , Apr 24 3:17 PM
                  View Source
                  • 0 Attachment
                    Here's a proposed diff for the POSTSCREEN_README:

                    rob0@harrier:~/stuff/postscreen.dnswl$ diff -Nru POSTSCREEN_README*
                    --- POSTSCREEN_README 2013-04-12 03:34:16.000000000 +0000
                    +++ POSTSCREEN_README.new 2013-04-24 21:04:06.155395154 +0000
                    @@ -245,6 +245,7 @@

                    * Pregreet test
                    * DNS White/blacklist test
                    + * Skipping other tests for whitelisted clients
                    * When tests fail before the 220 SMTP server greeting

                    Pregreet test
                    @@ -315,6 +316,17 @@
                    the combined DNSBL score is equal to or greater than the threshold. See "When
                    tests fail before the 220 SMTP server greeting" below.

                    +Skipping other tests for whitelisted clients
                    +
                    +The postscreen_skip_tests parameter lists the short names of tests which will
                    +be skipped if a client's combined DNSBL score is less than or equal to
                    +postscreen_skip_tests_threshold. This only makes sense when using whitelists
                    +with negative weights in the postscreen_dnsbl_sites list.
                    +
                    +The tests which can be skipped are all but the DNSBL test itself. The default
                    +is to perform the blacklist and MX policy tests, but skip the greet test and
                    +all the "deep protocol" tests, described below.
                    +
                    When tests fail before the 220 SMTP server greeting

                    When the client address matches the permanent blacklist, or when the client
                    @@ -612,6 +624,7 @@
                    postscreen_dnsbl_threshold = 2
                    postscreen_dnsbl_sites = zen.spamhaus.org*2
                    bl.spamcop.net*1 b.barracudacentral.org*1
                    + list.dnswl.org*-1 swl.spamhaus.org*-1

                    Note: if your DNSBL queries have a "secret" in the domain name, you must
                    censor this information from the postscreen(8) SMTP replies. For example:
                    --
                    http://rob0.nodns4.us/ -- system administration and consulting
                    Offlist GMX mail is seen only if "/dev/rob0" is in the Subject:
                  • Wietse Venema
                    ... This will not skip (or otherwise overrule) the *static* access list that is queried before DNS lookup. Normally a client must pass *dynamic* test X to
                    Message 9 of 9 , Apr 24 4:28 PM
                    View Source
                    • 0 Attachment
                      /dev/rob0:
                      > Here's a proposed diff for the POSTSCREEN_README:
                      >
                      > rob0@harrier:~/stuff/postscreen.dnswl$ diff -Nru POSTSCREEN_README*
                      > --- POSTSCREEN_README 2013-04-12 03:34:16.000000000 +0000
                      > +++ POSTSCREEN_README.new 2013-04-24 21:04:06.155395154 +0000
                      > @@ -245,6 +245,7 @@
                      >
                      > * Pregreet test
                      > * DNS White/blacklist test
                      > + * Skipping other tests for whitelisted clients

                      This will not "skip" (or otherwise overrule) the *static* access
                      list that is queried before DNS lookup.

                      Normally a client must "pass *dynamic* test X" to become temporarily
                      whitelisted for test X. Instead of passing test X, the new feature
                      temporarily whitelists the client for test X based on DNSBL score.

                      In theory there could be different DNSBL thresholds for different
                      tests. But I expect that no sane person would use Postfix that
                      way.

                      Wietse
                    Your message has been successfully submitted and would be delivered to recipients shortly.