Loading ...
Sorry, an error occurred while loading the content.

Re: Three trivial filtering questions

Expand Messages
  • Ronald F. Guilmette
    In message , ... Correct me if I m wrong, but I think you just made the case for the value of such a feature. ... Some. ...
    Message 1 of 18 , Aug 5, 2013
    • 0 Attachment
      In message <51FF9E18.9050701@...>,
      Noel Jones <njones@...> wrote:

      >I use a pcre table to reject any HELO that starts with a bracket or
      >looks like an IP. Legit hosts that use this form are very rare here
      >-- maybe one every couple years.
      >...
      >There is no built-in postfix restriction to compare the HELO to the
      >client hostname, and I would question the value of such a feature.

      Correct me if I'm wrong, but I think you just made the case for
      the value of such a feature.

      >Do you see lots of spam with incorrect IP in the HELO?

      Some.

      >Do you see
      >significant numbers of legit hosts using a bracketed IP HELO?

      None so far.


      Regards,
      rfg
    • Noel Jones
      ... No. Here, near-zero legit clients use bracketed HELO. Looks as if I ve whitelisted 2 clients in the last ~5 years (I see one of them has fixed their HELO
      Message 2 of 18 , Aug 5, 2013
      • 0 Attachment
        On 8/5/2013 12:54 PM, Ronald F. Guilmette wrote:
        > In message <51FF9E18.9050701@...>,
        > Noel Jones <njones@...> wrote:
        >
        >> I use a pcre table to reject any HELO that starts with a bracket or
        >> looks like an IP. Legit hosts that use this form are very rare here
        >> -- maybe one every couple years.
        >> ...
        >> There is no built-in postfix restriction to compare the HELO to the
        >> client hostname, and I would question the value of such a feature.
        >
        > Correct me if I'm wrong, but I think you just made the case for
        > the value of such a feature.

        No. Here, near-zero legit clients use bracketed HELO. Looks as if
        I've whitelisted 2 clients in the last ~5 years (I see one of them
        has fixed their HELO sometime since then). That's close enough to
        zero for me.

        My solution is to reject everyone that has a bracketed IP in the
        HELO, using a simple check_helo_access pcre map. I don't care if a
        spambot is RFC compliant, I still don't want their mail.

        I see zero value in testing to see if the HELO IP is forged, since
        using any IP seems to be a very strong spambot indicator.

        I know my spam is not your spam, so maybe you see something
        different. Provide some evidence if you think this is useful.

        To make a case that any new feature is needed, it must be of
        widespread benefit, and provide something that cannot (easily) be
        done using existing tools. Including sample code and documentation
        helps.


        I will note that I'm referring to random internet clients and not
        authorized SMTP AUTH or mynetworks clients. Desktop mail clients
        send all manner of cruft as their HELO, and doing *any* kind of HELO
        tests on authorized clients is foolish.


        >> Do you see
        >> significant numbers of legit hosts using a bracketed IP HELO?
        >
        > None so far.

        The defense rests.


        >
        >
        > Regards,
        > rfg
        >



        -- Noel Jones
      • Ronald F. Guilmette
        In message , ... I agree. ... We appear to be in violent agreement. ... OK. Works for me! I just wish that it wasn t
        Message 3 of 18 , Aug 5, 2013
        • 0 Attachment
          In message <51FFF9C5.9070409@...>,
          Noel Jones <njones@...> wrote:

          >No. Here, near-zero legit clients use bracketed HELO. Looks as if
          >I've whitelisted 2 clients in the last ~5 years (I see one of them
          >has fixed their HELO sometime since then). That's close enough to
          >zero for me.

          I agree.

          >My solution is to reject everyone that has a bracketed IP in the
          >HELO, using a simple check_helo_access pcre map. I don't care if a
          >spambot is RFC compliant, I still don't want their mail.

          We appear to be in violent agreement.

          >I see zero value in testing to see if the HELO IP is forged, since
          >using any IP seems to be a very strong spambot indicator.

          OK. Works for me! I just wish that it wasn't necessary to
          have to run an external PCRE to catch it, and that the
          reject_non_fqdn_helo_hostname verb actually did what it's name
          intutively implies, and what the documentation says it does.

          [A.B.C.D] is distinctly _not_ an FQDN.


          Regards,
          rfg
        • Noel Jones
          ... I can see where one might get confused. I ll submit a one-line doc patch rather than argue the point. -- Noel Jones
          Message 4 of 18 , Aug 5, 2013
          • 0 Attachment
            On 8/5/2013 4:16 PM, Ronald F. Guilmette wrote:

            >> I see zero value in testing to see if the HELO IP is forged, since
            >> using any IP seems to be a very strong spambot indicator.
            >
            > OK. Works for me! I just wish that it wasn't necessary to
            > have to run an external PCRE to catch it, and that the
            > reject_non_fqdn_helo_hostname verb actually did what it's name
            > intutively implies, and what the documentation says it does.
            >
            > [A.B.C.D] is distinctly _not_ an FQDN.

            I can see where one might get confused. I'll submit a one-line doc
            patch rather than argue the point.


            -- Noel Jones


            >
            >
            > Regards,
            > rfg
            >
          • Ronald F. Guilmette
            In message , ... I have no wish to belabor the point either, however for the record I am not confused. An FQDN is an FQDN,
            Message 5 of 18 , Aug 5, 2013
            • 0 Attachment
              In message <520023B2.1070709@...>,
              Noel Jones <njones@...> wrote:

              >On 8/5/2013 4:16 PM, Ronald F. Guilmette wrote:
              >
              >>> I see zero value in testing to see if the HELO IP is forged, since
              >>> using any IP seems to be a very strong spambot indicator.
              >>
              >> OK. Works for me! I just wish that it wasn't necessary to
              >> have to run an external PCRE to catch it, and that the
              >> reject_non_fqdn_helo_hostname verb actually did what it's name
              >> intutively implies, and what the documentation says it does.
              >>
              >> [A.B.C.D] is distinctly _not_ an FQDN.
              >
              >I can see where one might get confused. I'll submit a one-line doc
              >patch rather than argue the point.

              I have no wish to belabor the point either, however for the record
              I am not confused. An FQDN is an FQDN, and [A.B.C.D] quite certainly
              isn't one.

              I don't know much but I do know that.


              Regards,
              rfg
            • Stan Hoeppner
              ... Fair enough. In the mean time I ll share my experience with dnsbl and rhsbl restrictions. YMMV. I use the following *BL restrictions after all other
              Message 6 of 18 , Aug 6, 2013
              • 0 Attachment
                On 8/5/2013 2:52 AM, Ronald F. Guilmette wrote:

                > Actually, having adjusted my smtpd_recipient_restrictions rather
                > dramatically today, and looking now at the day's maillog file,
                > I think that I am entirely less sure that the problem is what
                > I said it was earlier. I am now getting at least _some_
                > rejects based on SURBL and URIBL, whereas earlier I had not
                > yet seen any at all in the maillog.
                >
                > I think that I should ask everyone to ignore my earlier question
                > about this until I have time to gather more data and see if I can
                > see what's actually going on. Perhaps there's nothing wrong at
                > all!

                Fair enough. In the mean time I'll share my experience with dnsbl and
                rhsbl restrictions. YMMV. I use the following *BL restrictions after
                all other main.cf restrictions, which is why the absolute numbers are so
                low. RBL checks are the most latency expensive built-in smtpd
                restrictions so I evaluate them last.

                reject_rbl_client zen.spamhaus.org
                reject_rbl_client b.barracudacentral.org
                reject_rhsbl_reverse_client dbl.spamhaus.org
                reject_rhsbl_sender dbl.spamhaus.org
                reject_rhsbl_helo dbl.spamhaus.org

                'gcml' is my simple keystroke saving script that greps the mail log for
                an input string.

                ~$ gcml reject|wc -l
                9957
                ~$ gcml reject|grep zen|wc -l
                149
                ~$ gcml reject|grep barracudacentral|wc -l
                118
                ~$ gcml reject|grep dbl|grep Unverified|wc -l
                15
                ~$ gcml reject|grep dbl|grep 'Sender address'|wc -l
                30

                Here we see the relative effectiveness of *BLs in general on this MX,
                accounting for only 3% of rejections. rhsbls account for 0.3% of
                rejections. 45 of 9957 messages may seem not worth the effort, but
                that's 45 fewer msgs going through Spamassassin spamd. Body scanning
                those 45 eats far more resources than rejecting the other 9912
                connections combined, so for me it's worth the extra 3 lines in main.cf.


                On 8/5/2013 3:11 AM, Ronald F. Guilmette wrote:

                > Last, first, does the order make any difference in the end?


                I'm responding a bit out of context to what you were asking with this in
                your other post, but the question is a perfect lead-in to demonstrate
                why main.cf restriction order matters.

                Note above that reject_rhsbl_helo didn't reject a single connection.
                This is most likely due to the fact that snowshoe spammers using $domain
                in HELO also use $domain in the rDNS string and/or the envelop sender
                address. So if we rearrange the restriction order to something like this

                reject_rhsbl_helo dbl.spamhaus.org
                reject_rhsbl_reverse_client dbl.spamhaus.org
                reject_rhsbl_sender dbl.spamhaus.org
                reject_rbl_client zen.spamhaus.org
                reject_rbl_client b.barracudacentral.org

                then we'll likely see the rhsbl checks rejecting many more connections.
                If I put this entire block near the top of my restrictions the numbers
                would skyrocket with a much higher percent of that 9957 being rejected
                by *BL Restrictions.

                With all restrictions under smtpd_recipient_restrictions they are
                processed in strict order. The first match wins, so if the rhsbl_helo
                check evaluates to true the msg is rejected and no other smtpd
                restrictions below it are processed. This strict ordering currently
                allows 97% of the spam rejected in smtpd to be killed off with the
                fewest resources consumed, first via inbuilt Postfix checks, then by my
                various custom tables, and finally by *BL restrictions. Any spam that
                runs this smtpd restriction gauntlet is then rejected by header_checks
                and if not enters the queue and is piped to spamd for tagging.

                This strict evaluation ordering allows one to optimize resource
                consumption per spam connection by using the least expensive
                restrictions first and most expensive last. This is not only critical
                for very high volume MTAs, but also for low power machines or those that
                serve multiple functions such as SOHO servers. *BL checks won't
                introduce sufficient latency to cause serious delays on the latter class
                of machines, but they certainly can on high volume MTAs, even if using a
                local resolver and/or rbldnsd. Obviously it's best to reject spam
                connections with the least number of DNS queries possible, from a
                latency/throughput and/or resource consumption perspective.

                I'm sure already know most of this Ron. I'm using this opportunity to
                get a recent post into the archives covering these topics, hence the
                subject change with added search keywords.

                --
                Stan
              • Stan Hoeppner
                ... PCRE tables don t run externally . They re simply tables. Postfix smtpd processes them by calling libpcre to which it is linked, just as it is linked
                Message 7 of 18 , Aug 6, 2013
                • 0 Attachment
                  On 8/5/2013 6:16 PM, Ronald F. Guilmette wrote:
                  > In message <520023B2.1070709@...>,
                  > Noel Jones <njones@...> wrote:

                  >>> OK. Works for me! I just wish that it wasn't necessary to
                  >>> have to run an external PCRE to catch it, and that the

                  PCRE tables don't "run externally". They're simply tables. Postfix
                  smtpd processes them by calling libpcre to which it is linked, just as
                  it is linked with other libraries including libc, libresolv, libssl, etc
                  that handle other functions:

                  ~$ pmap 10296
                  10296: smtpd -n smtp -t inet -u -c -o stress= -s 2 ...
                  ...
                  b7104000 200K r-x-- /lib/libpcre.so.3.12.1
                  b7136000 4K rw--- /lib/libpcre.so.3.12.1
                  b7137000 12K r-x-- /usr/lib/postfix/dict_pcre.so
                  b713a000 4K r---- /usr/lib/postfix/dict_pcre.so
                  b713b000 4K rw--- /usr/lib/postfix/dict_pcre.so
                  ...
                  b74f3000 88K r-x-- /usr/lib/libsasl2.so.2.0.23
                  b7509000 4K rw--- /usr/lib/libsasl2.so.2.0.23
                  ...
                  b773e000 16K r-x-- /var/spool/postfix/lib/libnss_dns-2.11.3.so
                  b7742000 4K r---- /var/spool/postfix/lib/libnss_dns-2.11.3.so
                  b7743000 4K rw--- /var/spool/postfix/lib/libnss_dns-2.11.3.so
                  ...
                  b7763000 184K r-x-- /usr/lib/postfix/smtpd
                  b7792000 8K r---- /usr/lib/postfix/smtpd
                  b7794000 4K rw--- /usr/lib/postfix/smtpd
                  b9178000 264K rw--- [ anon ]
                  bfd19000 132K rw--- [ stack ]
                  total 7664K

                  >> I can see where one might get confused. I'll submit a one-line doc
                  >> patch rather than argue the point.
                  >
                  > I have no wish to belabor the point either, however for the record
                  > I am not confused. An FQDN is an FQDN, and [A.B.C.D] quite certainly
                  > isn't one.

                  Indeed. You're not the first to be stumped by the discrepancy between
                  this parameter's behavior and that described in the docs. I commend
                  Noel for submitting this long overdue patch.

                  --
                  Stan
                Your message has been successfully submitted and would be delivered to recipients shortly.