Loading ...
Sorry, an error occurred while loading the content.

Re: Three trivial filtering questions

Expand Messages
  • Stan Hoeppner
    ... BTW, if you want to maximize potential hits on RHSBLs just short of doing body checks, you may want to give Sahil Tandon s TCP server based RHSBL header
    Message 1 of 18 , Aug 4, 2013
    • 0 Attachment
      On 8/4/2013 10:13 PM, Ronald F. Guilmette wrote:

      > Do I need to use that if I want to perform RHSBL checks?

      BTW, if you want to maximize potential hits on RHSBLs just short of
      doing body checks, you may want to give Sahil Tandon's TCP server based
      RHSBL header checker a spin. It grabs domains from headers and checks
      them against the 3 most popular RHSBLs: DBL, SURBL, and URIBL. You can
      add or subtract from the default server list if you choose.

      http://www.hardwarefreak.com/checkdbl.pl.txt

      He may have a newer version somewhere. This is the last version I used.
      I quit using it because it wasn't stopping much with my small mail
      stream, after all of my smtpd restrictions. With larger flows it may be
      useful for catching some that may otherwise make it to your content
      filter, or make it through entirely if you don't use one.

      --
      Stan
    • Ronald F. Guilmette
      In message , ... As I mentioned, I am not using postscreen at the present time. ... Nope. But thank you for mentioning
      Message 2 of 18 , Aug 5, 2013
      • 0 Attachment
        In message <51FF1BBA.9000904@...>,
        Stan Hoeppner <stan@...> wrote:

        >> Doing RBL client checks in postscreen?
        >
        >That would be one cause.

        As I mentioned, I am not using postscreen at the present time.

        >Another could be having duplicate
        >reject_rbl_client statements in smtpd_client_restrictions.

        Nope. But thank you for mentioning that possibility.

        Actually, having adjusted my smtpd_recipient_restrictions rather
        dramatically today, and looking now at the day's maillog file,
        I think that I am entirely less sure that the problem is what
        I said it was earlier. I am now getting at least _some_
        rejects based on SURBL and URIBL, whereas earlier I had not
        yet seen any at all in the maillog.

        I think that I should ask everyone to ignore my earlier question
        about this until I have time to gather more data and see if I can
        see what's actually going on. Perhaps there's nothing wrong at
        all!


        Regards,
        rfg
      • Ronald F. Guilmette
        In message , ... Did I say I was having a problem? There s a difference between Yea, I could probably spend half an
        Message 3 of 18 , Aug 5, 2013
        • 0 Attachment
          In message <51FF2563.1070708@...>,
          Stan Hoeppner <stan@...> wrote:

          >> If not maybe a new restriction
          >> verb would be useful to perform this exact check.
          >
          >Maybe you should explain why you're having a problem rejecting spamware
          >that HELO's with an IP literal.

          Did I say I was having a problem?

          There's a difference between "Yea, I could probably spend half an afternoon
          hacking up something external that will perform this parcticular check" and
          "There's an already-built-in Postfix verb for that."

          The latter appeals to my maximally lazy self, but the former doesn't
          quite rise to the level of something that I would characterize as
          "a problem".

          >If rejecting based on a HELO string is
          >your last line of defense you're in trouble Ron.

          Last, first, does the order make any difference in the end?

          >Surely a spamfighter
          >of your experience isn't pinning his hopes on HELO. ;)

          HELO is _very_ informative.

          In the first hour after I re-jiggered my main.cf today, I could already
          see spammers trying to HELO with [A.B.C.D]. In contrast to that, I
          personally am not aware at the present time of any serious mail server
          that I care to receive mail from that HELOs with the [A.B.C.D] style...
          even if the RFC does allow it (which we both know it does).

          (At some point, everyone running a mail server realizes that the old
          admonition to "be liberal in what you accept" has already gone the
          way of the dinoasaur some time ago.)

          >If your IP literal HELO problem is indeed bot ware, then using
          >Postscreen will stop these clients, before they have a chance to HELO.

          I don't have any data to tell me what they are, exactly, just yet, and
          actually, I don't even mind if they HELO. I'd just like the simplest
          and quickest thing to reject based on HELO with bracketed IP address,
          and I'm not real eager to work on setting up postscreen today. But
          thanks for the suggestion.

          If I can't reject on bracketed IP in HELO/EHLO then at least I would
          have expected Postfix to provide some verb which would have the effect
          of at least making sure the bracketed IP is correct. Oh well. :-(

          >> I am not using postscreen at the present time.
          >>
          >> Do I need to use that if I want to perform RHSBL checks?
          >
          >No, they are independent of one another.

          OK, good.

          >But if you want to easily stop
          >bots Postscreen is the way to go.

          Why?

          The combination of 6 or so of the best RBLs, together with SURBL, URIBL,
          and Spamhaus DBL seem to be taking care of pretty much everything as of
          now, bot or otherwise. So who am I to argue with success?

          >With your current setup and described problem you could simply remark
          >all of your reject_rbl_client statements temporarily and see if your
          >reject_rhsbl_* statements catch anything.

          Good point! I think 'll try that. Thanks!


          Regards,
          rfg
        • Ronald F. Guilmette
          In message , ... Thank you. I have just looked at that but I can t see what it does that makes it in any way superior to
          Message 4 of 18 , Aug 5, 2013
          • 0 Attachment
            In message <51FF2AD2.2080607@...>,
            Stan Hoeppner <stan@...> wrote:

            >BTW, if you want to maximize potential hits on RHSBLs just short of
            >doing body checks, you may want to give Sahil Tandon's TCP server based
            >RHSBL header checker a spin. It grabs domains from headers and checks
            >them against the 3 most popular RHSBLs: DBL, SURBL, and URIBL.

            Thank you. I have just looked at that but I can't see what it does
            that makes it in any way superior to the built-in restriction verbs
            that I can (and have) already put in main.cf.


            Regards,
            rfg
          • Noel Jones
            ... I use a pcre table to reject any HELO that starts with a bracket or looks like an IP. Legit hosts that use this form are very rare here -- maybe one every
            Message 5 of 18 , Aug 5, 2013
            • 0 Attachment
              On 8/4/2013 10:13 PM, Ronald F. Guilmette wrote:
              > In message <51FF13EB.8090105@...>,
              > Noel Jones <njones@...> wrote:
              >
              >> On 8/4/2013 8:06 PM, Ronald F. Guilmette wrote:
              >>> Does reject_non_fqdn_helo_hostname, when placed in the
              >>> smtpd_helo_restrictions, permit clients to HELO/EHLO
              >>> with a square-bracket enclosed dotted quad IPv4 address?
              >>
              >> Yes.
              >
              > The documentatation should probably be adjusted to make that more clear.
              > Right now it reads:
              >
              > Reject the request when the HELO or EHLO hostname is not in fully-
              > qualified domain form, as required by the RFC.
              >
              >>> If so, is the dotted quad checked to see that it properly
              >>> represents the actual IP address of the actual current client?
              >>
              >> No.
              >
              > Is there any restriction verb that would cause a HELO/EHLO which specifies
              > a square-bracketed dotted quad IPv4 address to be rejected when & if the
              > dotted quad does not match the actual current client IP address?

              I use a pcre table to reject any HELO that starts with a bracket or
              looks like an IP. Legit hosts that use this form are very rare here
              -- maybe one every couple years.

              >
              > Would reject_unknown_helo_hostname do it? If not maybe a new restriction
              > verb would be useful to perform this exact check.

              There is no built-in postfix restriction to compare the HELO to the
              client hostname, and I would question the value of such a feature.

              Do you see lots of spam with incorrect IP in the HELO? Do you see
              significant numbers of legit hosts using a bracketed IP HELO?


              >
              >>> Certainly, some spam
              >>> that I believe should have been rejected on the basis of one or another
              >>> of the above RHS filters I am instead seeing (in my maillog file) being
              >>> rejected instead by one or another of the subsequent reject_rbl_client
              >>> filters. What could I be doing wrong?
              >>

              You'll need too show evidence for further help on this.


              >>
              >> Doing RBL client checks in postscreen?
              >
              > I am not using postscreen at the present time.
              >
              > Do I need to use that if I want to perform RHSBL checks?

              RHSBL checks work without postscreen. If you use postscreen, it
              will reject clients before the smtpd_*_restrictions (and the smtpd
              program itself) are ever run.

              http://www.postfix.org/POSTSCREEN_README.html


              -- Noel Jones

              >
              >
              > Regards,
              > rfg
              >
            • Noel Jones
              ... The built-in restrictions can check envelope information for RBL/RHSBL listings. Sahil s lightweight TCP server can also check message headers such as
              Message 6 of 18 , Aug 5, 2013
              • 0 Attachment
                On 8/5/2013 3:16 AM, Ronald F. Guilmette wrote:
                > In message <51FF2AD2.2080607@...>,
                > Stan Hoeppner <stan@...> wrote:
                >
                >> BTW, if you want to maximize potential hits on RHSBLs just short of
                >> doing body checks, you may want to give Sahil Tandon's TCP server based
                >> RHSBL header checker a spin. It grabs domains from headers and checks
                >> them against the 3 most popular RHSBLs: DBL, SURBL, and URIBL.
                >
                > Thank you. I have just looked at that but I can't see what it does
                > that makes it in any way superior to the built-in restriction verbs
                > that I can (and have) already put in main.cf.
                >
                >
                > Regards,
                > rfg
                >


                The built-in restrictions can check envelope information for
                RBL/RHSBL listings.

                Sahil's lightweight TCP server can also check message headers such
                as Message-ID: and From: header for RHSBL listed domains. Other
                than this clever TCP table, they only other way to check these are
                with a milter or content_filter.

                This used to catch some extra spam, but hasn't been very effective
                for me lately due to changing spammer tactics. YMMV.



                -- Noel Jones
              • Ronald F. Guilmette
                In message , ... Correct me if I m wrong, but I think you just made the case for the value of such a feature. ... Some. ...
                Message 7 of 18 , Aug 5, 2013
                • 0 Attachment
                  In message <51FF9E18.9050701@...>,
                  Noel Jones <njones@...> wrote:

                  >I use a pcre table to reject any HELO that starts with a bracket or
                  >looks like an IP. Legit hosts that use this form are very rare here
                  >-- maybe one every couple years.
                  >...
                  >There is no built-in postfix restriction to compare the HELO to the
                  >client hostname, and I would question the value of such a feature.

                  Correct me if I'm wrong, but I think you just made the case for
                  the value of such a feature.

                  >Do you see lots of spam with incorrect IP in the HELO?

                  Some.

                  >Do you see
                  >significant numbers of legit hosts using a bracketed IP HELO?

                  None so far.


                  Regards,
                  rfg
                • Noel Jones
                  ... No. Here, near-zero legit clients use bracketed HELO. Looks as if I ve whitelisted 2 clients in the last ~5 years (I see one of them has fixed their HELO
                  Message 8 of 18 , Aug 5, 2013
                  • 0 Attachment
                    On 8/5/2013 12:54 PM, Ronald F. Guilmette wrote:
                    > In message <51FF9E18.9050701@...>,
                    > Noel Jones <njones@...> wrote:
                    >
                    >> I use a pcre table to reject any HELO that starts with a bracket or
                    >> looks like an IP. Legit hosts that use this form are very rare here
                    >> -- maybe one every couple years.
                    >> ...
                    >> There is no built-in postfix restriction to compare the HELO to the
                    >> client hostname, and I would question the value of such a feature.
                    >
                    > Correct me if I'm wrong, but I think you just made the case for
                    > the value of such a feature.

                    No. Here, near-zero legit clients use bracketed HELO. Looks as if
                    I've whitelisted 2 clients in the last ~5 years (I see one of them
                    has fixed their HELO sometime since then). That's close enough to
                    zero for me.

                    My solution is to reject everyone that has a bracketed IP in the
                    HELO, using a simple check_helo_access pcre map. I don't care if a
                    spambot is RFC compliant, I still don't want their mail.

                    I see zero value in testing to see if the HELO IP is forged, since
                    using any IP seems to be a very strong spambot indicator.

                    I know my spam is not your spam, so maybe you see something
                    different. Provide some evidence if you think this is useful.

                    To make a case that any new feature is needed, it must be of
                    widespread benefit, and provide something that cannot (easily) be
                    done using existing tools. Including sample code and documentation
                    helps.


                    I will note that I'm referring to random internet clients and not
                    authorized SMTP AUTH or mynetworks clients. Desktop mail clients
                    send all manner of cruft as their HELO, and doing *any* kind of HELO
                    tests on authorized clients is foolish.


                    >> Do you see
                    >> significant numbers of legit hosts using a bracketed IP HELO?
                    >
                    > None so far.

                    The defense rests.


                    >
                    >
                    > Regards,
                    > rfg
                    >



                    -- Noel Jones
                  • Ronald F. Guilmette
                    In message , ... I agree. ... We appear to be in violent agreement. ... OK. Works for me! I just wish that it wasn t
                    Message 9 of 18 , Aug 5, 2013
                    • 0 Attachment
                      In message <51FFF9C5.9070409@...>,
                      Noel Jones <njones@...> wrote:

                      >No. Here, near-zero legit clients use bracketed HELO. Looks as if
                      >I've whitelisted 2 clients in the last ~5 years (I see one of them
                      >has fixed their HELO sometime since then). That's close enough to
                      >zero for me.

                      I agree.

                      >My solution is to reject everyone that has a bracketed IP in the
                      >HELO, using a simple check_helo_access pcre map. I don't care if a
                      >spambot is RFC compliant, I still don't want their mail.

                      We appear to be in violent agreement.

                      >I see zero value in testing to see if the HELO IP is forged, since
                      >using any IP seems to be a very strong spambot indicator.

                      OK. Works for me! I just wish that it wasn't necessary to
                      have to run an external PCRE to catch it, and that the
                      reject_non_fqdn_helo_hostname verb actually did what it's name
                      intutively implies, and what the documentation says it does.

                      [A.B.C.D] is distinctly _not_ an FQDN.


                      Regards,
                      rfg
                    • Noel Jones
                      ... I can see where one might get confused. I ll submit a one-line doc patch rather than argue the point. -- Noel Jones
                      Message 10 of 18 , Aug 5, 2013
                      • 0 Attachment
                        On 8/5/2013 4:16 PM, Ronald F. Guilmette wrote:

                        >> I see zero value in testing to see if the HELO IP is forged, since
                        >> using any IP seems to be a very strong spambot indicator.
                        >
                        > OK. Works for me! I just wish that it wasn't necessary to
                        > have to run an external PCRE to catch it, and that the
                        > reject_non_fqdn_helo_hostname verb actually did what it's name
                        > intutively implies, and what the documentation says it does.
                        >
                        > [A.B.C.D] is distinctly _not_ an FQDN.

                        I can see where one might get confused. I'll submit a one-line doc
                        patch rather than argue the point.


                        -- Noel Jones


                        >
                        >
                        > Regards,
                        > rfg
                        >
                      • Ronald F. Guilmette
                        In message , ... I have no wish to belabor the point either, however for the record I am not confused. An FQDN is an FQDN,
                        Message 11 of 18 , Aug 5, 2013
                        • 0 Attachment
                          In message <520023B2.1070709@...>,
                          Noel Jones <njones@...> wrote:

                          >On 8/5/2013 4:16 PM, Ronald F. Guilmette wrote:
                          >
                          >>> I see zero value in testing to see if the HELO IP is forged, since
                          >>> using any IP seems to be a very strong spambot indicator.
                          >>
                          >> OK. Works for me! I just wish that it wasn't necessary to
                          >> have to run an external PCRE to catch it, and that the
                          >> reject_non_fqdn_helo_hostname verb actually did what it's name
                          >> intutively implies, and what the documentation says it does.
                          >>
                          >> [A.B.C.D] is distinctly _not_ an FQDN.
                          >
                          >I can see where one might get confused. I'll submit a one-line doc
                          >patch rather than argue the point.

                          I have no wish to belabor the point either, however for the record
                          I am not confused. An FQDN is an FQDN, and [A.B.C.D] quite certainly
                          isn't one.

                          I don't know much but I do know that.


                          Regards,
                          rfg
                        • Stan Hoeppner
                          ... Fair enough. In the mean time I ll share my experience with dnsbl and rhsbl restrictions. YMMV. I use the following *BL restrictions after all other
                          Message 12 of 18 , Aug 6, 2013
                          • 0 Attachment
                            On 8/5/2013 2:52 AM, Ronald F. Guilmette wrote:

                            > Actually, having adjusted my smtpd_recipient_restrictions rather
                            > dramatically today, and looking now at the day's maillog file,
                            > I think that I am entirely less sure that the problem is what
                            > I said it was earlier. I am now getting at least _some_
                            > rejects based on SURBL and URIBL, whereas earlier I had not
                            > yet seen any at all in the maillog.
                            >
                            > I think that I should ask everyone to ignore my earlier question
                            > about this until I have time to gather more data and see if I can
                            > see what's actually going on. Perhaps there's nothing wrong at
                            > all!

                            Fair enough. In the mean time I'll share my experience with dnsbl and
                            rhsbl restrictions. YMMV. I use the following *BL restrictions after
                            all other main.cf restrictions, which is why the absolute numbers are so
                            low. RBL checks are the most latency expensive built-in smtpd
                            restrictions so I evaluate them last.

                            reject_rbl_client zen.spamhaus.org
                            reject_rbl_client b.barracudacentral.org
                            reject_rhsbl_reverse_client dbl.spamhaus.org
                            reject_rhsbl_sender dbl.spamhaus.org
                            reject_rhsbl_helo dbl.spamhaus.org

                            'gcml' is my simple keystroke saving script that greps the mail log for
                            an input string.

                            ~$ gcml reject|wc -l
                            9957
                            ~$ gcml reject|grep zen|wc -l
                            149
                            ~$ gcml reject|grep barracudacentral|wc -l
                            118
                            ~$ gcml reject|grep dbl|grep Unverified|wc -l
                            15
                            ~$ gcml reject|grep dbl|grep 'Sender address'|wc -l
                            30

                            Here we see the relative effectiveness of *BLs in general on this MX,
                            accounting for only 3% of rejections. rhsbls account for 0.3% of
                            rejections. 45 of 9957 messages may seem not worth the effort, but
                            that's 45 fewer msgs going through Spamassassin spamd. Body scanning
                            those 45 eats far more resources than rejecting the other 9912
                            connections combined, so for me it's worth the extra 3 lines in main.cf.


                            On 8/5/2013 3:11 AM, Ronald F. Guilmette wrote:

                            > Last, first, does the order make any difference in the end?


                            I'm responding a bit out of context to what you were asking with this in
                            your other post, but the question is a perfect lead-in to demonstrate
                            why main.cf restriction order matters.

                            Note above that reject_rhsbl_helo didn't reject a single connection.
                            This is most likely due to the fact that snowshoe spammers using $domain
                            in HELO also use $domain in the rDNS string and/or the envelop sender
                            address. So if we rearrange the restriction order to something like this

                            reject_rhsbl_helo dbl.spamhaus.org
                            reject_rhsbl_reverse_client dbl.spamhaus.org
                            reject_rhsbl_sender dbl.spamhaus.org
                            reject_rbl_client zen.spamhaus.org
                            reject_rbl_client b.barracudacentral.org

                            then we'll likely see the rhsbl checks rejecting many more connections.
                            If I put this entire block near the top of my restrictions the numbers
                            would skyrocket with a much higher percent of that 9957 being rejected
                            by *BL Restrictions.

                            With all restrictions under smtpd_recipient_restrictions they are
                            processed in strict order. The first match wins, so if the rhsbl_helo
                            check evaluates to true the msg is rejected and no other smtpd
                            restrictions below it are processed. This strict ordering currently
                            allows 97% of the spam rejected in smtpd to be killed off with the
                            fewest resources consumed, first via inbuilt Postfix checks, then by my
                            various custom tables, and finally by *BL restrictions. Any spam that
                            runs this smtpd restriction gauntlet is then rejected by header_checks
                            and if not enters the queue and is piped to spamd for tagging.

                            This strict evaluation ordering allows one to optimize resource
                            consumption per spam connection by using the least expensive
                            restrictions first and most expensive last. This is not only critical
                            for very high volume MTAs, but also for low power machines or those that
                            serve multiple functions such as SOHO servers. *BL checks won't
                            introduce sufficient latency to cause serious delays on the latter class
                            of machines, but they certainly can on high volume MTAs, even if using a
                            local resolver and/or rbldnsd. Obviously it's best to reject spam
                            connections with the least number of DNS queries possible, from a
                            latency/throughput and/or resource consumption perspective.

                            I'm sure already know most of this Ron. I'm using this opportunity to
                            get a recent post into the archives covering these topics, hence the
                            subject change with added search keywords.

                            --
                            Stan
                          • Stan Hoeppner
                            ... PCRE tables don t run externally . They re simply tables. Postfix smtpd processes them by calling libpcre to which it is linked, just as it is linked
                            Message 13 of 18 , Aug 6, 2013
                            • 0 Attachment
                              On 8/5/2013 6:16 PM, Ronald F. Guilmette wrote:
                              > In message <520023B2.1070709@...>,
                              > Noel Jones <njones@...> wrote:

                              >>> OK. Works for me! I just wish that it wasn't necessary to
                              >>> have to run an external PCRE to catch it, and that the

                              PCRE tables don't "run externally". They're simply tables. Postfix
                              smtpd processes them by calling libpcre to which it is linked, just as
                              it is linked with other libraries including libc, libresolv, libssl, etc
                              that handle other functions:

                              ~$ pmap 10296
                              10296: smtpd -n smtp -t inet -u -c -o stress= -s 2 ...
                              ...
                              b7104000 200K r-x-- /lib/libpcre.so.3.12.1
                              b7136000 4K rw--- /lib/libpcre.so.3.12.1
                              b7137000 12K r-x-- /usr/lib/postfix/dict_pcre.so
                              b713a000 4K r---- /usr/lib/postfix/dict_pcre.so
                              b713b000 4K rw--- /usr/lib/postfix/dict_pcre.so
                              ...
                              b74f3000 88K r-x-- /usr/lib/libsasl2.so.2.0.23
                              b7509000 4K rw--- /usr/lib/libsasl2.so.2.0.23
                              ...
                              b773e000 16K r-x-- /var/spool/postfix/lib/libnss_dns-2.11.3.so
                              b7742000 4K r---- /var/spool/postfix/lib/libnss_dns-2.11.3.so
                              b7743000 4K rw--- /var/spool/postfix/lib/libnss_dns-2.11.3.so
                              ...
                              b7763000 184K r-x-- /usr/lib/postfix/smtpd
                              b7792000 8K r---- /usr/lib/postfix/smtpd
                              b7794000 4K rw--- /usr/lib/postfix/smtpd
                              b9178000 264K rw--- [ anon ]
                              bfd19000 132K rw--- [ stack ]
                              total 7664K

                              >> I can see where one might get confused. I'll submit a one-line doc
                              >> patch rather than argue the point.
                              >
                              > I have no wish to belabor the point either, however for the record
                              > I am not confused. An FQDN is an FQDN, and [A.B.C.D] quite certainly
                              > isn't one.

                              Indeed. You're not the first to be stumped by the discrepancy between
                              this parameter's behavior and that described in the docs. I commend
                              Noel for submitting this long overdue patch.

                              --
                              Stan
                            Your message has been successfully submitted and would be delivered to recipients shortly.