Loading ...
Sorry, an error occurred while loading the content.
 

Re: FILTER_README suggestions

Expand Messages
  • Wietse Venema
    ... Thanks to all who recommended the tag and deliver approach. ... If you have examples to share, it will save me time. Wietse
    Message 1 of 16 , Sep 1, 2006
      Eray Aslan:
      > On Fri, September 1, 2006 5:53 pm, Wietse Venema wrote:
      > > What do people use:
      > >
      > > - Have the filter return a distinct exit status that says "discard"?
      > >
      > > - Have the filter insert a "badness" indicator in a message header,
      > > and dispose of bad mail with Postfix HOLD/DISCARD actions, maildrop
      > > rules, cyrus sieves, or procmail filters?
      > >
      > > - Something completely different? Maybe no-one uses the pipe+sendmail
      > > example and we can drop it from the documentation.
      >
      > We use "tag & deliver" a.k.a second option. For some clients, maildrop
      > places the mail in appropriate folder. For others, we let the MUA decide
      > what to do with tagged mail.

      Thanks to all who recommended the "tag and deliver" approach.

      > Basically, quarantining is too much trouble for our end users and we don't
      > discard email for fear of false positives.
      >
      > I suggest adding some simple maildrop/procmail recipies as examples to
      > help the uninitiated.

      If you have examples to share, it will save me time.

      Wietse
    • /dev/rob0
      ... Interesting. As readily-available as your address is, I would think spam would be a major problem for you. ... Amavisd-new with local(8) and address
      Message 2 of 16 , Sep 1, 2006
        On Friday 01 September 2006 09:53, Wietse Venema wrote:
        > Postfix's FILTER_README was written long before backscatter became
        > a problem. The first example (see below signature) has a warning
        > not to reject mail:
        >
        > Note: in this time of mail worms and spam, it is a BAD IDEA to
        > send known viruses or spam back to the sender, because that
        > address is likely to be forged. It is safer to discard known
        > to be bad content and to quarantine suspicious content so that
        > it can be inspected by a human being.
        >
        > Unfortunately, the text gives no example of how one would implement
        > this advice. Personally, I use no external filter so I have a hard
        > time coming up with field-tested examples.

        Interesting. As readily-available as your address is, I would think
        spam would be a major problem for you.

        > What do people use:

        Amavisd-new with local(8) and address tagging, with ~/.forward+spam
        redirecting to a user-accessible Quarantine folder. I'm very pleased
        with the results. It's smooth and simple, no arcane and difficult
        procmail syntax. It's something I could put within the reach of an
        ordinary user, with a simple example in "/etc/skel/.forward+spam".

        I do change the amavisd-new defaults to D_PASS for all categories but
        virus (and I have not yet had a virus pass through Postfix's defenses.)

        > - Have the filter return a distinct exit status that says "discard"?
        >
        > - Have the filter insert a "badness" indicator in a message header,
        > and dispose of bad mail with Postfix HOLD/DISCARD actions, maildrop
        > rules, cyrus sieves, or procmail filters?
        >
        > - Something completely different? Maybe no-one uses the pipe+sendmail
        > example and we can drop it from the documentation.

        In actual practice I bet few sites are using the simple content filter
        example. However, it is potentially useful as a starting point for one
        who wants to roll his/her own filter.

        > 24 # Specify your content filter here.
        > 25 # filter <in.$$ || {
        > 26 # echo Message content rejected; exit $EX_UNAVAILABLE; }

        You could just expand the comments in the script to list the filter's
        choices. :)
        --
        Offlist mail to this address is discarded unless
        "/dev/rob0" or "not-spam" is in Subject: header
      • Darron Froese
        ... We reject some mail directly at SMTP (reject_unknown_sender_domain, reject_non_fqdn_sender, check_helo_access, reject_unknown_recipient_domain,
        Message 3 of 16 , Sep 1, 2006
          On 1-Sep-06, at 8:53 AM, Wietse Venema wrote:

          > Unfortunately, the text gives no example of how one would implement
          > this advice. Personally, I use no external filter so I have a hard
          > time coming up with field-tested examples.
          >
          > What do people use:

          We reject some mail directly at SMTP (reject_unknown_sender_domain,
          reject_non_fqdn_sender, check_helo_access,
          reject_unknown_recipient_domain, reject_unauth_destination,
          reject_unlisted_recipient and a manual domain name blacklist) but
          after that:

          1. Everything that matches a cable, DSL or dialup connection gets
          greylisted with tumgreyspf - configurable with a manual whitelist for
          broken mail servers.
          2. All mail with known viruses get silently discarded (amavisd-new
          and clamav)
          3. Mail gets tagged with SpamAssassin (through amavisd-new and a
          bunch of extra SA plugins)
          4. Clients have the option to discard above a certain SA score or
          filter through a web interface .

          I did up a diagram of it to figure it all out in my mind while I was
          building:

          http://nonfiction.ca/mail-arch.jpg

          Works great for us - brought down my spam levels from 300 / day with
          old Postfix and SA 2.6 down to about 2 or 3 / day.
          --
          darron froese
          principal
          nonfiction studios inc.
          t 403.686.8887
          c 403.819.7887
          f 403.313.9233
          w http://nonfiction.ca/
          e darron@...
        • Victor Duchovni
          ... I use badness headers, then pump all labeled mail into the hold queue of a downstream Postfix instance. From the hold queue messages are moved into
          Message 4 of 16 , Sep 1, 2006
            On Fri, Sep 01, 2006 at 10:53:49AM -0400, Wietse Venema wrote:

            > - Have the filter insert a "badness" indicator in a message header,
            > and dispose of bad mail with Postfix HOLD/DISCARD actions, maildrop
            > rules, cyrus sieves, or procmail filters?

            I use "badness" headers, then pump all labeled mail into the "hold" queue
            of a downstream Postfix instance. From the "hold" queue messages are
            moved into medium term storage (quarantine) and users receive periodic
            notices of quarantined mail, and are able to click on links to release
            selected messages. The tools in question are not publically available,
            perhaps someday...

            --
            Viktor.

            Disclaimer: off-list followups get on-list replies or get ignored.
            Please do not ignore the "Reply-To" header.

            To unsubscribe from the postfix-users list, visit
            http://www.postfix.org/lists.html or click the link below:
            <mailto:majordomo@...?body=unsubscribe%20postfix-users>

            If my response solves your problem, the best way to thank me is to not
            send an "it worked, thanks" follow-up. If you must respond, please put
            "It worked, thanks" in the "Subject" so I can delete these quickly.
          • Tony Earnshaw
            fr den 01.09.2006 Klokka 12:40 (-0400) skreiv Wietse Venema: [...] ... A couple of dead simple stanzas from a maildrop/dspam setup: in /etc/maildroprc ($VHOME
            Message 5 of 16 , Sep 1, 2006
              fr den 01.09.2006 Klokka 12:40 (-0400) skreiv Wietse Venema:

              [...]

              > > We use "tag & deliver" a.k.a second option. For some clients, maildrop
              > > places the mail in appropriate folder. For others, we let the MUA decide
              > > what to do with tagged mail.
              >
              > Thanks to all who recommended the "tag and deliver" approach.
              >
              > > Basically, quarantining is too much trouble for our end users and we don't
              > > discard email for fear of false positives.
              > >
              > > I suggest adding some simple maildrop/procmail recipies as examples to
              > > help the uninitiated.
              >
              > If you have examples to share, it will save me time.

              A couple of dead simple stanzas from a maildrop/dspam setup:

              in /etc/maildroprc ($VHOME is a variable defined at the head of
              maildroprc):

              `test -d $VHOME/Maildir/.dspam-quarantine/`
              if( $RETURNCODE == 1 )
              {
              `/usr/bin/maildirmake "$VHOME/Maildir/.dspam-quarantine"/`
              }
              __________________________

              if ( /^X-DSPAM-Result:[:space:]+Spam$/ )
              {
              to "$VHOME/Maildir/.dspam-quarantine"
              }

              --Tonni

              --
              Tony Earnshaw
              reservebergenser
            • Darron Froese
              ... This is a Sieve recipe that is available to our users: # Delete SpamAssassin blacklisted mails. if header :comparator i;ascii-casemap :contains
              Message 6 of 16 , Sep 1, 2006
                On 1-Sep-06, at 10:40 AM, Wietse Venema wrote:

                > If you have examples to share, it will save me time.

                This is a Sieve recipe that is available to our users:

                # Delete SpamAssassin blacklisted mails.
                if header :comparator "i;ascii-casemap" :contains "X-Spam-Status"
                "BLACKLISTED" {
                discard;
                stop;
                }
                # Delete high score SPAM above 14 SA score (configurable).
                if allof ( header :comparator "i;ascii-casemap" :matches "Subject"
                "[SPAM]*", header :value "gt" :comparator "i;ascii-numeric" "X-Spam-
                Score" ["14"] ) {
                discard;
                stop;
                }
                # Filter Lower Scored Spam for checking.
                if header :comparator "i;ascii-casemap" :matches "Subject" "[SPAM]*" {
                fileinto "INBOX.spam";
                stop;
                }

                It's their choice what they want to do for their own accounts - this
                is my personal setup - all managed via a web interface.
                --
                darron froese
                principal
                nonfiction studios inc.
                t 403.686.8887
                c 403.819.7887
                f 403.313.9233
                w http://nonfiction.ca/
                e darron@...
              • Eray Aslan
                ... Here is a sample maildroprc file: maildirmake=/path/to/maildirmake MAILDIR=$DEFAULT JUNK_FOLDER=.Spam _JUNK_DEST=$MAILDIR/$JUNK_FOLDER/ #automatically
                Message 7 of 16 , Sep 1, 2006
                  On Fri, September 1, 2006 7:40 pm, Wietse Venema wrote:
                  > If you have examples to share, it will save me time.

                  Here is a sample maildroprc file:

                  maildirmake=/path/to/maildirmake
                  MAILDIR=$DEFAULT
                  JUNK_FOLDER=.Spam

                  _JUNK_DEST=$MAILDIR/$JUNK_FOLDER/


                  #automatically create the Junk folder
                  `test -d $_JUNK_DEST`
                  if ($RETURNCODE != 0 )
                  {
                  `$maildirmake $_JUNK_DEST`
                  # auto subscribe. the following works for courier-imap
                  `echo $_JUNK_DEST >> $MAILDIR/courierimapsubscribed`
                  }

                  # Spam gets tagged with X-Spam-Flag
                  if ( /^X-Spam-Flag: YES/:h )
                  {
                  exception {
                  to "$_JUNK_DEST"
                  }
                  }
                  else
                  {
                  exception {
                  to "$MAILDIR/"
                  }
                  }

                  --
                  Eray
                • Aaron Bennett
                  ... I can t speak highly enough of Maia Mailguard. It s based on amavisd-new but allows per-user bayes training, white and blacklists. We re a middling-volume
                  Message 8 of 16 , Sep 1, 2006
                    Wietse Venema wrote:
                    > What do people use:
                    >
                    >

                    I can't speak highly enough of Maia Mailguard. It's based on
                    amavisd-new but allows per-user bayes training, white and blacklists.
                    We're a middling-volume site (about 85K messages per day) and have two
                    dual-xeon mail relays running Maia and uvscan/clamd. The database
                    component of Maia sits on a third box. In our older environment, we had
                    amavisd-new + spamassassin running without per-user training or
                    quarantining or the other benefits of Maia and as a result, we had to be
                    conservative with our SA thresholds to avoid false-positives. Our
                    number 1 complaint was spam -- since we list all or most staff and
                    faculty email on our web page. Now with Maia, we are heroes. Since
                    mid-july when we rolled it out, we've blocked over 600,000 spams with a
                    false positive rate of 0.06% and a false-negative rate of 6.57%. The
                    power is the user-training; it's easy and simple and lets users take
                    effective action against spam. It really lets SpamAssassin shine.



                    --
                    Aaron Bennett
                    Sr. Unix Systems Administrator
                    Clark University ITS
                    abennett@... | 508.781.7315
                  • Eray Aslan
                    ... For the record, here is a working maildroprc file. Please disregard the previous one. maildirmake=/usr/bin/maildirmake MAILDIR=$DEFAULT JUNK_FOLDER=.Spam
                    Message 9 of 16 , Sep 1, 2006
                      On Fri, September 1, 2006 8:30 pm, Eray Aslan wrote:
                      > On Fri, September 1, 2006 7:40 pm, Wietse Venema wrote:
                      >> If you have examples to share, it will save me time.
                      >
                      > Here is a sample maildroprc file:

                      For the record, here is a working maildroprc file. Please disregard the
                      previous one.


                      maildirmake=/usr/bin/maildirmake
                      MAILDIR=$DEFAULT
                      JUNK_FOLDER=.Spam

                      _JUNK_DEST=$MAILDIR/$JUNK_FOLDER/


                      #automatically create the Junk folder
                      `test -d $_JUNK_DEST`
                      if ($RETURNCODE != 0 )
                      {
                      `$maildirmake $_JUNK_DEST`
                      # auto subscribe. the following works for courier-imap
                      `echo INBOX$JUNK_FOLDER >> $MAILDIR/courierimapsubscribed`
                      }
                      # Spam gets tagged with X-Spam-Flag
                      if ( /^X-Spam-Flag: YES/:h )
                      {
                      exception {
                      to "$_JUNK_DEST"
                      }
                      }
                      else
                      {
                      exception {
                      to "$MAILDIR/"
                      }
                      }

                      --
                      Eray
                    • mouss
                      ... I favour tag & deliver, be that to a Junk folder or somewhere else. So I configure the filters to tag all mail (spam or not), and use these in maildrop to
                      Message 10 of 16 , Sep 1, 2006
                        Wietse Venema wrote:
                        > Postfix's FILTER_README was written long before backscatter became
                        > a problem. The first example (see below signature) has a warning
                        > not to reject mail:
                        >
                        > Note: in this time of mail worms and spam, it is a BAD IDEA to
                        > send known viruses or spam back to the sender, because that
                        > address is likely to be forged. It is safer to discard known
                        > to be bad content and to quarantine suspicious content so that
                        > it can be inspected by a human being.
                        >
                        > Unfortunately, the text gives no example of how one would implement
                        > this advice. Personally, I use no external filter so I have a hard
                        > time coming up with field-tested examples.
                        >
                        > What do people use:
                        >
                        > - Have the filter return a distinct exit status that says "discard"?
                        >
                        > - Have the filter insert a "badness" indicator in a message header,
                        > and dispose of bad mail with Postfix HOLD/DISCARD actions, maildrop
                        > rules, cyrus sieves, or procmail filters?
                        >
                        > - Something completely different? Maybe no-one uses the pipe+sendmail
                        > example and we can drop it from the documentation.
                        >
                        >

                        I favour tag & deliver, be that to a Junk folder or somewhere else. So I
                        configure the filters to tag all mail (spam or not), and use these in
                        maildrop to deliver to a Junk folder.

                        here is an example with spamassassin + courier-imap (the .folder
                        notation below) + maildrop 2.x

                        if (/^X-Spam-Flag:\s*YES/)
                        {
                        exception {
                        to "$DEFAULT/.Junk/";
                        }
                        }

                        with dspam, this would be
                        if (/^X-DSPAM-Result: Spam/)
                        ...

                        for amavisd-new banned attachments:
                        if (/^X-Amavis-Alert:\s*BANNED/)
                        ...

                        for bogofilter, one would use
                        /^X-Bogosity:\s*(\S+),.*\s+spamicity=([\d\.]+)/
                        BOGO_STATUS="${MATCH1}"
                        BOGO_SCORE="${MATCH2}"

                        and decide based on these vars (bogofilter has "unsure" result, so the
                        decision here is not binary).

                        ...

                        This may be either per-site (maildroprc) or per-user (.mailfilter).

                        PS. One can implement a "commitee" where the final disposition is a
                        function of the individual filter tags (if SA and bogo agree, you have
                        more confidence in the result, ... etc). This however needs real
                        measurements to get any justification...
                      • Jorey Bump
                        ... I tag, then reject/hold/deliver based on the score in a message header (rejecting only in a before-queue content filter). I prefer not to alter anything
                        Message 11 of 16 , Sep 1, 2006
                          Wietse Venema wrote:

                          > - Have the filter insert a "badness" indicator in a message header,
                          > and dispose of bad mail with Postfix HOLD/DISCARD actions, maildrop
                          > rules, cyrus sieves, or procmail filters?

                          I tag, then reject/hold/deliver based on the score in a message header
                          (rejecting only in a before-queue content filter). I prefer not to alter
                          anything normally visible to the user (like the subject), but they are
                          able to act on the header information of anything that gets through.

                          I use header_checks for filtering, selectively uncommenting and
                          adjusting the following:

                          # HOLD messages marked as spam by SpamAssassin, for later inspection
                          #/^X-Spam-Flag: YES/ HOLD Identified as spam by SpamAssassin.

                          # REJECT messages marked as spam by SpamAssassin
                          # Use this with a before-queue content filter, only!
                          #/^X-Spam-Flag: YES/ REJECT Identified as spam by SpamAssassin.

                          # REJECT only high scores
                          # Use this with a before-queue content filter, only!
                          /X-Spam-Level: \*{7,}/ REJECT Identified as spam by SpamAssassin.

                          # HOLD messages with score in specified range for inspection
                          /X-Spam-Level: \*{5,6}$/ HOLD Identified as spam by SpamAssassin.


                          Only the HOLD rules are relevant to FILTER_README, but I include the
                          rest because I will typically set up an after-queue filter first, then
                          configure the before-queue filter. This way, I can easily switch back to
                          the after-queue filter if there are any problems.
                        • o2 - Marcin Wasilewski
                          ... From: Wietse Venema To: Postfix users Sent: Friday, September 01, 2006 4:53 PM Subject:
                          Message 12 of 16 , Sep 4, 2006
                            ----- Original Message -----
                            From: "Wietse Venema" <wietse@...>
                            To: "Postfix users" <postfix-users@...>
                            Sent: Friday, September 01, 2006 4:53 PM
                            Subject: FILTER_README suggestions


                            > Postfix's FILTER_README was written long before backscatter became
                            > a problem. The first example (see below signature) has a warning
                            > not to reject mail:
                            >
                            > Note: in this time of mail worms and spam, it is a BAD IDEA to
                            > send known viruses or spam back to the sender, because that
                            > address is likely to be forged. It is safer to discard known
                            > to be bad content and to quarantine suspicious content so that
                            > it can be inspected by a human being.
                            >
                            > Unfortunately, the text gives no example of how one would implement
                            > this advice. Personally, I use no external filter so I have a hard
                            > time coming up with field-tested examples.
                            >
                            > What do people use:
                            >
                            > - Have the filter return a distinct exit status that says "discard"?
                            >
                            > - Have the filter insert a "badness" indicator in a message header,
                            > and dispose of bad mail with Postfix HOLD/DISCARD actions, maildrop
                            > rules, cyrus sieves, or procmail filters?
                            >
                            > - Something completely different? Maybe no-one uses the pipe+sendmail
                            > example and we can drop it from the documentation.
                            >
                            > Wietse
                            >
                            > 1 #!/bin/sh
                            > 2
                            > 3 # Simple shell-based filter. It is meant to be invoked as follows:
                            > 4 # /path/to/script -f sender recipients...
                            > 5
                            > 6 # Localize these. The -G option does nothing before Postfix 2.3.
                            > 7 INSPECT_DIR=/var/spool/filter
                            > 8 SENDMAIL="/usr/sbin/sendmail -G -i" # NEVER NEVER NEVER use "-t" here.
                            > 9
                            > 10 # Exit codes from <sysexits.h>
                            > 11 EX_TEMPFAIL=75
                            > 12 EX_UNAVAILABLE=69
                            > 13
                            > 14 # Clean up when done or when aborting.
                            > 15 trap "rm -f in.$$" 0 1 2 3 15
                            > 16
                            > 17 # Start processing.
                            > 18 cd $INSPECT_DIR || {
                            > 19 echo $INSPECT_DIR does not exist; exit $EX_TEMPFAIL; }
                            > 20
                            > 21 cat >in.$$ || {
                            > 22 echo Cannot save mail to file; exit $EX_TEMPFAIL; }
                            > 23
                            > 24 # Specify your content filter here.
                            > 25 # filter <in.$$ || {
                            > 26 # echo Message content rejected; exit $EX_UNAVAILABLE; }
                            > 27
                            > 28 $SENDMAIL "$@" <in.$$
                            > 29
                            > 30 exit $?
                            >

                            Hello,

                            what do You think about this example: sa_quarantine.sh
                            #!/bin/bash
                            #Marcin Wasilewski, 20060904
                            QUARANTINE_ABOVE=6
                            REJECT_ABOVE=15
                            SCORE=0

                            INSPECT_DIR=/proxsmtp
                            QUARANTINE_DIR=/proxsmtp/QUARANTINE
                            EX_TEMPFAIL=75

                            # Start processing.
                            cat | spamc | tee $INSPECT_DIR/in.$$ || {
                            echo -e "Cannot save mail to file"; exit $EX_TEMPFAIL; }

                            SCORE=`grep '^X-Spam-Status' $INSPECT_DIR/in.$$ | sed 's/.* score=//' | sed
                            's/ .*//' |sed 's/\..*//'`

                            if [ "$SCORE" -ge $REJECT_ABOVE ]; then
                            echo "550 Sorry, your message was flagged as spam and rejected!" >&2
                            rm -f $INSPECT_DIR/in.$$
                            exit 1
                            fi

                            if [ "$SCORE" -ge $QUARANTINE_ABOVE ]; then
                            echo "550 Sorry, your message was flagged as spam and quarantined!" >&2
                            mv $INSPECT_DIR/in.$$ $QUARANTINE_DIR/`date +%Y%m%d-%H:%M:%S`_$RANDOM.eml
                            exit 1
                            fi

                            exit 0
                            #######################
                            it is a beta verison that I just create and it works.
                            It requires: http://memberwebs.com/nielsen/software/proxsmtp/ to run
                            proxsmtpd -f proxsmtpd.conf -d 4
                            where proxsmtpd.conf is:
                            OutAddress: 10025
                            FilterCommand: /proxsmtp/sa_quarantine.sh
                            FilterType: pipe
                            Listen: 127.0.0.1:10024

                            and Postfix's master.cf should be:
                            smtp inet n - - - - smtpd
                            -o smtpd_proxy_filter=127.0.0.1:10024

                            # Re-injection after content filter
                            127.0.0.1:10025 inet n - n - - smtpd
                            -o smtpd_authorized_xforward_hosts=127.0.0.0/8
                            -o smtpd_client_restrictions=
                            -o smtpd_helo_restrictions=
                            -o smtpd_sender_restrictions=
                            -o smtpd_recipient_restrictions=permit_mynetworks,reject
                            -o smtpd_data_restrictions=
                            -o receive_override_options=no_unknown_recipient_checks

                            I didn't test it strong enough to move it to production environment but I
                            think it may be a good example.
                            If someone could look at sa_quarantine.sh and correct my sed exp. and other
                            errors - because I'm sure it can be done in better way :)

                            Best regards
                            Marcin
                          Your message has been successfully submitted and would be delivered to recipients shortly.