Loading ...
Sorry, an error occurred while loading the content.

Per user SPF and Anti Virus checks

Expand Messages
  • Paul G. Allen
    I am using a PostgreSQL database to store hosted e-mail account information. I am using AVG for Virus, Spam, and RBL checks. I am using policyd-spf for SPF
    Message 1 of 11 , Mar 31 4:35 AM
    • 0 Attachment
      I am using a PostgreSQL database to store hosted e-mail account
      information. I am using AVG for Virus, Spam, and RBL checks. I am using
      policyd-spf for SPF checks.

      Each e-mail user can enable/disable SPF, Virus, and Spam checks through
      our user interface. Doing so sets a flag in the PostgreSQL database to
      enable (1) the option, or disable (0) the option.

      Currently I have the Postfix configuration set to run all e-mail through
      AVG and SPF. I would like to configure it to run AVG or perform SPF
      checks based upon the setting in the database on a per-recipient basis,
      but I am not sure how to do it.

      TIA,

      PGA
      --
      Paul G. Allen, BSIT/SE
      Network Administrator
      Greenest Host
      www.greenesthost.com
    • mouss
      ... for the policy service, it s as easy as check_recipient_access $yourmap and let $yorumap return check_policy_service .... when the user opted in SPF
      Message 2 of 11 , Mar 31 5:20 AM
      • 0 Attachment
        Paul G. Allen wrote:
        > I am using a PostgreSQL database to store hosted e-mail account
        > information. I am using AVG for Virus, Spam, and RBL checks. I am
        > using policyd-spf for SPF checks.
        >
        > Each e-mail user can enable/disable SPF, Virus, and Spam checks
        > through our user interface. Doing so sets a flag in the PostgreSQL
        > database to enable (1) the option, or disable (0) the option.
        >
        > Currently I have the Postfix configuration set to run all e-mail
        > through AVG and SPF. I would like to configure it to run AVG or
        > perform SPF checks based upon the setting in the database on a
        > per-recipient basis, but I am not sure how to do it.

        for the policy service, it's as easy as
        check_recipient_access $yourmap

        and let $yorumap return "check_policy_service ...." when the user opted
        in SPF checks.

        for AV and spam, you need to do that in your content_filter (easy with
        amavisd-new) or use multiple instances of postfix and use transport_maps
        (and not content_filter) to select the filter depending on the recipient.
      • Ronald MacDonald
        ... Indeed, as Mouss says, the quickest - and easiest - way of getting this done on an AV level is by setting up amavisd-new to do this. Ronald. -- Ronald
        Message 3 of 11 , Mar 31 5:32 AM
        • 0 Attachment
          On 31/03/2008, mouss <mouss@...> wrote:

          > for AV and spam, you need to do that in your content_filter (easy with
          > amavisd-new) or use multiple instances of postfix and use transport_maps
          > (and not content_filter) to select the filter depending on the recipient.
          >


          Indeed, as Mouss says, the quickest - and easiest - way of getting
          this done on an AV level is by setting up amavisd-new to do this.


          Ronald.

          --
          Ronald MacDonald
          http://www.rmacd.com/
          0777 235 1655
        • Ronald MacDonald
          ... I presume that when you re storing email account information already on your databases, you re already doing at least a couple of lookups per user per
          Message 4 of 11 , Mar 31 5:56 AM
          • 0 Attachment
            On 31/03/2008, Paul G. Allen <paul.allen@...> wrote:
            > Ronald MacDonald wrote:
            > > Indeed, as Mouss says, the quickest - and easiest - way of getting
            > > this done on an AV level is by setting up amavisd-new to do this.
            > >
            >
            >
            > What affect will it have on performance with hundreds of e-mail accounts?
            >

            I presume that when you're storing email account information already
            on your databases, you're already doing at least a couple of lookups
            per user per email. This is normal. In honesty, a query to find if
            'scan for viruses/spam' will take miliseconds. Contrary to this, the
            process of actually scanning the mail may take many seconds. On a
            small 533MHz system, 512 RAM, I sometimes see mails through
            AMaVisd+Spam Assassin taking up to 8-10 seconds each.

            How long is a piece of string?

            Ronald.

            --
            Ronald MacDonald
            http://www.rmacd.com/
            0777 235 1655
          • mouss
            ... I think he is concerned about amavisd-new. amavisd-new uses spamassassin for spam filtering. Multiple instances may be worth the effort. they allow using
            Message 5 of 11 , Mar 31 6:08 AM
            • 0 Attachment
              Ronald MacDonald wrote:
              > On 31/03/2008, Paul G. Allen <paul.allen@...> wrote:
              >
              >> [snip]
              >>
              >> What affect will it have on performance with hundreds of e-mail accounts?
              >>
              >>
              >
              >

              I think he is concerned about amavisd-new. amavisd-new uses spamassassin
              for spam filtering.

              Multiple instances may be worth the effort. they allow using whatever
              filter he wants, and make it easy to use multiple machines for filtering
              (each machine would filter mail for a set of users) when needed.

              Note that dspam also have an opt-in/opt-out mechanism. but as said in
              aprevious thread, it requires some work to get it going.

              > [snip]
              >
            • Paul G. Allen
              ... Yes I am as the AVG documentation states that there may be performance issues when using it. We are expanding our customer base every week and the load
              Message 6 of 11 , Mar 31 4:31 PM
              • 0 Attachment
                mouss wrote:
                > Ronald MacDonald wrote:
                >> On 31/03/2008, Paul G. Allen <paul.allen@...> wrote:
                >>
                >>> [snip]
                >>>
                >>> What affect will it have on performance with hundreds of e-mail
                >>> accounts?
                >>>
                >>>
                >>
                >>
                >
                > I think he is concerned about amavisd-new. amavisd-new uses spamassassin
                > for spam filtering.

                Yes I am as the AVG documentation states that there may be performance
                issues when using it. We are expanding our customer base every week and
                the load will only increase.

                >
                > Multiple instances may be worth the effort. they allow using whatever
                > filter he wants, and make it easy to use multiple machines for filtering
                > (each machine would filter mail for a set of users) when needed.

                In the future this may be (well, hopefully we get to a large enough
                customer base where we need multiple servers :) ) necessary in the
                future. Currently AVG and policyd-spf are running on the Postfix server.

                >
                > Note that dspam also have an opt-in/opt-out mechanism. but as said in
                > aprevious thread, it requires some work to get it going.
                >

                Before the responses thus far, I was able to sort of get it working
                using transport_maps (instead of the content_filter that I had). The
                only problem is that it was caught in a recursive loop where the e-mails
                are being sent to AVG, the response is returned to Postfix, then it's
                sent to AVG again, etc., etc. (NOTE: Postfix does properly read the DB
                and only sends e-mails to AVG for accounts with Antispam enabled, so I
                at least got that part right. :D )

                It seems I read something somewhere about the possibility of this
                happening and how to properly configure such a filter so it won't, but I
                can't remember where it was, what it was, or how to do it.

                TIA,

                PGA
                --
                Paul G. Allen, BSIT/SE
                Network Administrator
                Greenest Host
                www.greenesthost.com
              • Scott Kitterman
                ... There are several SPF policy servers. If you are using my Python policy server, it supports using restriction classes in combination with the policy
                Message 7 of 11 , Mar 31 4:45 PM
                • 0 Attachment
                  On Monday 31 March 2008 19:31:18 Paul G. Allen wrote:
                  > mouss wrote:
                  > > Ronald MacDonald wrote:
                  > >> On 31/03/2008, Paul G. Allen <paul.allen@...> wrote:
                  > >>> [snip]
                  > >>>
                  > >>> What affect will it have on performance with hundreds of e-mail
                  > >>> accounts?
                  > >
                  > > I think he is concerned about amavisd-new. amavisd-new uses spamassassin
                  > > for spam filtering.
                  >
                  > Yes I am as the AVG documentation states that there may be performance
                  > issues when using it. We are expanding our customer base every week and
                  > the load will only increase.
                  >
                  > > Multiple instances may be worth the effort. they allow using whatever
                  > > filter he wants, and make it easy to use multiple machines for filtering
                  > > (each machine would filter mail for a set of users) when needed.
                  >
                  > In the future this may be (well, hopefully we get to a large enough
                  > customer base where we need multiple servers :) ) necessary in the
                  > future. Currently AVG and policyd-spf are running on the Postfix server.

                  There are several SPF policy servers. If you are using my Python policy
                  server, it supports using restriction classes in combination with the policy
                  server to support different actions based on SPF result and per user. I
                  don't know that it's a sufficiently scalable approach for your needs, but it
                  might be worth a look. There's a per user README included with the package
                  that at least roughs out the concept:

                  http://www.openspf.org/Software#python-postfix-policyd-spf

                  Scott K
                • Paul G. Allen
                  ... Yes I am using it. I read an article that it works well enough for a few thousand users (at least for the person that wrote the article). I have not yet
                  Message 8 of 11 , Mar 31 5:03 PM
                  • 0 Attachment
                    Scott Kitterman wrote:

                    >
                    > There are several SPF policy servers. If you are using my Python policy
                    > server, it supports using restriction classes in combination with the policy
                    > server to support different actions based on SPF result and per user. I
                    > don't know that it's a sufficiently scalable approach for your needs, but it
                    > might be worth a look. There's a per user README included with the package
                    > that at least roughs out the concept:
                    >
                    > http://www.openspf.org/Software#python-postfix-policyd-spf
                    >

                    Yes I am using it. I read an article that it works well enough for a few
                    thousand users (at least for the person that wrote the article). I have
                    not yet gotten to per user for SPF yet as I am still working on getting
                    the per user AVG working. (I prefer to screw up my configuration in one
                    place at a time if you know what I mean. :D ) Once I get AVG working
                    properly, then I'll move to per user SPF. (Currently all e-mail goes
                    through SPF checks, except for mail from localhost or mail from trusted
                    domains such as greenesthost.com.)

                    Thanks,

                    PGA
                    --
                    Paul G. Allen, BSIT/SE
                    Network Administrator
                    Greenest Host
                    www.greenesthost.com
                  • mouss
                    ... you used a single instance. ... transports are global in an instance. you need to run multiple instances of postfix: run postfix multiple times (not just
                    Message 9 of 11 , Apr 1, 2008
                    • 0 Attachment
                      Paul G. Allen wrote:
                      > [snip]
                      >>
                      >
                      > Before the responses thus far, I was able to sort of get it working
                      > using transport_maps (instead of the content_filter that I had). The
                      > only problem is that it was caught in a recursive loop where the
                      > e-mails are being sent to AVG, the response is returned to Postfix,
                      > then it's sent to AVG again, etc., etc. (NOTE: Postfix does properly
                      > read the DB and only sends e-mails to AVG for accounts with Antispam
                      > enabled, so I at least got that part right. :D )

                      you used a single instance.
                      >
                      > It seems I read something somewhere about the possibility of this
                      > happening and how to properly configure such a filter so it won't, but
                      > I can't remember where it was, what it was, or how to do it.

                      transports are global in an instance. you need to run multiple instances
                      of postfix: run postfix multiple times (not just edit one master.cf).

                      - one postfix handles mail on port 25 and uses transport_maps to
                      redirect users to the corresponding filter. mail that should not be
                      filtered is passed to the second instance (below) via relayhost for
                      instance.

                      - the other postfix uses a "standard" transport_maps (no filtering) and
                      handles delivery.
                    • Paul G. Allen
                      ... I was finally able to get back to finishing this server and wanted to put closure on this thread. Thanks for the advice and help. What I did was setup a
                      Message 10 of 11 , Apr 8, 2008
                      • 0 Attachment
                        mouss wrote:

                        >
                        > transports are global in an instance. you need to run multiple instances
                        > of postfix: run postfix multiple times (not just edit one master.cf).
                        >
                        > - one postfix handles mail on port 25 and uses transport_maps to
                        > redirect users to the corresponding filter. mail that should not be
                        > filtered is passed to the second instance (below) via relayhost for
                        > instance.
                        >
                        > - the other postfix uses a "standard" transport_maps (no filtering) and
                        > handles delivery.
                        >

                        I was finally able to get back to finishing this server and wanted to
                        put closure on this thread.

                        Thanks for the advice and help.

                        What I did was setup a second instance of Postfix to handle the
                        filtering. The first (primary) instance checks the PostgreSQL database
                        to see if the recipient has enabled Anti-Spam through the use of
                        transport_maps. If they have, it sends the incoming message to the
                        second instance listening on a local port.

                        The second instance uses AVG to scan the message (content_filter). AVG
                        returns the message back to the second instance, and it in turn relays
                        the message to the qmail server.

                        Note that the first instance also performs initial "inexpensive" checks
                        on incoming e-mails to help weed out the bulk of spam before it's even
                        queued. The way all this is configured each scanner (AVG and the SPF
                        policy server, as well as our optional local blacklist) and each e-mail
                        server could be run on different machines easily when traffic increases
                        to the point tat it's necessary. Users have complete control over
                        whether e-mail is scanned or not, and unlike the qmail server, the user
                        settings actually work.

                        During all this, I've decided I really like Postfix and qmail sucks. :)

                        PGA
                        --
                        Paul G. Allen, BSIT/SE
                        Network Administrator
                        Greenest Host
                        www.greenesthost.com
                      • mouss
                        ... if you do no rewrite on postfix, this is ok. if you do rewrite, you d better pass all mail through the second instance, possibly via spam filter (for users
                        Message 11 of 11 , Apr 8, 2008
                        • 0 Attachment
                          Paul G. Allen wrote:
                          > mouss wrote:
                          >
                          >>
                          >> transports are global in an instance. you need to run multiple
                          >> instances of postfix: run postfix multiple times (not just edit one
                          >> master.cf).
                          >>
                          >> - one postfix handles mail on port 25 and uses transport_maps to
                          >> redirect users to the corresponding filter. mail that should not be
                          >> filtered is passed to the second instance (below) via relayhost for
                          >> instance.
                          >>
                          >> - the other postfix uses a "standard" transport_maps (no filtering)
                          >> and handles delivery.
                          >>
                          >
                          > I was finally able to get back to finishing this server and wanted to
                          > put closure on this thread.
                          >
                          > Thanks for the advice and help.
                          >
                          > What I did was setup a second instance of Postfix to handle the
                          > filtering. The first (primary) instance checks the PostgreSQL database
                          > to see if the recipient has enabled Anti-Spam through the use of
                          > transport_maps. If they have, it sends the incoming message to the
                          > second instance listening on a local port.

                          if you do no rewrite on postfix, this is ok. if you do rewrite, you'd
                          better pass all mail through the second instance, possibly via spam
                          filter (for users that need filtering). This way you can do all rewrite
                          in the second instance (so that the filter gets "original" addresses).
                          in short, the first instance would have a relayhost pointing to the
                          second instance, and you'll have transport entries for users who need
                          filtering pointing to the filter. the filter then passes back all mail
                          to the second instance. all like if you had multiple machines with
                          different roles.

                          >
                          > The second instance uses AVG to scan the message (content_filter). AVG
                          > returns the message back to the second instance, and it in turn relays
                          > the message to the qmail server.
                          >
                          > Note that the first instance also performs initial "inexpensive"
                          > checks on incoming e-mails to help weed out the bulk of spam before
                          > it's even queued. The way all this is configured each scanner (AVG and
                          > the SPF policy server, as well as our optional local blacklist) and
                          > each e-mail server could be run on different machines easily when
                          > traffic increases to the point tat it's necessary. Users have complete
                          > control over whether e-mail is scanned or not, and unlike the qmail
                          > server, the user settings actually work.
                          >
                          > During all this, I've decided I really like Postfix and qmail sucks. :)
                          >
                          > PGA
                        Your message has been successfully submitted and would be delivered to recipients shortly.