Loading ...
Sorry, an error occurred while loading the content.

Re: Stripping Received: headers

Expand Messages
  • Wietse Venema
    ... As documented header_checks is not an smtpd(8) feature, it is a cleanup(8) feature. The easiest way to give separate treatment to mail from the internal
    Message 1 of 16 , Apr 11 7:58 AM
    • 0 Attachment
      Geoff Shang:
      > submission inet n - - - - smtpd
      > -o smtpd_enforce_tls=yes
      > -o smtpd_sasl_auth_enable=yes
      > -o smtpd_client_restrictions=permit_sasl_authenticated,reject
      > -o milter_macro_daemon_name=ORIGINATING
      > -o header_checks=pcre:/etc/postfix/header_checks

      As documented header_checks is not an smtpd(8) feature, it is
      a cleanup(8) feature.

      The easiest way to give separate treatment to mail from the
      internal network versus mail from outside is to use separate
      Postfix instances.

      Otherwise,

      submission inet n - - - - smtpd
      -o cleanup_service=submission_cleanup

      submission_cleanup unix n ............................ cleanup
      -o header_checks=pcre:/etc/postfix/header_checks

      would do the job.

      Wietse
    • Benny Pedersen
      ... header_checks is incomming on smtpd, but you use submission so you must change to to smtp_header_checks http://www.postfix.org/header_checks.5.html ...
      Message 2 of 16 , Apr 11 8:05 AM
      • 0 Attachment
        Geoff Shang skrev den 2013-04-11 16:33:
        > Hi,
        >
        > I'm trying to strip Received: headers from mail at various parts of
        > our processing, for security reasons.
        >
        > I'm starting with mail that comes in from authenticated clients. I
        > tried doing the following:
        >
        > master.cf:
        >
        > submission inet n - - - - smtpd
        > -o smtpd_enforce_tls=yes
        > -o smtpd_sasl_auth_enable=yes
        > -o smtpd_client_restrictions=permit_sasl_authenticated,reject
        > -o milter_macro_daemon_name=ORIGINATING
        > -o header_checks=pcre:/etc/postfix/header_checks

        header_checks is incomming on smtpd, but you use submission

        so you must change to to smtp_header_checks

        http://www.postfix.org/header_checks.5.html

        >
        > /etc/postfix/header_checks:
        >
        > /^Received:/ IGNORE

        this one is to gready, dont use it on header_checks

        >
        > I ran this through Postmap with a query from a message I sent myself,
        > and the IGNORE key is correctly returned. But if I actually send
        > myself a message, it comes through with the Received: line intact.
        >
        > I did some searching and found
        > http://marc.info/?l=postfix-users&m=122106227124195&w=2
        >
        > I'm curious to know why this would work and the above wouldn't. Am I
        > just trying to do it too early in the process?
        >
        > A related question, is it possible to prevent Postfix from generating
        > lines like this?

        what problems do you like to resolve ?

        >
        > Geoff.

        --
        senders that put my email into body content will deliver it to my own
        trashcan, so if you like to get reply, dont do it
      • Noel Jones
        ... No, header_checks are performed on all incoming mail. As already explained, the problem above is that -o header_checks=... has no effect on smtpd(5). ...
        Message 3 of 16 , Apr 11 9:29 AM
        • 0 Attachment
          On 4/11/2013 10:05 AM, Benny Pedersen wrote:
          > Geoff Shang skrev den 2013-04-11 16:33:
          >> Hi,
          >>
          >> I'm trying to strip Received: headers from mail at various parts of
          >> our processing, for security reasons.
          >>
          >> I'm starting with mail that comes in from authenticated clients. I
          >> tried doing the following:
          >>
          >> master.cf:
          >>
          >> submission inet n - - - - smtpd
          >> -o smtpd_enforce_tls=yes
          >> -o smtpd_sasl_auth_enable=yes
          >> -o smtpd_client_restrictions=permit_sasl_authenticated,reject
          >> -o milter_macro_daemon_name=ORIGINATING
          >> -o header_checks=pcre:/etc/postfix/header_checks
          >
          > header_checks is incomming on smtpd, but you use submission

          No, header_checks are performed on all incoming mail.

          As already explained, the problem above is that "-o
          header_checks=..." has no effect on smtpd(5).

          >
          > so you must change to to smtp_header_checks

          smtp_header_checks are performed on outgoing mail during smtp(5)
          delivery.

          But you're sort of on the right track. You can use
          smtp_header_checks to remove the Received: headers from
          authenticated mail before external delivery with something like:
          /^Received: .*by myserver.example.com \(Postfix\) with ESMTPS?A
          id.*$/ IGNORE

          Geoff, please note I've seen some overanxious anti-spam systems that
          consider mail with no Received: headers as spam.



          -- Noel Jones




          >
          > http://www.postfix.org/header_checks.5.html
          >
          >>
          >> /etc/postfix/header_checks:
          >>
          >> /^Received:/ IGNORE
          >
          > this one is to gready, dont use it on header_checks
          >
          >>
          >> I ran this through Postmap with a query from a message I sent myself,
          >> and the IGNORE key is correctly returned. But if I actually send
          >> myself a message, it comes through with the Received: line intact.
          >>
          >> I did some searching and found
          >> http://marc.info/?l=postfix-users&m=122106227124195&w=2
          >>
          >> I'm curious to know why this would work and the above wouldn't. Am I
          >> just trying to do it too early in the process?
          >>
          >> A related question, is it possible to prevent Postfix from generating
          >> lines like this?
          >
          > what problems do you like to resolve ?
          >
          >>
          >> Geoff.
          >
        • Benny Pedersen
          ... +1 ... yes it included as it used all incomming, but not directly with smtpd ... is submission not using smtp_header_checks ? ... yep will soon try to
          Message 4 of 16 , Apr 11 9:55 AM
          • 0 Attachment
            Noel Jones skrev den 2013-04-11 18:29:

            > No, header_checks are performed on all incoming mail.

            +1

            > As already explained, the problem above is that "-o
            > header_checks=..." has no effect on smtpd(5).

            yes it included as it used all incomming, but not directly with smtpd

            >> so you must change to to smtp_header_checks
            >
            > smtp_header_checks are performed on outgoing mail during smtp(5)
            > delivery.

            is submission not using smtp_header_checks ?

            > But you're sort of on the right track. You can use
            > smtp_header_checks to remove the Received: headers from
            > authenticated mail before external delivery with something like:
            > /^Received: .*by myserver.example.com \(Postfix\) with ESMTPS?A
            > id.*$/ IGNORE

            yep will soon try to apply it here

            > Geoff, please note I've seen some overanxious anti-spam systems that
            > consider mail with no Received: headers as spam.

            can one show an example main.cf that remove all recieved headers on
            remote senders ?, there would allways be one last hop imho

            --
            senders that put my email into body content will deliver it to my own
            trashcan, so if you like to get reply, dont do it
          • DTNX Postmaster
            ... In our case, the problem was with overzealous ones that filter on all Received: headers, and therefore block legitimate mail because the authenticated
            Message 5 of 16 , Apr 11 10:10 AM
            • 0 Attachment
              On Apr 11, 2013, at 18:29, Noel Jones <njones@...> wrote:

              >> so you must change to to smtp_header_checks
              >
              > smtp_header_checks are performed on outgoing mail during smtp(5)
              > delivery.
              >
              > But you're sort of on the right track. You can use
              > smtp_header_checks to remove the Received: headers from
              > authenticated mail before external delivery with something like:
              > /^Received: .*by myserver.example.com \(Postfix\) with ESMTPS?A
              > id.*$/ IGNORE
              >
              > Geoff, please note I've seen some overanxious anti-spam systems that
              > consider mail with no Received: headers as spam.

              In our case, the problem was with overzealous ones that filter on all
              Received: headers, and therefore block legitimate mail because the
              authenticated client is connecting from an access provider range listed
              by Spamhaus, or something similar.

              Our solution so far is to strip a few of the internal Received:
              headers, and 'REPLACE' the one that contains the connecting IP with a
              'Received: by hostname.domain.tld (from authenticated client)' header.
              Since the submission hosts never send directly, it will always have at
              least three or four Received: headers when offered to the destination
              MX.

              Since the regular expression is fairly specific, this is done with
              'header_checks' in our case.

              HTH,
              Jona
            • Reindl Harald
              ... has your submission service smtp or smtpd in master.cf? mine has smtpd as all other working ones out there
              Message 6 of 16 , Apr 11 10:20 AM
              • 0 Attachment
                Am 11.04.2013 18:55, schrieb Benny Pedersen:
                >> smtp_header_checks are performed on outgoing mail during smtp(5)
                >> delivery.
                >
                > is submission not using smtp_header_checks?

                has your submission service smtp or smtpd in master.cf?
                mine has smtpd as all other working ones out there
              • Reindl Harald
                ... to make it clear: submission is nothing else as smtpd on port 587 and if you want not rely on /etc/services you would even write 587 instead submission the
                Message 7 of 16 , Apr 11 10:44 AM
                • 0 Attachment
                  Am 11.04.2013 19:20, schrieb Reindl Harald:
                  >
                  >
                  > Am 11.04.2013 18:55, schrieb Benny Pedersen:
                  >>> smtp_header_checks are performed on outgoing mail during smtp(5)
                  >>> delivery.
                  >>
                  >> is submission not using smtp_header_checks?
                  >
                  > has your submission service smtp or smtpd in master.cf?
                  > mine has smtpd as all other working ones out there

                  to make it clear:

                  submission is nothing else as smtpd on port 587
                  and if you want not rely on /etc/services you would
                  even write 587 instead submission

                  the only difference between port 25 and 587 is
                  usually that you require authentication on 587

                  [harry@srv-rhsoft:~]$ cat /etc/services | grep submission
                  submission 587/tcp msa # mail message submission
                  submission 587/udp msa # mail message submission
                • Noel Jones
                  ... No. submission uses the smtpd(5) service to receive mail, which uses header_checks (indirectly, through the cleanup service). smtp_header_checks are used
                  Message 8 of 16 , Apr 11 1:12 PM
                  • 0 Attachment
                    On 4/11/2013 11:55 AM, Benny Pedersen wrote:
                    > Noel Jones skrev den 2013-04-11 18:29:

                    >> smtp_header_checks are performed on outgoing mail during smtp(5)
                    >> delivery.
                    >
                    > is submission not using smtp_header_checks ?

                    No.

                    submission uses the smtpd(5) service to receive mail, which uses
                    header_checks (indirectly, through the cleanup service).

                    smtp_header_checks are used by the smtp(5) transport when sending
                    mail to remote systems.


                    http://www.postfix.org/OVERVIEW.html


                    -- Noel Jones
                  • Geoff Shang
                    ... Oh duh! Thanks for pointing this out. ... Thanks Wietse. I think I will opt for this latter option. Some have suggested smtp_header_checks, and I may use
                    Message 9 of 16 , Apr 12 6:04 AM
                    • 0 Attachment
                      On Thu, 11 Apr 2013, Wietse Venema wrote:

                      > Geoff Shang:
                      >> submission inet n - - - - smtpd
                      >> -o smtpd_enforce_tls=yes
                      >> -o smtpd_sasl_auth_enable=yes
                      >> -o smtpd_client_restrictions=permit_sasl_authenticated,reject
                      >> -o milter_macro_daemon_name=ORIGINATING
                      >> -o header_checks=pcre:/etc/postfix/header_checks
                      >
                      > As documented header_checks is not an smtpd(8) feature, it is
                      > a cleanup(8) feature.

                      Oh duh! Thanks for pointing this out.

                      > The easiest way to give separate treatment to mail from the
                      > internal network versus mail from outside is to use separate
                      > Postfix instances.
                      >
                      > Otherwise,
                      >
                      > submission inet n - - - - smtpd
                      > -o cleanup_service=submission_cleanup
                      >
                      > submission_cleanup unix n ............................ cleanup
                      > -o header_checks=pcre:/etc/postfix/header_checks
                      >
                      > would do the job.

                      Thanks Wietse. I think I will opt for this latter option.

                      Some have suggested smtp_header_checks, and I may use this in some places.
                      But since this box will deliver some mail locally as well as externally, I
                      think I will implement the separate cleanup process.

                      Thanks everyone for your input.

                      Geoff.
                    • Geoff Shang
                      ... hmm. This didn t work. I m a bit stuck as to why. I thought that perhaps it might be running before the Received: header is created, but in that case, I
                      Message 10 of 16 , Apr 12 6:56 AM
                      • 0 Attachment
                        On Fri, 12 Apr 2013, Geoff Shang wrote:

                        >> submission inet n - - - - smtpd
                        >> -o cleanup_service=submission_cleanup
                        >>
                        >> submission_cleanup unix n ............................ cleanup
                        >> -o header_checks=pcre:/etc/postfix/header_checks
                        >>
                        >> would do the job.
                        >
                        > Thanks Wietse. I think I will opt for this latter option.

                        hmm. This didn't work. I'm a bit stuck as to why. I thought that
                        perhaps it might be running before the Received: header is created, but in
                        that case, I don't know why the example I linked to earlier that searches
                        for an authenticated header would work, while this would not.

                        Geoff.
                      • Wietse Venema
                        ... You need to do postfix reload after editing master.cf. The submission_cleanup service will see the Received: header that was prepended by the submission
                        Message 11 of 16 , Apr 12 7:20 AM
                        • 0 Attachment
                          Geoff Shang:
                          > On Fri, 12 Apr 2013, Geoff Shang wrote:
                          >
                          > >> submission inet n - - - - smtpd
                          > >> -o cleanup_service=submission_cleanup
                          > >>
                          > >> submission_cleanup unix n ............................ cleanup
                          > >> -o header_checks=pcre:/etc/postfix/header_checks
                          > >>
                          > >> would do the job.
                          > >
                          > > Thanks Wietse. I think I will opt for this latter option.
                          >
                          > hmm. This didn't work. I'm a bit stuck as to why. I thought that
                          > perhaps it might be running before the Received: header is created, but in
                          > that case, I don't know why the example I linked to earlier that searches
                          > for an authenticated header would work, while this would not.

                          You need to do "postfix reload" after editing master.cf.

                          The submission_cleanup service will see the Received: header that
                          was prepended by the submission server.

                          However, if your Milter adds headers then those aren't seen by
                          header_checks; you would need to use milter_header_checks.

                          Wietse
                        • Geoff Shang
                          ... I did. I did it again for good measure - no difference. ... Is there any way I can be sure that the special cleanup agent is running? I see the socket
                          Message 12 of 16 , Apr 12 7:49 AM
                          • 0 Attachment
                            On Fri, 12 Apr 2013, Wietse Venema wrote:

                            > You need to do "postfix reload" after editing master.cf.

                            I did. I did it again for good measure - no difference.

                            > The submission_cleanup service will see the Received: header that
                            > was prepended by the submission server.

                            Is there any way I can be sure that the special cleanup agent is running?
                            I see the socket /var/spool/postfix/public/submission_cleanup

                            > However, if your Milter adds headers then those aren't seen by
                            > header_checks; you would need to use milter_header_checks.

                            We don't appear to be using any milters, despite the
                            'milter_macro_daemon_name=ORIGINATING'

                            Here's what I did in case I messed up:

                            master.cf:

                            # service type private unpriv chroot wakeup maxproc command + args
                            # (yes) (yes) (yes) (never) (100)
                            #
                            ==========================================================================
                            smtp inet n - - - - smtpd
                            submission inet n - - - - smtpd
                            -o smtpd_enforce_tls=yes
                            -o smtpd_sasl_auth_enable=yes
                            -o smtpd_client_restrictions=permit_sasl_authenticated,reject
                            -o milter_macro_daemon_name=ORIGINATING
                            # Use a special cleanup service so we can strip headers.
                            -o cleanup_service=submission_cleanup

                            smtps inet n - - - - smtpd
                            -o smtpd_tls_wrappermode=yes
                            -o smtpd_sasl_auth_enable=yes
                            -o smtpd_client_restrictions=permit_sasl_authenticated,reject
                            -o milter_macro_daemon_name=ORIGINATING
                            # Use a special cleanup service so we can strip headers.
                            -o cleanup_service=submission_cleanup

                            submission_cleanup unix n - - - - cleanup
                            # Strip Received: lines from authenticated mail
                            -o header_checks=pcre:/etc/postfix/header_checks



                            /etc/postfix/header_checks:

                            # Remove any Received: headers from authenticated mail.
                            /^Received:/ IGNORE



                            An example message. The line is matched if I run it through postmap.
                            Some details have to be obscured, sorry. I'm on holiday so I'm not
                            worried about letting the hostname through, you can all get it from my
                            headers anyway. Obviously I'm not posting from my work address.

                            Return-Path: <my.address@...>
                            X-Original-To: my.address@...
                            Delivered-To: my.address@...
                            Received: from [192.168.0.20] (dsl-mlibrasgw2-50de1c-161.dhcp.inet.fi
                            [80.222.28.161])
                            by mail.example.com (Postfix) with ESMTPSA id DED281C40E9
                            for <my.address@...>; Fri, 12 Apr 2013 14:35:47
                            +0000 (UTC)
                            Date: Fri, 12 Apr 2013 17:35:44 +0300 (EEST)
                            From: Geoff Shang <my.address@...>
                            X-X-Sender: geoff@...
                            To: my.address@...
                            Subject: test
                            Message-ID: <alpine.DEB.2.02.1304121735310.14582@...>
                            User-Agent: Alpine 2.02 (DEB 1266 2009-07-14)
                            MIME-Version: 1.0
                            Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII

                            Geoff.\
                          • /dev/rob0
                            A word at the outset here: I predict this will come back to bite you in a most painful way. As Noel suggested, you re going to run afoul of some clueless spam
                            Message 13 of 16 , Apr 12 8:03 AM
                            • 0 Attachment
                              A word at the outset here: I predict this will come back to bite you
                              in a most painful way. As Noel suggested, you're going to run afoul
                              of some clueless spam checks. Some years back I know that Hotmail/MSN
                              actually *discarded* such mail silently!

                              Note also that Postfix itself uses Received: headers as a protection
                              against mail loops. Let's hope you don't get a loop going!

                              On Fri, Apr 12, 2013 at 05:49:47PM +0300, Geoff Shang wrote:
                              > Is there any way I can be sure that the special cleanup agent
                              > is running? I see the socket
                              > /var/spool/postfix/public/submission_cleanup

                              It's running. To see what it does:

                              > master.cf:

                              > submission_cleanup unix n - - - - cleanup
                              > # Strip Received: lines from authenticated mail
                              > -o header_checks=pcre:/etc/postfix/header_checks
                              -o syslog_name=postfix/submission/cleanup

                              Every non-default service should have its own syslog_name to enhance
                              your log searches.

                              > /etc/postfix/header_checks:
                              >
                              > # Remove any Received: headers from authenticated mail.
                              > /^Received:/ IGNORE
                              /./ WARN

                              That might get too noisy in the logs, but at least you will know your
                              alternate cleanup service is being used.
                              --
                              http://rob0.nodns4.us/ -- system administration and consulting
                              Offlist GMX mail is seen only if "/dev/rob0" is in the Subject:
                            • Wietse Venema
                              ... Are you using receive_override_options? in main.cf or master.cf? Wietse
                              Message 14 of 16 , Apr 12 10:17 AM
                              • 0 Attachment
                                Geoff Shang:
                                > On Fri, 12 Apr 2013, Wietse Venema wrote:
                                >
                                > > You need to do "postfix reload" after editing master.cf.
                                >
                                > I did. I did it again for good measure - no difference.

                                Are you using receive_override_options? in main.cf or master.cf?

                                Wietse
                              • Geoff Shang
                                ... No. Geoff.
                                Message 15 of 16 , Apr 15 7:15 AM
                                • 0 Attachment
                                  On Fri, 12 Apr 2013, Wietse Venema wrote:

                                  > Geoff Shang:
                                  >> On Fri, 12 Apr 2013, Wietse Venema wrote:
                                  >>
                                  >>> You need to do "postfix reload" after editing master.cf.
                                  >>
                                  >> I did. I did it again for good measure - no difference.
                                  >
                                  > Are you using receive_override_options? in main.cf or master.cf?

                                  No.

                                  Geoff.
                                Your message has been successfully submitted and would be delivered to recipients shortly.