Loading ...
Sorry, an error occurred while loading the content.

Re: How to copy all incoming and outgoing messages

Expand Messages
  • Curtis Vaughan
    ... This is great question, as I ve been wondering how people meet gov. requirements on storing corp. emails with Postfix. As it is our problem is that some
    Message 1 of 14 , May 1 4:17 PM
    • 0 Attachment
      D Hill wrote:
      > On Thu, 1 May 2008 at 22:22 -0000, forums@...
      > confabulated:
      >
      >> Hi Everyone,
      >>
      >> How do I make it so postfix will copy all incoming and outgoing
      >> emails to another email invisibly? It's a SOX requirement actually. I
      >> have to save all the emails.
      >>
      >> So, I create one account called "log". Every email that goes through
      >> the mail server has to be invisibly copied to that user whether
      >> incoming or outgoing. Thanks for the help. I'm new to postfix.
      >
      > Postfix configuration parameters:
      >
      > always_bcc, sender_bcc_maps, recipient_bcc_maps
      >
      > control what you are asking for. You can do a search on this page:
      >
      > http://www.postfix.org/postconf.5.html
      >
      > for further explination.

      This is great question, as I've been wondering how people meet gov.
      requirements on storing corp. emails with Postfix.
      As it is our problem is that some people use POP3 and delete as they
      download (I've been fighting this for years, but I can't control the
      foreign office!).
      Anyhow.... Even with IMAP, although most mail is saved, still there is
      they can delete emails from Trash and then those emails are lost.

      If all mail were just sent to another address, then it would make it
      difficult to find emails in the event we need to recover specific ones.
      It seems like there should be a way to say, regardless of whether
      someone wants to delete -- well, don't delete it from the server. Maybe
      do something else with it within the same users profile. I don't know.
      I should note that we are not a large company. Only 60-odd employees
      worldwide. And the budget for me to have continual backups doesn't
      exist. We do nightly backups using BackupPC. So we have about a weeks
      retention.

      Any ideas are most welcome, but please take into consideration our
      constrictions (budgetary).
    • Victor Duchovni
      ... You can get a fair wait with regexp based recipient_bcc_maps, which allow you to capture the original envelope recipient. Merge message copies are not
      Message 2 of 14 , May 1 4:24 PM
      • 0 Attachment
        On Thu, May 01, 2008 at 04:17:26PM -0700, Curtis Vaughan wrote:

        > This is great question, as I've been wondering how people meet gov.
        > requirements on storing corp. emails with Postfix.
        > As it is our problem is that some people use POP3 and delete as they
        > download (I've been fighting this for years, but I can't control the
        > foreign office!).
        > Anyhow.... Even with IMAP, although most mail is saved, still there is
        > they can delete emails from Trash and then those emails are lost.

        You can get a fair wait with regexp based recipient_bcc_maps, which
        allow you to capture the original envelope recipient. Merge message
        copies are not sufficient IMHO as headers are not accurate/complete.

        I usee a "tee" proxy that sends an encapsulated archive copy and the real
        message in parallel (archive "." immediately precedes message "." and
        blocks message delivery on failure).

        --
        Viktor.

        Disclaimer: off-list followups get on-list replies or get ignored.
        Please do not ignore the "Reply-To" header.

        To unsubscribe from the postfix-users list, visit
        http://www.postfix.org/lists.html or click the link below:
        <mailto:majordomo@...?body=unsubscribe%20postfix-users>

        If my response solves your problem, the best way to thank me is to not
        send an "it worked, thanks" follow-up. If you must respond, please put
        "It worked, thanks" in the "Subject" so I can delete these quickly.
      • Sahil Tandon
        ... Is there publicly available documentation on how to implement the tee proxy backup solution in Postfix? -- Sahil Tandon
        Message 3 of 14 , May 1 5:04 PM
        • 0 Attachment
          * Victor Duchovni <Victor.Duchovni@...> [2008-05-01 19:24:17 -0400]:

          > On Thu, May 01, 2008 at 04:17:26PM -0700, Curtis Vaughan wrote:
          >
          > > This is great question, as I've been wondering how people meet gov.
          > > requirements on storing corp. emails with Postfix.
          > > As it is our problem is that some people use POP3 and delete as they
          > > download (I've been fighting this for years, but I can't control the
          > > foreign office!).
          > > Anyhow.... Even with IMAP, although most mail is saved, still there is
          > > they can delete emails from Trash and then those emails are lost.
          >
          > You can get a fair wait with regexp based recipient_bcc_maps, which
          > allow you to capture the original envelope recipient. Merge message
          > copies are not sufficient IMHO as headers are not accurate/complete.
          >
          > I usee a "tee" proxy that sends an encapsulated archive copy and the real
          > message in parallel (archive "." immediately precedes message "." and
          > blocks message delivery on failure).

          Is there publicly available documentation on how to implement the tee proxy
          backup solution in Postfix?

          --
          Sahil Tandon <sahil@...>
        • Victor Duchovni
          ... 1. Write a tee proxy 2. Set content_filter to the tee proxy -- Viktor. Disclaimer: off-list followups get on-list replies or get ignored. Please do not
          Message 4 of 14 , May 1 6:50 PM
          • 0 Attachment
            On Thu, May 01, 2008 at 08:04:17PM -0400, Sahil Tandon wrote:

            > Is there publicly available documentation on how to implement the tee proxy
            > backup solution in Postfix?

            1. Write a tee proxy
            2. Set content_filter to the tee proxy

            --
            Viktor.

            Disclaimer: off-list followups get on-list replies or get ignored.
            Please do not ignore the "Reply-To" header.

            To unsubscribe from the postfix-users list, visit
            http://www.postfix.org/lists.html or click the link below:
            <mailto:majordomo@...?body=unsubscribe%20postfix-users>

            If my response solves your problem, the best way to thank me is to not
            send an "it worked, thanks" follow-up. If you must respond, please put
            "It worked, thanks" in the "Subject" so I can delete these quickly.
          • Tandon, Sahil (IM)
            ... Thanks, that is very helpful. Sahil ... NOTICE: If received in error, please destroy and notify sender. Sender does not intend to waive confidentiality or
            Message 5 of 14 , May 2 6:19 AM
            • 0 Attachment
              > > Is there publicly available documentation on how to
              > implement the tee
              > > proxy backup solution in Postfix?
              >
              > 1. Write a tee proxy
              > 2. Set content_filter to the tee proxy

              Thanks, that is very helpful.

              Sahil
              --------------------------------------------------------

              NOTICE: If received in error, please destroy and notify sender. Sender does not intend to waive confidentiality or privilege. Use of this email is prohibited when received in error.
            • Victor Duchovni
              ... It s the best I can do until I find the cycles to fully document and then release the tee proxy I am using. I don t think it is appropriate to release it
              Message 6 of 14 , May 2 6:40 AM
              • 0 Attachment
                On Fri, May 02, 2008 at 09:19:34AM -0400, Tandon, Sahil (IM) wrote:

                > > > Is there publicly available documentation on how to
                > > implement the tee
                > > > proxy backup solution in Postfix?
                > >
                > > 1. Write a tee proxy
                > > 2. Set content_filter to the tee proxy
                >
                > Thanks, that is very helpful.

                It's the best I can do until I find the cycles to fully document and
                then release the tee proxy I am using. I don't think it is appropriate
                to release it in an undocumented state.

                --
                Viktor.

                Disclaimer: off-list followups get on-list replies or get ignored.
                Please do not ignore the "Reply-To" header.

                To unsubscribe from the postfix-users list, visit
                http://www.postfix.org/lists.html or click the link below:
                <mailto:majordomo@...?body=unsubscribe%20postfix-users>

                If my response solves your problem, the best way to thank me is to not
                send an "it worked, thanks" follow-up. If you must respond, please put
                "It worked, thanks" in the "Subject" so I can delete these quickly.
              • Charles Marcus
                ... Understandable, and I m sure I and many others look forward to this... With the advancing requirements of burdensome regulations, this (some kind of basic
                Message 7 of 14 , May 2 7:41 AM
                • 0 Attachment
                  On 5/2/2008, Victor Duchovni (Victor.Duchovni@...) wrote:
                  > It's the best I can do until I find the cycles to fully document and
                  > then release the tee proxy I am using. I don't think it is appropriate
                  > to release it in an undocumented state.

                  Understandable, and I'm sure I and many others look forward to this...

                  With the advancing requirements of burdensome regulations, this (some
                  kind of basic archiving capability) will become more and more important,
                  and in my opinion, it is not unreasonable for postfix to provide an
                  integrated/built-in method of performing this function, to ensure a
                  robust and safe - ie, don't bounce if there is a problem with the
                  mirror, but queue until it is back up, etc - functionality.

                  It would also be nice if it could easily deliver to an appropriate
                  sub-folder - ie, one named after the local address part of the original
                  envelope recipient (the one(s) being tested for during recipient
                  validation stage when the primary server accepted the message for final
                  delivery)...

                  --

                  Best regards,

                  Charles
                • Victor Duchovni
                  ... I don t expect the proxy in question to ever be part of Postfix. Postfix supports at least 4 extension mechanisms: - Post-queue content filters - Pre-queue
                  Message 8 of 14 , May 2 9:10 AM
                  • 0 Attachment
                    On Fri, May 02, 2008 at 10:41:40AM -0400, Charles Marcus wrote:

                    > On 5/2/2008, Victor Duchovni (Victor.Duchovni@...) wrote:
                    > >It's the best I can do until I find the cycles to fully document and
                    > >then release the tee proxy I am using. I don't think it is appropriate
                    > >to release it in an undocumented state.
                    >
                    > Understandable, and I'm sure I and many others look forward to this...
                    >
                    > With the advancing requirements of burdensome regulations, this (some
                    > kind of basic archiving capability) will become more and more important,
                    > and in my opinion, it is not unreasonable for postfix to provide an
                    > integrated/built-in method of performing this function, to ensure a
                    > robust and safe - ie, don't bounce if there is a problem with the
                    > mirror, but queue until it is back up, etc - functionality.

                    I don't expect the proxy in question to ever be part of Postfix. Postfix
                    supports at least 4 extension mechanisms:

                    - Post-queue content filters
                    - Pre-queue proxy filters
                    - Milters
                    - Policy servers

                    It is up to the Postfix community and vendors to create add-on tools
                    that make use of these features. So I don't see a "buit-in" archive
                    feature any time soon.

                    > It would also be nice if it could easily deliver to an appropriate
                    > sub-folder - ie, one named after the local address part of the original
                    > envelope recipient (the one(s) being tested for during recipient
                    > validation stage when the primary server accepted the message for final
                    > delivery)...

                    The right mechanism leaves this choice to the administrator, the
                    archive copy is created and queued, after that you can deliver it
                    where-ever you want (configure the archive Postfix instance transport
                    rules accoringly). It would be wrong to make delivery decisions in the
                    archive module, they would never be sufficiently comprehensive.

                    --
                    Viktor.

                    Disclaimer: off-list followups get on-list replies or get ignored.
                    Please do not ignore the "Reply-To" header.

                    To unsubscribe from the postfix-users list, visit
                    http://www.postfix.org/lists.html or click the link below:
                    <mailto:majordomo@...?body=unsubscribe%20postfix-users>

                    If my response solves your problem, the best way to thank me is to not
                    send an "it worked, thanks" follow-up. If you must respond, please put
                    "It worked, thanks" in the "Subject" so I can delete these quickly.
                  • Michael Katz
                    ... You can do exactly this with MPP. I get a million lashes for mentioning our solution on the list, but until someone funds my life we have to charge for
                    Message 9 of 14 , May 2 9:53 AM
                    • 0 Attachment
                      Forums wrote:
                      > Hi Everyone,
                      >
                      > How do I make it so postfix will copy all incoming and outgoing emails to another email invisibly? It's a SOX requirement actually. I have to save all the emails.
                      >
                      > So, I create one account called "log". Every email that goes through the mail server has to be invisibly copied to that user whether incoming or outgoing. Thanks for the help. I'm new to postfix.
                      >
                      > Peter

                      You can do exactly this with MPP. I get a million lashes for mentioning
                      our solution on the list, but until someone funds my life we have to
                      charge for the software. messagepartners.com .

                      M Katz

                      >
                      >
                      >
                    • Wietse Venema
                      ... If we can agree on a usable MIME encapsulation, then it should be possible to spawn off a message in the cleanup server, after the Milter processing has
                      Message 10 of 14 , May 2 10:04 AM
                      • 0 Attachment
                        Victor Duchovni:
                        > On Fri, May 02, 2008 at 10:41:40AM -0400, Charles Marcus wrote:
                        >
                        > > On 5/2/2008, Victor Duchovni (Victor.Duchovni@...) wrote:
                        > > >It's the best I can do until I find the cycles to fully document and
                        > > >then release the tee proxy I am using. I don't think it is appropriate
                        > > >to release it in an undocumented state.
                        > >
                        > > Understandable, and I'm sure I and many others look forward to this...
                        > >
                        > > With the advancing requirements of burdensome regulations, this (some
                        > > kind of basic archiving capability) will become more and more important,
                        > > and in my opinion, it is not unreasonable for postfix to provide an
                        > > integrated/built-in method of performing this function, to ensure a
                        > > robust and safe - ie, don't bounce if there is a problem with the
                        > > mirror, but queue until it is back up, etc - functionality.
                        >
                        > I don't expect the proxy in question to ever be part of Postfix. Postfix
                        > supports at least 4 extension mechanisms:
                        >
                        > - Post-queue content filters
                        > - Pre-queue proxy filters
                        > - Milters
                        > - Policy servers
                        >
                        > It is up to the Postfix community and vendors to create add-on tools
                        > that make use of these features. So I don't see a "buit-in" archive
                        > feature any time soon.
                        >
                        > > It would also be nice if it could easily deliver to an appropriate
                        > > sub-folder - ie, one named after the local address part of the original
                        > > envelope recipient (the one(s) being tested for during recipient
                        > > validation stage when the primary server accepted the message for final
                        > > delivery)...
                        >
                        > The right mechanism leaves this choice to the administrator, the
                        > archive copy is created and queued, after that you can deliver it
                        > where-ever you want (configure the archive Postfix instance transport
                        > rules accoringly). It would be wrong to make delivery decisions in the
                        > archive module, they would never be sufficiently comprehensive.

                        If we can agree on a usable MIME encapsulation, then it should be
                        possible to spawn off a message in the cleanup server, after the
                        Milter processing has happened, and before the (SMTP) client is
                        notified that the mail transaction is complete.

                        However, Postfix is a general-purpose MTA, and you can already
                        configure dedicated delivery channels (with transport maps and
                        master.cf) that have soft-bounce turned on, so there is no need
                        for built-in special delivery modes that never bounce. Just set
                        the maximal queue time large enough.

                        Wietse
                      • Victor Duchovni
                        ... This is a bit tricky, because the is not necessarily a right answer. - IMHO, The natural format for an archive message is a success DSN, with the
                        Message 11 of 14 , May 2 10:37 AM
                        • 0 Attachment
                          On Fri, May 02, 2008 at 01:04:16PM -0400, Wietse Venema wrote:

                          > If we can agree on a usable MIME encapsulation, then it should be
                          > possible to spawn off a message in the cleanup server, after the
                          > Milter processing has happened, and before the (SMTP) client is
                          > notified that the mail transaction is complete.

                          This is a bit tricky, because the is not necessarily a "right" answer.

                          - IMHO, The natural format for an archive message is a success DSN,
                          with the original message attached in full (not just headers). Having
                          the format defined by a standard is IMHO rather attractive.

                          - Sendmail have for many years been supplying "copier" milter that
                          is perhaps a defacto standard for such encapsulation. This is a
                          multipart/mixed, with the envelope in the first part and the message
                          in the second. The first part encodes the sender and recipients one
                          per line as follows:

                          --boundary
                          Content-Type: text/plain
                          Content-Transfer-Encoding: 7bit

                          Attached is a copy of a message being sent by Sendmail Message Copier
                          Sendmail Copier.
                          via the proxy on <hostname.without.the.angle.brackets>.

                          Original sender: sender@...
                          Original recipient(s): rcpt1@...
                          rcpt2@...
                          rcpt3@...

                          Original message is attached.

                          --boundary

                          It would be far more natural to drop the verbiage and prefix each
                          address with a type:

                          Sender: <address>
                          Recipient: <address>
                          ...

                          The second part is the attached message.

                          - Microsoft Exchange has an archive format called
                          Exchange Envelope Journalling. This too is not ideal, but is
                          broadly implemented.

                          Any format we would choose would either be ugly (ad-hoc format from
                          existing vendor products) or Postfix-specific (at least initially).

                          The archive module for my 'tee' proxy punts the issue by using
                          format templates for the envelope part.

                          --boundary
                          Content-Type: text/plain
                          Content-Transfer-Encoding: 7bit

                          $topmatter
                          `printf "$senderfmt", $sender`
                          `printf "$1strcptfmt", $rcpt1`
                          `printf "$nxtrcptfmt", $rcpt2`
                          ...
                          $footer

                          --boundary

                          So I can generate the Sendmail compatible format without hard-coding
                          it.

                          This can generate any 2-part envelope/message encapsulation with some
                          stuff above the envelope, then the sender address in some form, then
                          the recipients with the 1st formatted differently if need be, a footer
                          and a separately attached message.

                          I also capture "ORCPT" values, but don't currently distinguish between
                          these and real recipients, because I don't accept "ORCPT" from outside,
                          rather ORCPT is the recipient just before my own virtual address
                          expansion. So capture of ORCPT would need to be optional and probably
                          deserves its own format template.

                          $topmatter
                          `printf "$senderfmt", $sender`
                          `printf "$1strcptfmt", $rcpt1`
                          [`printf "$orcptfmt", $orcpt1`]
                          `printf "$nxtrcptfmt", $rcpt2`
                          [`printf "$orcptfmt", $orcpt2`]
                          ...
                          $footer

                          With so much rope, is this still a reasonable Postfix feature?

                          --
                          Viktor.

                          Disclaimer: off-list followups get on-list replies or get ignored.
                          Please do not ignore the "Reply-To" header.

                          To unsubscribe from the postfix-users list, visit
                          http://www.postfix.org/lists.html or click the link below:
                          <mailto:majordomo@...?body=unsubscribe%20postfix-users>

                          If my response solves your problem, the best way to thank me is to not
                          send an "it worked, thanks" follow-up. If you must respond, please put
                          "It worked, thanks" in the "Subject" so I can delete these quickly.
                        • Wietse Venema
                          [About forking off an acrhive-copy message just before the cleanup server commits the queue file transaction] ... It should be able to provide one text/plain
                          Message 12 of 14 , May 2 10:57 AM
                          • 0 Attachment
                            [About forking off an acrhive-copy message just before the cleanup
                            server commits the queue file transaction]

                            Victor Duchovni:
                            > With so much rope, is this still a reasonable Postfix feature?

                            It should be able to provide one text/plain MIME segment with
                            original sender, sender, dsn original recipient, Postfix original
                            recipient; and one message/rfc822 MIME segment with the content.

                            As long as the fields in the first MIME segment have distinct
                            labels, I don't see this as particularly challenging. Given the
                            bounce templates as an example, the first MIME segment could even
                            be made configurable. But it would in all likelihood be a separate
                            daemon process.

                            Wietse
                          Your message has been successfully submitted and would be delivered to recipients shortly.