Loading ...
Sorry, an error occurred while loading the content.

Bounce Processing "Best Practices?"

Expand Messages
  • Steve Jenkins
    Happy New Year to you all. This is a best practices question for other Postfix users who may be using Postfix to send email to large opt-in mailing lists. We
    Message 1 of 3 , Jan 1, 2011
    • 0 Attachment
      Happy New Year to you all.

      This is a "best practices" question for other Postfix users who may be using
      Postfix to send email to large opt-in mailing lists.

      We have a subscriber list of 1MM+ registered members of a popular video game
      website. The vast majority of them are also opted in for our monthly
      newsletter (a separate checkbox during the registration process), but we
      haven't sent out a newsletter in a few years. We've only been sending
      transactional email to the "active" website members, as well as a few
      thousand daily "new content" alerts to the users that have asked for them in
      their notification options.

      In getting our newsletter campaigns ready to send, we believe we've done
      everything all the big ESPs ask: a relatively up-to-date version of Postfix
      (2.3.3), a dedicated IP with a good reputation and no blacklists, valid SPF
      record, valid DKIM signing, and we've signed up for every FBL we could find.

      Before sending a newsletter to that large of a group, we decided to first
      send out an account confirmation message to all our opted-in subscribers
      which includes their name, login information, subscription date, and links
      to unsubscribe and/or otherwise manage their delivery settings. The two
      goals of this message were to a) do some much-needed housekeeping on our
      list to clean out old addresses that bounce, and b) verify that the
      addresses which don't bounce still want to hear from us and give them a
      chance to unsubscribe if they don't.

      We used Postfix to set up a number of virtual mailboxes for the FBL accounts
      and one to receive unsubscribe requests from clients that support the
      List-Unsubscribe: header. We also used Postfix to set up a virtual mail host
      with a catchall address @... using virtual_mailbox_maps =
      hash:/etc/postfix/vmailbox. We then sent out the account confirmation
      messages using a custom Return-Path: for each message that is a combination
      of the subscriber's user ID, then a plus sign as a delimiter, then an MD5
      string (1234567+4s6d5f4a9s8d7f6a5d4@...). This is different
      than the approach we've been taking with our transactional and alerts mails,
      where we've been using a static Return-Path: to a virtual mailbox file
      @... and then grep it nightly and unsubscribe any bounces we find
      in there.

      Sending is happening right now, and so far, everything seems to be working
      nicely (we've only got a couple hundred thousand messages to go). Watching
      the maillog in real-time, the ESPs are accepting our mails for delivery, the
      FBL virtual mailboxes are receiving a moderate number of messages, the
      List-Unsubscribe virtual mailbox is receiving a few, and the bounce catchall
      is receiving A LOT of bounces. As expected, the older accounts are bouncing
      the most.

      So with all that explained, I have few questions:

      1) What's the optimal way for us to process the bounces? Our plan is to grep
      the catchall file and unsubscribe based on the user ID in the original
      Return-Path. But will that get them ALL?

      2) Should we also be grepping our /var/log/maillog* files and looking for
      5xx errors? Or will those all end up in the catchall eventually, after
      Postfix gives up trying to deliver? Should we be looking in our logs for
      anything else to unsubscribe users?

      3) Would we be better off using VERP instead of our custom Return-Path? I've
      read through http://www.postfix.org/VERP_README.html but can't tell if that
      gives us any advantage over the approach we're using.

      4) We want to be good netizens and responsible senders, so can anyone see
      any additional steps (general or specific) that we're NOT doing but should
      be?

      Thanks and Happy New Year,

      Steve
    • Jerry
      On Sat, 1 Jan 2011 21:36:58 -0800 Steve Jenkins articulated: [snip] ... I think you may seriously want to reexamine the term:
      Message 2 of 3 , Jan 2, 2011
      • 0 Attachment
        On Sat, 1 Jan 2011 21:36:58 -0800
        Steve Jenkins <steve@...> articulated:

        [snip]

        > In getting our newsletter campaigns ready to send, we believe we've
        > done everything all the big ESPs ask: a relatively up-to-date version
        > of Postfix (2.3.3), a dedicated IP with a good reputation and no
        > blacklists, valid SPF record, valid DKIM signing, and we've signed up
        > for every FBL we could find.

        I think you may seriously want to reexamine the term: "relatively
        up-to-date". Unless there is some over riding reason for using this
        version, I would strongly recommend updating to the latest stable
        version.

        > 3) Would we be better off using VERP instead of our custom
        > Return-Path? I've read through
        > http://www.postfix.org/VERP_README.html but can't tell if that gives
        > us any advantage over the approach we're using.

        On the mailing lists that I have worked on, we have always employed
        "VERP". I has worked very well and is usually easily understood. Most
        mailing list management programs can easily implement it.

        --
        Jerry ✌
        postfix-user@...
        _____________________________________________________________________
        TO REPORT A PROBLEM see http://www.postfix.org/DEBUG_README.html#mail
        TO (UN)SUBSCRIBE see http://www.postfix.org/lists.html
      • Andrew Beverley
        ... ... I don t see why that wouldn t get them all, but I personally pipe the bounces straight into a perl script from Postfix and remove them using
        Message 3 of 3 , Jan 2, 2011
        • 0 Attachment
          On Sat, 2011-01-01 at 21:36 -0800, Steve Jenkins wrote:

          > This is a "best practices" question for other Postfix users who may be using
          > Postfix to send email to large opt-in mailing lists.
          >

          <snip>

          > So with all that explained, I have few questions:
          >
          > 1) What's the optimal way for us to process the bounces? Our plan is to grep
          > the catchall file and unsubscribe based on the user ID in the original
          > Return-Path. But will that get them ALL?

          I don't see why that wouldn't get them all, but I personally pipe the
          bounces straight into a perl script from Postfix and remove them using
          that. This means that they get removed straight away automatically.

          > 2) Should we also be grepping our /var/log/maillog* files and looking for
          > 5xx errors? Or will those all end up in the catchall eventually, after
          > Postfix gives up trying to deliver? Should we be looking in our logs for
          > anything else to unsubscribe users?

          Just processing all the undeliverables should capture everything.

          > 3) Would we be better off using VERP instead of our custom Return-Path? I've
          > read through http://www.postfix.org/VERP_README.html but can't tell if that
          > gives us any advantage over the approach we're using.

          I can't think of any specific advantages that it would give you, but the
          technique is already built-in to Postfix and recognised as a fairly
          standard way of processing bounces.

          > 4) We want to be good netizens and responsible senders, so can anyone see
          > any additional steps (general or specific) that we're NOT doing but should
          > be?
          >

          All looks good to me. The other things that you might want to consider
          are:

          - Applying for whitelist status from those email providers that operate
          one (such as AOL).

          - Use different IP addresses for different types of mailings (for
          example one IP address for your known good recipients and one for your
          initial mailing)

          - Use a whitelisting agent SuretyMail

          Andy
        Your message has been successfully submitted and would be delivered to recipients shortly.