Loading ...
Sorry, an error occurred while loading the content.

Re: Programmatic access to the showq daemon/data

Expand Messages
  • Victor Duchovni
    ... Fix this, instead of processing the consequences, fix the real problem, eliminate the source of bounces. -- Viktor. Disclaimer: off-list followups get
    Message 1 of 8 , Oct 1, 2007
    • 0 Attachment
      On Mon, Oct 01, 2007 at 05:21:58PM +0100, Ward, Martin wrote:

      > Hi,
      >
      > Is there any kind of documentation on how to access the mail queue list
      > from within a program? Perl or C would be nice!
      >
      > I ask because I have a number of mail servers that regularly have mail
      > queues > 300,000 emails and a lot of these queues are bounces and
      > double-bounces.

      Fix this, instead of processing the consequences, fix the real problem,
      eliminate the source of bounces.

      --
      Viktor.

      Disclaimer: off-list followups get on-list replies or get ignored.
      Please do not ignore the "Reply-To" header.

      To unsubscribe from the postfix-users list, visit
      http://www.postfix.org/lists.html or click the link below:
      <mailto:majordomo@...?body=unsubscribe%20postfix-users>

      If my response solves your problem, the best way to thank me is to not
      send an "it worked, thanks" follow-up. If you must respond, please put
      "It worked, thanks" in the "Subject" so I can delete these quickly.
    • Wietse Venema
      ... No. The mailq command (actually, postqueue -p) provides sendmail compatibility. It is not good for programmatic processing. The ad-hoc postqueue/showq
      Message 2 of 8 , Oct 1, 2007
      • 0 Attachment
        Ward, Martin:
        > Hi,
        >
        > Is there any kind of documentation on how to access the mail queue list
        > from within a program? Perl or C would be nice!

        No.

        The mailq command (actually, postqueue -p) provides sendmail
        compatibility. It is not good for programmatic processing.

        The ad-hoc postqueue/showq protocol needs to be replaced by one
        that is more suitable for programmatic processing, and then the
        legacy mailq needs to be implemented on top of that. you can
        then plug in your scripts without having to parse mailq output.

        > I ask because I have a number of mail servers that regularly have mail
        > queues > 300,000 emails and a lot of these queues are bounces and
        > double-bounces. Right now I have a simple script that someone wrote
        > (possibly it even came with the Postfix installation since I didn't
        > perform the installation) which runs the "postqueue -p" command,
        > grep/awks out the message ID of the unwanted emails and runs "postsuper
        > -d" for each ID.

        EEEEKS!

        Don't accept mail for bogus senders, so that you don't have to
        send bounces later!

        Talk about fixing the wrong problem at the wrong end!

        > The postqueue(1) command takes the longest to run so is the most obvious
        > starting point for trying to speed this process up, and it struck me
        > that if I could write a program that could directly access the mail
        > queue and get the IDs of the unwanted emails that way, it might be
        > quicker.

        The postqueue command takes long because it has to examine every
        queue file. There is no way that you can speed that up except by
        strong the entire mail queue metadata in a database.

        Wietse
      • Ward, Martin
        Re: Programmatic access to the showq daemon/data Unfortunately I can t do that without more power than most countries have, or at least a few assassins... I
        Message 3 of 8 , Oct 1, 2007
        • 0 Attachment
          Re: Programmatic access to the showq daemon/data
          Unfortunately I can't do that without more power than most countries have, or at least a few assassins...
           
          I work for an ISP so a lot of mail we try and route is junk (I don't like it but I have to live with it). The customer, quite rightly, won't accept it but since we already have we must deal with the concequences.
           
          |\/|artin


          From: Victor Duchovni [mailto:Victor.Duchovni@...]
          Sent: Mon 01-Oct-07 17:36
          To: Ward, Martin
          Cc: postfix-users@...
          Subject: Re: Programmatic access to the showq daemon/data

          On Mon, Oct 01, 2007 at 05:21:58PM +0100, Ward, Martin wrote:

          > Hi,
          >
          > Is there any kind of documentation on how
          to access the mail queue list
          > from within a program? Perl or C would be
          nice!
          >
          > I ask because I have a number of mail servers that
          regularly have mail
          > queues > 300,000 emails and a lot of these queues
          are bounces and
          > double-bounces.

          Fix this, instead of processing the consequences, fix the real problem,
          eliminate the source of bounces.

          --
                  Viktor.

          Disclaimer: off-list followups get on-list replies or get ignored.
          Please do not ignore the "Reply-To" header.

          To unsubscribe from the postfix-users list, visit
          http://www.postfix.org/lists.html or click the link below:
          <mailto:majordomo@...?body=unsubscribe%20postfix-users>

          If my response solves your problem, the best way to thank me is to not
          send an "it worked, thanks" follow-up. If you must respond, please put
          "It worked, thanks" in the "Subject" so I can delete these quickly.



          *************************************************************************************
          The message is intended for the named addressee only and may not be disclosed to or used by anyone else, nor may it be copied in any way.

          The contents of this message and its attachments are confidential and may also be subject to legal privilege. If you are not the named addressee and/or have received this message in error, please advise us by e-mailing security@... and delete the message and any attachments without retaining any copies.

          Internet communications are not secure and COLT does not accept responsibility for this message, its contents nor responsibility for any viruses.

          No contracts can be created or varied on behalf of COLT Telecommunications, its subsidiaries or affiliates ("COLT") and any other party by email Communications unless expressly agreed in writing with such other party.

          Please note that incoming emails will be automatically scanned to eliminate potential viruses and unsolicited promotional emails. For more information refer to www.colt.net or contact us on +44(0)20 7390 3900.
        • Victor Duchovni
          ... Are you a backup MX provider? If so, phase out this service for customers who don t provide valid recipient lists. Consider enabling recipient verification
          Message 4 of 8 , Oct 1, 2007
          • 0 Attachment
            On Tue, Oct 02, 2007 at 12:05:42AM +0100, Ward, Martin wrote:

            > Unfortunately I can't do that without more power than most countries have,
            > or at least a few assassins...
            >
            > I work for an ISP so a lot of mail we try and route is junk (I don't like
            > it but I have to live with it). The customer, quite rightly, won't accept
            > it but since we already have we must deal with the concequences.

            Are you a backup MX provider? If so, phase out this service for customers
            who don't provide valid recipient lists. Consider enabling recipient
            verification for relay domains that don't provide valid user lists,
            but reject email for invalid addresses.

            Finally, if clients reject messages from your relay even for valid
            recipients, based on spam scores, ... phase out service to them also,
            one must not filter traffic from the backup MX, if the backup MX's
            anti-spam policy is not satisfactory, one must drop the find or build
            a more suitable backup MX whose ingress policy is satisfactory.

            --
            Viktor.

            Disclaimer: off-list followups get on-list replies or get ignored.
            Please do not ignore the "Reply-To" header.

            To unsubscribe from the postfix-users list, visit
            http://www.postfix.org/lists.html or click the link below:
            <mailto:majordomo@...?body=unsubscribe%20postfix-users>

            If my response solves your problem, the best way to thank me is to not
            send an "it worked, thanks" follow-up. If you must respond, please put
            "It worked, thanks" in the "Subject" so I can delete these quickly.
          • Ward, Martin
            ... mail ... Unfortunately I didn t design or implement this system and I have to work with what I have been given, a complete rewrite of the System is not
            Message 5 of 8 , Oct 2, 2007
            • 0 Attachment
              > > I ask because I have a number of mail servers that regularly have
              mail
              > > queues 300,000 emails and a lot of these queues are bounces and
              > > double-bounces. Right now I have a simple script that someone wrote
              > > (possibly it even came with the Postfix installation since I didn't
              > > perform the installation) which runs the "postqueue -p" command,
              > > grep/awks out the message ID of the unwanted emails and runs
              > > "postsuper -d" for each ID.
              >
              > EEEEKS!
              >
              > Don't accept mail for bogus senders, so that you don't have
              > to send bounces later!
              >
              > Talk about fixing the wrong problem at the wrong end!
              >

              Unfortunately I didn't design or implement this system and I have
              to work with what I have been given, a complete rewrite of the
              System is not going to happen! As for bogus senders, this is
              something that will be looked at but how can I verify them?

              1. DNS check of the mail domain, but mail domains are easily
              faked.
              2. SPF. Not implemented widely enough to be useful, but is
              certianly one thing I will be looking at.
              3. Forward/reverse resolution of the name/IP address of the
              sending server is not a good test by any means given the
              number of hosts that will route emails for other domains
              (my company offers such a service).

              How else can I use Postfix to verify the sender address?

              > The postqueue command takes long because it has to examine
              > every queue file. There is no way that you can speed that up
              > except by strong the entire mail queue metadata in a database.
              >

              Yes, I was afraid that was the case. I wanted to see if there
              was programmatic access to the postqueue/showq functionality
              so that I could at least limit the amount of processing involved
              In generating a list of mail senders.

              Thanks for your thoughts.

              |\/|artin


              *************************************************************************************
              The message is intended for the named addressee only and may not be disclosed to or used by anyone else, nor may it be copied in any way.

              The contents of this message and its attachments are confidential and may also be subject to legal privilege. If you are not the named addressee and/or have received this message in error, please advise us by e-mailing security@... and delete the message and any attachments without retaining any copies.

              Internet communications are not secure and COLT does not accept responsibility for this message, its contents nor responsibility for any viruses.

              No contracts can be created or varied on behalf of COLT Telecommunications, its subsidiaries or affiliates ("COLT") and any other party by email Communications unless expressly agreed in writing with such other party.

              Please note that incoming emails will be automatically scanned to eliminate potential viruses and unsolicited promotional emails. For more information refer to www.colt.net or contact us on +44(0)20 7390 3900.
            • Victor Duchovni
              ... Not bogus senders, incoming mail to bogus recipients, which you accept, but then cannot deliver and thus bounce, generating gobs of backscatter at joe-job
              Message 6 of 8 , Oct 2, 2007
              • 0 Attachment
                On Tue, Oct 02, 2007 at 09:40:16AM +0100, Ward, Martin wrote:

                > > > I ask because I have a number of mail servers that regularly have
                > mail
                > > > queues 300,000 emails and a lot of these queues are bounces and
                > > > double-bounces. Right now I have a simple script that someone wrote
                > > > (possibly it even came with the Postfix installation since I didn't
                > > > perform the installation) which runs the "postqueue -p" command,
                > > > grep/awks out the message ID of the unwanted emails and runs
                > > > "postsuper -d" for each ID.
                > >
                > > EEEEKS!
                > >
                > > Don't accept mail for bogus senders, so that you don't have
                > > to send bounces later!
                > >
                > > Talk about fixing the wrong problem at the wrong end!
                > >
                >
                > Unfortunately I didn't design or implement this system and I have
                > to work with what I have been given, a complete rewrite of the
                > System is not going to happen! As for bogus senders, this is
                > something that will be looked at but how can I verify them?

                Not bogus senders, incoming mail to bogus recipients, which you accept,
                but then cannot deliver and thus bounce, generating gobs of backscatter
                at joe-job victims and DoSing your own queue.

                > How else can I use Postfix to verify the sender address?

                The *recipient* address for downstream relay mail via:

                reject_unverified_recipient

                and suitable negative/positive cache lifetimes.

                http://www.postfix.org/ADDRESS_VERIFICATION_README.html#recipient

                --
                Viktor.

                Disclaimer: off-list followups get on-list replies or get ignored.
                Please do not ignore the "Reply-To" header.

                To unsubscribe from the postfix-users list, visit
                http://www.postfix.org/lists.html or click the link below:
                <mailto:majordomo@...?body=unsubscribe%20postfix-users>

                If my response solves your problem, the best way to thank me is to not
                send an "it worked, thanks" follow-up. If you must respond, please put
                "It worked, thanks" in the "Subject" so I can delete these quickly.
              • MacShane, Tracy
                ... mail ... You have to remember that most spammers are stupid. Simply by implementing reject_non_fqdn_hostname reject_invalid_hostname
                Message 7 of 8 , Oct 2, 2007
                • 0 Attachment
                  > -----Original Message-----
                  > From: owner-postfix-users@...
                  > [mailto:owner-postfix-users@...] On Behalf Of Ward, Martin
                  > Sent: Tuesday, 2 October 2007 6:40 PM
                  > To: Postfix users
                  > Subject: RE: Programmatic access to the showq daemon/data
                  >
                  > > > I ask because I have a number of mail servers that regularly have
                  mail
                  > > > queues 300,000 emails and a lot of these queues are bounces and
                  > > > double-bounces.
                  > >
                  > > EEEEKS!
                  > >
                  > > Don't accept mail for bogus senders, so that you don't have to send
                  > > bounces later!
                  > >
                  > > Talk about fixing the wrong problem at the wrong end!
                  > >
                  >
                  > Unfortunately I didn't design or implement this system and I
                  > have to work with what I have been given, a complete rewrite
                  > of the System is not going to happen! As for bogus senders,
                  > this is something that will be looked at but how can I verify them?
                  >
                  > 1. DNS check of the mail domain, but mail domains are easily
                  > faked.

                  You have to remember that most spammers are stupid. Simply by
                  implementing

                  reject_non_fqdn_hostname
                  reject_invalid_hostname
                  reject_non_fqdn_recipient (after permit_mynetworks, in our environment)
                  reject_unknown_recipient_domain (probably not needed since we
                  (obviously!) have reject_unauth_destination and good relay_recipient
                  maps, but what the hey)
                  reject_non_fqdn_sender
                  reject_unknown_sender_domain
                  reject_non_fqdn_recipient
                  reject_unauth_pipelining,
                  reject_multi_recipient_bounce

                  in the smtpd restrictions and "strict_rfc821_envelopes = yes", we cut
                  the amount of spam traffic we processed by 40%. Having a
                  "check_client_access" map and blocking the top 20 networks that
                  generated traffic to us (according to Pflogsumm reports - the biggies
                  for us are comcast.net, tpnet.net, res.rr.com, verizon.net, etc) reduced
                  the traffic we processed by another 20%. Adding the MAPS RBL (now owned
                  by Trend Micro) means that we now only accept 20% of the traffic we did
                  before we migrated to Postfix. Only 15% of that 20% is tagged as
                  malware/spam by our content scanner.


                  > 2. SPF. Not implemented widely enough to be useful, but is
                  > certianly one thing I will be looking at.

                  We have it implemented - we reject less than a dozen messages a day by
                  using it, though.

                  > 3. Forward/reverse resolution of the name/IP address of the
                  > sending server is not a good test by any means given the
                  > number of hosts that will route emails for other domains
                  > (my company offers such a service).

                  See above, re (mostly) "stupid spammers".

                  >
                  > How else can I use Postfix to verify the sender address?

                  Whatever you do, don't implement sender verification. One of the smaller
                  ISPs here in Australia implemented it, and promptly ended up on the MAPS
                  RBL (due to the fact their outbound connections doubled (or more)
                  overnight), so we were rejecting their mail. And we couldn't send *them*
                  mail, because naturally we were rejecting their verification
                  connections.

                  For another strategy, you might want to look into "greylisting" (tons of
                  docos on the web, including at Postfix.org). Most spammers seem not to
                  have gotten the hang of that one yet either.
                Your message has been successfully submitted and would be delivered to recipients shortly.