Loading ...
Sorry, an error occurred while loading the content.
 

Re: Programmatic access to the showq daemon/data

Expand Messages
  • Wietse Venema
    ... No. The mailq command (actually, postqueue -p) provides sendmail compatibility. It is not good for programmatic processing. The ad-hoc postqueue/showq
    Message 1 of 8 , Oct 1, 2007
      Ward, Martin:
      > Hi,
      >
      > Is there any kind of documentation on how to access the mail queue list
      > from within a program? Perl or C would be nice!

      No.

      The mailq command (actually, postqueue -p) provides sendmail
      compatibility. It is not good for programmatic processing.

      The ad-hoc postqueue/showq protocol needs to be replaced by one
      that is more suitable for programmatic processing, and then the
      legacy mailq needs to be implemented on top of that. you can
      then plug in your scripts without having to parse mailq output.

      > I ask because I have a number of mail servers that regularly have mail
      > queues > 300,000 emails and a lot of these queues are bounces and
      > double-bounces. Right now I have a simple script that someone wrote
      > (possibly it even came with the Postfix installation since I didn't
      > perform the installation) which runs the "postqueue -p" command,
      > grep/awks out the message ID of the unwanted emails and runs "postsuper
      > -d" for each ID.

      EEEEKS!

      Don't accept mail for bogus senders, so that you don't have to
      send bounces later!

      Talk about fixing the wrong problem at the wrong end!

      > The postqueue(1) command takes the longest to run so is the most obvious
      > starting point for trying to speed this process up, and it struck me
      > that if I could write a program that could directly access the mail
      > queue and get the IDs of the unwanted emails that way, it might be
      > quicker.

      The postqueue command takes long because it has to examine every
      queue file. There is no way that you can speed that up except by
      strong the entire mail queue metadata in a database.

      Wietse
    • Ward, Martin
      Re: Programmatic access to the showq daemon/data Unfortunately I can t do that without more power than most countries have, or at least a few assassins... I
      Message 2 of 8 , Oct 1, 2007
        Re: Programmatic access to the showq daemon/data
        Unfortunately I can't do that without more power than most countries have, or at least a few assassins...
         
        I work for an ISP so a lot of mail we try and route is junk (I don't like it but I have to live with it). The customer, quite rightly, won't accept it but since we already have we must deal with the concequences.
         
        |\/|artin


        From: Victor Duchovni [mailto:Victor.Duchovni@...]
        Sent: Mon 01-Oct-07 17:36
        To: Ward, Martin
        Cc: postfix-users@...
        Subject: Re: Programmatic access to the showq daemon/data

        On Mon, Oct 01, 2007 at 05:21:58PM +0100, Ward, Martin wrote:

        > Hi,
        >
        > Is there any kind of documentation on how
        to access the mail queue list
        > from within a program? Perl or C would be
        nice!
        >
        > I ask because I have a number of mail servers that
        regularly have mail
        > queues > 300,000 emails and a lot of these queues
        are bounces and
        > double-bounces.

        Fix this, instead of processing the consequences, fix the real problem,
        eliminate the source of bounces.

        --
                Viktor.

        Disclaimer: off-list followups get on-list replies or get ignored.
        Please do not ignore the "Reply-To" header.

        To unsubscribe from the postfix-users list, visit
        http://www.postfix.org/lists.html or click the link below:
        <mailto:majordomo@...?body=unsubscribe%20postfix-users>

        If my response solves your problem, the best way to thank me is to not
        send an "it worked, thanks" follow-up. If you must respond, please put
        "It worked, thanks" in the "Subject" so I can delete these quickly.



        *************************************************************************************
        The message is intended for the named addressee only and may not be disclosed to or used by anyone else, nor may it be copied in any way.

        The contents of this message and its attachments are confidential and may also be subject to legal privilege. If you are not the named addressee and/or have received this message in error, please advise us by e-mailing security@... and delete the message and any attachments without retaining any copies.

        Internet communications are not secure and COLT does not accept responsibility for this message, its contents nor responsibility for any viruses.

        No contracts can be created or varied on behalf of COLT Telecommunications, its subsidiaries or affiliates ("COLT") and any other party by email Communications unless expressly agreed in writing with such other party.

        Please note that incoming emails will be automatically scanned to eliminate potential viruses and unsolicited promotional emails. For more information refer to www.colt.net or contact us on +44(0)20 7390 3900.
      • Victor Duchovni
        ... Are you a backup MX provider? If so, phase out this service for customers who don t provide valid recipient lists. Consider enabling recipient verification
        Message 3 of 8 , Oct 1, 2007
          On Tue, Oct 02, 2007 at 12:05:42AM +0100, Ward, Martin wrote:

          > Unfortunately I can't do that without more power than most countries have,
          > or at least a few assassins...
          >
          > I work for an ISP so a lot of mail we try and route is junk (I don't like
          > it but I have to live with it). The customer, quite rightly, won't accept
          > it but since we already have we must deal with the concequences.

          Are you a backup MX provider? If so, phase out this service for customers
          who don't provide valid recipient lists. Consider enabling recipient
          verification for relay domains that don't provide valid user lists,
          but reject email for invalid addresses.

          Finally, if clients reject messages from your relay even for valid
          recipients, based on spam scores, ... phase out service to them also,
          one must not filter traffic from the backup MX, if the backup MX's
          anti-spam policy is not satisfactory, one must drop the find or build
          a more suitable backup MX whose ingress policy is satisfactory.

          --
          Viktor.

          Disclaimer: off-list followups get on-list replies or get ignored.
          Please do not ignore the "Reply-To" header.

          To unsubscribe from the postfix-users list, visit
          http://www.postfix.org/lists.html or click the link below:
          <mailto:majordomo@...?body=unsubscribe%20postfix-users>

          If my response solves your problem, the best way to thank me is to not
          send an "it worked, thanks" follow-up. If you must respond, please put
          "It worked, thanks" in the "Subject" so I can delete these quickly.
        • Ward, Martin
          ... mail ... Unfortunately I didn t design or implement this system and I have to work with what I have been given, a complete rewrite of the System is not
          Message 4 of 8 , Oct 2, 2007
            > > I ask because I have a number of mail servers that regularly have
            mail
            > > queues 300,000 emails and a lot of these queues are bounces and
            > > double-bounces. Right now I have a simple script that someone wrote
            > > (possibly it even came with the Postfix installation since I didn't
            > > perform the installation) which runs the "postqueue -p" command,
            > > grep/awks out the message ID of the unwanted emails and runs
            > > "postsuper -d" for each ID.
            >
            > EEEEKS!
            >
            > Don't accept mail for bogus senders, so that you don't have
            > to send bounces later!
            >
            > Talk about fixing the wrong problem at the wrong end!
            >

            Unfortunately I didn't design or implement this system and I have
            to work with what I have been given, a complete rewrite of the
            System is not going to happen! As for bogus senders, this is
            something that will be looked at but how can I verify them?

            1. DNS check of the mail domain, but mail domains are easily
            faked.
            2. SPF. Not implemented widely enough to be useful, but is
            certianly one thing I will be looking at.
            3. Forward/reverse resolution of the name/IP address of the
            sending server is not a good test by any means given the
            number of hosts that will route emails for other domains
            (my company offers such a service).

            How else can I use Postfix to verify the sender address?

            > The postqueue command takes long because it has to examine
            > every queue file. There is no way that you can speed that up
            > except by strong the entire mail queue metadata in a database.
            >

            Yes, I was afraid that was the case. I wanted to see if there
            was programmatic access to the postqueue/showq functionality
            so that I could at least limit the amount of processing involved
            In generating a list of mail senders.

            Thanks for your thoughts.

            |\/|artin


            *************************************************************************************
            The message is intended for the named addressee only and may not be disclosed to or used by anyone else, nor may it be copied in any way.

            The contents of this message and its attachments are confidential and may also be subject to legal privilege. If you are not the named addressee and/or have received this message in error, please advise us by e-mailing security@... and delete the message and any attachments without retaining any copies.

            Internet communications are not secure and COLT does not accept responsibility for this message, its contents nor responsibility for any viruses.

            No contracts can be created or varied on behalf of COLT Telecommunications, its subsidiaries or affiliates ("COLT") and any other party by email Communications unless expressly agreed in writing with such other party.

            Please note that incoming emails will be automatically scanned to eliminate potential viruses and unsolicited promotional emails. For more information refer to www.colt.net or contact us on +44(0)20 7390 3900.
          • Victor Duchovni
            ... Not bogus senders, incoming mail to bogus recipients, which you accept, but then cannot deliver and thus bounce, generating gobs of backscatter at joe-job
            Message 5 of 8 , Oct 2, 2007
              On Tue, Oct 02, 2007 at 09:40:16AM +0100, Ward, Martin wrote:

              > > > I ask because I have a number of mail servers that regularly have
              > mail
              > > > queues 300,000 emails and a lot of these queues are bounces and
              > > > double-bounces. Right now I have a simple script that someone wrote
              > > > (possibly it even came with the Postfix installation since I didn't
              > > > perform the installation) which runs the "postqueue -p" command,
              > > > grep/awks out the message ID of the unwanted emails and runs
              > > > "postsuper -d" for each ID.
              > >
              > > EEEEKS!
              > >
              > > Don't accept mail for bogus senders, so that you don't have
              > > to send bounces later!
              > >
              > > Talk about fixing the wrong problem at the wrong end!
              > >
              >
              > Unfortunately I didn't design or implement this system and I have
              > to work with what I have been given, a complete rewrite of the
              > System is not going to happen! As for bogus senders, this is
              > something that will be looked at but how can I verify them?

              Not bogus senders, incoming mail to bogus recipients, which you accept,
              but then cannot deliver and thus bounce, generating gobs of backscatter
              at joe-job victims and DoSing your own queue.

              > How else can I use Postfix to verify the sender address?

              The *recipient* address for downstream relay mail via:

              reject_unverified_recipient

              and suitable negative/positive cache lifetimes.

              http://www.postfix.org/ADDRESS_VERIFICATION_README.html#recipient

              --
              Viktor.

              Disclaimer: off-list followups get on-list replies or get ignored.
              Please do not ignore the "Reply-To" header.

              To unsubscribe from the postfix-users list, visit
              http://www.postfix.org/lists.html or click the link below:
              <mailto:majordomo@...?body=unsubscribe%20postfix-users>

              If my response solves your problem, the best way to thank me is to not
              send an "it worked, thanks" follow-up. If you must respond, please put
              "It worked, thanks" in the "Subject" so I can delete these quickly.
            • MacShane, Tracy
              ... mail ... You have to remember that most spammers are stupid. Simply by implementing reject_non_fqdn_hostname reject_invalid_hostname
              Message 6 of 8 , Oct 2, 2007
                > -----Original Message-----
                > From: owner-postfix-users@...
                > [mailto:owner-postfix-users@...] On Behalf Of Ward, Martin
                > Sent: Tuesday, 2 October 2007 6:40 PM
                > To: Postfix users
                > Subject: RE: Programmatic access to the showq daemon/data
                >
                > > > I ask because I have a number of mail servers that regularly have
                mail
                > > > queues 300,000 emails and a lot of these queues are bounces and
                > > > double-bounces.
                > >
                > > EEEEKS!
                > >
                > > Don't accept mail for bogus senders, so that you don't have to send
                > > bounces later!
                > >
                > > Talk about fixing the wrong problem at the wrong end!
                > >
                >
                > Unfortunately I didn't design or implement this system and I
                > have to work with what I have been given, a complete rewrite
                > of the System is not going to happen! As for bogus senders,
                > this is something that will be looked at but how can I verify them?
                >
                > 1. DNS check of the mail domain, but mail domains are easily
                > faked.

                You have to remember that most spammers are stupid. Simply by
                implementing

                reject_non_fqdn_hostname
                reject_invalid_hostname
                reject_non_fqdn_recipient (after permit_mynetworks, in our environment)
                reject_unknown_recipient_domain (probably not needed since we
                (obviously!) have reject_unauth_destination and good relay_recipient
                maps, but what the hey)
                reject_non_fqdn_sender
                reject_unknown_sender_domain
                reject_non_fqdn_recipient
                reject_unauth_pipelining,
                reject_multi_recipient_bounce

                in the smtpd restrictions and "strict_rfc821_envelopes = yes", we cut
                the amount of spam traffic we processed by 40%. Having a
                "check_client_access" map and blocking the top 20 networks that
                generated traffic to us (according to Pflogsumm reports - the biggies
                for us are comcast.net, tpnet.net, res.rr.com, verizon.net, etc) reduced
                the traffic we processed by another 20%. Adding the MAPS RBL (now owned
                by Trend Micro) means that we now only accept 20% of the traffic we did
                before we migrated to Postfix. Only 15% of that 20% is tagged as
                malware/spam by our content scanner.


                > 2. SPF. Not implemented widely enough to be useful, but is
                > certianly one thing I will be looking at.

                We have it implemented - we reject less than a dozen messages a day by
                using it, though.

                > 3. Forward/reverse resolution of the name/IP address of the
                > sending server is not a good test by any means given the
                > number of hosts that will route emails for other domains
                > (my company offers such a service).

                See above, re (mostly) "stupid spammers".

                >
                > How else can I use Postfix to verify the sender address?

                Whatever you do, don't implement sender verification. One of the smaller
                ISPs here in Australia implemented it, and promptly ended up on the MAPS
                RBL (due to the fact their outbound connections doubled (or more)
                overnight), so we were rejecting their mail. And we couldn't send *them*
                mail, because naturally we were rejecting their verification
                connections.

                For another strategy, you might want to look into "greylisting" (tons of
                docos on the web, including at Postfix.org). Most spammers seem not to
                have gotten the hang of that one yet either.
              Your message has been successfully submitted and would be delivered to recipients shortly.