Loading ...
Sorry, an error occurred while loading the content.

Monitoring queue depth?

Expand Messages
  • David Wolfskill
    I recently started using the qshape script to help understand what s going on with some of our SMTP relays. I find it useful, but I think it would be helpful
    Message 1 of 16 , Oct 19, 2006
    • 0 Attachment
      I recently started using the qshape script to help understand what's
      going on with some of our SMTP relays.

      I find it useful, but I think it would be helpful to be able to just
      get a measure of the queue depth for the queues before deciding to use
      qshape to get more detail. Eventually, I'd like to be able to make
      "Postfix queue depth" something that is amenable to monitoring with
      (e.g.) Nagios.

      Perhaps I'm confused (maybe I shouldn't be wanting to do this?), but I
      don't see a straightforward way to get that type of information easily
      via an automated process.

      I'd appreciate suggestions and/or pointers. I'm not convinced that
      hacking qshape to report the information is especially useful or
      productive. :-}

      Thanks!

      Peace,
      david
      --
      David Wolfskill Trend Micro San Jose dhw@...
      cell: (650) 400-2312 office: (408) 625-1076 or (408) 453-6277 x124

      --
      David Wolfskill Trend Micro San Jose dhw@...
      cell: (650) 400-2312 office: (408) 625-1076 or (408) 453-6277 x124
    • Ralf Hildebrandt
      ... What is the queue depth? Simply counting the number of files in deferred/active? -- Ralf Hildebrandt (Ralf.Hildebrandt@charite.de)
      Message 2 of 16 , Oct 19, 2006
      • 0 Attachment
        * David Wolfskill <dhw@...>:

        > I find it useful, but I think it would be helpful to be able to just
        > get a measure of the queue depth for the queues before deciding to use
        > qshape to get more detail. Eventually, I'd like to be able to make
        > "Postfix queue depth" something that is amenable to monitoring with
        > (e.g.) Nagios.

        What is the queue depth?
        Simply counting the number of files in deferred/active?

        --
        Ralf Hildebrandt (Ralf.Hildebrandt@...) spamtrap@...
        Postfix - Einrichtung, Betrieb und Wartung Tel. +49 (0)30-450 570-155
        http://www.postfix-buch.com
        Why you can't find your system administrators:
        The admin electroculted themselves whilst installing some cabling near the main power cable. Since the admin's body is the only thing keeping the power flowing, the management boarded up the body and pretended they still had a system administrator.
      • David Wolfskill
        ... In this context, I believe so. It s the number that qshape reports as total when it s told to look at a single queue. And determining that number should
        Message 3 of 16 , Oct 19, 2006
        • 0 Attachment
          On Thu, Oct 19, 2006 at 10:24:04PM +0200, Ralf Hildebrandt wrote:
          >...
          >What is the queue depth?
          >Simply counting the number of files in deferred/active?

          In this context, I believe so. It's the number that qshape reports as
          "total" when it's told to look at a single queue.

          And determining that number should (I would think) be a rather low-cost,
          quick operation. It ought not require actually opening any of the files
          in a queue directory.

          Peace,
          david
          --
          David Wolfskill Trend Micro San Jose dhw@...
          cell: (650) 400-2312 office: (408) 625-1076 or (408) 453-6277 x124
        • Victor Duchovni
          ... # for q in maildrop incoming active deferred do echo $q $(find /var/spool/postfix/$q ! -type d -print | wc -l) done Yes, this is *much* faster than qshape,
          Message 4 of 16 , Oct 19, 2006
          • 0 Attachment
            On Thu, Oct 19, 2006 at 01:41:22PM -0700, David Wolfskill wrote:

            > On Thu, Oct 19, 2006 at 10:24:04PM +0200, Ralf Hildebrandt wrote:
            > >...
            > >What is the queue depth?
            > >Simply counting the number of files in deferred/active?
            >
            > In this context, I believe so. It's the number that qshape reports as
            > "total" when it's told to look at a single queue.
            >
            > And determining that number should (I would think) be a rather low-cost,
            > quick operation. It ought not require actually opening any of the files
            > in a queue directory.

            # for q in maildrop incoming active deferred
            do
            echo $q $(find /var/spool/postfix/$q ! -type d -print | wc -l)
            done

            Yes, this is *much* faster than qshape, but at least in the case of the
            deferred queue much less informative...

            --
            Viktor.

            Disclaimer: off-list followups get on-list replies or get ignored.
            Please do not ignore the "Reply-To" header.

            To unsubscribe from the postfix-users list, visit
            http://www.postfix.org/lists.html or click the link below:
            <mailto:majordomo@...?body=unsubscribe%20postfix-users>

            If my response solves your problem, the best way to thank me is to not
            send an "it worked, thanks" follow-up. If you must respond, please put
            "It worked, thanks" in the "Subject" so I can delete these quickly.
          • David Wolfskill
            ... OK. :-) ... Aye, though it s much easier for something like Nagios to cope with, and folks can bring out tools like qshape only when there s evidence that
            Message 5 of 16 , Oct 19, 2006
            • 0 Attachment
              On Thu, Oct 19, 2006 at 04:47:54PM -0400, Victor Duchovni wrote:
              >...
              > # for q in maildrop incoming active deferred
              > do
              > echo $q $(find /var/spool/postfix/$q ! -type d -print | wc -l)
              > done

              OK. :-)

              >Yes, this is *much* faster than qshape, but at least in the case of the
              >deferred queue much less informative...

              Aye, though it's much easier for something like Nagios to cope with, and
              folks can bring out tools like qshape only when there's evidence that
              such an effort is warranted.

              Thanks!

              Peace,
              david
              --
              David Wolfskill Trend Micro San Jose dhw@...
              cell: (650) 400-2312 office: (408) 625-1076 or (408) 453-6277 x124
            • Ralf Hildebrandt
              ... You could also use mailq to get a queue count, and then have a program tail the log to find out how much mails went into the system and how many left. Hmm,
              Message 6 of 16 , Oct 19, 2006
              • 0 Attachment
                * David Wolfskill <dhw@...>:

                > Aye, though it's much easier for something like Nagios to cope with, and
                > folks can bring out tools like qshape only when there's evidence that
                > such an effort is warranted.

                You could also use mailq to get a queue count, and then have a program
                tail the log to find out how much mails went into the system and how
                many left.

                Hmm, the qmgr could keep that information in memory as well!
                Something like "postfix queuestats" could output the number of mails
                in the different queues.

                --
                Ralf Hildebrandt (Ralf.Hildebrandt@...) spamtrap@...
                Postfix - Einrichtung, Betrieb und Wartung Tel. +49 (0)30-450 570-155
                http://www.postfix-buch.com
                "I think there is a world market for maybe five computers."
                - Thomas Watson, chairman of IBM, 1943.
              • David Wolfskill
                ... Could, though the quick census of files under each queue directory (cf. Victor s response) appears to be rather faster & less complicated -- at the risk of
                Message 7 of 16 , Oct 19, 2006
                • 0 Attachment
                  On Thu, Oct 19, 2006 at 11:01:14PM +0200, Ralf Hildebrandt wrote:
                  >...
                  >You could also use mailq to get a queue count, and then have a program
                  >tail the log to find out how much mails went into the system and how
                  >many left.

                  Could, though the quick census of files under each queue directory
                  (cf. Victor's response) appears to be rather faster & less complicated
                  -- at the risk of using an interface that probably isn't intended
                  for the purpose. :-}

                  >Hmm, the qmgr could keep that information in memory as well!

                  I expect it may well (though I haven't actually looked at the code).

                  >Something like "postfix queuestats" could output the number of mails
                  >in the different queues.

                  That would be rather handy, I'd think -- but then, perhaps other folks may
                  not find it of interest or use. For our situation, I believe it would
                  be quite useful indeed.

                  I'll probably cobble up something based on Victor's shell script snippet
                  to use for now.

                  Thanks for your (collective) help; back to lurking. :-)

                  Peace,
                  david

                  --
                  David Wolfskill Trend Micro San Jose dhw@...
                  cell: (650) 400-2312 office: (408) 625-1076 or (408) 453-6277 x124
                • Wietse Venema
                  ... The amount of memory used by Postfix must NEVER be proportional with the amount of mail in the queue, or the size of an individual email message. Wietse
                  Message 8 of 16 , Oct 20, 2006
                  • 0 Attachment
                    Ralf Hildebrandt:
                    > Hmm, the qmgr could keep that information in memory as well!
                    > Something like "postfix queuestats" could output the number of mails
                    > in the different queues.

                    The amount of memory used by Postfix must NEVER be proportional
                    with the amount of mail in the queue, or the size of an individual
                    email message.

                    Wietse
                  • Ralf Hildebrandt
                    ... How would a counter affect the memory in this way? There s only a limited number of queues (active, deferred, hold, incoming) and one counter (64bit?) for
                    Message 9 of 16 , Oct 20, 2006
                    • 0 Attachment
                      * Wietse Venema <wietse@...>:
                      > Ralf Hildebrandt:
                      > > Hmm, the qmgr could keep that information in memory as well!
                      > > Something like "postfix queuestats" could output the number of mails
                      > > in the different queues.
                      >
                      > The amount of memory used by Postfix must NEVER be proportional
                      > with the amount of mail in the queue, or the size of an individual
                      > email message.

                      How would a counter affect the memory in this way? There's only a
                      limited number of queues (active, deferred, hold, incoming) and one
                      counter (64bit?) for each queue.

                      --
                      Ralf Hildebrandt (Ralf.Hildebrandt@...) spamtrap@...
                      Postfix - Einrichtung, Betrieb und Wartung Tel. +49 (0)30-450 570-155
                      http://www.postfix-buch.com
                      If we can dispel the delusion that learning about computers should be
                      an activity of fiddling with array indexes and worrying whether X is
                      an integer or a real number, we can begin to focus on programming as a
                      source of ideas.
                    • Anton Yuzhaninov
                      Hello, Ralf. ... Yes, such counters in qmgr will very useful. -- WBR, Anton Yuzhaninov Rambler
                      Message 10 of 16 , Oct 20, 2006
                      • 0 Attachment
                        Hello, Ralf.

                        You wrote on Friday, October 20, 2006, 11:27:25 AM:

                        > * Wietse Venema <wietse@...>:
                        >> Ralf Hildebrandt:
                        >> > Hmm, the qmgr could keep that information in memory as well!
                        >> > Something like "postfix queuestats" could output the number of mails
                        >> > in the different queues.
                        >>
                        >> The amount of memory used by Postfix must NEVER be proportional
                        >> with the amount of mail in the queue, or the size of an individual
                        >> email message.

                        > How would a counter affect the memory in this way? There's only a
                        > limited number of queues (active, deferred, hold, incoming) and one
                        > counter (64bit?) for each queue.

                        Yes, such counters in qmgr will very useful.

                        --
                        WBR,
                        Anton Yuzhaninov
                        Rambler
                      • Cami
                        ... Quite useful. Having to run mailq|tail -1 or use find can be painfully long, especially on heavily loaded systems. Cami
                        Message 11 of 16 , Oct 20, 2006
                        • 0 Attachment
                          Anton Yuzhaninov wrote:
                          > Hello, Ralf.
                          >
                          > You wrote on Friday, October 20, 2006, 11:27:25 AM:
                          >
                          >> * Wietse Venema <wietse@...>:
                          >>> Ralf Hildebrandt:
                          >>>
                          >>>> Hmm, the qmgr could keep that information in memory as well!
                          >>>> Something like "postfix queuestats" could output the number of mails
                          >>>> in the different queues.
                          >>>
                          >>> The amount of memory used by Postfix must NEVER be proportional
                          >>> with the amount of mail in the queue, or the size of an individual
                          >>> email message.
                          >>
                          >> How would a counter affect the memory in this way? There's only a
                          >> limited number of queues (active, deferred, hold, incoming) and one
                          >> counter (64bit?) for each queue.

                          Quite useful. Having to run 'mailq|tail -1' or use 'find' can be
                          painfully long, especially on heavily loaded systems.

                          Cami
                        • Leandro Santi
                          ... The find recipe should be fast enough. Moreover, GNU find does leaf optimization: remove the -type f and no stat() syscall will be issued for leaf
                          Message 12 of 16 , Oct 20, 2006
                          • 0 Attachment
                            On 10/20/06, Cami <camis@...> wrote:

                            > Quite useful. Having to run 'mailq|tail -1' or use 'find' can be
                            > painfully long, especially on heavily loaded systems.

                            The 'find' recipe should be fast enough. Moreover, GNU find
                            does leaf optimization: remove the -type f and no stat() syscall
                            will be issued for leaf directory entries. This will accelerate
                            things further.

                            But, if you remove -type f, how'd you detect directories from
                            regular files, and avoid a false queue length count? Fortunately,
                            Postfix has a clever filesystem structure: queue sub-directories
                            have one-character names, and queue entries does not (they
                            have longer names):

                            # time mailq | tail -1; time /usr/local/bin/find
                            /var/spool/postfix/{incoming,active,deferred} -printf %f\\n | perl
                            -lne 'print if length > 1;' | wc -l
                            -- 30296 Kbytes in 1309 Requests.

                            real 0m0.902s
                            user 0m0.040s
                            sys 0m0.090s
                            1328

                            real 0m0.170s
                            user 0m0.100s
                            sys 0m0.110s

                            Leandro
                          • Wietse Venema
                            ... Hello, Ralf. ... Obviously, you need more than counters if you want to report more than just numbers. ... The queue manager by design does not know: 1) How
                            Message 13 of 16 , Oct 20, 2006
                            • 0 Attachment
                              Cami:
                              > Anton Yuzhaninov wrote:
                              Hello, Ralf.

                              >>> Hmm, the qmgr could keep that information in memory as well!
                              >>> Something like "postfix queuestats" could output the number of mails
                              >>> in the different queues.
                              >>
                              >> The amount of memory used by Postfix must NEVER be proportional
                              >> with the amount of mail in the queue, or the size of an individual
                              >> email message.

                              Obviously, you need more than counters if you want to report more
                              than just numbers.

                              > How would a counter affect the memory in this way? There's only a
                              > limited number of queues (active, deferred, hold, incoming) and one
                              > counter (64bit?) for each queue.

                              The queue manager by design does not know:

                              1) How much mail is in the deferred queue. It only knows whether
                              or not a deferred queue scan comes up empty handed.

                              2) How much mail is in the incoming or hold queue. It only knows
                              whether or not an incoming queue scan comes up empty handed.

                              3) How much mail is removed by the postsuper command.

                              4) How much mail the flush daemon moves to the incoming queue
                              after ETRN or equivalent.

                              I think it is bad design to insist that the queue manager knows
                              where everything is at all times.

                              Wietse
                            • Mark Martinec
                              Leandro, ... Good advice, thanks! ... Squeezing it further for fun (and for those of us not having -printf in find): find
                              Message 14 of 16 , Oct 20, 2006
                              • 0 Attachment
                                Leandro,

                                > The 'find' recipe should be fast enough. Moreover, GNU find
                                > does leaf optimization: remove the -type f and no stat() syscall
                                > will be issued for leaf directory entries. This will accelerate
                                > things further.

                                Good advice, thanks!

                                > find /var/spool/postfix/{incoming,active,deferred} -printf %f\\n | perl
                                > -lne 'print if length > 1;' | wc -l

                                Squeezing it further for fun (and for those of us not having -printf in find):

                                find /var/spool/postfix/{incoming,active,deferred} | \
                                awk -F/ '$NF~/../{print $NF}' | wc -l

                                Mark
                              • Leandro Santi
                                ... How about dropping the trailing wc: $ find /var/spool/postfix/{incoming,active,deferred} | awk -F/ { if (length($NF) 2) { num++; } } END { print num;
                                Message 15 of 16 , Oct 20, 2006
                                • 0 Attachment
                                  On 10/20/06, Mark Martinec <Mark.Martinec+postfix@...> wrote:

                                  > > find /var/spool/postfix/{incoming,active,deferred} -printf %f\\n | perl
                                  > > -lne 'print if length > 1;' | wc -l
                                  >
                                  > Squeezing it further for fun (and for those of us not having -printf in find):
                                  >
                                  > find /var/spool/postfix/{incoming,active,deferred} | \
                                  > awk -F/ '$NF~/../{print $NF}' | wc -l

                                  How about dropping the trailing wc:

                                  $ find /var/spool/postfix/{incoming,active,deferred} | awk -F/ '{ if
                                  (length($NF) > 2) { num++; } } END { print num; }'; find
                                  /var/spool/postfix/{incoming,active,deferred} | awk -F/
                                  '$NF~/../{print $NF}' | wc -l
                                  1200
                                  1199

                                  Does this code do what it should? I don't know about
                                  awk programming, handle it with extreme care :-)

                                  More stuff comes to mind: the {incoming,active,deferred}
                                  is non-portable, so it'd be nice to rewrite it using more
                                  portable bourne shell code.

                                  Leandro

                                  ps: I learnt the one-character trick from an old post
                                  by Victor, a few years back. I used a similar (i.e. leaf
                                  optimization-based) variation for measuring queue
                                  length for a custom monitoring, but Victor's trick is
                                  nicer IMO.
                                • David Wolfskill
                                  ... I realize it s been a couple of months since that discussion, but I ve managed to cobble up a (Perl) script to do that monitoring that appears to behave
                                  Message 16 of 16 , Dec 28, 2006
                                  • 0 Attachment
                                    On Thu, Oct 19, 2006 at 12:50:18PM -0700, DavidWolfskill< wrote:
                                    >...
                                    >I find [qshape] useful, but I think it would be helpful to be able to just
                                    >get a measure of the queue depth for the queues before deciding to use
                                    >qshape to get more detail. Eventually, I'd like to be able to make
                                    >"Postfix queue depth" something that is amenable to monitoring with
                                    >(e.g.) Nagios.
                                    >...
                                    >I'd appreciate suggestions and/or pointers. I'm not convinced that
                                    >hacking qshape to report the information is especially useful or
                                    >productive. :-}

                                    I realize it's been a couple of months since that discussion, but
                                    I've managed to cobble up a (Perl) script to do that monitoring
                                    that appears to behave pretty much as I'd like it to behave. Although
                                    we haven't started using it in conjunction with Nagios, the script
                                    itself seems to be suitable for invocation via a Nagios plug-in, as well
                                    as being invoked "manually" (in a way reminiscent of vmstat(8) or
                                    iostat(8)).

                                    Largely as a result of this lattter usage, I have been calling it
                                    "qstat." (Well, that, and my lack of creativity when it comes to
                                    names for things.) Despite the name similarity, there is one
                                    significant (and possibly misleading) difference: qstat merely shows a
                                    "snapshot" of the census of the specified queues, rather than indicating
                                    how many messages passed through the given queue(s) during the reported
                                    interval.

                                    The script does need to run under the auspices of a user who has
                                    permission to read the specified queue directories (and any
                                    subdirectories under them); it does not attempt to open or read the
                                    non-directory files within those hierarchies.

                                    I am indebted to several folks who contributed to the discussion on this
                                    list, and thus influenced its development; of course, bugs and other
                                    errors are strictly my doing.

                                    I'd like to make the script available for folks who are interested; it
                                    weighs in at a little under 6 KB (including comments & documentation, of
                                    course). I don't consider this something worth spamming postfix-devel,
                                    but am open to suggestions.

                                    So for now, I'll send copies of the script to folks who send me
                                    (private) mail requesting that; there's no need to spam this list with
                                    such requests.

                                    Peace,
                                    david
                                    --
                                    David Wolfskill Trend Micro San Jose dhw@...
                                    cell: (650) 400-2312 office: (408) 625-1076 or (408) 453-6277 x124
                                  Your message has been successfully submitted and would be delivered to recipients shortly.