Loading ...
Sorry, an error occurred while loading the content.

Re: Out: 452 Insufficient system storage

Expand Messages
  • Wietse Venema
    ... It may be worthwhile to run the Postfix fsspace test program. - Download any Postfix source code that compiles on your system. - cd into the source tree,
    Message 1 of 13 , Mar 2, 2010
    • 0 Attachment
      donovan jeffrey j:
      >
      > On Mar 2, 2010, at 7:31 PM, Daniel V. Reinhardt wrote:
      >
      > >> this is default on all my systems.
      > >
      > >>
      > >> MX1
      > >> /dev/disk1s3 77G 51G 26G 66% /
      > >>
      > >> MX2
      > >> /dev/disk0s3 234G 46G 187G 20% /
      > >>
      > >>

      It may be worthwhile to run the Postfix fsspace test program.

      - Download any Postfix source code that compiles on your system.

      - cd into the source tree, then execute the following commands:

      make makefiles
      cd src/util
      make fsspace
      ./fsspace /var/spool/postfix

      and report if the numbers look wrong.

      Postfix uses the fsspace routine to determine the amount of
      free space in the queue file system.

      Wietse
    • Stan Hoeppner
      ... What filesystem are you using? Are you running out of inodes? /$ df -i -- Stan
      Message 2 of 13 , Mar 2, 2010
      • 0 Attachment
        donovan jeffrey j put forth on 3/1/2010 8:06 AM:
        > Greetings
        >
        > I had several of these on my primary MX this weekend and one just popped
        > up. Can someone explain where this Insufficient system storage is ?

        What filesystem are you using? Are you running out of inodes?

        /$ df -i

        --
        Stan
      • Wietse Venema
        ... Not needed. The Postfix SMTP server logs this with verbose mode turned on: msg_info( %s: blocks %lu avail %lu min_free %lu msg_size_limit %lu , myname,
        Message 3 of 13 , Mar 2, 2010
        • 0 Attachment
          Wietse Venema:
          > donovan jeffrey j:
          > >
          > > On Mar 2, 2010, at 7:31 PM, Daniel V. Reinhardt wrote:
          > >
          > > >> this is default on all my systems.
          > > >
          > > >>
          > > >> MX1
          > > >> /dev/disk1s3 77G 51G 26G 66% /
          > > >>
          > > >> MX2
          > > >> /dev/disk0s3 234G 46G 187G 20% /
          > > >>
          > > >>
          >
          > It may be worthwhile to run the Postfix fsspace test program.

          Not needed. The Postfix SMTP server logs this with verbose mode
          turned on:

          msg_info("%s: blocks %lu avail %lu min_free %lu msg_size_limit %lu",
          myname,
          (unsigned long) fsbuf.block_size,
          (unsigned long) fsbuf.block_free,
          (unsigned long) var_queue_minfree,
          (unsigned long) var_message_limit);

          That is, it logs the file system blocksize, the number of free
          blocks, and the "queue_minfree" and "message_size_limit" Postfix
          parameter values.

          Based on these, it decides if the number of free blocks is more
          than queue_minfree, and if the message size limit in blocks is less
          than the free blocks/1.5.

          Wietse

          > - Download any Postfix source code that compiles on your system.
          >
          > - cd into the source tree, then execute the following commands:
          >
          > make makefiles
          > cd src/util
          > make fsspace
          > ./fsspace /var/spool/postfix
          >
          > and report if the numbers look wrong.
          >
          > Postfix uses the fsspace routine to determine the amount of
          > free space in the queue file system.
          >
          > Wietse
          >
          >
        • donovan jeffrey j
          ... mx1:/usr/local/postfix-2.7.0/src/util root# ./fsspace /var/spool/postfix /var/spool/postfix: block size 4096, blocks free 6836216
          Message 4 of 13 , Mar 2, 2010
          • 0 Attachment
            On Mar 2, 2010, at 9:03 PM, Wietse Venema wrote:

            > It may be worthwhile to run the Postfix fsspace test program.
            >
            > - Download any Postfix source code that compiles on your system.
            >
            > - cd into the source tree, then execute the following commands:
            >
            > make makefiles
            > cd src/util
            > make fsspace
            > ./fsspace /var/spool/postfix
            >
            > and report if the numbers look wrong.
            >
            > Postfix uses the fsspace routine to determine the amount of
            > free space in the queue file system.
            >
            > Wietse

            mx1:/usr/local/postfix-2.7.0/src/util root# ./fsspace /var/spool/postfix
            /var/spool/postfix: block size 4096, blocks free 6836216
            mx1:/usr/local/postfix-2.7.0/src/util root# df -i
            Filesystem 512-blocks Used Avail Capacity
            iused ifree %iused Mounted on
            /dev/disk0s3 160574256 105372640 54689616 66%
            13235578 6836202 66% /
            devfs 198 198 0 100%
            590 0 100% /dev
            fdesc 2 2 0 100%
            4 253 2% /dev
            <volfs> 1024 1024 0 100%
            0 0 100% /.vol
            /dev/disk1s3 489972528 46944240 443028288 10% 5868028
            55378536 10% /Volumes/SSHD2
            automount -nsl [212] 0 0 0 100%
            0 0 100% /Network
            automount -fstab [223] 0 0 0 100%
            0 0 100% /automount/Servers
            automount -static [223] 0 0 0 100%
            0 0 100% /automount/static
            /dev/disk2s3 489972528 105210640 384761888 21% 13151328
            48095236 21% /Volumes/MX1



            I was sitting on a memory upgrade, and I through some in , possibly
            some Virtual memory foo in the OS. i also did a block level clone to
            another larger drive /dev/disk2s3 as a backup. did a reboot. nothing
            yet. I'lll have to wait and see if maybe the reboot makes this vanish.
            still running on original drive. maybe the drive isn't so S.M.A.R.T

            stay tuned. :)

            -j
          • Wietse Venema
            ... That would be 26 GBytes. ... And that s 26 GBytes as well. It would be interesting to see what Postfix smtpd logs. You can turn it on selectively postconf
            Message 5 of 13 , Mar 2, 2010
            • 0 Attachment
              donovan jeffrey j:
              >
              > On Mar 2, 2010, at 9:03 PM, Wietse Venema wrote:
              >
              > > It may be worthwhile to run the Postfix fsspace test program.
              > >
              > > - Download any Postfix source code that compiles on your system.
              > >
              > > - cd into the source tree, then execute the following commands:
              > >
              > > make makefiles
              > > cd src/util
              > > make fsspace
              > > ./fsspace /var/spool/postfix
              > >
              > > and report if the numbers look wrong.
              > >
              > > Postfix uses the fsspace routine to determine the amount of
              > > free space in the queue file system.
              > >
              > > Wietse
              >
              > mx1:/usr/local/postfix-2.7.0/src/util root# ./fsspace /var/spool/postfix
              > /var/spool/postfix: block size 4096, blocks free 6836216

              That would be 26 GBytes.

              > Filesystem 512-blocks Used Avail Capacity
              > iused ifree %iused Mounted on
              > /dev/disk0s3 160574256 105372640 54689616 66%
              > 13235578 6836202 66% /

              And that's 26 GBytes as well.

              It would be interesting to see what Postfix smtpd logs. You can
              turn it on selectively

              postconf -e debug_peer_list=127.0.0.1
              postfix reload

              Then do "telnet 127.0.0.1 25", and "grep smtpd_check_queue /the/maillog/file".

              You can kill the logging with

              postconf -e debug_peer_list=

              Wietse
            • donovan jeffrey j
              ... i just tried this, i wasn t sure i was reading logs correctly. maillog is also being used by amavis, but i greped out postfix messages to a file, then
              Message 6 of 13 , Mar 3, 2010
              • 0 Attachment
                On Mar 3, 2010, at 12:09 AM, Wietse Venema wrote:

                > donovan jeffrey j:
                >> <SNIP> 54689616 66%
                >> 13235578 6836202 66% /
                >
                > And that's 26 GBytes as well.
                >
                > It would be interesting to see what Postfix smtpd logs. You can
                > turn it on selectively
                >
                > postconf -e debug_peer_list=127.0.0.1
                > postfix reload
                >
                > Then do "telnet 127.0.0.1 25", and "grep smtpd_check_queue /the/
                > maillog/file".
                >
                > You can kill the logging with
                >
                > postconf -e debug_peer_list=
                >
                > Wietse
                >

                i just tried this, i wasn't sure i was reading logs correctly. maillog
                is also being used by amavis, but i greped out postfix messages to a
                file, then tailed that file.
                in the short time I ran this I did not see the smtpd_check_queue . I
                am out of my maintenance window so I will try it later and let it run
                longer.

                Mar 3 09:49:59 mx1 postfix/smtp[1054]: name_mask: resource
                Mar 3 09:49:59 mx1 postfix/smtp[1054]: name_mask: software
                Mar 3 09:49:59 mx1 postfix/qmgr[603]: 0529299C4604: removed
                Mar 3 09:49:59 mx1 postfix/smtp[1054]: < 127.0.0.1[127.0.0.1]: 220
                [127.0.0.1] ESMTP amavisd-new service ready
                Mar 3 09:49:59 mx1 postfix/smtp[1054]: > 127.0.0.1[127.0.0.1]: EHLO
                mx1.beth.k12.pa.us
                Mar 3 09:49:59 mx1 postfix/smtp[1054]: < 127.0.0.1[127.0.0.1]: 250-
                [127.0.0.1]
                Mar 3 09:49:59 mx1 postfix/smtp[1054]: < 127.0.0.1[127.0.0.1]: 250-
                PIPELINING
                Mar 3 09:49:59 mx1 postfix/smtp[1054]: < 127.0.0.1[127.0.0.1]: 250-SIZE
                Mar 3 09:49:59 mx1 postfix/smtp[1054]: < 127.0.0.1[127.0.0.1]:
                250-8BITMIME
                Mar 3 09:49:59 mx1 postfix/smtp[1054]: < 127.0.0.1[127.0.0.1]: 250-
                ENHANCEDSTATUSCODES
                Mar 3 09:49:59 mx1 postfix/smtp[1054]: < 127.0.0.1[127.0.0.1]: 250
                XFORWARD NAME ADDR PROTO HELO
                Mar 3 09:49:59 mx1 postfix/smtp[1054]: server features: 0x78f size 0
                Mar 3 09:49:59 mx1 postfix/smtp[1054]: Using ESMTP PIPELINING, TCP
                send buffer size is 4096
                Mar 3 09:49:59 mx1 postfix/smtp[1054]: > 127.0.0.1[127.0.0.1]:
                XFORWARD NAME=mx2.beth.k12.pa.us ADDR=192.168.1.10
                Mar 3 09:49:59 mx1 postfix/smtp[1054]: > 127.0.0.1[127.0.0.1]:
                XFORWARD PROTO=ESMTP HELO=mx2.beth.k12.pa.us
                Mar 3 09:49:59 mx1 postfix/smtp[1054]: > 127.0.0.1[127.0.0.1]: MAIL
                FROM:<owner-macscrpt@...> SIZE=3652 BODY=8BITMIME
                Mar 3 09:49:59 mx1 postfix/smtp[1054]: > 127.0.0.1[127.0.0.1]: RCPT
                TO:<jdonovan@...>
                Mar 3 09:49:59 mx1 postfix/smtp[1054]: > 127.0.0.1[127.0.0.1]: DATA
                Mar 3 09:49:59 mx1 postfix/smtp[1054]: < 127.0.0.1[127.0.0.1]: 250
                2.5.0 Ok
                Mar 3 09:49:59 mx1 postfix/smtp[1054]: < 127.0.0.1[127.0.0.1]: 250
                2.5.0 Ok
                Mar 3 09:49:59 mx1 postfix/smtp[1054]: < 127.0.0.1[127.0.0.1]: 250
                2.1.0 Sender owner-macscrpt@... OK
                Mar 3 09:49:59 mx1 postfix/smtp[1054]: < 127.0.0.1[127.0.0.1]: 250
                2.1.5 Recipient jdonovan@... OK
                Mar 3 09:49:59 mx1 postfix/smtp[1054]: < 127.0.0.1[127.0.0.1]: 354
                End data with <CR><LF>.<CR><LF>
                Mar 3 09:49:59 mx1 postfix/smtp[1054]: > 127.0.0.1[127.0.0.1]: .
                Mar 3 09:49:59 mx1 postfix/smtp[1054]: > 127.0.0.1[127.0.0.1]: QUIT
                Mar 3 09:49:59 mx1 postfix/smtp[1066]: < 127.0.0.1[127.0.0.1]: 250
                2.6.0 Ok, id=01261-05, from MTA: 250 Ok: queued as 4DDF323BA248
                Mar 3 09:49:59 mx1 postfix/smtp[1066]: E1AEF99C4624: to=<droman@...
                >, relay=127.0.0.1[127.0.0.1], delay=3, status=sent (250 2.6.0 Ok,
                id=01261-05, from MTA: 250 Ok: queued as 4DDF323BA248)
                Mar 3 09:49:59 mx1 postfix/smtp[1066]: name_mask: resource
                Mar 3 09:49:59 mx1 postfix/smtp[1066]: name_mask: software
                Mar 3 09:49:59 mx1 postfix/qmgr[603]: E1AEF99C4624: removed
                Mar 3 09:49:59 mx1 postfix/smtp[1066]: < 127.0.0.1[127.0.0.1]: 220
                [127.0.0.1] ESMTP amavisd-new service ready
                Mar 3 09:49:59 mx1 postfix/smtp[1066]: > 127.0.0.1[127.0.0.1]: EHLO
                mx1.beth.k12.pa.us
                Mar 3 09:49:59 mx1 postfix/smtp[1066]: < 127.0.0.1[127.0.0.1]: 250-
                [127.0.0.1]
                Mar 3 09:49:59 mx1 postfix/smtp[1066]: < 127.0.0.1[127.0.0.1]: 250-
                PIPELINING
                Mar 3 09:49:59 mx1 postfix/smtp[1066]: < 127.0.0.1[127.0.0.1]: 250-SIZE
                Mar 3 09:49:59 mx1 postfix/smtp[1066]: < 127.0.0.1[127.0.0.1]:
                250-8BITMIME
                Mar 3 09:49:59 mx1 postfix/smtp[1066]: < 127.0.0.1[127.0.0.1]: 250-
                ENHANCEDSTATUSCODES
                Mar 3 09:49:59 mx1 postfix/smtp[1066]: < 127.0.0.1[127.0.0.1]: 250
                XFORWARD NAME ADDR PROTO HELO
                Mar 3 09:49:59 mx1 postfix/smtp[1066]: server features: 0x78f size 0
                Mar 3 09:49:59 mx1 postfix/smtp[1066]: Using ESMTP PIPELINING, TCP
                send buffer size is 4096
                Mar 3 09:49:59 mx1 postfix/smtp[1066]: > 127.0.0.1[127.0.0.1]:
                XFORWARD NAME=mx2.beth.k12.pa.us ADDR=192.168.1.10
                Mar 3 09:49:59 mx1 postfix/smtp[1066]: > 127.0.0.1[127.0.0.1]:
                XFORWARD PROTO=ESMTP HELO=mail2.beth.k12.pa.us
                Mar 3 09:49:59 mx1 postfix/smtp[1066]: > 127.0.0.1[127.0.0.1]: MAIL
                FROM:<Gordon.Moll@...> SIZE=6185
                Mar 3 09:49:59 mx1 postfix/smtp[1066]: > 127.0.0.1[127.0.0.1]: RCPT
                TO:<jmoll@...>
                Mar 3 09:49:59 mx1 postfix/smtp[1066]: > 127.0.0.1[127.0.0.1]: DATA
                Mar 3 09:49:59 mx1 postfix/smtp[1066]: < 127.0.0.1[127.0.0.1]: 250
                2.5.0 Ok
                Mar 3 09:49:59 mx1 postfix/smtp[1066]: < 127.0.0.1[127.0.0.1]: 250
                2.5.0 Ok
                Mar 3 09:49:59 mx1 postfix/smtp[1066]: < 127.0.0.1[127.0.0.1]: 250
                2.1.0 Sender Gordon.Moll@... OK
                Mar 3 09:49:59 mx1 postfix/smtp[1066]: < 127.0.0.1[127.0.0.1]: 250
                2.1.5 Recipient jmoll@... OK
                Mar 3 09:49:59 mx1 postfix/smtp[1066]: < 127.0.0.1[127.0.0.1]: 354
                End data with <CR><LF>.<CR><LF>
                Mar 3 09:49:59 mx1 postfix/smtp[1066]: > 127.0.0.1[127.0.0.1]: .
                Mar 3 09:49:59 mx1 postfix/smtp[1066]: > 127.0.0.1[127.0.0.1]: QUIT
                Mar 3 09:50:00 mx1 postfix/postfix-script: refreshing the Postfix
                mail system
                Mar 3 09:50:00 mx1 postfix/master[55]: reload configuration


                thank you for your time.
                -j
              • Wietse Venema
                ... You need to instrument the smtpd process after the content filter, Wietse
                Message 7 of 13 , Mar 3, 2010
                • 0 Attachment
                  donovan jeffrey j:
                  > Mar 3 09:49:59 mx1 postfix/smtp[1054]: name_mask: resource
                  > Mar 3 09:49:59 mx1 postfix/smtp[1054]: name_mask: software
                  > Mar 3 09:49:59 mx1 postfix/qmgr[603]: 0529299C4604: removed
                  > Mar 3 09:49:59 mx1 postfix/smtp[1054]: < 127.0.0.1[127.0.0.1]: 220
                  > [127.0.0.1] ESMTP amavisd-new service ready

                  You need to instrument the smtpd process after the content filter,

                  Wietse
                • donovan
                  ... This turned out to be an Autowhitelist growing out of control in /var/ clamav/.spamassassin 68816850944 Mar 6 16:39 auto-whitelist I suspect something is
                  Message 8 of 13 , Mar 6, 2010
                  • 0 Attachment
                    On Mar 3, 2010, at 5:24 PM, Wietse Venema wrote:

                    > donovan jeffrey j:
                    >> Mar 3 09:49:59 mx1 postfix/smtp[1054]: name_mask: resource
                    >> Mar 3 09:49:59 mx1 postfix/smtp[1054]: name_mask: software
                    >> Mar 3 09:49:59 mx1 postfix/qmgr[603]: 0529299C4604: removed
                    >> Mar 3 09:49:59 mx1 postfix/smtp[1054]: < 127.0.0.1[127.0.0.1]: 220
                    >> [127.0.0.1] ESMTP amavisd-new service ready
                    >
                    > You need to instrument the smtpd process after the content filter,
                    >
                    > Wietse
                    >

                    This turned out to be an Autowhitelist growing out of control in /var/
                    clamav/.spamassassin

                    68816850944 Mar 6 16:39 auto-whitelist

                    I suspect something is up with this heap.
                    -j
                  Your message has been successfully submitted and would be delivered to recipients shortly.