Loading ...
Sorry, an error occurred while loading the content.

Re: Out: 452 Insufficient system storage

Expand Messages
  • Stan Hoeppner
    ... What filesystem are you using? Are you running out of inodes? /$ df -i -- Stan
    Message 1 of 13 , Mar 2 6:12 PM
    • 0 Attachment
      donovan jeffrey j put forth on 3/1/2010 8:06 AM:
      > Greetings
      >
      > I had several of these on my primary MX this weekend and one just popped
      > up. Can someone explain where this Insufficient system storage is ?

      What filesystem are you using? Are you running out of inodes?

      /$ df -i

      --
      Stan
    • Wietse Venema
      ... Not needed. The Postfix SMTP server logs this with verbose mode turned on: msg_info( %s: blocks %lu avail %lu min_free %lu msg_size_limit %lu , myname,
      Message 2 of 13 , Mar 2 6:28 PM
      • 0 Attachment
        Wietse Venema:
        > donovan jeffrey j:
        > >
        > > On Mar 2, 2010, at 7:31 PM, Daniel V. Reinhardt wrote:
        > >
        > > >> this is default on all my systems.
        > > >
        > > >>
        > > >> MX1
        > > >> /dev/disk1s3 77G 51G 26G 66% /
        > > >>
        > > >> MX2
        > > >> /dev/disk0s3 234G 46G 187G 20% /
        > > >>
        > > >>
        >
        > It may be worthwhile to run the Postfix fsspace test program.

        Not needed. The Postfix SMTP server logs this with verbose mode
        turned on:

        msg_info("%s: blocks %lu avail %lu min_free %lu msg_size_limit %lu",
        myname,
        (unsigned long) fsbuf.block_size,
        (unsigned long) fsbuf.block_free,
        (unsigned long) var_queue_minfree,
        (unsigned long) var_message_limit);

        That is, it logs the file system blocksize, the number of free
        blocks, and the "queue_minfree" and "message_size_limit" Postfix
        parameter values.

        Based on these, it decides if the number of free blocks is more
        than queue_minfree, and if the message size limit in blocks is less
        than the free blocks/1.5.

        Wietse

        > - Download any Postfix source code that compiles on your system.
        >
        > - cd into the source tree, then execute the following commands:
        >
        > make makefiles
        > cd src/util
        > make fsspace
        > ./fsspace /var/spool/postfix
        >
        > and report if the numbers look wrong.
        >
        > Postfix uses the fsspace routine to determine the amount of
        > free space in the queue file system.
        >
        > Wietse
        >
        >
      • donovan jeffrey j
        ... mx1:/usr/local/postfix-2.7.0/src/util root# ./fsspace /var/spool/postfix /var/spool/postfix: block size 4096, blocks free 6836216
        Message 3 of 13 , Mar 2 7:54 PM
        • 0 Attachment
          On Mar 2, 2010, at 9:03 PM, Wietse Venema wrote:

          > It may be worthwhile to run the Postfix fsspace test program.
          >
          > - Download any Postfix source code that compiles on your system.
          >
          > - cd into the source tree, then execute the following commands:
          >
          > make makefiles
          > cd src/util
          > make fsspace
          > ./fsspace /var/spool/postfix
          >
          > and report if the numbers look wrong.
          >
          > Postfix uses the fsspace routine to determine the amount of
          > free space in the queue file system.
          >
          > Wietse

          mx1:/usr/local/postfix-2.7.0/src/util root# ./fsspace /var/spool/postfix
          /var/spool/postfix: block size 4096, blocks free 6836216
          mx1:/usr/local/postfix-2.7.0/src/util root# df -i
          Filesystem 512-blocks Used Avail Capacity
          iused ifree %iused Mounted on
          /dev/disk0s3 160574256 105372640 54689616 66%
          13235578 6836202 66% /
          devfs 198 198 0 100%
          590 0 100% /dev
          fdesc 2 2 0 100%
          4 253 2% /dev
          <volfs> 1024 1024 0 100%
          0 0 100% /.vol
          /dev/disk1s3 489972528 46944240 443028288 10% 5868028
          55378536 10% /Volumes/SSHD2
          automount -nsl [212] 0 0 0 100%
          0 0 100% /Network
          automount -fstab [223] 0 0 0 100%
          0 0 100% /automount/Servers
          automount -static [223] 0 0 0 100%
          0 0 100% /automount/static
          /dev/disk2s3 489972528 105210640 384761888 21% 13151328
          48095236 21% /Volumes/MX1



          I was sitting on a memory upgrade, and I through some in , possibly
          some Virtual memory foo in the OS. i also did a block level clone to
          another larger drive /dev/disk2s3 as a backup. did a reboot. nothing
          yet. I'lll have to wait and see if maybe the reboot makes this vanish.
          still running on original drive. maybe the drive isn't so S.M.A.R.T

          stay tuned. :)

          -j
        • Wietse Venema
          ... That would be 26 GBytes. ... And that s 26 GBytes as well. It would be interesting to see what Postfix smtpd logs. You can turn it on selectively postconf
          Message 4 of 13 , Mar 2 9:09 PM
          • 0 Attachment
            donovan jeffrey j:
            >
            > On Mar 2, 2010, at 9:03 PM, Wietse Venema wrote:
            >
            > > It may be worthwhile to run the Postfix fsspace test program.
            > >
            > > - Download any Postfix source code that compiles on your system.
            > >
            > > - cd into the source tree, then execute the following commands:
            > >
            > > make makefiles
            > > cd src/util
            > > make fsspace
            > > ./fsspace /var/spool/postfix
            > >
            > > and report if the numbers look wrong.
            > >
            > > Postfix uses the fsspace routine to determine the amount of
            > > free space in the queue file system.
            > >
            > > Wietse
            >
            > mx1:/usr/local/postfix-2.7.0/src/util root# ./fsspace /var/spool/postfix
            > /var/spool/postfix: block size 4096, blocks free 6836216

            That would be 26 GBytes.

            > Filesystem 512-blocks Used Avail Capacity
            > iused ifree %iused Mounted on
            > /dev/disk0s3 160574256 105372640 54689616 66%
            > 13235578 6836202 66% /

            And that's 26 GBytes as well.

            It would be interesting to see what Postfix smtpd logs. You can
            turn it on selectively

            postconf -e debug_peer_list=127.0.0.1
            postfix reload

            Then do "telnet 127.0.0.1 25", and "grep smtpd_check_queue /the/maillog/file".

            You can kill the logging with

            postconf -e debug_peer_list=

            Wietse
          • donovan jeffrey j
            ... i just tried this, i wasn t sure i was reading logs correctly. maillog is also being used by amavis, but i greped out postfix messages to a file, then
            Message 5 of 13 , Mar 3 10:00 AM
            • 0 Attachment
              On Mar 3, 2010, at 12:09 AM, Wietse Venema wrote:

              > donovan jeffrey j:
              >> <SNIP> 54689616 66%
              >> 13235578 6836202 66% /
              >
              > And that's 26 GBytes as well.
              >
              > It would be interesting to see what Postfix smtpd logs. You can
              > turn it on selectively
              >
              > postconf -e debug_peer_list=127.0.0.1
              > postfix reload
              >
              > Then do "telnet 127.0.0.1 25", and "grep smtpd_check_queue /the/
              > maillog/file".
              >
              > You can kill the logging with
              >
              > postconf -e debug_peer_list=
              >
              > Wietse
              >

              i just tried this, i wasn't sure i was reading logs correctly. maillog
              is also being used by amavis, but i greped out postfix messages to a
              file, then tailed that file.
              in the short time I ran this I did not see the smtpd_check_queue . I
              am out of my maintenance window so I will try it later and let it run
              longer.

              Mar 3 09:49:59 mx1 postfix/smtp[1054]: name_mask: resource
              Mar 3 09:49:59 mx1 postfix/smtp[1054]: name_mask: software
              Mar 3 09:49:59 mx1 postfix/qmgr[603]: 0529299C4604: removed
              Mar 3 09:49:59 mx1 postfix/smtp[1054]: < 127.0.0.1[127.0.0.1]: 220
              [127.0.0.1] ESMTP amavisd-new service ready
              Mar 3 09:49:59 mx1 postfix/smtp[1054]: > 127.0.0.1[127.0.0.1]: EHLO
              mx1.beth.k12.pa.us
              Mar 3 09:49:59 mx1 postfix/smtp[1054]: < 127.0.0.1[127.0.0.1]: 250-
              [127.0.0.1]
              Mar 3 09:49:59 mx1 postfix/smtp[1054]: < 127.0.0.1[127.0.0.1]: 250-
              PIPELINING
              Mar 3 09:49:59 mx1 postfix/smtp[1054]: < 127.0.0.1[127.0.0.1]: 250-SIZE
              Mar 3 09:49:59 mx1 postfix/smtp[1054]: < 127.0.0.1[127.0.0.1]:
              250-8BITMIME
              Mar 3 09:49:59 mx1 postfix/smtp[1054]: < 127.0.0.1[127.0.0.1]: 250-
              ENHANCEDSTATUSCODES
              Mar 3 09:49:59 mx1 postfix/smtp[1054]: < 127.0.0.1[127.0.0.1]: 250
              XFORWARD NAME ADDR PROTO HELO
              Mar 3 09:49:59 mx1 postfix/smtp[1054]: server features: 0x78f size 0
              Mar 3 09:49:59 mx1 postfix/smtp[1054]: Using ESMTP PIPELINING, TCP
              send buffer size is 4096
              Mar 3 09:49:59 mx1 postfix/smtp[1054]: > 127.0.0.1[127.0.0.1]:
              XFORWARD NAME=mx2.beth.k12.pa.us ADDR=192.168.1.10
              Mar 3 09:49:59 mx1 postfix/smtp[1054]: > 127.0.0.1[127.0.0.1]:
              XFORWARD PROTO=ESMTP HELO=mx2.beth.k12.pa.us
              Mar 3 09:49:59 mx1 postfix/smtp[1054]: > 127.0.0.1[127.0.0.1]: MAIL
              FROM:<owner-macscrpt@...> SIZE=3652 BODY=8BITMIME
              Mar 3 09:49:59 mx1 postfix/smtp[1054]: > 127.0.0.1[127.0.0.1]: RCPT
              TO:<jdonovan@...>
              Mar 3 09:49:59 mx1 postfix/smtp[1054]: > 127.0.0.1[127.0.0.1]: DATA
              Mar 3 09:49:59 mx1 postfix/smtp[1054]: < 127.0.0.1[127.0.0.1]: 250
              2.5.0 Ok
              Mar 3 09:49:59 mx1 postfix/smtp[1054]: < 127.0.0.1[127.0.0.1]: 250
              2.5.0 Ok
              Mar 3 09:49:59 mx1 postfix/smtp[1054]: < 127.0.0.1[127.0.0.1]: 250
              2.1.0 Sender owner-macscrpt@... OK
              Mar 3 09:49:59 mx1 postfix/smtp[1054]: < 127.0.0.1[127.0.0.1]: 250
              2.1.5 Recipient jdonovan@... OK
              Mar 3 09:49:59 mx1 postfix/smtp[1054]: < 127.0.0.1[127.0.0.1]: 354
              End data with <CR><LF>.<CR><LF>
              Mar 3 09:49:59 mx1 postfix/smtp[1054]: > 127.0.0.1[127.0.0.1]: .
              Mar 3 09:49:59 mx1 postfix/smtp[1054]: > 127.0.0.1[127.0.0.1]: QUIT
              Mar 3 09:49:59 mx1 postfix/smtp[1066]: < 127.0.0.1[127.0.0.1]: 250
              2.6.0 Ok, id=01261-05, from MTA: 250 Ok: queued as 4DDF323BA248
              Mar 3 09:49:59 mx1 postfix/smtp[1066]: E1AEF99C4624: to=<droman@...
              >, relay=127.0.0.1[127.0.0.1], delay=3, status=sent (250 2.6.0 Ok,
              id=01261-05, from MTA: 250 Ok: queued as 4DDF323BA248)
              Mar 3 09:49:59 mx1 postfix/smtp[1066]: name_mask: resource
              Mar 3 09:49:59 mx1 postfix/smtp[1066]: name_mask: software
              Mar 3 09:49:59 mx1 postfix/qmgr[603]: E1AEF99C4624: removed
              Mar 3 09:49:59 mx1 postfix/smtp[1066]: < 127.0.0.1[127.0.0.1]: 220
              [127.0.0.1] ESMTP amavisd-new service ready
              Mar 3 09:49:59 mx1 postfix/smtp[1066]: > 127.0.0.1[127.0.0.1]: EHLO
              mx1.beth.k12.pa.us
              Mar 3 09:49:59 mx1 postfix/smtp[1066]: < 127.0.0.1[127.0.0.1]: 250-
              [127.0.0.1]
              Mar 3 09:49:59 mx1 postfix/smtp[1066]: < 127.0.0.1[127.0.0.1]: 250-
              PIPELINING
              Mar 3 09:49:59 mx1 postfix/smtp[1066]: < 127.0.0.1[127.0.0.1]: 250-SIZE
              Mar 3 09:49:59 mx1 postfix/smtp[1066]: < 127.0.0.1[127.0.0.1]:
              250-8BITMIME
              Mar 3 09:49:59 mx1 postfix/smtp[1066]: < 127.0.0.1[127.0.0.1]: 250-
              ENHANCEDSTATUSCODES
              Mar 3 09:49:59 mx1 postfix/smtp[1066]: < 127.0.0.1[127.0.0.1]: 250
              XFORWARD NAME ADDR PROTO HELO
              Mar 3 09:49:59 mx1 postfix/smtp[1066]: server features: 0x78f size 0
              Mar 3 09:49:59 mx1 postfix/smtp[1066]: Using ESMTP PIPELINING, TCP
              send buffer size is 4096
              Mar 3 09:49:59 mx1 postfix/smtp[1066]: > 127.0.0.1[127.0.0.1]:
              XFORWARD NAME=mx2.beth.k12.pa.us ADDR=192.168.1.10
              Mar 3 09:49:59 mx1 postfix/smtp[1066]: > 127.0.0.1[127.0.0.1]:
              XFORWARD PROTO=ESMTP HELO=mail2.beth.k12.pa.us
              Mar 3 09:49:59 mx1 postfix/smtp[1066]: > 127.0.0.1[127.0.0.1]: MAIL
              FROM:<Gordon.Moll@...> SIZE=6185
              Mar 3 09:49:59 mx1 postfix/smtp[1066]: > 127.0.0.1[127.0.0.1]: RCPT
              TO:<jmoll@...>
              Mar 3 09:49:59 mx1 postfix/smtp[1066]: > 127.0.0.1[127.0.0.1]: DATA
              Mar 3 09:49:59 mx1 postfix/smtp[1066]: < 127.0.0.1[127.0.0.1]: 250
              2.5.0 Ok
              Mar 3 09:49:59 mx1 postfix/smtp[1066]: < 127.0.0.1[127.0.0.1]: 250
              2.5.0 Ok
              Mar 3 09:49:59 mx1 postfix/smtp[1066]: < 127.0.0.1[127.0.0.1]: 250
              2.1.0 Sender Gordon.Moll@... OK
              Mar 3 09:49:59 mx1 postfix/smtp[1066]: < 127.0.0.1[127.0.0.1]: 250
              2.1.5 Recipient jmoll@... OK
              Mar 3 09:49:59 mx1 postfix/smtp[1066]: < 127.0.0.1[127.0.0.1]: 354
              End data with <CR><LF>.<CR><LF>
              Mar 3 09:49:59 mx1 postfix/smtp[1066]: > 127.0.0.1[127.0.0.1]: .
              Mar 3 09:49:59 mx1 postfix/smtp[1066]: > 127.0.0.1[127.0.0.1]: QUIT
              Mar 3 09:50:00 mx1 postfix/postfix-script: refreshing the Postfix
              mail system
              Mar 3 09:50:00 mx1 postfix/master[55]: reload configuration


              thank you for your time.
              -j
            • Wietse Venema
              ... You need to instrument the smtpd process after the content filter, Wietse
              Message 6 of 13 , Mar 3 2:24 PM
              • 0 Attachment
                donovan jeffrey j:
                > Mar 3 09:49:59 mx1 postfix/smtp[1054]: name_mask: resource
                > Mar 3 09:49:59 mx1 postfix/smtp[1054]: name_mask: software
                > Mar 3 09:49:59 mx1 postfix/qmgr[603]: 0529299C4604: removed
                > Mar 3 09:49:59 mx1 postfix/smtp[1054]: < 127.0.0.1[127.0.0.1]: 220
                > [127.0.0.1] ESMTP amavisd-new service ready

                You need to instrument the smtpd process after the content filter,

                Wietse
              • donovan
                ... This turned out to be an Autowhitelist growing out of control in /var/ clamav/.spamassassin 68816850944 Mar 6 16:39 auto-whitelist I suspect something is
                Message 7 of 13 , Mar 6 4:36 PM
                • 0 Attachment
                  On Mar 3, 2010, at 5:24 PM, Wietse Venema wrote:

                  > donovan jeffrey j:
                  >> Mar 3 09:49:59 mx1 postfix/smtp[1054]: name_mask: resource
                  >> Mar 3 09:49:59 mx1 postfix/smtp[1054]: name_mask: software
                  >> Mar 3 09:49:59 mx1 postfix/qmgr[603]: 0529299C4604: removed
                  >> Mar 3 09:49:59 mx1 postfix/smtp[1054]: < 127.0.0.1[127.0.0.1]: 220
                  >> [127.0.0.1] ESMTP amavisd-new service ready
                  >
                  > You need to instrument the smtpd process after the content filter,
                  >
                  > Wietse
                  >

                  This turned out to be an Autowhitelist growing out of control in /var/
                  clamav/.spamassassin

                  68816850944 Mar 6 16:39 auto-whitelist

                  I suspect something is up with this heap.
                  -j
                Your message has been successfully submitted and would be delivered to recipients shortly.