Loading ...
Sorry, an error occurred while loading the content.

A weird puzzle

Expand Messages
  • Philip Mak
    About 10 hours ago, my website started getting really slow. My MRTG graph shows the server s bandwidth usage dropping to 1/4 of its previous amount:
    Message 1 of 5 , Feb 2, 2004
    • 0 Attachment
      About 10 hours ago, my website started getting really slow. My MRTG
      graph shows the server's bandwidth usage dropping to 1/4 of its
      previous amount:

      http://lina.aaanime.net/~pmak/dropoff.png

      When I got on the server, the load average was above 7. The shell
      prompt response time was very slow, the webpages took forever to load,
      but the CPU was over 90% idle, the swap used was less than the
      physical memory free, and the si/so/bi/bo values in "vmstat 1" were
      mostly 0.

      I couldn't figure out why it was lagging so badly. Then on a whim, I
      tried "rm -rf" on the StateDir. An empty StateDir was re-created by
      Apache::ASP, and then the lag problems mysteriously evaporated.

      Any ideas on how I can prevent this problem from reoccurring? (Damn, I
      should have saved the StateDir for further analysis instead of
      deleting it.)

      ---------------------------------------------------------------------
      To unsubscribe, e-mail: asp-unsubscribe@...
      For additional commands, e-mail: asp-help@...
    • Josh Chamas
      ... I have never seen this before. My advice would be to write a program to monitor your system load & send you an alert when its over a certain value. I
      Message 2 of 5 , Feb 2, 2004
      • 0 Attachment
        Philip Mak wrote:
        > About 10 hours ago, my website started getting really slow. My MRTG
        > graph shows the server's bandwidth usage dropping to 1/4 of its
        > previous amount:
        >
        > http://lina.aaanime.net/~pmak/dropoff.png
        >
        > When I got on the server, the load average was above 7. The shell
        > prompt response time was very slow, the webpages took forever to load,
        > but the CPU was over 90% idle, the swap used was less than the
        > physical memory free, and the si/so/bi/bo values in "vmstat 1" were
        > mostly 0.
        >
        > I couldn't figure out why it was lagging so badly. Then on a whim, I
        > tried "rm -rf" on the StateDir. An empty StateDir was re-created by
        > Apache::ASP, and then the lag problems mysteriously evaporated.
        >
        > Any ideas on how I can prevent this problem from reoccurring? (Damn, I
        > should have saved the StateDir for further analysis instead of
        > deleting it.)
        >

        I have never seen this before. My advice would be to write a program
        to monitor your system load & send you an alert when its over a certain value.
        I would use the 5 minute load value & not 1 minute as 1 minute can be
        fairly volatile.

        Yes, next time, please save the StateDir for possibly reproducing this.
        It may have been something like your site getting slammed by a bot/spider,
        and then the StateDir having to cleanup all the garbage sessions &/or
        deal with really huge internal databases tracking timeouts & such.
        Maybe it was some quirk to do with SDBM_File used for the internal
        database that is used with the session manager. I am not sure we will
        ever know now.

        Regards,

        Josh
        ________________________________________________________________
        Josh Chamas, Founder phone:925-552-0128
        Chamas Enterprises Inc. http://www.chamas.com
        NodeWorks Link Checker http://www.nodeworks.com


        ---------------------------------------------------------------------
        To unsubscribe, e-mail: asp-unsubscribe@...
        For additional commands, e-mail: asp-help@...
      • Philip Mak
        Gah, I just got the daily LogWatch e-mail for that server, and it s apparent why. I wonder if the disk is bad. I just had it replaced last month! ... WARNING:
        Message 3 of 5 , Feb 3, 2004
        • 0 Attachment
          Gah, I just got the daily LogWatch e-mail for that server, and it's
          apparent why.

          I wonder if the disk is bad. I just had it replaced last month!

          --------------------- Kernel Begin ------------------------

          WARNING: Kernel Errors Present
          end_request: I/O error, dev 03:02 (hda)...: 4507Time(s)
          end_request: I/O error, dev 03BAsect=29...: 9Time(s)
          hda: dma_intr: error=0x40 { Uncorrect...: 4296Time(s)
          hda: dma_intr: status=0x51 { DriveReady SeekComplete Error }...: 4304Time(s)
          hda: error waiting for DMA...: 2Time(s)
          hda: read_intr: error=0x40 { Uncorrect...: 192Time(s)
          hda: read_intr: status=0x59 { DriveReady SeekComplete DataRequest Error }...: 193Time(s)

          ---------------------- Kernel End -------------------------

          Feb 2 14:41:34 hda: dma_intr: status=0x51 { DriveReady SeekComplete Error }
          Feb 2 14:41:34 hda: dma_intr: error=0x40 { UncorrectableError }, LBAsect=293485, high=0, low=293485, sector=84632
          Feb 2 14:41:34 end_request: I/O error, dev 03:02 (hda), sector 84632
          Feb 2 14:41:34 hda: dma_intr: status=0x51 { DriveReady SeekComplete Error }
          Feb 2 14:41:40 hda: dma_intr: error=0x40 { UncorrectableError }, LBAsect=293485, high=0, low=293485, sector=84640
          Feb 2 14:41:41 end_request: I/O error, dev 03:02 (hda), sector 84640
          Feb 2 14:41:41 hda: dma_intr: status=0x51 { DriveReady SeekComplete Error }
          Feb 2 14:41:41 hda: dma_intr: error=0x40 { UncorrectableError }, LBAsect=293485, high=0, low=293485, sector=84640
          Feb 2 14:41:41 end_request: I/O error, dev 03:02 (hda), sector 84640
          Feb 2 14:41:49 hda: dma_intr: status=0x51 { DriveReady SeekComplete Error }
          Feb 2 14:41:49 hda: dma_intr: error=0x40 { UncorrectableError }, LBAsect=293485, high=0, low=293485, sector=84640

          (the rest all looks the same, just with later timestamps)

          ---------------------------------------------------------------------
          To unsubscribe, e-mail: asp-unsubscribe@...
          For additional commands, e-mail: asp-help@...
        • Chris Marschall
          Hi all, Can anyone tell me how to get the mmm-asp-perl.el file to work for syntax highlighting etc under Emacs. Thanks in advance. Regards, Chris. ... To
          Message 4 of 5 , Feb 3, 2004
          • 0 Attachment
            Hi all,

            Can anyone tell me how to get the mmm-asp-perl.el file to work for syntax
            highlighting etc under Emacs.

            Thanks in advance.

            Regards,

            Chris.


            ---------------------------------------------------------------------
            To unsubscribe, e-mail: asp-unsubscribe@...
            For additional commands, e-mail: asp-help@...
          • Josh Chamas
            ... It should be as easy as copying the text into your .emacs file, but this assumes that you already have mmm mode installed for emacs. For this see:
            Message 5 of 5 , Feb 3, 2004
            • 0 Attachment
              Chris Marschall wrote:
              > Hi all,
              >
              > Can anyone tell me how to get the mmm-asp-perl.el file to work for syntax
              > highlighting etc under Emacs.
              >

              It should be as easy as copying the text into your .emacs file, but this
              assumes that you already have mmm mode installed for emacs. For this see:

              http://mmm-mode.sourceforge.net/

              Regards,

              Josh

              ________________________________________________________________
              Josh Chamas, Founder phone:925-552-0128
              Chamas Enterprises Inc. http://www.chamas.com
              NodeWorks Link Checker http://www.nodeworks.com


              ---------------------------------------------------------------------
              To unsubscribe, e-mail: asp-unsubscribe@...
              For additional commands, e-mail: asp-help@...
            Your message has been successfully submitted and would be delivered to recipients shortly.