Loading ...
Sorry, an error occurred while loading the content.
 

Apache::ASP locking up

Expand Messages
  • Warren Young
    We have a site where Apache::ASP appears to be locking up every few days. If we rm -r /tmp/asp, the system begins responding again. We ve upgraded to the
    Message 1 of 10 , Nov 18, 2004
      We have a site where Apache::ASP appears to be locking up every few
      days. If we rm -r /tmp/asp, the system begins responding again.

      We've upgraded to the latest Apache::ASP, and it doesn't help. The
      system is a Red Hat Linux 7.3 machine running Apache 1.3.27-4. None of
      our many other sites are having this problem, to the best of our
      knowledge. (They're not under our direct control.) On the other hand,
      we have an incredible diversity in our installed systems. So, there's
      no telling whether the problem is the machine's software configuration,
      or it's part of the environment the machine is being used in, or it's
      the users.

      Here's what 'ls -lR /tmp/asp' says:

      [root@Meadowlawn root]# ls -lR /tmp/asp
      /tmp/asp:
      total 48
      drwxr-x--- 2 apache apache 4096 Nov 18 05:03 00/
      drwxr-x--- 2 apache apache 4096 Nov 18 08:55 01/
      drwxr-x--- 2 apache apache 4096 Nov 18 09:31 02/
      drwxr-x--- 2 apache apache 4096 Nov 17 15:48 03/
      drwxr-x--- 2 apache apache 4096 Nov 18 09:24 04/
      drwxr-x--- 2 apache apache 4096 Nov 18 09:16 05/
      drwxr-x--- 2 apache apache 4096 Nov 18 09:24 06/
      drwxr-x--- 2 apache apache 4096 Nov 18 09:26 09/
      drwxr-x--- 2 apache apache 4096 Nov 18 09:16 0c/
      drwxr-x--- 2 apache apache 4096 Nov 18 09:16 0e/
      drwxr-x--- 2 apache apache 4096 Nov 18 08:51 0f/
      drwxr-x--- 2 apache apache 4096 Nov 18 08:51 server/

      /tmp/asp/00:
      total 0

      /tmp/asp/01:
      total 0

      /tmp/asp/02:
      total 16
      -rw-r----- 1 apache apache 0 Nov 18 09:31
      0245e15d91fa6bc24f45455ebb1b118c.dir
      -rw-rw-rw- 1 apache apache 0 Nov 18 09:31
      0245e15d91fa6bc24f45455ebb1b118c.lock
      -rw-r----- 1 apache apache 8192 Nov 18 09:31
      0245e15d91fa6bc24f45455ebb1b118c.pag
      -rw-r----- 1 apache apache 0 Nov 18 09:25
      02cc91f1cd27bbe3dcd8d2da1bfd5c6f.dir
      -rw-rw-rw- 1 apache apache 0 Nov 18 09:35
      02cc91f1cd27bbe3dcd8d2da1bfd5c6f.lock
      -rw-r----- 1 apache apache 8192 Nov 18 09:35
      02cc91f1cd27bbe3dcd8d2da1bfd5c6f.pag

      /tmp/asp/03:
      total 0

      /tmp/asp/04:
      total 0

      /tmp/asp/05:
      total 0

      /tmp/asp/06:
      total 0

      /tmp/asp/09:
      total 0

      /tmp/asp/0c:
      total 16
      -rw-r----- 1 apache apache 0 Nov 18 09:01
      0cafb734b4fee71793b02eac8e49682c.dir
      -rw-rw-rw- 1 apache apache 0 Nov 18 09:34
      0cafb734b4fee71793b02eac8e49682c.lock
      -rw-r----- 1 apache apache 8192 Nov 18 09:34
      0cafb734b4fee71793b02eac8e49682c.pag
      -rw-r----- 1 apache apache 0 Nov 18 09:16
      0cdaf20928718a3c45b05a8a6fdf8d70.dir
      -rw-rw-rw- 1 apache apache 0 Nov 18 09:16
      0cdaf20928718a3c45b05a8a6fdf8d70.lock
      -rw-r----- 1 apache apache 8192 Nov 18 09:16
      0cdaf20928718a3c45b05a8a6fdf8d70.pag

      /tmp/asp/0e:
      total 0

      /tmp/asp/0f:
      total 0

      /tmp/asp/server:
      total 16
      -rw-r----- 1 apache apache 0 Nov 18 08:51 application.dir
      -rw-r--r-- 1 apache apache 0 Nov 18 09:35 application.lock
      -rw-r----- 1 apache apache 8192 Nov 18 09:35 application.pag
      -rw-r----- 1 apache apache 0 Nov 17 15:05 internal.dir
      -rw-r--r-- 1 apache apache 0 Nov 18 11:38 internal.lock
      -rw-r----- 1 apache apache 8192 Nov 18 09:35 internal.pag


      The only thing that looks out of place to me is that the *.pag files
      here are larger than some others I looked at. This site gets an average
      of a about 5500 hits per day.

      ---------------------------------------------------------------------
      To unsubscribe, e-mail: asp-unsubscribe@...
      For additional commands, e-mail: asp-help@...
    • Christopher Hicks
      ... You might be tripping over a bug in the db code. Can you try the Apache::ASP on a Fedora Core 2 or 3 machine? That s what I ve been developing and
      Message 2 of 10 , Nov 18, 2004
        On Thu, 18 Nov 2004, Warren Young wrote:

        > The only thing that looks out of place to me is that the *.pag files
        > here are larger than some others I looked at. This site gets an average
        > of a about 5500 hits per day.

        You might be tripping over a bug in the db code. Can you try the
        Apache::ASP on a Fedora Core 2 or 3 machine? That's what I've been
        developing and deploying on and its worked well. 7.3 is long long out of
        maintenance too so you really should upgrade it anyway. (Apache2 is nice,
        but it'll take a bit of tweaking to migrate your config so try this out on
        a nonproduction box.)

        --
        </chris>

        "Fans of Mozilla's free, open-source Firefox browser make the
        ardent Apple faithful look like a bunch of slackers."
        - Rebecca Lieb at clickz.com

        ---------------------------------------------------------------------
        To unsubscribe, e-mail: asp-unsubscribe@...
        For additional commands, e-mail: asp-help@...
      • Warren Young
        ... A bug in which db code? If it s part of the open source underpinnings of Apache::ASP, why is the bug still there? Why haven t you submitted a patch? ...
        Message 3 of 10 , Nov 19, 2004
          Christopher Hicks wrote:

          > You might be tripping over a bug in the db code.

          A bug in which db code? If it's part of the open source underpinnings
          of Apache::ASP, why is the bug still there? Why haven't you submitted a
          patch?

          > Can you try the Apache::ASP on a Fedora Core 2 or 3 machine?

          We have no evidence that correlates the problem with platform. Before
          we send a person out to a remote site to do an OS upgrade, we'd have to
          have a pretty good idea that it will fix the problem.

          > try this out on a nonproduction box.)

          Since we don't know how the box is being locked up, how could we test
          whether the OS upgrade helped? Again, we have many, many other
          sites...surely if it was as simple as an OS issue, some of our other
          RH7.3 systems would also be locking up.

          ---------------------------------------------------------------------
          To unsubscribe, e-mail: asp-unsubscribe@...
          For additional commands, e-mail: asp-help@...
        • Josh Chamas
          ... What is the state of the system when it locks up? Is the machine under load, is a process spinning out of control, is there any errors in the error_log
          Message 4 of 10 , Nov 19, 2004
            Warren Young wrote:
            > We have a site where Apache::ASP appears to be locking up every few
            > days. If we rm -r /tmp/asp, the system begins responding again.
            >
            > We've upgraded to the latest Apache::ASP, and it doesn't help. The
            > system is a Red Hat Linux 7.3 machine running Apache 1.3.27-4. None of
            > our many other sites are having this problem, to the best of our
            > knowledge. (They're not under our direct control.) On the other hand,
            > we have an incredible diversity in our installed systems. So, there's
            > no telling whether the problem is the machine's software configuration,
            > or it's part of the environment the machine is being used in, or it's
            > the users.
            >

            What is the state of the system when it locks up? Is the machine
            under load, is a process spinning out of control, is there any
            errors in the error_log that look interesting? Or does it just freeze?

            Does restarting the web server help things without removing the /tmp/asp
            entirely, or is removing that directory the only fix? If the details
            you provided about /tmp/asp are before removing but after a freeze,
            then I would say there is nothing that looks out of the ordinary there
            that would raise concern on my side.

            Regards,

            Josh

            ---------------------------------------------------------------------
            To unsubscribe, e-mail: asp-unsubscribe@...
            For additional commands, e-mail: asp-help@...
          • Warren Young
            ... These systems are installed in schools, so they re only being used from roughly 7am to 4pm. This problem is happening within that window, while it is
            Message 5 of 10 , Nov 19, 2004
              Josh Chamas wrote:

              > What is the state of the system when it locks up?

              These systems are installed in schools, so they're only being used from
              roughly 7am to 4pm. This problem is happening within that window, while
              it is being used during the day. We have never seen it lock up in this
              way over night.

              > Is the machine under load

              To the extent that 5500 hits per day is "load", yes. :)

              There seems to be a loose correlation between the problem and load: this
              site is one of the heaviest users. We have heavier users, though, that
              aren't seeing the problem.

              > is a process spinning out of control,

              We don't notice a busied out CPU, if that's what you mean. And, I
              wouldn't think anything is truly deadlocked, since all it takes is
              removing the ASP state directory to fix the problem.

              > is there any errors in the error_log that look interesting?

              No, but then we send a lot of stuff to error_log ourselves. And, we
              don't have Debug turned up. Should we?

              > Does restarting the web server help things without removing the /tmp/asp
              > entirely,

              No.

              > or is removing that directory the only fix?

              Yes.

              > If the details
              > you provided about /tmp/asp are before removing but after a freeze,
              > then I would say there is nothing that looks out of the ordinary there
              > that would raise concern on my side.

              Hmmm... Would having the files help?

              ---------------------------------------------------------------------
              To unsubscribe, e-mail: asp-unsubscribe@...
              For additional commands, e-mail: asp-help@...
            • Fagyal Csongor
              Warren, Are you sure you don t put something wrong into $Application / $Session? Might it be that it is one of your ASP applications that freezes Apache,
              Message 6 of 10 , Nov 19, 2004
                Warren,

                Are you sure you don't put "something wrong" into $Application /
                $Session? Might it be that it is one of your ASP applications that
                freezes Apache, because of something you store in the above scopes?

                You might also try to write a small script that dumps your *full* perl
                namespace in mod_perl, to see if something is "leaking"...

                Regards,
                - Csongor

                >> What is the state of the system when it locks up?
                >
                >
                > These systems are installed in schools, so they're only being used
                > from roughly 7am to 4pm. This problem is happening within that
                > window, while it is being used during the day. We have never seen it
                > lock up in this way over night.
                >
                >> Is the machine under load
                >
                >
                > To the extent that 5500 hits per day is "load", yes. :)
                >
                > There seems to be a loose correlation between the problem and load:
                > this site is one of the heaviest users. We have heavier users,
                > though, that aren't seeing the problem.
                >
                >> is a process spinning out of control,
                >
                >
                > We don't notice a busied out CPU, if that's what you mean. And, I
                > wouldn't think anything is truly deadlocked, since all it takes is
                > removing the ASP state directory to fix the problem.
                >
                >> is there any errors in the error_log that look interesting?
                >
                >
                > No, but then we send a lot of stuff to error_log ourselves. And, we
                > don't have Debug turned up. Should we?
                >
                >> Does restarting the web server help things without removing the /tmp/asp
                >> entirely,
                >
                >
                > No.
                >
                >> or is removing that directory the only fix?
                >
                >
                > Yes.
                >
                >> If the details
                >> you provided about /tmp/asp are before removing but after a freeze,
                >> then I would say there is nothing that looks out of the ordinary there
                >> that would raise concern on my side.
                >
                >
                > Hmmm... Would having the files help?
                >


                ---------------------------------------------------------------------
                To unsubscribe, e-mail: asp-unsubscribe@...
                For additional commands, e-mail: asp-help@...
              • Warren Young
                ... Could be, but if so, why is only this site locking up? ... Do you remember the incantation for that? I m sure I can look it up if you don t have it off
                Message 7 of 10 , Nov 19, 2004
                  Fagyal Csongor wrote:
                  >
                  > Are you sure you don't put "something wrong" into $Application /
                  > $Session?

                  Could be, but if so, why is only this site locking up?

                  > You might also try to write a small script that dumps your *full* perl
                  > namespace in mod_perl, to see if something is "leaking"...

                  Do you remember the incantation for that? I'm sure I can look it up if
                  you don't have it off the top of your head.

                  ---------------------------------------------------------------------
                  To unsubscribe, e-mail: asp-unsubscribe@...
                  For additional commands, e-mail: asp-help@...
                • Christopher Hicks
                  ... The db that s used for storing the sessions which seems to be the files you re deleting. There are a variety of dbm implementations and I m not sure which
                  Message 8 of 10 , Nov 20, 2004
                    On Fri, 19 Nov 2004, Warren Young wrote:
                    > Christopher Hicks wrote:
                    >> You might be tripping over a bug in the db code.
                    >
                    > A bug in which db code?

                    The db that's used for storing the sessions which seems to be the files
                    you're deleting. There are a variety of dbm implementations and I'm not
                    sure which Apache::ASP ends up using on RH7.3.

                    > If it's part of the open source underpinnings of Apache::ASP, why is the
                    > bug still there? Why haven't you submitted a patch?

                    The reason I suggested it might be a problem in the db code is that you
                    could try recompiling that without much trouble. (Taking a more current
                    rpm and rebuilding it for 7.3 would be the ideal solution, but if you're
                    not familiar with tweaking the rpm build .spec files going from source
                    would be easier.) I haven't run into any db problems with Apache::ASP,
                    but we don't have any Red Hat 7, 8, or 9 boxes left in production at this
                    point.

                    >> Can you try the Apache::ASP on a Fedora Core 2 or 3 machine?
                    >
                    > We have no evidence that correlates the problem with platform. Before we
                    > send a person out to a remote site to do an OS upgrade, we'd have to have a
                    > pretty good idea that it will fix the problem.

                    Considering that Red Hat 7.3 is long out of having patch support for it
                    upgrading would seem to be a very good idea for reasons that have nothing
                    to do with fixing this problem.

                    >> try this out on a nonproduction box.)
                    >
                    > Since we don't know how the box is being locked up, how could we test
                    > whether the OS upgrade helped?

                    Try replicating the problem in a test environment?

                    > Again, we have many, many other sites...surely if it was as simple as an
                    > OS issue, some of our other RH7.3 systems would also be locking up.

                    Not necessarily. If this site is your heaviest site then they may be
                    tripping over a database locking issue that just doesn't have as good an
                    odds of happening at he other sites.

                    --
                    </chris>

                    "Fans of Mozilla's free, open-source Firefox browser make the
                    ardent Apple faithful look like a bunch of slackers."
                    - Rebecca Lieb at clickz.com

                    ---------------------------------------------------------------------
                    To unsubscribe, e-mail: asp-unsubscribe@...
                    For additional commands, e-mail: asp-help@...
                  • Josh Chamas
                    Hi Warren, At this point, I would recommend setting Debug -1 and then stop/start apache, and send me error_log from start until freeze. Then stop apache, and
                    Message 9 of 10 , Nov 21, 2004
                      Hi Warren,

                      At this point, I would recommend setting Debug -1 and then
                      stop/start apache, and send me error_log from start until freeze.
                      Then stop apache, and get me a tar/gzip copy of the directory /tmp/asp
                      *unless* there is sensitive data in the data structures in
                      which case you should not be sending me that.

                      It may be something that is being stored in $Session or $Application
                      that is killing things, and is only specific to that DBM format.

                      Also, if all this debugging is just taking too long, then you
                      might do something like this just for this one server:

                      # httpd.conf
                      <Perl>
                      use Apache::ASP::StateManager;
                      $Apache::ASP::State::DefaultStateDB = 'DB_File';
                      </Perl>
                      PerlSetVar StateDB DB_File
                      # end httpd.conf

                      By default Apache::ASP uses SDBM_File for the internal database
                      storage. You can also try the new PerlSetVar without the <Perl>
                      block. You can also try GDBM_File without using DB_File, either
                      way as you may only have one or the other installed on your system.

                      Note these things in <Perl> blocks in not officially supported,
                      but this is a worst case scenario where SDBM_File just isn't working
                      for you for the internal database, and I think this should work to
                      override it. Note also, if you use the <Perl> method, you need
                      to delete the /tmp/asp directory between changing the DefaultStateDB
                      setting, as Apache::ASP is not used to this being changed at runtime,
                      its a hard coded global in the code.

                      Regards,

                      Josh


                      Warren Young wrote:
                      > Josh Chamas wrote:
                      >
                      >> What is the state of the system when it locks up?
                      >
                      >
                      > These systems are installed in schools, so they're only being used from
                      > roughly 7am to 4pm. This problem is happening within that window, while
                      > it is being used during the day. We have never seen it lock up in this
                      > way over night.
                      >
                      >> Is the machine under load
                      >
                      >
                      > To the extent that 5500 hits per day is "load", yes. :)
                      >
                      > There seems to be a loose correlation between the problem and load: this
                      > site is one of the heaviest users. We have heavier users, though, that
                      > aren't seeing the problem.
                      >
                      >> is a process spinning out of control,
                      >
                      >
                      > We don't notice a busied out CPU, if that's what you mean. And, I
                      > wouldn't think anything is truly deadlocked, since all it takes is
                      > removing the ASP state directory to fix the problem.
                      >
                      >> is there any errors in the error_log that look interesting?
                      >
                      >
                      > No, but then we send a lot of stuff to error_log ourselves. And, we
                      > don't have Debug turned up. Should we?
                      >
                      >> Does restarting the web server help things without removing the /tmp/asp
                      >> entirely,
                      >
                      >
                      > No.
                      >
                      >> or is removing that directory the only fix?
                      >
                      >
                      > Yes.
                      >
                      >> If the details
                      >> you provided about /tmp/asp are before removing but after a freeze,
                      >> then I would say there is nothing that looks out of the ordinary there
                      >> that would raise concern on my side.
                      >
                      >
                      > Hmmm... Would having the files help?
                      >
                      > ---------------------------------------------------------------------
                      > To unsubscribe, e-mail: asp-unsubscribe@...
                      > For additional commands, e-mail: asp-help@...
                      >
                      >

                      ---------------------------------------------------------------------
                      To unsubscribe, e-mail: asp-unsubscribe@...
                      For additional commands, e-mail: asp-help@...
                    • Warren Young
                      ... Okay, I ll do that. School is out this week at the problem site, so I don t expect it to die this week. ... It does slow things down, but we can tolerate
                      Message 10 of 10 , Nov 22, 2004
                        Josh Chamas wrote:
                        >
                        > At this point, I would recommend setting Debug -1 and then

                        Okay, I'll do that. School is out this week at the problem site, so I
                        don't expect it to die this week.

                        > Also, if all this debugging is just taking too long, then you
                        > might do something like this just for this one server:

                        It does slow things down, but we can tolerate it.

                        Thanks!

                        ---------------------------------------------------------------------
                        To unsubscribe, e-mail: asp-unsubscribe@...
                        For additional commands, e-mail: asp-help@...
                      Your message has been successfully submitted and would be delivered to recipients shortly.