Loading ...
Sorry, an error occurred while loading the content.

Re: how to trap connection reset/abort in Apache::Filter?

Expand Messages
  • Stas Bekman
    ... I think the first step is be able to reliably reproduce that. Let me see if I can figure out how to do that. --
    Message 1 of 13 , May 2, 2005
    • 0 Attachment
      Jeff Ambrosino wrote:
      > Ok, I tried this approach, and it only partially worked; unfortunately
      > my httpd procs are still dying, and APR::Error doesn't seem to pass
      > anything. Here's the error I got _before_ wrapping $f->print in the
      > eval:
      >
      > [Sun May 01 05:12:21 2005] [error] [client X.X.X.X]
      > Apache::Filter::print: (104)Connection reset by peer at
      > /var/httpd/lib/perl/MyFilter.pm line
      > 453n\tMyFilter::handler('Apache::Filter=SCALAR0x87106a8)',
      > 'APR::Brigade=SCALAR(0x86eac6c)') called at -e line 0\n\teval {...}
      > called at -e line 0\n
      > Apache::Filter: (103) Software caused connection abort at -e line 0
      >
      > And here's how it looks in the error+log _after_ putting in the eval wrapper:
      >
      > Apache::Filter: (103) Software caused connection abort at -e line 0
      >
      > Notice there's no timestamp next to the Apache::Filter error, so it
      > appears that something is writing that message directly to STDERR.
      >
      > The main problem is that with or without the eval, the httpd process
      > is getting killed, and it seems to be taking a while for the httpd
      > parent to realize this and start new procs (I'm using MPM prefork.)
      >
      > Here's the code I'm using to trap the $f->print failure:
      >
      > if ( defined($f->ctx) ) {
      > eval { $f->print ($f->ctx) };
      >
      > if ($@ && ref $@) {
      > $r->log_error("WARNING " . "\$f->print
      > exception: $@");
      > }
      > }
      >
      > Any suggestions?
      >
      > And FWIW, I haven't been able to reproduce these conditions
      > artificially, I just have to wait until it occurs in the wild. A
      > "wild" event that occurs regularly, however, is a daily vulnerability
      > scan by ScanAlert (a service that we pay for.) They throw all sorts
      > of potential http exploits at our server and notify us of any problems
      > - I'm guessing some of their tests do some funky socket operations
      > that cause Apache::FIlter to choke.

      I think the first step is be able to reliably reproduce that. Let me see
      if I can figure out how to do that.

      --
      __________________________________________________________________
      Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
      http://stason.org/ mod_perl Guide ---> http://perl.apache.org
      mailto:stas@... http://use.perl.org http://apacheweek.com
      http://modperlbook.org http://apache.org http://ticketmaster.com
    • Jeff Ambrosino
      Hi Stas & MP list, I tried causing this error by opening sockets to the web server, then killing (-9) the client, but that didn t work. However, every morning
      Message 2 of 13 , May 3, 2005
      • 0 Attachment
        Hi Stas & MP list,

        I tried causing this error by opening sockets to the web server, then
        killing (-9) the client, but that didn't work. However, every morning
        like clockwork the scanning service (ScanAlert) hits our site and
        causes this problem... so, while I cannot intentionally cause the
        error by myself, I know when our scanning service will do so, and I
        could put some additional code in place to watch for it. If there are
        additional debug/traps that you could suggest, I'd like to try. Any
        ideas?

        thanks
        Jeff


        On 5/2/05, Stas Bekman <stas@...> wrote:
        > Jeff Ambrosino wrote:
        > > Ok, I tried this approach, and it only partially worked; unfortunately
        > > my httpd procs are still dying, and APR::Error doesn't seem to pass
        > > anything. Here's the error I got _before_ wrapping $f->print in the
        > > eval:
        > >
        > > [Sun May 01 05:12:21 2005] [error] [client X.X.X.X]
        > > Apache::Filter::print: (104)Connection reset by peer at
        > > /var/httpd/lib/perl/MyFilter.pm line
        > > 453n\tMyFilter::handler('Apache::Filter=SCALAR0x87106a8)',
        > > 'APR::Brigade=SCALAR(0x86eac6c)') called at -e line 0\n\teval {...}
        > > called at -e line 0\n
        > > Apache::Filter: (103) Software caused connection abort at -e line 0
        > >
        > > And here's how it looks in the error+log _after_ putting in the eval wrapper:
        > >
        > > Apache::Filter: (103) Software caused connection abort at -e line 0
        > >
        > > Notice there's no timestamp next to the Apache::Filter error, so it
        > > appears that something is writing that message directly to STDERR.
        > >
        > > The main problem is that with or without the eval, the httpd process
        > > is getting killed, and it seems to be taking a while for the httpd
        > > parent to realize this and start new procs (I'm using MPM prefork.)
        > >
        > > Here's the code I'm using to trap the $f->print failure:
        > >
        > > if ( defined($f->ctx) ) {
        > > eval { $f->print ($f->ctx) };
        > >
        > > if ($@ && ref $@) {
        > > $r->log_error("WARNING " . "\$f->print
        > > exception: $@");
        > > }
        > > }
        > >
        > > Any suggestions?
        > >
        > > And FWIW, I haven't been able to reproduce these conditions
        > > artificially, I just have to wait until it occurs in the wild. A
        > > "wild" event that occurs regularly, however, is a daily vulnerability
        > > scan by ScanAlert (a service that we pay for.) They throw all sorts
        > > of potential http exploits at our server and notify us of any problems
        > > - I'm guessing some of their tests do some funky socket operations
        > > that cause Apache::FIlter to choke.
        >
        > I think the first step is be able to reliably reproduce that. Let me see
        > if I can figure out how to do that.
        >
        > --
        > __________________________________________________________________
        > Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
        > http://stason.org/ mod_perl Guide ---> http://perl.apache.org
        > mailto:stas@... http://use.perl.org http://apacheweek.com
        > http://modperlbook.org http://apache.org http://ticketmaster.com
        >
      • Stas Bekman
        ... I m playing with this issue. I ve notice that we already have a reliably reproducable issue with t/protocol/echo_nonblock.t which calls
        Message 3 of 13 , May 11, 2005
        • 0 Attachment
          Jeff Ambrosino wrote:
          > Hi Stas & MP list,
          >
          > I tried causing this error by opening sockets to the web server, then
          > killing (-9) the client, but that didn't work. However, every morning
          > like clockwork the scanning service (ScanAlert) hits our site and
          > causes this problem... so, while I cannot intentionally cause the
          > error by myself, I know when our scanning service will do so, and I
          > could put some additional code in place to watch for it. If there are
          > additional debug/traps that you could suggest, I'd like to try. Any
          > ideas?

          I'm playing with this issue. I've notice that we already have a reliably
          reproducable issue with t/protocol/echo_nonblock.t which calls
          t/protocol/TestProtocol/echo_nonblock.pm. Could you please try the current
          svn (if you go for the snapshot please wait for 6 hours, since I've just
          committed things) [1]

          There I now use the code:

          die $@ unless ref $@ eq 'APR::Error'
          && (APR::Status::is_ECONNABORTED($@) ||
          APR::Status::is_ECONNRESET($@));

          can you also see that die doesn't appear in the error_log of the test
          suite? and if you just do

          die $@;

          it logs the error?

          t/TEST -v t/protocol/echo_nonblock.t

          and check t/logs/error_log

          [1]
          http://perl.apache.org/download/source.html#Development_mod_perl_2_0_Source_Distribution

          --
          __________________________________________________________________
          Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
          http://stason.org/ mod_perl Guide ---> http://perl.apache.org
          mailto:stas@... http://use.perl.org http://apacheweek.com
          http://modperlbook.org http://apache.org http://ticketmaster.com
        • Jeff Ambrosino
          Ok, this is going to be tough for me to test because I m still on RC4, so it s not yet converted to the new Apache2 naming. I ll still try to make/test the
          Message 4 of 13 , May 11, 2005
          • 0 Attachment
            Ok, this is going to be tough for me to test because I'm still on RC4,
            so it's not yet converted to the new Apache2 naming. I'll still try
            to make/test the svn snapshot, though (which also gives me a good
            reason to get more familiar with subversion.) I'll email what I find
            out.

            thanks Stas!

            Jeff




            On 5/11/05, Stas Bekman <stas@...> wrote:
            > I'm playing with this issue. I've notice that we already have a reliably
            > reproducable issue with t/protocol/echo_nonblock.t which calls
            > t/protocol/TestProtocol/echo_nonblock.pm. Could you please try the current
            > svn (if you go for the snapshot please wait for 6 hours, since I've just
            > committed things) [1]
            >
            > There I now use the code:
            [.......]
          • Stas Bekman
            ... OK, may be just try comparing to: $@ == APR::Const::ECONNABORTED or $@ == APR::Const::ECONNRESET The new subs just supposed to work on more platforms,
            Message 5 of 13 , May 11, 2005
            • 0 Attachment
              Jeff Ambrosino wrote:
              > Ok, this is going to be tough for me to test because I'm still on RC4,
              > so it's not yet converted to the new Apache2 naming. I'll still try
              > to make/test the svn snapshot, though (which also gives me a good
              > reason to get more familiar with subversion.) I'll email what I find
              > out.
              >
              > thanks Stas!

              OK, may be just try comparing to:

              $@ == APR::Const::ECONNABORTED or $@ == APR::Const::ECONNRESET

              The new subs just supposed to work on more platforms, where other similar
              codes may be sent by the system.

              You can still play with the test t/protocol/TestProtocol/echo_nonblock.pm
              from RC4.

              but looking again at your report, I can see that the issue is elsewhere.

              I believe it's coming from src/modules/perl/modperl_filter.c:

              if (filter->mode == MP_INPUT_FILTER_MODE) {
              ...
              }
              else {
              MP_RUN_CROAK(modperl_output_filter_flush(filter),
              "Apache2::Filter");
              }

              is that it? if you change the "Apache2::Filter" string (drop the 2 in your
              case) to something else?

              --
              __________________________________________________________________
              Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
              http://stason.org/ mod_perl Guide ---> http://perl.apache.org
              mailto:stas@... http://use.perl.org http://apacheweek.com
              http://modperlbook.org http://apache.org http://ticketmaster.com
            • Stas Bekman
              ... If this is the case, please try this patch, I hope it applies to RC4 (I ve dropped 2 for you) Index: src/modules/perl/modperl_filter.c ... +++
              Message 6 of 13 , May 11, 2005
              • 0 Attachment
                Stas Bekman wrote:
                > Jeff Ambrosino wrote:
                >
                >> Ok, this is going to be tough for me to test because I'm still on RC4,
                >> so it's not yet converted to the new Apache2 naming. I'll still try
                >> to make/test the svn snapshot, though (which also gives me a good
                >> reason to get more familiar with subversion.) I'll email what I find
                >> out.
                >>
                >> thanks Stas!
                >
                >
                > OK, may be just try comparing to:
                >
                > $@ == APR::Const::ECONNABORTED or $@ == APR::Const::ECONNRESET
                >
                > The new subs just supposed to work on more platforms, where other
                > similar codes may be sent by the system.
                >
                > You can still play with the test
                > t/protocol/TestProtocol/echo_nonblock.pm from RC4.
                >
                > but looking again at your report, I can see that the issue is elsewhere.
                >
                > I believe it's coming from src/modules/perl/modperl_filter.c:
                >
                > if (filter->mode == MP_INPUT_FILTER_MODE) {
                > ...
                > }
                > else {
                > MP_RUN_CROAK(modperl_output_filter_flush(filter),
                > "Apache2::Filter");
                > }
                >
                > is that it? if you change the "Apache2::Filter" string (drop the 2 in
                > your case) to something else?

                If this is the case, please try this patch, I hope it applies to RC4 (I've
                dropped '2' for you)

                Index: src/modules/perl/modperl_filter.c
                ===================================================================
                --- src/modules/perl/modperl_filter.c (revision 169708)
                +++ src/modules/perl/modperl_filter.c (working copy)
                @@ -535,11 +535,15 @@
                filter->bb_in = NULL;
                }
                MP_RUN_CROAK(modperl_input_filter_flush(filter),
                - "Apache::Filter");
                + "Apache::Filter internal flush");
                }
                else {
                - MP_RUN_CROAK(modperl_output_filter_flush(filter),
                - "Apache::Filter");
                + apr_status_t rc = modperl_output_filter_flush(filter);
                + if (!(rc == APR_SUCCESS ||
                + APR_STATUS_IS_ECONNRESET(rc) ||
                + APR_STATUS_IS_ECONNABORTED(rc))) {
                + modperl_croak(aTHX_ rc, "Apache::Filter internal flush");
                + }
                }

                MP_FILTER_RESTORE_ERRSV(errsv);

                I need to figure out how to write a test for this, so I can decide what's
                the right way to handle this.

                --
                __________________________________________________________________
                Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
                http://stason.org/ mod_perl Guide ---> http://perl.apache.org
                mailto:stas@... http://use.perl.org http://apacheweek.com
                http://modperlbook.org http://apache.org http://ticketmaster.com
              • Stas Bekman
                Jeff? Could you please try that patch that I ve sent earlier? I think it s right, but I was still not very successful at reproducing your problem on the HTTP
                Message 7 of 13 , May 23, 2005
                • 0 Attachment
                  Jeff? Could you please try that patch that I've sent earlier? I think it's
                  right, but I was still not very successful at reproducing your problem on
                  the HTTP level. Thanks.

                  Stas Bekman wrote:
                  > Stas Bekman wrote:
                  >
                  >> Jeff Ambrosino wrote:
                  >>
                  >>> Ok, this is going to be tough for me to test because I'm still on RC4,
                  >>> so it's not yet converted to the new Apache2 naming. I'll still try
                  >>> to make/test the svn snapshot, though (which also gives me a good
                  >>> reason to get more familiar with subversion.) I'll email what I find
                  >>> out.
                  >>>
                  >>> thanks Stas!
                  >>
                  >>
                  >>
                  >> OK, may be just try comparing to:
                  >>
                  >> $@ == APR::Const::ECONNABORTED or $@ == APR::Const::ECONNRESET
                  >>
                  >> The new subs just supposed to work on more platforms, where other
                  >> similar codes may be sent by the system.
                  >>
                  >> You can still play with the test
                  >> t/protocol/TestProtocol/echo_nonblock.pm from RC4.
                  >>
                  >> but looking again at your report, I can see that the issue is elsewhere.
                  >>
                  >> I believe it's coming from src/modules/perl/modperl_filter.c:
                  >>
                  >> if (filter->mode == MP_INPUT_FILTER_MODE) {
                  >> ...
                  >> }
                  >> else {
                  >> MP_RUN_CROAK(modperl_output_filter_flush(filter),
                  >> "Apache2::Filter");
                  >> }
                  >>
                  >> is that it? if you change the "Apache2::Filter" string (drop the 2 in
                  >> your case) to something else?
                  >
                  >
                  > If this is the case, please try this patch, I hope it applies to RC4
                  > (I've dropped '2' for you)
                  >
                  > Index: src/modules/perl/modperl_filter.c
                  > ===================================================================
                  > --- src/modules/perl/modperl_filter.c (revision 169708)
                  > +++ src/modules/perl/modperl_filter.c (working copy)
                  > @@ -535,11 +535,15 @@
                  > filter->bb_in = NULL;
                  > }
                  > MP_RUN_CROAK(modperl_input_filter_flush(filter),
                  > - "Apache::Filter");
                  > + "Apache::Filter internal flush");
                  > }
                  > else {
                  > - MP_RUN_CROAK(modperl_output_filter_flush(filter),
                  > - "Apache::Filter");
                  > + apr_status_t rc = modperl_output_filter_flush(filter);
                  > + if (!(rc == APR_SUCCESS ||
                  > + APR_STATUS_IS_ECONNRESET(rc) ||
                  > + APR_STATUS_IS_ECONNABORTED(rc))) {
                  > + modperl_croak(aTHX_ rc, "Apache::Filter internal flush");
                  > + }
                  > }
                  >
                  > MP_FILTER_RESTORE_ERRSV(errsv);
                  >
                  > I need to figure out how to write a test for this, so I can decide
                  > what's the right way to handle this.
                  >


                  --
                  __________________________________________________________________
                  Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
                  http://stason.org/ mod_perl Guide ---> http://perl.apache.org
                  mailto:stas@... http://use.perl.org http://apacheweek.com
                  http://modperlbook.org http://apache.org http://ticketmaster.com
                • Stas Bekman
                  ... Excellent. Now committed. Thanks Jeff. -- __________________________________________________________________ Stas Bekman JAm_pH ------ Just
                  Message 8 of 13 , Jun 28, 2005
                  • 0 Attachment
                    Jeff Ambrosino wrote:
                    > Stas, just a quick "closure"... the patch has survived several ScanAlert
                    > scans without any aborted httpd children. This was the original way I
                    > discovered the problem (www.scanalert.com - they periodically throw
                    > all sorts of weird hack/exploit traffic at our server as an audit.)
                    > So from my end, the problem is officially fixed.

                    Excellent. Now committed.

                    Thanks Jeff.


                    --
                    __________________________________________________________________
                    Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
                    http://stason.org/ mod_perl Guide ---> http://perl.apache.org
                    mailto:stas@... http://use.perl.org http://apacheweek.com
                    http://modperlbook.org http://apache.org http://ticketmaster.com
                  • Stas Bekman
                    ... Excellent. Now committed. Thanks Jeff. -- __________________________________________________________________ Stas Bekman JAm_pH ------ Just
                    Message 9 of 13 , Jun 28, 2005
                    • 0 Attachment
                      Jeff Ambrosino wrote:
                      > Stas, just a quick "closure"... the patch has survived several ScanAlert
                      > scans without any aborted httpd children. This was the original way I
                      > discovered the problem (www.scanalert.com - they periodically throw
                      > all sorts of weird hack/exploit traffic at our server as an audit.)
                      > So from my end, the problem is officially fixed.

                      Excellent. Now committed.

                      Thanks Jeff.


                      --
                      __________________________________________________________________
                      Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
                      http://stason.org/ mod_perl Guide ---> http://perl.apache.org
                      mailto:stas@... http://use.perl.org http://apacheweek.com
                      http://modperlbook.org http://apache.org http://ticketmaster.com
                    Your message has been successfully submitted and would be delivered to recipients shortly.