Loading ...
Sorry, an error occurred while loading the content.

Re: how to trap connection reset/abort in Apache::Filter?

Expand Messages
  • Jeff Ambrosino
    Ok, I tried this approach, and it only partially worked; unfortunately my httpd procs are still dying, and APR::Error doesn t seem to pass anything. Here s
    Message 1 of 13 , May 2, 2005
    • 0 Attachment
      Ok, I tried this approach, and it only partially worked; unfortunately
      my httpd procs are still dying, and APR::Error doesn't seem to pass
      anything. Here's the error I got _before_ wrapping $f->print in the
      eval:

      [Sun May 01 05:12:21 2005] [error] [client X.X.X.X]
      Apache::Filter::print: (104)Connection reset by peer at
      /var/httpd/lib/perl/MyFilter.pm line
      453n\tMyFilter::handler('Apache::Filter=SCALAR0x87106a8)',
      'APR::Brigade=SCALAR(0x86eac6c)') called at -e line 0\n\teval {...}
      called at -e line 0\n
      Apache::Filter: (103) Software caused connection abort at -e line 0

      And here's how it looks in the error+log _after_ putting in the eval wrapper:

      Apache::Filter: (103) Software caused connection abort at -e line 0

      Notice there's no timestamp next to the Apache::Filter error, so it
      appears that something is writing that message directly to STDERR.

      The main problem is that with or without the eval, the httpd process
      is getting killed, and it seems to be taking a while for the httpd
      parent to realize this and start new procs (I'm using MPM prefork.)

      Here's the code I'm using to trap the $f->print failure:

      if ( defined($f->ctx) ) {
      eval { $f->print ($f->ctx) };

      if ($@ && ref $@) {
      $r->log_error("WARNING " . "\$f->print
      exception: $@");
      }
      }

      Any suggestions?

      And FWIW, I haven't been able to reproduce these conditions
      artificially, I just have to wait until it occurs in the wild. A
      "wild" event that occurs regularly, however, is a daily vulnerability
      scan by ScanAlert (a service that we pay for.) They throw all sorts
      of potential http exploits at our server and notify us of any problems
      - I'm guessing some of their tests do some funky socket operations
      that cause Apache::FIlter to choke.

      thanks
      Jeff


      On 5/2/05, Stas Bekman <stas@...> wrote:
      > > Actually, it looks like using APR::Error [1] and wrapping $f->print in
      > > an eval would do the trick...?
      >
      > That's correct, Jeff. eval and check for APR::Const::ECONNRESET as
      > explained in [1]
      >
      > > [1] http://perl.apache.org/docs/2.0/api/APR/Error.html
      >
      > Please report whether this did the trick. We ought to document this one,
      > since many will encounter this issue.
    • Stas Bekman
      ... I think the first step is be able to reliably reproduce that. Let me see if I can figure out how to do that. --
      Message 2 of 13 , May 2, 2005
      • 0 Attachment
        Jeff Ambrosino wrote:
        > Ok, I tried this approach, and it only partially worked; unfortunately
        > my httpd procs are still dying, and APR::Error doesn't seem to pass
        > anything. Here's the error I got _before_ wrapping $f->print in the
        > eval:
        >
        > [Sun May 01 05:12:21 2005] [error] [client X.X.X.X]
        > Apache::Filter::print: (104)Connection reset by peer at
        > /var/httpd/lib/perl/MyFilter.pm line
        > 453n\tMyFilter::handler('Apache::Filter=SCALAR0x87106a8)',
        > 'APR::Brigade=SCALAR(0x86eac6c)') called at -e line 0\n\teval {...}
        > called at -e line 0\n
        > Apache::Filter: (103) Software caused connection abort at -e line 0
        >
        > And here's how it looks in the error+log _after_ putting in the eval wrapper:
        >
        > Apache::Filter: (103) Software caused connection abort at -e line 0
        >
        > Notice there's no timestamp next to the Apache::Filter error, so it
        > appears that something is writing that message directly to STDERR.
        >
        > The main problem is that with or without the eval, the httpd process
        > is getting killed, and it seems to be taking a while for the httpd
        > parent to realize this and start new procs (I'm using MPM prefork.)
        >
        > Here's the code I'm using to trap the $f->print failure:
        >
        > if ( defined($f->ctx) ) {
        > eval { $f->print ($f->ctx) };
        >
        > if ($@ && ref $@) {
        > $r->log_error("WARNING " . "\$f->print
        > exception: $@");
        > }
        > }
        >
        > Any suggestions?
        >
        > And FWIW, I haven't been able to reproduce these conditions
        > artificially, I just have to wait until it occurs in the wild. A
        > "wild" event that occurs regularly, however, is a daily vulnerability
        > scan by ScanAlert (a service that we pay for.) They throw all sorts
        > of potential http exploits at our server and notify us of any problems
        > - I'm guessing some of their tests do some funky socket operations
        > that cause Apache::FIlter to choke.

        I think the first step is be able to reliably reproduce that. Let me see
        if I can figure out how to do that.

        --
        __________________________________________________________________
        Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
        http://stason.org/ mod_perl Guide ---> http://perl.apache.org
        mailto:stas@... http://use.perl.org http://apacheweek.com
        http://modperlbook.org http://apache.org http://ticketmaster.com
      • Jeff Ambrosino
        Hi Stas & MP list, I tried causing this error by opening sockets to the web server, then killing (-9) the client, but that didn t work. However, every morning
        Message 3 of 13 , May 3, 2005
        • 0 Attachment
          Hi Stas & MP list,

          I tried causing this error by opening sockets to the web server, then
          killing (-9) the client, but that didn't work. However, every morning
          like clockwork the scanning service (ScanAlert) hits our site and
          causes this problem... so, while I cannot intentionally cause the
          error by myself, I know when our scanning service will do so, and I
          could put some additional code in place to watch for it. If there are
          additional debug/traps that you could suggest, I'd like to try. Any
          ideas?

          thanks
          Jeff


          On 5/2/05, Stas Bekman <stas@...> wrote:
          > Jeff Ambrosino wrote:
          > > Ok, I tried this approach, and it only partially worked; unfortunately
          > > my httpd procs are still dying, and APR::Error doesn't seem to pass
          > > anything. Here's the error I got _before_ wrapping $f->print in the
          > > eval:
          > >
          > > [Sun May 01 05:12:21 2005] [error] [client X.X.X.X]
          > > Apache::Filter::print: (104)Connection reset by peer at
          > > /var/httpd/lib/perl/MyFilter.pm line
          > > 453n\tMyFilter::handler('Apache::Filter=SCALAR0x87106a8)',
          > > 'APR::Brigade=SCALAR(0x86eac6c)') called at -e line 0\n\teval {...}
          > > called at -e line 0\n
          > > Apache::Filter: (103) Software caused connection abort at -e line 0
          > >
          > > And here's how it looks in the error+log _after_ putting in the eval wrapper:
          > >
          > > Apache::Filter: (103) Software caused connection abort at -e line 0
          > >
          > > Notice there's no timestamp next to the Apache::Filter error, so it
          > > appears that something is writing that message directly to STDERR.
          > >
          > > The main problem is that with or without the eval, the httpd process
          > > is getting killed, and it seems to be taking a while for the httpd
          > > parent to realize this and start new procs (I'm using MPM prefork.)
          > >
          > > Here's the code I'm using to trap the $f->print failure:
          > >
          > > if ( defined($f->ctx) ) {
          > > eval { $f->print ($f->ctx) };
          > >
          > > if ($@ && ref $@) {
          > > $r->log_error("WARNING " . "\$f->print
          > > exception: $@");
          > > }
          > > }
          > >
          > > Any suggestions?
          > >
          > > And FWIW, I haven't been able to reproduce these conditions
          > > artificially, I just have to wait until it occurs in the wild. A
          > > "wild" event that occurs regularly, however, is a daily vulnerability
          > > scan by ScanAlert (a service that we pay for.) They throw all sorts
          > > of potential http exploits at our server and notify us of any problems
          > > - I'm guessing some of their tests do some funky socket operations
          > > that cause Apache::FIlter to choke.
          >
          > I think the first step is be able to reliably reproduce that. Let me see
          > if I can figure out how to do that.
          >
          > --
          > __________________________________________________________________
          > Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
          > http://stason.org/ mod_perl Guide ---> http://perl.apache.org
          > mailto:stas@... http://use.perl.org http://apacheweek.com
          > http://modperlbook.org http://apache.org http://ticketmaster.com
          >
        • Stas Bekman
          ... I m playing with this issue. I ve notice that we already have a reliably reproducable issue with t/protocol/echo_nonblock.t which calls
          Message 4 of 13 , May 11, 2005
          • 0 Attachment
            Jeff Ambrosino wrote:
            > Hi Stas & MP list,
            >
            > I tried causing this error by opening sockets to the web server, then
            > killing (-9) the client, but that didn't work. However, every morning
            > like clockwork the scanning service (ScanAlert) hits our site and
            > causes this problem... so, while I cannot intentionally cause the
            > error by myself, I know when our scanning service will do so, and I
            > could put some additional code in place to watch for it. If there are
            > additional debug/traps that you could suggest, I'd like to try. Any
            > ideas?

            I'm playing with this issue. I've notice that we already have a reliably
            reproducable issue with t/protocol/echo_nonblock.t which calls
            t/protocol/TestProtocol/echo_nonblock.pm. Could you please try the current
            svn (if you go for the snapshot please wait for 6 hours, since I've just
            committed things) [1]

            There I now use the code:

            die $@ unless ref $@ eq 'APR::Error'
            && (APR::Status::is_ECONNABORTED($@) ||
            APR::Status::is_ECONNRESET($@));

            can you also see that die doesn't appear in the error_log of the test
            suite? and if you just do

            die $@;

            it logs the error?

            t/TEST -v t/protocol/echo_nonblock.t

            and check t/logs/error_log

            [1]
            http://perl.apache.org/download/source.html#Development_mod_perl_2_0_Source_Distribution

            --
            __________________________________________________________________
            Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
            http://stason.org/ mod_perl Guide ---> http://perl.apache.org
            mailto:stas@... http://use.perl.org http://apacheweek.com
            http://modperlbook.org http://apache.org http://ticketmaster.com
          • Jeff Ambrosino
            Ok, this is going to be tough for me to test because I m still on RC4, so it s not yet converted to the new Apache2 naming. I ll still try to make/test the
            Message 5 of 13 , May 11, 2005
            • 0 Attachment
              Ok, this is going to be tough for me to test because I'm still on RC4,
              so it's not yet converted to the new Apache2 naming. I'll still try
              to make/test the svn snapshot, though (which also gives me a good
              reason to get more familiar with subversion.) I'll email what I find
              out.

              thanks Stas!

              Jeff




              On 5/11/05, Stas Bekman <stas@...> wrote:
              > I'm playing with this issue. I've notice that we already have a reliably
              > reproducable issue with t/protocol/echo_nonblock.t which calls
              > t/protocol/TestProtocol/echo_nonblock.pm. Could you please try the current
              > svn (if you go for the snapshot please wait for 6 hours, since I've just
              > committed things) [1]
              >
              > There I now use the code:
              [.......]
            • Stas Bekman
              ... OK, may be just try comparing to: $@ == APR::Const::ECONNABORTED or $@ == APR::Const::ECONNRESET The new subs just supposed to work on more platforms,
              Message 6 of 13 , May 11, 2005
              • 0 Attachment
                Jeff Ambrosino wrote:
                > Ok, this is going to be tough for me to test because I'm still on RC4,
                > so it's not yet converted to the new Apache2 naming. I'll still try
                > to make/test the svn snapshot, though (which also gives me a good
                > reason to get more familiar with subversion.) I'll email what I find
                > out.
                >
                > thanks Stas!

                OK, may be just try comparing to:

                $@ == APR::Const::ECONNABORTED or $@ == APR::Const::ECONNRESET

                The new subs just supposed to work on more platforms, where other similar
                codes may be sent by the system.

                You can still play with the test t/protocol/TestProtocol/echo_nonblock.pm
                from RC4.

                but looking again at your report, I can see that the issue is elsewhere.

                I believe it's coming from src/modules/perl/modperl_filter.c:

                if (filter->mode == MP_INPUT_FILTER_MODE) {
                ...
                }
                else {
                MP_RUN_CROAK(modperl_output_filter_flush(filter),
                "Apache2::Filter");
                }

                is that it? if you change the "Apache2::Filter" string (drop the 2 in your
                case) to something else?

                --
                __________________________________________________________________
                Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
                http://stason.org/ mod_perl Guide ---> http://perl.apache.org
                mailto:stas@... http://use.perl.org http://apacheweek.com
                http://modperlbook.org http://apache.org http://ticketmaster.com
              • Stas Bekman
                ... If this is the case, please try this patch, I hope it applies to RC4 (I ve dropped 2 for you) Index: src/modules/perl/modperl_filter.c ... +++
                Message 7 of 13 , May 11, 2005
                • 0 Attachment
                  Stas Bekman wrote:
                  > Jeff Ambrosino wrote:
                  >
                  >> Ok, this is going to be tough for me to test because I'm still on RC4,
                  >> so it's not yet converted to the new Apache2 naming. I'll still try
                  >> to make/test the svn snapshot, though (which also gives me a good
                  >> reason to get more familiar with subversion.) I'll email what I find
                  >> out.
                  >>
                  >> thanks Stas!
                  >
                  >
                  > OK, may be just try comparing to:
                  >
                  > $@ == APR::Const::ECONNABORTED or $@ == APR::Const::ECONNRESET
                  >
                  > The new subs just supposed to work on more platforms, where other
                  > similar codes may be sent by the system.
                  >
                  > You can still play with the test
                  > t/protocol/TestProtocol/echo_nonblock.pm from RC4.
                  >
                  > but looking again at your report, I can see that the issue is elsewhere.
                  >
                  > I believe it's coming from src/modules/perl/modperl_filter.c:
                  >
                  > if (filter->mode == MP_INPUT_FILTER_MODE) {
                  > ...
                  > }
                  > else {
                  > MP_RUN_CROAK(modperl_output_filter_flush(filter),
                  > "Apache2::Filter");
                  > }
                  >
                  > is that it? if you change the "Apache2::Filter" string (drop the 2 in
                  > your case) to something else?

                  If this is the case, please try this patch, I hope it applies to RC4 (I've
                  dropped '2' for you)

                  Index: src/modules/perl/modperl_filter.c
                  ===================================================================
                  --- src/modules/perl/modperl_filter.c (revision 169708)
                  +++ src/modules/perl/modperl_filter.c (working copy)
                  @@ -535,11 +535,15 @@
                  filter->bb_in = NULL;
                  }
                  MP_RUN_CROAK(modperl_input_filter_flush(filter),
                  - "Apache::Filter");
                  + "Apache::Filter internal flush");
                  }
                  else {
                  - MP_RUN_CROAK(modperl_output_filter_flush(filter),
                  - "Apache::Filter");
                  + apr_status_t rc = modperl_output_filter_flush(filter);
                  + if (!(rc == APR_SUCCESS ||
                  + APR_STATUS_IS_ECONNRESET(rc) ||
                  + APR_STATUS_IS_ECONNABORTED(rc))) {
                  + modperl_croak(aTHX_ rc, "Apache::Filter internal flush");
                  + }
                  }

                  MP_FILTER_RESTORE_ERRSV(errsv);

                  I need to figure out how to write a test for this, so I can decide what's
                  the right way to handle this.

                  --
                  __________________________________________________________________
                  Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
                  http://stason.org/ mod_perl Guide ---> http://perl.apache.org
                  mailto:stas@... http://use.perl.org http://apacheweek.com
                  http://modperlbook.org http://apache.org http://ticketmaster.com
                • Stas Bekman
                  Jeff? Could you please try that patch that I ve sent earlier? I think it s right, but I was still not very successful at reproducing your problem on the HTTP
                  Message 8 of 13 , May 23, 2005
                  • 0 Attachment
                    Jeff? Could you please try that patch that I've sent earlier? I think it's
                    right, but I was still not very successful at reproducing your problem on
                    the HTTP level. Thanks.

                    Stas Bekman wrote:
                    > Stas Bekman wrote:
                    >
                    >> Jeff Ambrosino wrote:
                    >>
                    >>> Ok, this is going to be tough for me to test because I'm still on RC4,
                    >>> so it's not yet converted to the new Apache2 naming. I'll still try
                    >>> to make/test the svn snapshot, though (which also gives me a good
                    >>> reason to get more familiar with subversion.) I'll email what I find
                    >>> out.
                    >>>
                    >>> thanks Stas!
                    >>
                    >>
                    >>
                    >> OK, may be just try comparing to:
                    >>
                    >> $@ == APR::Const::ECONNABORTED or $@ == APR::Const::ECONNRESET
                    >>
                    >> The new subs just supposed to work on more platforms, where other
                    >> similar codes may be sent by the system.
                    >>
                    >> You can still play with the test
                    >> t/protocol/TestProtocol/echo_nonblock.pm from RC4.
                    >>
                    >> but looking again at your report, I can see that the issue is elsewhere.
                    >>
                    >> I believe it's coming from src/modules/perl/modperl_filter.c:
                    >>
                    >> if (filter->mode == MP_INPUT_FILTER_MODE) {
                    >> ...
                    >> }
                    >> else {
                    >> MP_RUN_CROAK(modperl_output_filter_flush(filter),
                    >> "Apache2::Filter");
                    >> }
                    >>
                    >> is that it? if you change the "Apache2::Filter" string (drop the 2 in
                    >> your case) to something else?
                    >
                    >
                    > If this is the case, please try this patch, I hope it applies to RC4
                    > (I've dropped '2' for you)
                    >
                    > Index: src/modules/perl/modperl_filter.c
                    > ===================================================================
                    > --- src/modules/perl/modperl_filter.c (revision 169708)
                    > +++ src/modules/perl/modperl_filter.c (working copy)
                    > @@ -535,11 +535,15 @@
                    > filter->bb_in = NULL;
                    > }
                    > MP_RUN_CROAK(modperl_input_filter_flush(filter),
                    > - "Apache::Filter");
                    > + "Apache::Filter internal flush");
                    > }
                    > else {
                    > - MP_RUN_CROAK(modperl_output_filter_flush(filter),
                    > - "Apache::Filter");
                    > + apr_status_t rc = modperl_output_filter_flush(filter);
                    > + if (!(rc == APR_SUCCESS ||
                    > + APR_STATUS_IS_ECONNRESET(rc) ||
                    > + APR_STATUS_IS_ECONNABORTED(rc))) {
                    > + modperl_croak(aTHX_ rc, "Apache::Filter internal flush");
                    > + }
                    > }
                    >
                    > MP_FILTER_RESTORE_ERRSV(errsv);
                    >
                    > I need to figure out how to write a test for this, so I can decide
                    > what's the right way to handle this.
                    >


                    --
                    __________________________________________________________________
                    Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
                    http://stason.org/ mod_perl Guide ---> http://perl.apache.org
                    mailto:stas@... http://use.perl.org http://apacheweek.com
                    http://modperlbook.org http://apache.org http://ticketmaster.com
                  • Stas Bekman
                    ... Excellent. Now committed. Thanks Jeff. -- __________________________________________________________________ Stas Bekman JAm_pH ------ Just
                    Message 9 of 13 , Jun 28, 2005
                    • 0 Attachment
                      Jeff Ambrosino wrote:
                      > Stas, just a quick "closure"... the patch has survived several ScanAlert
                      > scans without any aborted httpd children. This was the original way I
                      > discovered the problem (www.scanalert.com - they periodically throw
                      > all sorts of weird hack/exploit traffic at our server as an audit.)
                      > So from my end, the problem is officially fixed.

                      Excellent. Now committed.

                      Thanks Jeff.


                      --
                      __________________________________________________________________
                      Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
                      http://stason.org/ mod_perl Guide ---> http://perl.apache.org
                      mailto:stas@... http://use.perl.org http://apacheweek.com
                      http://modperlbook.org http://apache.org http://ticketmaster.com
                    • Stas Bekman
                      ... Excellent. Now committed. Thanks Jeff. -- __________________________________________________________________ Stas Bekman JAm_pH ------ Just
                      Message 10 of 13 , Jun 28, 2005
                      • 0 Attachment
                        Jeff Ambrosino wrote:
                        > Stas, just a quick "closure"... the patch has survived several ScanAlert
                        > scans without any aborted httpd children. This was the original way I
                        > discovered the problem (www.scanalert.com - they periodically throw
                        > all sorts of weird hack/exploit traffic at our server as an audit.)
                        > So from my end, the problem is officially fixed.

                        Excellent. Now committed.

                        Thanks Jeff.


                        --
                        __________________________________________________________________
                        Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
                        http://stason.org/ mod_perl Guide ---> http://perl.apache.org
                        mailto:stas@... http://use.perl.org http://apacheweek.com
                        http://modperlbook.org http://apache.org http://ticketmaster.com
                      Your message has been successfully submitted and would be delivered to recipients shortly.