Loading ...
Sorry, an error occurred while loading the content.

RE: [mp2] CGI.pm param's always empty from only some clients (bug or mistake on my part?)

Expand Messages
  • Scott Beuker
    Hi Geoff, ... I was using 2.25, but at this point I have upgraded to the latest (3.00) at your advice but to no avail. ... run(); # That s it for the main
    Message 1 of 9 , Oct 30, 2003
    • 0 Attachment
      Hi Geoff,

      > can you let us know specifically the version of CGI.pm you
      > were using. also,
      > please try with the latest CGI.pm from CPAN and see if that helps you.

      I was using 2.25, but at this point I have upgraded to the latest (3.00)
      at your advice but to no avail.

      > at any rate, if you could post a snippet that shows pretty
      > much how you use
      > CGI.pm to get the POST data, I'll use that approach for the
      > tests to help
      > reduce it some (the import statement, calls to $q->param, etc).

      Sure, here goes, with some comments thrown in:
      ----- perl ------
      run();
      # That's it for the main routine, this was originally not a mod_perl app
      #
      # This small sub basically determines what the user is doing and calls an
      # appropriate sub in turn, passing the CGI and DB handles. The problem
      # crops up right away because the users who are hitting the 'submit' button
      # in my app are being sent the initial page again, or in other words it is
      # page_select() being called (which is the initial page a user is shown)
      # instead of page_graphs() which is what the user should be shown. Among
      # my tests I used a javascript alert to check the value of submit_type in
      # on "onClick" and the value is indeed "instances" at that point. But then
      # on the back end, the value is found to be undefined.
      sub run()
      {
      ...
      my $q = new CGI;
      ...
      if ($q->param('submit_type') eq "instances") {
      $TABLE = $q->param('f_table');
      $TIMESHIFT = $q->param('f_timeshift');
      page_graphs($q, $dbh);
      } elsif (defined($q->param('f_table'))) {
      $TABLE = $q->param('f_table');
      $TIMESHIFT = $q->param('f_timeshift') if
      (defined($q->param('f_timeshift')));
      page_select($q, $dbh);
      } else {
      page_select($q, $dbh);
      }
      exit(0);
      }
      ----- /perl -----

      Please let me know if there is anything more I can do. I hope this helps.
      I thought about writing a small example script myself but given the amount
      of time I have spent on this issue already, I have to be careful here at
      work.

      Cheers,
      Scott Beuker
    • Scott Beuker
      ... A little extra information that may help to narrow it down; I sniffed the HTTP sessions to the server and got a copy of the headers in each of a working
      Message 2 of 9 , Oct 30, 2003
      • 0 Attachment
        > Please let me know if there is anything more I can do. I hope
        > this helps.
        > I thought about writing a small example script myself but
        > given the amount
        > of time I have spent on this issue already, I have to be
        > careful here at
        > work.

        A little extra information that may help to narrow it down; I
        sniffed the HTTP sessions to the server and got a copy of the
        headers in each of a working and non-working computer. I did
        not see any differences of significance so I was going to try
        changing the various fields one at a time to figure out which
        caused it not to work, by telneting into the server and then
        sending the query manually.

        Instead what I found was that the headers in the http request
        do not matter, it seems to simply be a matter of what client
        computer you are coming from. When I pasted in the request of
        a machine that did not work from a machine that works, it did
        still work, and vice versa.

        Unfortunately for me, given this information I have hit the
        end of the troubleshooting I can do. I don't have the skills
        required to reverse engineer CGI.pm or mp2. I think I may
        have to revert to Apache1/MP1 for the forseeable future. I
        really hope you are able to track this one down.

        Thanks,
        Scott Beuker
      • Geoffrey Young
        ... yike, that s not a good sign. unfortunately, it also limits our ability to help. we can only do so much investigation when we can t reproduce an issue -
        Message 3 of 9 , Oct 30, 2003
        • 0 Attachment
          Scott Beuker wrote:
          >
          >
          >>Please let me know if there is anything more I can do. I hope
          >>this helps.
          >>I thought about writing a small example script myself but
          >>given the amount
          >>of time I have spent on this issue already, I have to be
          >>careful here at
          >>work.
          >
          >
          > A little extra information that may help to narrow it down; I
          > sniffed the HTTP sessions to the server and got a copy of the
          > headers in each of a working and non-working computer. I did
          > not see any differences of significance so I was going to try
          > changing the various fields one at a time to figure out which
          > caused it not to work, by telneting into the server and then
          > sending the query manually.
          >
          > Instead what I found was that the headers in the http request
          > do not matter, it seems to simply be a matter of what client
          > computer you are coming from. When I pasted in the request of
          > a machine that did not work from a machine that works, it did
          > still work, and vice versa.

          yike, that's not a good sign.

          unfortunately, it also limits our ability to help. we can only do so much
          investigation when we can't reproduce an issue - if you can't reliably
          reproduce it across your own machines it's even harder.

          >
          > Unfortunately for me, given this information I have hit the
          > end of the troubleshooting I can do.

          one way to help would be to start stripping away your environment, piece by
          piece. for instance, start by removing all non-essential processing from
          your script, then by removing excess httpd.conf so that you just have enough
          to run one request, and all that request does is something that exhibits the
          bug.

          then try reproducing that httpd.conf and script within Apache-Test. you can
          use this as a starting point.

          http://perl.apache.org/~geoff/bug-mp2.tar.gz

          if you can boil everything down so that anyone can untar test, run it, and
          reproduce it, we'll be much more likely to be able to fix it.

          I know it sounds like a lot. sorry :)

          --Geoff
        • Scott Beuker
          ... I tried this first, no luck. ... I did this, and the result is interesting to me. First, as a side note, when I put PerlTrace o in the httpd.conf right
          Message 4 of 9 , Oct 31, 2003
          • 0 Attachment
            > There are two things I'd suggest to try:
            >
            > 1) pass $r to CGI->new($r), in which case it won't rely on the global
            > Apache->request. And you can switch then to 'PerlHandler modperl'.

            I tried this first, no luck.

            > 2) use the IO trace to debug what's going on:
            >
            > you need to rebuild mp with MP_TRACE=1 and then add to httpd.conf:
            >
            > PerlTrace o
            >
            > http://perl.apache.org/docs/2.0/user/config/config.html#C_PerlTrace_
            >
            > compare the traces you get on the good and the bad machines.

            I did this, and the result is interesting to me.

            First, as a side note, when I put 'PerlTrace o' in the httpd.conf right off
            the bat, Apache segfaulted on restart. So then I instead put the following
            in the
            <Perl> section: ' $ENV{MOD_PERL_TRACE} = "o";' and it started fine.

            Oh and another side note... today I discovered that I am now seeing the
            issue
            from my own client here on my windows workstation *sometimes*. So it's no
            longer
            a per client issue.

            The trace is quote noisy considering the size of the pages I am returning,
            but
            let me try to paraphrase the juicy bits (at least to my untrained eyes):

            ----- error_log ------
            [Fri Oct 31 10:12:58 2003] [notice] SIGHUP received. Attempting to restart
            [Fri Oct 31 10:12:58 2003] [notice] seg fault or similar nasty error
            detected in the parent process
            mod_perl trace flags dump:
            <SNIP (mp flags, IO is on everything else is off)>
            <NOTE (this is where Apache segfaulted the first time, I made the httpd.conf
            change, and restarted)>
            [Fri Oct 31 10:16:13 2003] [warn] pid file /usr/local/apache/logs/httpd.pid
            overwritten -- Unclean shutdown of previous Apache run?
            [Fri Oct 31 10:16:13 2003] [notice] Apache/2.0.48 (Unix) mod_perl/1.99_10
            Perl/v5.8.0 configured -- resuming normal operations
            <SNIP (some seemingly inconsequential stuff)>
            <NOTE (this is a request from my machine that usually works but didn't work
            this time)>
            *** reading POST data: CGI/usr/lib/perl5/5.8.0/CGI.pm518 at (eval 18) line
            8.
            PerlIOApache_read: wanted 3362b, read 189b
            [f_table=R_rtrallint05m_&f_colour_availability_admin=%23F4E204&f_colour_avai
            lability_oper
            ational=%23F4B904&f_vars=bits_in&f_colour_bits_in=%23F4E204&f_vars=bits_out&
            f_colour_bits_out=%23F4B9;
            <SNIP (a bunch of perl code, see below... the brackets for the form data are
            not closed until AFTER the code below!)>
            PerlIOApache_write: 48 bytes [<HTML>
            <SNIP (Apache writing my page which indicates there was no form data)>
            <NOTE (this is the second request, again from my machine, again the
            variables were apparently not readable)>
            PerlIOApache_read: wanted 3361b, read 28b
            [f_table=R_rtrallint05m_&f_cotipart {
            <SNIP (the same code as per above, see below)
            <SNIP (Apache again writes my page, the one for when there is no form data)>
            <NOTE (this is the third request from me, which WORKS!)>
            *** reading POST data: CGI/usr/lib/perl5/5.8.0/CGI.pm518 at (eval 18) line
            8.
            PerlIOApache_read: wanted 3361b, read 3361b
            [f_table=R_rtrallint05m_&f_colour_availability_admin=%23F4E204&f_colour_avai
            lability_ope
            <SNIP (this time all my form data is there... notice it read all it wanted)>
            <SNIP (Apache writes my page, the CORRECT page, which indicates that the
            form data was available to my script>)
            ----- /error_log -----

            So in summary, it seems there is an issue reading all the form data on the
            requests that do not work. The variable that I use to determine which
            sub-routine to call is the submit button which, not surprisingly, comes last
            and so is being cut off. In case you missed it above, the square brackets
            which are seemingly supposed to contain the data that is read are not closed
            until AFTER the source code that appears. Buffer issues?

            Just in case it is of interest, here is the source that is printed during
            the failed read:

            ----- error_log ------
            PerlIOApache_read: wanted 3362b, read 189b
            [f_table=R_rtrallint05m_&f_colour_availability_admin=%23F4E204&f_colour_avai
            lability_oper
            ational=%23F4B904&f_vars=bits_in&f_colour_bits_in=%23F4E204&f_vars=bits_out&
            f_colour_bits_out=%23F4B9;
            my(%header,$body);
            my $filenumber = 0;
            while (!$buffer->eof) {
            %header = $buffer->readHeader;

            unless (%header) {
            $self->cgi_error("400 Bad request (malformed multipart POST)");
            return;
            }

            my($param)= $header{'Content-Disposition'}=~/ name="?([^\";]*)"?/;
            $param .= $TAINTED;

            # Bug: Netscape doesn't escape quotation marks in file names!!!
            my($filename) = $header{'Content-Disposition'}=~/
            filename="?([^\"]*)"?/;
            # Test for Opera's multiple upload feature
            my($multipart) = ( defined( $header{'Content-Type'} ) &&
            $header{'Content-Type'} =~ /multipart\/mixed/ ) ?
            1 : 0;

            # add this parameter to our list
            $self->add_parameter($param);

            # If no filename specified, then just read the data and assign it
            # to our parameter list.
            if ( ( !defined($filename) || $filename eq '' ) && !$multipart ) {
            my($value) = $buffer->readBody;
            $value .= $TAINTED;
            push(@{$self->{$param}},$value);
            next;
            }

            my ($tmpfile,$tmp,$filehandle);
            UPLOADS: {
            # If we get here, then we are dealing with a potentially large
            # uploaded form. Save the data to a temporary file, then open
            # the file for reading.

            # skip the file if uploads disabled
            if ($DISABLE_UPLOADS) {
            while (defined($data = $buffer->read)) { }
            last UPLOADS;
            }

            # set the filename to some recognizable value
            if ( ( !defined($filename) || $filename eq '' ) && $multipart ) {
            $filename = "multipart/mixed";
            }

            # choose a relatively unpredictable tmpfile sequence number
            my $seqno = unpack("%16C*",join('',localtime,values %ENV));
            for (my $cnt=10;$cnt>0;$cnt--) {
            next unless $tmpfile = new CGITempFile($seqno);
            $tmp = $tmpfile->as_string;
            last if defined($filehandle =
            Fh->new($filename,$tmp,$PRIVATE_TEMPFILES));
            $seqno += int rand(100);
            }
            die "CGI open of tmpfile: $!\n" unless defined $filehandle;
            $CGI::DefaultClass->binmode($filehandle) if $CGI::needs_binmode;

            # if this is an multipart/mixed attachment, save the header
            # together with the body for lateron parsing with an external
            # MIME parser module
            if ( $multipart ) {
            foreach ( keys %header ) {
            print $filehandle "$_: $header{$_}${CRLF}";
            }
            print $filehandle "${CRLF}";
            }

            my ($data);
            local($\) = '';
            while (defined($data = $buffer->read)) {
            print $filehandle $data;
            }

            # back up to beginning of file
            seek($filehandle,0,0);

            ## Close the filehandle if requested this allows a multipart MIME
            ## upload to contain many files, and we won't die due to too many
            ## open file handles. The user can access the files using the hash
            ## below.
            close $filehandle if $CLOSE_UPLOAD_FILES;
            $CGI::DefaultClass->binmode($filehandle) if $CGI::needs_binmode;

            # Save some information about the uploaded file where we can get
            # at it later.
            $self->{'.tmpfiles'}->{fileno($filehandle)}= {
            name => $tmpfile,
            info => {%header},
            };
            push(@{$self->{$param}},$filehandle);
            }
            }ðL]
            ----- /error_log -----

            That's a lot of data, so I'll stop there. Let me know if what I am providing
            is not clear or not what you are looking for.

            Thanks,
            Scott Beuker
          • Scott Beuker
            ... When I tried your patch as is, I got the following error: *** reading POST data: CGI/usr/lib/perl5/5.8.0/CGI.pm518 at (eval 18) line 8. [Fri Oct 31
            Message 5 of 9 , Oct 31, 2003
            • 0 Attachment
              > kyle's post got me thinking. it would be interesting to see
              > the output of
              > this along with those traces. this is against CGI.pm 3.00.
              > untested, but
              > you get the idea :)

              When I tried your patch as is, I got the following error:
              *** reading POST data: CGI/usr/lib/perl5/5.8.0/CGI.pm518 at (eval 18) line
              8.
              [Fri Oct 31 09:44:07 2003] [error] 30899: ModPerl::Registry: argument is not
              a blessed reference (expecting an APR::Table derived object) at (eval 18)
              line 9.

              So then I removed the second warn... I figure if it's reading
              twice, you will just see two warns (I was watching as the requests
              came in so there was no confusion):

              + if ($MOD_PERL) {
              + warn ('*** reading POST data: ', caller);
              + }

              The end result is I only ever saw one warning per POST for both
              the good and bad requests. You can see your warn's in amongst the
              other stuff in the error_log in my previous response to Stas.

              Cheers,
              Scott Beuker
            • Stas Bekman
              ... +++ CGI.pm 2003-10-31 13:58:28.000000000 -0800 @@ -515,8 +515,15 @@ } if ($meth eq POST ) { -
              Message 6 of 9 , Oct 31, 2003
              • 0 Attachment
                Scott, please try this patch:

                --- CGI.pm.orig 2003-10-31 13:45:06.000000000 -0800
                +++ CGI.pm 2003-10-31 13:58:28.000000000 -0800
                @@ -515,8 +515,15 @@
                }

                if ($meth eq 'POST') {
                - $self->read_from_client(\*STDIN,\$query_string,$content_length,0)
                - if $content_length > 0;
                + if ($content_length > 0) {
                + my $len = $content_length;
                + while ($len > 0) {
                + my $data = '';
                + my $read = $self->read_from_client(\*STDIN,\$data,$len,0);
                + $len -= $read;
                + $query_string .= $data if $read > 0;
                + }
                + }
                # Some people want to have their cake and eat it too!
                # Uncomment this line to have the contents of the query string
                # APPENDED to the POST data.


                __________________________________________________________________
                Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
                http://stason.org/ mod_perl Guide ---> http://perl.apache.org
                mailto:stas@... http://use.perl.org http://apacheweek.com
                http://modperlbook.org http://apache.org http://ticketmaster.com
              • Scott Beuker
                Stas, The patch seems to work based on one previously affected client! However, it s evening here and everyone has gone home. I ll unfortunately need to wait
                Message 7 of 9 , Oct 31, 2003
                • 0 Attachment
                  Stas,

                  The patch seems to work based on one previously affected client! However,
                  it's
                  evening here and everyone has gone home. I'll unfortunately need to wait
                  until
                  Monday to test more affected clients. I'll let you know then the results,
                  but
                  so far so good!

                  Thanks,
                  Scott




                  > Scott, please try this patch:
                  >
                  > --- CGI.pm.orig 2003-10-31 13:45:06.000000000 -0800
                  > +++ CGI.pm 2003-10-31 13:58:28.000000000 -0800
                  > @@ -515,8 +515,15 @@
                  > }
                  >
                  > if ($meth eq 'POST') {
                  > -
                  $self->read_from_client(\*STDIN,\$query_string,$content_length,0)
                  > - if $content_length > 0;
                  > + if ($content_length > 0) {
                  > + my $len = $content_length;
                  > + while ($len > 0) {
                  > + my $data = '';
                  > + my $read =
                  $self->read_from_client(\*STDIN,\$data,$len,0);
                  > + $len -= $read;
                  > + $query_string .= $data if $read > 0;
                  > + }
                  > + }
                  > # Some people want to have their cake and eat it too!
                  > # Uncomment this line to have the contents of the query string
                  > # APPENDED to the POST data.
                • Scott Beuker
                  Stas, I have confirmed that the patch has worked everywhere! Thank you very much for your help (and to Geoff too). Can I assume that you will be submitting the
                  Message 8 of 9 , Nov 3, 2003
                  • 0 Attachment
                    Stas,

                    I have confirmed that the patch has worked everywhere! Thank you very much
                    for
                    your help (and to Geoff too).

                    Can I assume that you will be submitting the patch to the CGI.pm maintainer?

                    Thanks,
                    Scott Beuker

                    ############################################################################
                    ####
                    > The patch seems to work based on one previously affected
                    > client! However, it's
                    > evening here and everyone has gone home. I'll unfortunately
                    > need to wait until
                    > Monday to test more affected clients. I'll let you know then
                    > the results, but
                    > so far so good!

                    --
                    Reporting bugs: http://perl.apache.org/bugs/
                    Mail list info: http://perl.apache.org/maillist/modperl.html
                  • Stas Bekman
                    ... Cool. ... I did already. However most likely the solution will be different. It s not clear whose responsibility it is to do the blocking read, the client
                    Message 9 of 9 , Nov 3, 2003
                    • 0 Attachment
                      Scott Beuker wrote:
                      > Stas,
                      >
                      > I have confirmed that the patch has worked everywhere! Thank you very much
                      > for
                      > your help (and to Geoff too).

                      Cool.

                      > Can I assume that you will be submitting the patch to the CGI.pm maintainer?

                      I did already. However most likely the solution will be different. It's not
                      clear whose responsibility it is to do the blocking read, the client (CGI.pm)
                      or the low-level read() (modperl). Lincoln says that modperl's, I want to ping
                      p5p to make sure that this is the right thing to do. Meanwhile please use that
                      patch remembering that it might be not the final one. I'll keep the list
                      posted on this issue.

                      __________________________________________________________________
                      Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
                      http://stason.org/ mod_perl Guide ---> http://perl.apache.org
                      mailto:stas@... http://use.perl.org http://apacheweek.com
                      http://modperlbook.org http://apache.org http://ticketmaster.com


                      --
                      Reporting bugs: http://perl.apache.org/bugs/
                      Mail list info: http://perl.apache.org/maillist/modperl.html
                    Your message has been successfully submitted and would be delivered to recipients shortly.