Loading ...
Sorry, an error occurred while loading the content.

Re: mod_perl2, HEAD request and Content-Length

Expand Messages
  • Boris Zentner
    Hi, ... But my point is, for a HEAD request, there is no data so apache should not touch my content-length header. I really dislike to generate the full data
    Message 1 of 25 , Aug 1, 2004
    • 0 Attachment
      Hi,

      Am Sonntag 01 August 2004 18:13 schrieb Stas Bekman:
      > Does anybody have any statistics on the percentage of HEAD requests vs.
      > GET? The purpose of HEAD request is not to save your CPU cycles, but to
      > avoid data transfer which is precisely what Apache 2.0 does, without
      > making you do the extra work to deal with HEAD requests.
      >
      > > also The comment implies, that I get a Content-Length header for a HEAD
      > > request, if I waste my time and generate the body of the message. I tried
      > > it and got _no_ Content-Length header!
      > >
      > >  $apr->content_type($media_type) unless $apr->main;
      > >   if ( $apr->header_only ) {
      > >     $apr->print( ' ' x $size ); # just to see if I got a content-length
      > > header return DONE;
      > >   }
      > >
      > > The only way to get the Content-Length header for a HEAD request is
      > >
      > >  $apr->headers_out->{'Content-Length'} = $size ;
      > >  $apr->content_type($media_type) unless $apr->main;
      > >   if ( $apr->header_only ) {
      > >     $apr->rflush;
      > >     return DONE;
      > >   }
      > >
      > > But I fail to see why this works, I thought the data walks the same
      > > filter chain only in more buckets.
      >
      > In dynamic scripts you never get the C-L header unless you set it
      > explicitly and some filter doesn't remove it. It's not possible for
      > Apache to add one, since headers are sent *before* it knows how much
      > data your handler is going to send. So HEAD or GET, you won't get it.

      But my point is, for a HEAD request, there is no data so apache should not
      touch my content-length header. I really dislike to generate the full data
      for the request and apache throws it away ( and even the I get no
      Content-Length header ).

      >
      > So HEAD on dynamic scripts that doesn't set their own C-L is a wasted
      > operation on behalf of the client. Unless you use some E-Tag (but I
      > haven't checked the RFC).
      >
      > BTW, what's wrong with $r->set_content_length()?

      Nothing, I searched for the function, but I did not find it! I know I should
      use mp2doc, but as I searched for it, I did not think on the tool. I tryed
      first with perldoc Apache::RequestRec and then I searched

      http://perl.apache.org/docs/2.0/api/Apache/RequestRec.html

      For some reason I thought if there is no function content_length then I have
      no function like this.

      > http://perl.apache.org/docs/2.0/api/Apache/Response.html#C_set_content_leng
      >th_ Neither you need to rflush. Neither you should return DONE, but OK.

      I tried it now and even with set_content_length and return OK I got no
      content-length header.

      PS: Much thanks for the pointer to Apache::Response, I found more functions
      that I missed already!

      --
      Boris

      --
      Report problems: http://perl.apache.org/bugs/
      Mail list info: http://perl.apache.org/maillist/modperl.html
      List etiquette: http://perl.apache.org/maillist/email-etiquette.html
    • Stas Bekman
      Boris Zentner wrote: [...] ... (sure, it was unrelated to the problem you were mentioned) let me try to write a test and I ll be back to you. Me thinking that
      Message 2 of 25 , Aug 1, 2004
      • 0 Attachment
        Boris Zentner wrote:
        [...]
        > Nothing, I searched for the function, but I did not find it! I know I should
        > use mp2doc, but as I searched for it, I did not think on the tool. I tryed
        > first with perldoc Apache::RequestRec and then I searched
        >
        > http://perl.apache.org/docs/2.0/api/Apache/RequestRec.html
        >
        > For some reason I thought if there is no function content_length then I have
        > no function like this.
        >
        >
        >>http://perl.apache.org/docs/2.0/api/Apache/Response.html#C_set_content_leng
        >>th_ Neither you need to rflush. Neither you should return DONE, but OK.
        >
        >
        > I tried it now and even with set_content_length and return OK I got no
        > content-length header.

        (sure, it was unrelated to the problem you were mentioned)

        let me try to write a test and I'll be back to you.

        Me thinking that we should drop the /^set_/ prefix in
        set_content_length, to be consistent with the rest of the API.

        > PS: Much thanks for the pointer to Apache::Response, I found more functions
        > that I missed already!

        All Apache::RequestRec classes now xref themselves from their
        Description sections, e.g.:
        http://perl.apache.org/docs/2.0/api/Apache/RequestRec.html#Description

        --
        __________________________________________________________________
        Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
        http://stason.org/ mod_perl Guide ---> http://perl.apache.org
        mailto:stas@... http://use.perl.org http://apacheweek.com
        http://modperlbook.org http://apache.org http://ticketmaster.com

        --
        Report problems: http://perl.apache.org/bugs/
        Mail list info: http://perl.apache.org/maillist/modperl.html
        List etiquette: http://perl.apache.org/maillist/email-etiquette.html
      • Stas Bekman
        ... I get indentical behavior with either of the two methods: my $body = This is a response string ; #$r- headers_out- { Content-Length } = length $body;
        Message 3 of 25 , Aug 1, 2004
        • 0 Attachment
          Stas Bekman wrote:

          > let me try to write a test and I'll be back to you.

          I get indentical behavior with either of the two methods:

          my $body = "This is a response string";
          #$r->headers_out->{'Content-Length'} = length $body;
          $r->set_content_length(length $body);

          $r->print('a');

          if the one-char-print line is commented out, I *never* get the C-L
          header for HEAD requests. Otherwise I always get the correct C-L header.

          Normally, Apache is not able to figure out how much data the content
          handler is going to send, since it needs to send headers early. However
          in the case where your handler returns w/o sending anything at all, the
          headers_out filter sees EOS and no data, so it probably thinks that C-L
          is wrong and drops it. My test with printing just one char makes it
          working. It's safe to send anything, since Apache will discard it for
          HEAD requests anyway. So here you have a workaround (but I can't promise
          that it will always continue to work).

          Please ask at the Apache list, whether there is a cleaner way to
          make Apache preserve the C-L header, w/o sending any data, and please
          let us know the verdict.

          p.s. I'm not sure how you've got a HEAD request with C-L header working,
          when using $r->headers_out->{'Content-Length'}; it doesn't work for me.
          (though I use Apache 2.0.50-dev so I won't be surprised if something has
          changed).

          --
          __________________________________________________________________
          Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
          http://stason.org/ mod_perl Guide ---> http://perl.apache.org
          mailto:stas@... http://use.perl.org http://apacheweek.com
          http://modperlbook.org http://apache.org http://ticketmaster.com

          --
          Report problems: http://perl.apache.org/bugs/
          Mail list info: http://perl.apache.org/maillist/modperl.html
          List etiquette: http://perl.apache.org/maillist/email-etiquette.html
        • Stas Bekman
          ... I ve just committed this test to the modperl-2.0 cvs rep: t/apache/head_request.t t/response/TestApache/head_request.pm please take a look --
          Message 4 of 25 , Aug 1, 2004
          • 0 Attachment
            Stas Bekman wrote:
            > Stas Bekman wrote:
            >
            >> let me try to write a test and I'll be back to you.

            I've just committed this test to the modperl-2.0 cvs rep:
            t/apache/head_request.t t/response/TestApache/head_request.pm
            please take a look

            --
            __________________________________________________________________
            Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
            http://stason.org/ mod_perl Guide ---> http://perl.apache.org
            mailto:stas@... http://use.perl.org http://apacheweek.com
            http://modperlbook.org http://apache.org http://ticketmaster.com

            --
            Report problems: http://perl.apache.org/bugs/
            Mail list info: http://perl.apache.org/maillist/modperl.html
            List etiquette: http://perl.apache.org/maillist/email-etiquette.html
          • Geoffrey Young
            ... for the record, this is fundamentally wrong. HEAD requests are supposed to be identical to GET requests in every way _except_ that there is no message
            Message 5 of 25 , Aug 2, 2004
            • 0 Attachment
              > But my point is, for a HEAD request, there is no data so apache should not
              > touch my content-length header. I really dislike to generate the full data
              > for the request and apache throws it away ( and even the I get no
              > Content-Length header ).

              for the record, this is fundamentally wrong. HEAD requests are supposed to
              be identical to GET requests in every way _except_ that there is no message
              body, which means that if a GET request for a specific resource does not
              have a C-L header then a HEAD request for the same resource also _must_ not
              have a C-L header. at least if you care about RFC compliance.

              --Geoff

              --
              Report problems: http://perl.apache.org/bugs/
              Mail list info: http://perl.apache.org/maillist/modperl.html
              List etiquette: http://perl.apache.org/maillist/email-etiquette.html
            • Boris Zentner
              Hi, ... ??? I do not know what you mean, my GET request _has_ a content-lenght header! For HEAD, I just do not calculate the expencive data for my body. I just
              Message 6 of 25 , Aug 2, 2004
              • 0 Attachment
                Hi,

                Am Montag 02 August 2004 15:28 schrieb Geoffrey Young:
                > > But my point is, for a HEAD request, there is no data so apache should
                > > not touch my content-length header. I really dislike to generate the full
                > > data for the request and apache throws it away ( and even the I get no
                > > Content-Length header ).
                >
                > for the record, this is fundamentally wrong. HEAD requests are supposed to
                > be identical to GET requests in every way _except_ that there is no message
                > body, which means that if a GET request for a specific resource does not
                > have a C-L header then a HEAD request for the same resource also _must_ not
                > have a C-L header. at least if you care about RFC compliance.
                >

                ??? I do not know what you mean, my GET request _has_ a content-lenght header!
                For HEAD, I just do not calculate the expencive data for my body.

                I just missed the content-length header on HEAD requests that is delivered on
                GET.

                > --Geoff

                --
                Boris

                --
                Report problems: http://perl.apache.org/bugs/
                Mail list info: http://perl.apache.org/maillist/modperl.html
                List etiquette: http://perl.apache.org/maillist/email-etiquette.html
              • Stas Bekman
                ... Geoff, see the t/apache/head_request.t test I ve added last night. If you don t send a body, Apache strips the C-L header for HEAD requests. Sending at
                Message 7 of 25 , Aug 2, 2004
                • 0 Attachment
                  Boris Zentner wrote:
                  > Hi,
                  >
                  > Am Montag 02 August 2004 15:28 schrieb Geoffrey Young:
                  >
                  >>>But my point is, for a HEAD request, there is no data so apache should
                  >>>not touch my content-length header. I really dislike to generate the full
                  >>>data for the request and apache throws it away ( and even the I get no
                  >>>Content-Length header ).
                  >>
                  >>for the record, this is fundamentally wrong. HEAD requests are supposed to
                  >>be identical to GET requests in every way _except_ that there is no message
                  >>body, which means that if a GET request for a specific resource does not
                  >>have a C-L header then a HEAD request for the same resource also _must_ not
                  >>have a C-L header. at least if you care about RFC compliance.
                  >>
                  >
                  >
                  > ??? I do not know what you mean, my GET request _has_ a content-lenght header!
                  > For HEAD, I just do not calculate the expencive data for my body.
                  >
                  > I just missed the content-length header on HEAD requests that is delivered on
                  > GET.

                  Geoff, see the t/apache/head_request.t test I've added last night. If
                  you don't send a body, Apache strips the C-L header for HEAD requests.
                  Sending at least 1 byte works as a workaround, but it smells like a bug
                  in Apache.


                  --
                  __________________________________________________________________
                  Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
                  http://stason.org/ mod_perl Guide ---> http://perl.apache.org
                  mailto:stas@... http://use.perl.org http://apacheweek.com
                  http://modperlbook.org http://apache.org http://ticketmaster.com

                  --
                  Report problems: http://perl.apache.org/bugs/
                  Mail list info: http://perl.apache.org/maillist/modperl.html
                  List etiquette: http://perl.apache.org/maillist/email-etiquette.html
                • Geoffrey Young
                  ... that is exactly what I mean - if you include a C-L header on a GET then you are supposed to have one for a HEAD request as well, expensive or not. HEAD is
                  Message 8 of 25 , Aug 2, 2004
                  • 0 Attachment
                    >> ??? I do not know what you mean, my GET request _has_ a content-lenght
                    >> header! For HEAD, I just do not calculate the expencive data for my body.

                    that is exactly what I mean - if you include a C-L header on a GET then you
                    are supposed to have one for a HEAD request as well, expensive or not. HEAD
                    is supposed to be exactly the same as a GET in all respects except that it
                    does not have a message body.

                    >>
                    >> I just missed the content-length header on HEAD requests that is
                    >> delivered on GET.
                    >
                    >
                    > Geoff, see the t/apache/head_request.t test I've added last night. If
                    > you don't send a body, Apache strips the C-L header for HEAD requests.
                    > Sending at least 1 byte works as a workaround, but it smells like a bug
                    > in Apache.

                    yes, clearly there is a bug in there, but not where you think - what is
                    important is that apache needs to do the same thing on GET as HEAD, not that
                    there is no C-L generated for contentless HEAD requests. this patch more
                    accurately represents the real bug which, IIRC, httpd is already aware of
                    (at least I'm recalling discussion about a C-L of 0).

                    --Geoff
                  • Jean-Michel Hiver
                    ... From RFC 2616: The HEAD method is identical to GET except that the server MUST NOT return a message-body in the response. The metainformation contained in
                    Message 9 of 25 , Aug 2, 2004
                    • 0 Attachment
                      Geoffrey Young wrote:
                      >>>??? I do not know what you mean, my GET request _has_ a content-lenght
                      >>>header! For HEAD, I just do not calculate the expencive data for my body.
                      >
                      > that is exactly what I mean - if you include a C-L header on a GET then you
                      > are supposed to have one for a HEAD request as well, expensive or not. HEAD
                      > is supposed to be exactly the same as a GET in all respects except that it
                      > does not have a message body.


                      From RFC 2616:

                      "The HEAD method is identical to GET except that the server MUST NOT
                      return a message-body in the response. The metainformation contained in
                      the HTTP headers in response to a HEAD request SHOULD be identical to
                      the information sent in response to a GET request."


                      From RFC 2119:

                      "SHOULD This word, or the adjective "RECOMMENDED", mean that there
                      may exist valid reasons in particular circumstances to ignore a
                      particular item, but the full implications must be understood and
                      carefully weighed before choosing a different course."


                      I think here we've hit a "valid reason in particular circumstance" to
                      strip the content-length header.

                      As for the implications, content-length is used for persistent HTTP
                      connections but that doesn't happen for HEAD requests - only for GETs.
                      Hence there should be no ill side effects by stripping the
                      Content-Length metadata for HEAD requests.


                      Cheers,
                      Jean-Michel.

                      --
                      Report problems: http://perl.apache.org/bugs/
                      Mail list info: http://perl.apache.org/maillist/modperl.html
                      List etiquette: http://perl.apache.org/maillist/email-etiquette.html
                    • Geoffrey Young
                      ... indeed :) ... yeah, this is the conversation that comes up all the time. from what I ve seen in various forums, despite this definition SHOULD essentially
                      Message 10 of 25 , Aug 2, 2004
                      • 0 Attachment
                        > From RFC 2616:
                        >
                        > "The HEAD method is identical to GET except that the server MUST NOT
                        > return a message-body in the response. The metainformation contained in
                        > the HTTP headers in response to a HEAD request SHOULD be identical to
                        > the information sent in response to a GET request."

                        indeed :)

                        >
                        >
                        > From RFC 2119:
                        >
                        > "SHOULD This word, or the adjective "RECOMMENDED", mean that there
                        > may exist valid reasons in particular circumstances to ignore a
                        > particular item, but the full implications must be understood and
                        > carefully weighed before choosing a different course."

                        yeah, this is the conversation that comes up all the time. from what I've
                        seen in various forums, despite this definition SHOULD essentially means
                        MUST. at least I haven't come across a circumstance where it hasn't :)

                        > I think here we've hit a "valid reason in particular circumstance" to
                        > strip the content-length header.

                        well, that depends. but the short of it is that almost nobody is HEAD
                        compliant (see a recent thread in the past month wrt mozilla and HEAD and
                        yahoo). and almost nobody is compliant because almost nobody understands
                        exactly what a HEAD request is supposed to entail, thus the reason why I was
                        belaboring the point. but if you go against the RFC "recommendation"
                        knowing full well what you are doing, I guess that's ok ;)

                        >
                        > As for the implications, content-length is used for persistent HTTP
                        > connections but that doesn't happen for HEAD requests - only for GETs.
                        > Hence there should be no ill side effects by stripping the
                        > Content-Length metadata for HEAD requests.

                        actually, I wonder how true that is. you would certainly think so, but if
                        you go back to that mozilla thread and read through the bugzilla entries the
                        mozilla dev guys make it sound as though they are using a HEAD request then
                        making a series of GETs over the same connection. but what do I know...

                        --Geoff

                        --
                        Report problems: http://perl.apache.org/bugs/
                        Mail list info: http://perl.apache.org/maillist/modperl.html
                        List etiquette: http://perl.apache.org/maillist/email-etiquette.html
                      • Stas Bekman
                        Geoffrey Young wrote: [...] ... Why do you say so? Your patch respresents a related, but a different issue. Boris has a problem with HEAD, and that s what the
                        Message 11 of 25 , Aug 2, 2004
                        • 0 Attachment
                          Geoffrey Young wrote:
                          [...]
                          >>>I just missed the content-length header on HEAD requests that is
                          >>>delivered on GET.
                          >>
                          >>
                          >>Geoff, see the t/apache/head_request.t test I've added last night. If
                          >>you don't send a body, Apache strips the C-L header for HEAD requests.
                          >>Sending at least 1 byte works as a workaround, but it smells like a bug
                          >>in Apache.
                          >
                          >
                          > yes, clearly there is a bug in there, but not where you think - what is
                          > important is that apache needs to do the same thing on GET as HEAD, not that
                          > there is no C-L generated for contentless HEAD requests. this patch more
                          > accurately represents the real bug which, IIRC, httpd is already aware of
                          > (at least I'm recalling discussion about a C-L of 0).

                          Why do you say so? Your patch respresents a related, but a different
                          issue. Boris has a problem with HEAD, and that's what the test was testing.

                          But, certainly please commit it, rename the files to something different
                          (content_length_header.t?) since it's not longer a HEAD request. And
                          please add a comment explaining the bug, ideally with xref to a bugzilla
                          for the bug you are talking about or discussion thread.

                          Thanks.

                          also please move that no strict thing, inside the block, where it is
                          needed. Thanks.

                          > +no strict qw(refs);
                          > +foreach my $method qw(GET HEAD) {

                          --
                          __________________________________________________________________
                          Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
                          http://stason.org/ mod_perl Guide ---> http://perl.apache.org
                          mailto:stas@... http://use.perl.org http://apacheweek.com
                          http://modperlbook.org http://apache.org http://ticketmaster.com

                          --
                          Report problems: http://perl.apache.org/bugs/
                          Mail list info: http://perl.apache.org/maillist/modperl.html
                          List etiquette: http://perl.apache.org/maillist/email-etiquette.html
                        • Geoffrey Young
                          ... ok, maybe I was misunderstanding the discussion and subsequent test. my points are these: - apache has essentially full reign over the C-L header - it
                          Message 12 of 25 , Aug 2, 2004
                          • 0 Attachment
                            >> yes, clearly there is a bug in there, but not where you think - what is
                            >> important is that apache needs to do the same thing on GET as HEAD,
                            >> not that
                            >> there is no C-L generated for contentless HEAD requests. this patch more
                            >> accurately represents the real bug which, IIRC, httpd is already aware of
                            >> (at least I'm recalling discussion about a C-L of 0).
                            >
                            >
                            > Why do you say so? Your patch respresents a related, but a different
                            > issue. Boris has a problem with HEAD, and that's what the test was testing.

                            ok, maybe I was misunderstanding the discussion and subsequent test. my
                            points are these:

                            - apache has essentially full reign over the C-L header - it decides
                            whether it is required or not, regardless of whether or not you call
                            ap_set_content_length

                            - more important than whether the C-L header shows up or not for a given
                            HEAD or GET request is whether apache is consistent with itself. that is,
                            regardless of what _you_ think it should do, apache _knows_ what it should
                            do and should do it consistently between GET and HEAD.

                            - of course, printing a C-L header to stdout works around all of this -
                            apache can only manage things when you use the official API, since it checks
                            the headers_out table and not the actual content.

                            so, the test, as it existed before, made me think that the problem you saw
                            was in that final test, that adding content magically made the C-L header
                            appear. my point was that as long as apache does whatever it does
                            consistently between GET and HEAD it's not a bug. but maybe I'm just not
                            seeing it, so if you guys grok all of that and still see an issue, let's
                            flesh it out.

                            >
                            > But, certainly please commit it, rename the files to something different
                            > (content_length_header.t?) since it's not longer a HEAD request. And
                            > please add a comment explaining the bug, ideally with xref to a bugzilla
                            > for the bug you are talking about or discussion thread.

                            will do.

                            > also please move that no strict thing, inside the block, where it is
                            > needed. Thanks.
                            >
                            >> +no strict qw(refs);
                            >> +foreach my $method qw(GET HEAD) {

                            d'oh!

                            --Geoff

                            --
                            Report problems: http://perl.apache.org/bugs/
                            Mail list info: http://perl.apache.org/maillist/modperl.html
                            List etiquette: http://perl.apache.org/maillist/email-etiquette.html
                          • Stas Bekman
                            ... See my explanation of what I think is the problem at the end of this email. ... eh? how exactly do you do that? without assbackwards ofcourse. ... Actually
                            Message 13 of 25 , Aug 2, 2004
                            • 0 Attachment
                              Geoffrey Young wrote:
                              >>>yes, clearly there is a bug in there, but not where you think - what is
                              >>>important is that apache needs to do the same thing on GET as HEAD,
                              >>>not that
                              >>>there is no C-L generated for contentless HEAD requests. this patch more
                              >>>accurately represents the real bug which, IIRC, httpd is already aware of
                              >>>(at least I'm recalling discussion about a C-L of 0).
                              >>
                              >>
                              >>Why do you say so? Your patch respresents a related, but a different
                              >>issue. Boris has a problem with HEAD, and that's what the test was testing.
                              >
                              >
                              > ok, maybe I was misunderstanding the discussion and subsequent test. my
                              > points are these:
                              >
                              > - apache has essentially full reign over the C-L header - it decides
                              > whether it is required or not, regardless of whether or not you call
                              > ap_set_content_length
                              >
                              > - more important than whether the C-L header shows up or not for a given
                              > HEAD or GET request is whether apache is consistent with itself. that is,
                              > regardless of what _you_ think it should do, apache _knows_ what it should
                              > do and should do it consistently between GET and HEAD.

                              See my explanation of what I think is the problem at the end of this email.

                              > - of course, printing a C-L header to stdout works around all of this -
                              > apache can only manage things when you use the official API, since it checks
                              > the headers_out table and not the actual content.

                              eh? how exactly do you do that? without assbackwards ofcourse.

                              > so, the test, as it existed before, made me think that the problem you saw
                              > was in that final test, that adding content magically made the C-L header
                              > appear.

                              Actually I the problem I saw was exactly what Boris was talking about:
                              The C-L header wasn't there. The test simply exercises 4 different
                              combinations of sending and not sending C-L header and content.

                              > my point was that as long as apache does whatever it does
                              > consistently between GET and HEAD it's not a bug. but maybe I'm just not
                              > seeing it, so if you guys grok all of that and still see an issue, let's
                              > flesh it out.

                              I think it's there. The test that shows that is:

                              {
                              # if the response handler sends no data, and sets C-L header,
                              # the client doesn't get C-L header
                              my $uri = "$location?set_content_length";
                              my $res = $method->($uri);
                              ok t_cmp $res->code, 200, "$method $uri code";
                              ok t_cmp $res->header('Content-Length'), undef, "$method $uri
                              C-L header";
                              ok t_cmp $res->content, "", "$method $uri content";
                              }

                              As I didn't know what's the right thing that should happen, I didn't
                              make it a failure. But it probabaly shouldn't be undef in the C-L check.

                              But the fact that sending a bogus char and a different C-L header works
                              with HEAD (gets the C-L header through), whereas not sending any output
                              breaks, seems like a bug to me. As mentioned earlier I think the bug is
                              in the headers_out filter, which received EOS (since on HEAD apache
                              scratches the response body) and no body. So it takes the liberty to
                              nuke the C-L header, which I'm not sure is a good thing. When we send
                              some body, headers_out sends the headers before seeing EOS and therefore
                              it can't tell whether the body is coming or not, so it leaves the C-L
                              header alone. Which is at least inconsistent.

                              --
                              __________________________________________________________________
                              Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
                              http://stason.org/ mod_perl Guide ---> http://perl.apache.org
                              mailto:stas@... http://use.perl.org http://apacheweek.com
                              http://modperlbook.org http://apache.org http://ticketmaster.com

                              --
                              Report problems: http://perl.apache.org/bugs/
                              Mail list info: http://perl.apache.org/maillist/modperl.html
                              List etiquette: http://perl.apache.org/maillist/email-etiquette.html
                            • Geoffrey Young
                              ... ok, I think I see what is going on, but it s different than what you see :) first, apache calls the C-L filter, which has no regard for any existing C-L
                              Message 14 of 25 , Aug 2, 2004
                              • 0 Attachment
                                > As mentioned earlier I think the bug is
                                > in the headers_out filter, which received EOS (since on HEAD apache
                                > scratches the response body) and no body. So it takes the liberty to
                                > nuke the C-L header, which I'm not sure is a good thing. When we send
                                > some body, headers_out sends the headers before seeing EOS and therefore
                                > it can't tell whether the body is coming or not, so it leaves the C-L
                                > header alone. Which is at least inconsistent.

                                ok, I think I see what is going on, but it's different than what you see :)

                                first, apache calls the C-L filter, which has no regard for any existing C-L
                                header and calculates its own if it can. so, in your test case the call to
                                $r->set_content_length(25) is overwritten by the real C-L of zero, as
                                calculated by the C-L filter.

                                next the header filter kicks in. inside the header filter is this bit of logic:

                                if (r->header_only
                                && (clheader = apr_table_get(r->headers_out, "Content-Length"))
                                && !strcmp(clheader, "0")) {
                                apr_table_unset(r->headers_out, "Content-Length");
                                }

                                which treats HEAD requests with a C-L of zero as a special case. and, in
                                fact, that is what you have sent it, at least in the test case. this is why
                                GET and HEAD differ in this instance, because you haven't sent any real
                                content and a C-L of 0 is special.

                                so, I don't see the bug. well, I do see it, except that it is intentionally
                                coded to be as such, which turns the bug into a feature.

                                --Geoff

                                --
                                Report problems: http://perl.apache.org/bugs/
                                Mail list info: http://perl.apache.org/maillist/modperl.html
                                List etiquette: http://perl.apache.org/maillist/email-etiquette.html
                              • Glenn Strauss
                                On Mon, Aug 02, 2004 at 11:55:56AM -0700, Stas Bekman wrote: [...] ... I mentioned to Geoff off-list about possibly using a flush bucket, since I don t have a
                                Message 15 of 25 , Aug 3, 2004
                                • 0 Attachment
                                  On Mon, Aug 02, 2004 at 11:55:56AM -0700, Stas Bekman wrote:
                                  [...]
                                  > Actually I the problem I saw was exactly what Boris was talking about:
                                  > The C-L header wasn't there. The test simply exercises 4 different
                                  > combinations of sending and not sending C-L header and content.
                                  >
                                  > >my point was that as long as apache does whatever it does
                                  > >consistently between GET and HEAD it's not a bug. but maybe I'm just not
                                  > >seeing it, so if you guys grok all of that and still see an issue, let's
                                  > >flesh it out.
                                  >
                                  > I think it's there. The test that shows that is:
                                  >
                                  > {
                                  > # if the response handler sends no data, and sets C-L header,
                                  > # the client doesn't get C-L header
                                  > my $uri = "$location?set_content_length";
                                  > my $res = $method->($uri);
                                  > ok t_cmp $res->code, 200, "$method $uri code";
                                  > ok t_cmp $res->header('Content-Length'), undef, "$method $uri
                                  > C-L header";
                                  > ok t_cmp $res->content, "", "$method $uri content";
                                  > }
                                  >
                                  > As I didn't know what's the right thing that should happen, I didn't
                                  > make it a failure. But it probabaly shouldn't be undef in the C-L check.
                                  >
                                  > But the fact that sending a bogus char and a different C-L header works
                                  > with HEAD (gets the C-L header through), whereas not sending any output
                                  > breaks, seems like a bug to me. As mentioned earlier I think the bug is
                                  > in the headers_out filter, which received EOS (since on HEAD apache
                                  > scratches the response body) and no body. So it takes the liberty to
                                  > nuke the C-L header, which I'm not sure is a good thing. When we send
                                  > some body, headers_out sends the headers before seeing EOS and therefore
                                  > it can't tell whether the body is coming or not, so it leaves the C-L
                                  > header alone. Which is at least inconsistent.


                                  I mentioned to Geoff off-list about possibly using a flush bucket,
                                  since I don't have a test setup ready (just replaced my dead
                                  laptop hard drive)

                                  I wrote:
                                  > > I used to make good use of $r->send_http_header() in my MP1 handlers.
                                  > > This method isn't available in MP2 (why?), though I could picture it
                                  > > replaced by sending a flush bucket down the filter chain.
                                  >
                                  On Tue, Aug 03, 2004 at 10:04:38AM -0400, Geoffrey Young wrote:
                                  > apache now sends the headers for you via the header filter. if you were
                                  > allowed to send your own headers you might send them before a filter ran
                                  > that would alter the headers. for instance, mod_include removes the ETag
                                  > header.

                                  Would a flush bucket be a great way to say "send headers now"?
                                  So if you're not going to send the actual content down the filter chain
                                  on a HEAD request, would sending a flush bucket make things happy?

                                  IIF this works, can I make a request the $r->send_http_header() be
                                  implemented in Apache2 to do just that? (create and pass a flush bucket)

                                  Cheers,
                                  Glenn

                                  --
                                  Report problems: http://perl.apache.org/bugs/
                                  Mail list info: http://perl.apache.org/maillist/modperl.html
                                  List etiquette: http://perl.apache.org/maillist/email-etiquette.html
                                • Stas Bekman
                                  ... Thanks Glenn, calling $r- rflush works perfectly fine to cause user-preset C-L header to be delivered to the client on HEAD request. ... I think this is
                                  Message 16 of 25 , Aug 4, 2004
                                  • 0 Attachment
                                    Glenn Strauss wrote:
                                    > On Mon, Aug 02, 2004 at 11:55:56AM -0700, Stas Bekman wrote:
                                    > [...]

                                    > I mentioned to Geoff off-list about possibly using a flush bucket,
                                    > since I don't have a test setup ready (just replaced my dead
                                    > laptop hard drive)

                                    Thanks Glenn, calling $r->rflush works perfectly fine to cause
                                    user-preset C-L header to be delivered to the client on HEAD request.

                                    > I wrote:
                                    >
                                    >>>I used to make good use of $r->send_http_header() in my MP1 handlers.
                                    >>>This method isn't available in MP2 (why?), though I could picture it
                                    >>>replaced by sending a flush bucket down the filter chain.
                                    >>
                                    > On Tue, Aug 03, 2004 at 10:04:38AM -0400, Geoffrey Young wrote:
                                    >
                                    >>apache now sends the headers for you via the header filter. if you were
                                    >>allowed to send your own headers you might send them before a filter ran
                                    >>that would alter the headers. for instance, mod_include removes the ETag
                                    >>header.

                                    I think this is not the reason why ap_send_http_header was removed from
                                    Apache2 API. It's simply not needed because the headers_out filter will
                                    inject the headers as soon as some body is sent. Your example of
                                    mod_include doesn't work if the response handler does:

                                    $r->print("foo bar");
                                    $r->rflush; # or if the previous print was big enough to cause flush
                                    $r->print(q[<!--#include virtual="/foo" -->]);

                                    > Would a flush bucket be a great way to say "send headers now"?
                                    > So if you're not going to send the actual content down the filter chain
                                    > on a HEAD request, would sending a flush bucket make things happy?

                                    It does.

                                    > IIF this works, can I make a request the $r->send_http_header() be
                                    > implemented in Apache2 to do just that? (create and pass a flush bucket)

                                    Absolutely. But it'd be nice if you could first ask the httpd-dev list:
                                    http://httpd.apache.org/lists.html#http-dev
                                    if they are interested in adding the C API to do that.

                                    If httpd-dev doesn't do it, I suppose we can still implement it in mp2.

                                    --
                                    __________________________________________________________________
                                    Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
                                    http://stason.org/ mod_perl Guide ---> http://perl.apache.org
                                    mailto:stas@... http://use.perl.org http://apacheweek.com
                                    http://modperlbook.org http://apache.org http://ticketmaster.com

                                    --
                                    Report problems: http://perl.apache.org/bugs/
                                    Mail list info: http://perl.apache.org/maillist/modperl.html
                                    List etiquette: http://perl.apache.org/maillist/email-etiquette.html
                                  • Stas Bekman
                                    Stas Bekman wrote: [...] ... On a second thought, I think it s better not to add this special method unless Apache does that. Since in 99.99% cases one should
                                    Message 17 of 25 , Aug 4, 2004
                                    • 0 Attachment
                                      Stas Bekman wrote:
                                      [...]
                                      >> Would a flush bucket be a great way to say "send headers now"?
                                      >> So if you're not going to send the actual content down the filter chain
                                      >> on a HEAD request, would sending a flush bucket make things happy?
                                      >
                                      >
                                      > It does.
                                      >
                                      >> IIF this works, can I make a request the $r->send_http_header() be
                                      >> implemented in Apache2 to do just that? (create and pass a flush bucket)
                                      >
                                      >
                                      > Absolutely. But it'd be nice if you could first ask the httpd-dev list:
                                      > http://httpd.apache.org/lists.html#http-dev
                                      > if they are interested in adding the C API to do that.
                                      >
                                      > If httpd-dev doesn't do it, I suppose we can still implement it in mp2.

                                      On a second thought, I think it's better not to add this special method
                                      unless Apache does that. Since in 99.99% cases one should not cause an
                                      early headers sending, and for those special cases which go against the
                                      Apache spec (sending no body for HEAD requests) we will simply document
                                      a workaround. This is because the workaround may stop working in the
                                      future and we will find ourselves in a bad situation.

                                      If Apache adds this method and documents it usage for the purposes
                                      similar to one discussed in this thread, then Apache will need to ensure
                                      that it'll work in the future. Therefore if you think send_http_header()
                                      is needed, please discuss that on the quoted above list, and depending
                                      on the outcome we will either provide the perl interface to it or simply
                                      document a workaround, which may or may not stop working in the future.

                                      --
                                      __________________________________________________________________
                                      Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
                                      http://stason.org/ mod_perl Guide ---> http://perl.apache.org
                                      mailto:stas@... http://use.perl.org http://apacheweek.com
                                      http://modperlbook.org http://apache.org http://ticketmaster.com

                                      --
                                      Report problems: http://perl.apache.org/bugs/
                                      Mail list info: http://perl.apache.org/maillist/modperl.html
                                      List etiquette: http://perl.apache.org/maillist/email-etiquette.html
                                    • Stas Bekman
                                      ... Thanks Geoff for looking at the source code and giving us the full story. I ve now converted it into the comments in the t/apache/content_length_header.t
                                      Message 18 of 25 , Aug 4, 2004
                                      • 0 Attachment
                                        Geoffrey Young wrote:
                                        >
                                        >>As mentioned earlier I think the bug is
                                        >>in the headers_out filter, which received EOS (since on HEAD apache
                                        >>scratches the response body) and no body. So it takes the liberty to
                                        >>nuke the C-L header, which I'm not sure is a good thing. When we send
                                        >>some body, headers_out sends the headers before seeing EOS and therefore
                                        >>it can't tell whether the body is coming or not, so it leaves the C-L
                                        >>header alone. Which is at least inconsistent.
                                        >
                                        >
                                        > ok, I think I see what is going on, but it's different than what you see :)
                                        >
                                        > first, apache calls the C-L filter, which has no regard for any existing C-L
                                        > header and calculates its own if it can. so, in your test case the call to
                                        > $r->set_content_length(25) is overwritten by the real C-L of zero, as
                                        > calculated by the C-L filter.
                                        >
                                        > next the header filter kicks in. inside the header filter is this bit of logic:
                                        >
                                        > if (r->header_only
                                        > && (clheader = apr_table_get(r->headers_out, "Content-Length"))
                                        > && !strcmp(clheader, "0")) {
                                        > apr_table_unset(r->headers_out, "Content-Length");
                                        > }
                                        >
                                        > which treats HEAD requests with a C-L of zero as a special case. and, in
                                        > fact, that is what you have sent it, at least in the test case. this is why
                                        > GET and HEAD differ in this instance, because you haven't sent any real
                                        > content and a C-L of 0 is special.
                                        >
                                        > so, I don't see the bug. well, I do see it, except that it is intentionally
                                        > coded to be as such, which turns the bug into a feature.

                                        Thanks Geoff for looking at the source code and giving us the full
                                        story. I've now converted it into the comments in the
                                        t/apache/content_length_header.t test and I also used Glenn's suggestion
                                        to use $r->rflush, which gives us:

                                        # 2. even though the spec says that content handlers should send an
                                        # identical response for GET and HEAD requests, some folks try to
                                        # avoid the overhead of generating the response body, which Apache is
                                        # going to discard anyway for HEAD requests. The following discussion
                                        # assumes that we deal with a HEAD request.
                                        #
                                        # When Apache sees EOS and no headers and no response body were sent,
                                        # ap_content_length_filter (httpd-2.0/server/protocol.c) sets C-L to
                                        # 0. Later on ap_http_header_filter
                                        # (httpd-2.0/modules/http/http_protocol.c) removes the C-L header for
                                        # the HEAD requests
                                        #
                                        # the workaround is to force the sending of the response headers,
                                        # before EOS was sent. The simplest solution is to use rflush():
                                        #
                                        # if ($r->header_only) { # HEAD
                                        # $body_len = calculate_body_len();
                                        # $r->set_content_length($body_len);
                                        # $r->rflush;
                                        # }
                                        # else { # GET
                                        # # generate and send the body
                                        # }
                                        #
                                        # now if the handler sets the C-L header it'll be delivered to the
                                        # client unmodified.


                                        --
                                        __________________________________________________________________
                                        Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
                                        http://stason.org/ mod_perl Guide ---> http://perl.apache.org
                                        mailto:stas@... http://use.perl.org http://apacheweek.com
                                        http://modperlbook.org http://apache.org http://ticketmaster.com

                                        --
                                        Report problems: http://perl.apache.org/bugs/
                                        Mail list info: http://perl.apache.org/maillist/modperl.html
                                        List etiquette: http://perl.apache.org/maillist/email-etiquette.html
                                      • Geoffrey Young
                                        ... I fully agree. ... right - apache expects you to not do anything special for HEAD requests anymore, so if you mess with that contract you are messing with
                                        Message 19 of 25 , Aug 4, 2004
                                        • 0 Attachment
                                          >> If httpd-dev doesn't do it, I suppose we can still implement it in mp2.
                                          >
                                          >
                                          > On a second thought, I think it's better not to add this special method
                                          > unless Apache does that.

                                          I fully agree.

                                          > Since in 99.99% cases one should not cause an
                                          > early headers sending, and for those special cases which go against the
                                          > Apache spec (sending no body for HEAD requests) we will simply document
                                          > a workaround. This is because the workaround may stop working in the
                                          > future and we will find ourselves in a bad situation.

                                          right - apache expects you to not do anything special for HEAD requests
                                          anymore, so if you mess with that contract you are messing with an
                                          unsupported feature, and mod_perl has no business supporting unsupported
                                          apache features through a new, mod_perl-specific API.

                                          --Geoff

                                          --
                                          Report problems: http://perl.apache.org/bugs/
                                          Mail list info: http://perl.apache.org/maillist/modperl.html
                                          List etiquette: http://perl.apache.org/maillist/email-etiquette.html
                                        • Stas Bekman
                                          This thread has been documented at: http://perl.apache.org/docs/2.0/user/handlers/http.html#C_Content_Length__Response_Header --
                                          Message 20 of 25 , Aug 8, 2004
                                          • 0 Attachment
                                            This thread has been documented at:
                                            http://perl.apache.org/docs/2.0/user/handlers/http.html#C_Content_Length__Response_Header

                                            --
                                            __________________________________________________________________
                                            Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
                                            http://stason.org/ mod_perl Guide ---> http://perl.apache.org
                                            mailto:stas@... http://use.perl.org http://apacheweek.com
                                            http://modperlbook.org http://apache.org http://ticketmaster.com

                                            --
                                            Report problems: http://perl.apache.org/bugs/
                                            Mail list info: http://perl.apache.org/maillist/modperl.html
                                            List etiquette: http://perl.apache.org/maillist/email-etiquette.html
                                          Your message has been successfully submitted and would be delivered to recipients shortly.