Loading ...
Sorry, an error occurred while loading the content.
 

Of HTTP/1.1 persistent connections and TCP Keepalive timers

Expand Messages
  • Jeff.Hodges@KingsMountain.com
    I m curious about how HTTP/1.1 [RFC2616] persistent connections typically work with respect to the typical browsers out in the wild today (Netscape & Microsoft
    Message 1 of 25 , Nov 1, 2000
      I'm curious about how HTTP/1.1 [RFC2616] persistent connections typically work
      with respect to the typical browsers out in the wild today (Netscape &
      Microsoft being the two I'm particularly interested in). If I cause a browser
      to send a GET request for a given URL (using HTTP/1.1) to a server, and the
      server doesn't encounter any errors in processing it and responding, and then
      I (say) don't touch the browser for hours, what *typically* happens to the
      established HTTP/1.1 (-over-TCP) connection?

      I note that RFC2616 says (in part)..


      :
      8 Connections

      8.1 Persistent Connections
      :
      HTTP implementations SHOULD implement persistent connections.
      :
      A significant difference between HTTP/1.1 and earlier versions of
      HTTP is that persistent connections are the default behavior of any
      HTTP connection. That is, unless otherwise indicated, the client
      SHOULD assume that the server will maintain a persistent connection,
      even after error responses from the server.
      :


      As it is written, this effectively puts the responsibility for closing the
      HTTP/1.1-cum-TCP connection on the client.

      In nosing around on this subject, I note that in [W.R.Stevens, TCP/IP
      Illustrated Vol 1, http://www.dqc.org/~chris/tcpip_ill/%5d, in chapter 23
      Stevens says that..

      1. "Keepalives are not part of the TCP specification. ... Nevertheless, many
      implementations provide the keep-alive timer."

      2. "If there is no activity on a given connection for 2 hours, the server
      sends a probe segment to the client. ... A perpetual question by people
      discovering the keepalive option is whether the 2-hour idle time value can be
      changed. They normally want it much lower, on the order of minutes. As we show
      in Appendix E, the value can usually be changed, but in all the systems
      described in this appendix, the keepalive interval is a system-wide value, so
      changing it affects all users of the option. "

      ..and in appendix E he shows kernel configuration parameters for several
      Unix-based TCP implementations, most all of which have a default 2-hour
      timeout *before* a keepalive packet will be sent.

      I also note that Microsoft shows a default value of 2 hour idletime for the
      keepalive timer in this doc:

      http://www.microsoft.com/technet/winnt/reskit/sur_tcp2.asp


      Some questions (again, in the case of HTTP/1.1 persistent connections):

      Q1. Do the popular browsers typically take the platform's OS's TCP defaults
      for
      the keepalive (if such capability is provided by the TCP/IP stack, and if it
      is actually used by the browser), or do they typically set this value to
      something in particular?


      Q2. What typical assumptions are made on the browsers' parts about an
      established connection to a web site in the absence of user actions? If a
      browser opened a HTTP/1.1 connection and the server is behaving as-specified
      by RFC2616, then it is up to the browser to close the connection. What do
      browsers typically do? I looked through the documented configuration
      parameters for Netscape Communicator..

      http://docs.iplanet.com/docs/manuals/communicator/newprefs/newprefn.html

      ..and could not find a timeout setting that's applicable for this particular
      case. How long will browsers, that are speaking HTTP/1.1, let this connection
      sit in the ESTABLISHED state?


      Q3. Are the popular browsers typically using HTTP/1.1, or HTTP/1.0? I didn't
      notice any config parameters that might have something to do with setting the
      default.

      thanks,

      JeffH
    • Joris Dobbelsteen
      ... HTTP/1.1 does not rely on the TCP timeout/keepalive. A HTTP/1.1 connected is alive only if the lower-layer connection is in the CONNECTED state. ... The
      Message 2 of 25 , Nov 2, 2000
        > -----Original Message-----
        > From: hodges@...
        > [mailto:hodges@...]On Behalf Of
        > Jeff.Hodges@...
        > Sent: Thursday, 02 November 2000 8:44
        > To: http-wg@...
        > Cc: Jeff.Hodges@...
        > Subject: Of HTTP/1.1 persistent connections and TCP Keepalive timers
        >
        >
        > I'm curious about how HTTP/1.1 [RFC2616] persistent
        > connections typically work
        > with respect to the typical browsers out in the wild today
        > (Netscape &
        > Microsoft being the two I'm particularly interested in). If I
        > cause a browser
        > to send a GET request for a given URL (using HTTP/1.1) to a
        > server, and the
        > server doesn't encounter any errors in processing it and
        > responding, and then
        > I (say) don't touch the browser for hours, what *typically*
        > happens to the
        > established HTTP/1.1 (-over-TCP) connection?
        >

        <<<SNIP>>>

        >
        > Some questions (again, in the case of HTTP/1.1 persistent
        > connections):
        >
        > Q1. Do the popular browsers typically take the platform's
        > OS's TCP defaults
        > for
        > the keepalive (if such capability is provided by the TCP/IP
        > stack, and if it
        > is actually used by the browser), or do they typically set
        > this value to
        > something in particular?

        HTTP/1.1 does not rely on the TCP timeout/keepalive. A HTTP/1.1 connected is alive only if the lower-layer connection is in the CONNECTED state.
        >
        >
        > Q2. What typical assumptions are made on the browsers' parts about an
        > established connection to a web site in the absence of user
        > actions? If a
        > browser opened a HTTP/1.1 connection and the server is
        > behaving as-specified
        > by RFC2616, then it is up to the browser to close the
        > connection. What do
        > browsers typically do? I looked through the documented configuration
        > parameters for Netscape Communicator..
        >
        >
        > http://docs.iplanet.com/docs/manuals/communicator/newprefs/new
        > prefn.html
        >
        > ..and could not find a timeout setting that's applicable for this particular
        > case. How long will browsers, that are speaking HTTP/1.1, let this connection
        > sit in the ESTABLISHED state?

        The browser can leave the connection open for an infinite ammount of time (if not restricted by lower-level protocols, like TCP).

        However, idle time-outs when the connection is CONNECTED are, in the real world, handled by the server instead of the client. This is because the server is not interrested in HTTP connections that are idle for an infinite ammount of time and just consuming bandwidth (the lower-level connection must be maintained)...

        In the real world, the server does close the connection (when it's idle) without an explicit message that the connection is going to be closed. The server also cannot do this on HTTP level.



        > Q3. Are the popular browsers typically using HTTP/1.1, or HTTP/1.0? I didn't
        > notice any config parameters that might have something to do with setting the
        > default.

        HTTP/1.1, this version indicates the browser is capable of handling features that are not supported in HTTP/1.0 (such as HTTP keepalive).

        MSIE (for Windows) supports HTTP/1.1, but also HTTP/1.0 (default for proxy connections). Here the settings can be changed...

        The version is also always send with a HTTP request or response (to indicate the browser/server capacties).

        >
        > thanks,
        >
        > JeffH
        >
        >


        Hope to inform you. Maybe comments from the WG?


        - Joris
      • James Lacey
        ... The web server will close the connection due to inactivity. You have to realize that web servers are trying to service literally thousands of clients with
        Message 3 of 25 , Nov 2, 2000
          Jeff.Hodges@... wrote:

          > I'm curious about how HTTP/1.1 [RFC2616] persistent connections typically work
          > with respect to the typical browsers out in the wild today (Netscape &
          > Microsoft being the two I'm particularly interested in). If I cause a browser
          > to send a GET request for a given URL (using HTTP/1.1) to a server, and the
          > server doesn't encounter any errors in processing it and responding, and then
          > I (say) don't touch the browser for hours, what *typically* happens to the
          > established HTTP/1.1 (-over-TCP) connection?

          The web server will close the connection due to inactivity.

          You have to realize that web servers are trying to service literally
          thousands of clients with only a relatively few TCP/IP connections.
          If you don't use it you lose it.

          Also, most web servers actively look for reasons to close a connection
          to a client. For example, if the web server generates any dynamic content
          for the client, then it will usually close the connection after the response it
          sent back. Regardless of whether or not the client supplied a Connection:
          Keep-Alive header or not. The reasoning behind this is that if the sever
          had to generate dynamic content on your behalf, then you've had your share
          and its time to give some other poor slob a turn.

          >
          >
          > I note that RFC2616 says (in part)..
          >
          > :
          > 8 Connections
          >
          > 8.1 Persistent Connections
          > :
          > HTTP implementations SHOULD implement persistent connections.
          > :
          > A significant difference between HTTP/1.1 and earlier versions of
          > HTTP is that persistent connections are the default behavior of any
          > HTTP connection. That is, unless otherwise indicated, the client
          > SHOULD assume that the server will maintain a persistent connection,
          > even after error responses from the server.
          > :
          >
          > As it is written, this effectively puts the responsibility for closing the
          > HTTP/1.1-cum-TCP connection on the client.

          Nope. See my comments above.

          Also, you have to realize that just cuz the client sends a
          Connection: Keep-Alive header it is in no way a guarantee
          that the server will not close the connection after the
          response is sent back.

          There is a HUGE difference between the way the HTTP spec
          is written and the way that web server's are actually designed
          to work on the internet.

          I'm not saying that the web server designers violated the
          HTTP protocol. Rather they have simply done what they
          have to do in order to protect their web server from
          attacks, deadlocks, and starving clients.

          Read further in the spec and you will see that the HTTP
          spec says that unsafe methods should not be pipelined.

          An unsafe method is a method that in some sense changes
          the state of the server and will not necessarily generate the
          same response every time it is executed.

          On the internet unsafe methods are typically used to
          represent a clients actions on the internet (i.e. I've just
          sent a request to buy product X with my credit card
          number aaaa-bbbb-cccc-dddd). Before submitting
          another unsafe method I should be allowed to get
          feedback about the current unsafe method and determine
          if I wish to proceed with the next unsafe method or not.
          So, when receiving an unsafe method (POST) most
          web servers will close the connection after the response
          is generated. Even if more unsafe methods have been sent
          into the pipe. They are simply discarded.

          >
          >
          > In nosing around on this subject, I note that in [W.R.Stevens, TCP/IP
          > Illustrated Vol 1, http://www.dqc.org/~chris/tcpip_ill/%5d, in chapter 23
          > Stevens says that..
          >
          > 1. "Keepalives are not part of the TCP specification. ... Nevertheless, many
          > implementations provide the keep-alive timer."
          >
          > 2. "If there is no activity on a given connection for 2 hours, the server
          > sends a probe segment to the client. ... A perpetual question by people
          > discovering the keepalive option is whether the 2-hour idle time value can be
          > changed. They normally want it much lower, on the order of minutes. As we show
          > in Appendix E, the value can usually be changed, but in all the systems
          > described in this appendix, the keepalive interval is a system-wide value, so
          > changing it affects all users of the option. "
          >
          > ..and in appendix E he shows kernel configuration parameters for several
          > Unix-based TCP implementations, most all of which have a default 2-hour
          > timeout *before* a keepalive packet will be sent.
          >
          > I also note that Microsoft shows a default value of 2 hour idletime for the
          > keepalive timer in this doc:
          >
          > http://www.microsoft.com/technet/winnt/reskit/sur_tcp2.asp

          Here I think you are confusing TCP/IP keep alive, with HTTP's
          Connection header and its Keep-Alive value.

          Trust me. These are two very different unrelatred things.

          >
          >
          > Some questions (again, in the case of HTTP/1.1 persistent connections):
          >
          > Q1. Do the popular browsers typically take the platform's OS's TCP defaults
          > for
          > the keepalive (if such capability is provided by the TCP/IP stack, and if it
          > is actually used by the browser), or do they typically set this value to
          > something in particular?

          Browsers do not use this at all I'm quite sure.

          >
          >
          > Q2. What typical assumptions are made on the browsers' parts about an
          > established connection to a web site in the absence of user actions? If a
          > browser opened a HTTP/1.1 connection and the server is behaving as-specified
          > by RFC2616, then it is up to the browser to close the connection. What do
          > browsers typically do? I looked through the documented configuration
          > parameters for Netscape Communicator..
          >
          > http://docs.iplanet.com/docs/manuals/communicator/newprefs/newprefn.html
          >
          > ..and could not find a timeout setting that's applicable for this particular
          > case. How long will browsers, that are speaking HTTP/1.1, let this connection
          > sit in the ESTABLISHED state?

          Until the server closes it IMHO. The server will close the connection due
          to inactivity; 30 seconds by default for iPlanet web server.

          >
          >
          > Q3. Are the popular browsers typically using HTTP/1.1, or HTTP/1.0? I didn't
          > notice any config parameters that might have something to do with setting the
          > default.
          >
          > thanks,
          >
          > JeffH
        • John Stracke
          ... Sane applications typically do not use TCP keepalives. TCP keepalives are a way of consuming bandwidth in order to make your application less reliable.
          Message 4 of 25 , Nov 2, 2000
            Jeff.Hodges@... wrote:

            > Q1. Do the popular browsers typically take the platform's OS's TCP defaults
            > for
            > the keepalive (if such capability is provided by the TCP/IP stack, and if it
            > is actually used by the browser), or do they typically set this value to
            > something in particular?

            Sane applications typically do not use TCP keepalives. TCP keepalives are a way
            of consuming bandwidth in order to make your application less reliable.

            See, keepalives are misnamed; idle TCP connections will be kept alive
            indefinitely by default. Using keepalives means you send an empty ACK every so
            often to see if the remote host is still up. The problem is that sending that
            keepalive will also mean you drop the connection if some intermediate router is
            down. But, if your link is idle, you don't care whether an intermediate router
            is down; you only care when you want to send data. So keepalives just add
            another way for your application to fail.

            --
            /==============================================================\
            |John Stracke | http://www.ecal.com |My opinions are my own.|
            |Chief Scientist |=============================================|
            |eCal Corp. |Two words that do not go together: "Memorial |
            |francis@...|Cookbook". |
            \==============================================================/
          • James Lacey
            ... The server-side will do whatever it wants no matter what you do on the client side. Believe me. I have been down this road. If you are assuming that just
            Message 5 of 25 , Nov 2, 2000
              Jeff.Hodges@... wrote:

              > Thanks for all the info about how web servers are typically implemented. My
              > apologies tho, I didn't really make my questions clear. What I'm really
              > interested in is how *browsers* are typically implemented, because, say, I'm
              > going to write my own HTTP/1.1-speaking gizmo for a very application-specific
              > purpose, and I have reasons to want it (the server side) to treat
              > client-initiated connections as persistent.
              >

              The server-side will do whatever it wants no matter what you do on
              the client side.

              Believe me. I have been down this road.

              If you are assuming that just because the HTTP protocol talks about
              persistent connections that you will be able to connect to a typical
              web server and have the connection stay up as long as you do things
              on the client side 'just right' you are mistaken.

              As I said before web servers actively look for reasons to close connections.
              There is _nothing_ that you can do on the client side to prevent this.
              Send Connection: Keep-Alive all you want. It will not keep the web
              server from closing the connection whenever it darn well feels like it.

              >
              > So this causes me to be curious about how BROWSERS will typically behave in
              > this context.
              >
              > Assuming one writes one's own HTTP/1.1-speaking gizmo (according to RFC2616),
              > then I have these questions...
              >
              > Q1. Do the popular BROWSERS typically take the platform's OS's TCP defaults
              > for the keepalive (IF such capability is provided by the TCP/IP stack, and IF
              > it is actually used by the browser), or do they typically set this value to
              > something in particular?

              Browsers do not use TCP/IP keep alive!

              Almost no one uses TCP/IP keep alive mechanisms.

              As John Stracke pointed out the use of TCP/IP keep
              alive actually makes your application less reliable
              and more likely to close an idle connection for no
              reason!

              >
              >
              > Q2. What typical assumptions are made on the BROWSERS' parts about an
              > established connection to a HTTP/1.1-speaking server in the absence of user
              > actions? If a browser opened a HTTP/1.1 connection and such a server is
              > behaving as-specified by RFC2616, then it is up to the browser to close the
              > connection. What do browsers typically do?
              >

              Typically, browsers do not close the connection.

              For example, suppose the server sends back an HTTP response to the client
              that does not have a Content-Length: header field and that it is not
              chunked.

              Then the only way the client knows that it has read the
              entire response off of the pipe is when the server closes the connection.
              When the server closes the connection the client will receive a
              zero-byte read which is socket layer's indication that the pipe
              is broken.

              >
              > I looked through the documented configuration parameters for Netscape
              > Communicator..
              >
              > http://docs.iplanet.com/docs/manuals/communicator/newprefs/newprefn.html
              >
              > ..and could not find a timeout setting that's applicable for this particular
              > case. How long will BROWSERS, that are speaking HTTP/1.1, let this connection
              > sit in the ESTABLISHED state?

              My guess is that until the server closes the connection.

              Which by the way, will be sooner than later.

              >
              >
              > Q3. Are the popular BROWSERS typically speaking HTTP/1.1, or HTTP/1.0? I
              > didn't
              > notice any config parameters that might have something to do with setting the
              > default.
              >

              I've done some snoops on browsers that are GETting HTML pages with
              lots of embedded links that are in the same realm as the original HTML
              page.

              Despite this obvious opportunity to take advantage of persistent connections
              the browser opens a connection for each subsequent GET.

              This is true even though the browser is advertising that it supports HTTP/1.1
              Most browsers seem to be supporting receiveing of chunked dynamic content
              rather than persistent connections ...

              >
              > thanks again,
              >
              > JeffH
            • Dave Kristol
              ... Actually, I think most responses recognized that you were interested in browsers. The problem is that the browser can only express its preference. But
              Message 6 of 25 , Nov 2, 2000
                Jeff.Hodges@... wrote:
                > Thanks for all the info about how web servers are typically implemented. My
                > apologies tho, I didn't really make my questions clear. What I'm really
                > interested in is how *browsers* are typically implemented, because, say, I'm
                > going to write my own HTTP/1.1-speaking gizmo for a very application-specific
                > purpose, and I have reasons to want it (the server side) to treat
                > client-initiated connections as persistent.

                Actually, I think most responses recognized that you were interested in
                browsers. The problem is that the browser can only express its
                preference. But the server is in command, and it can close a
                connection whenever it feels it must. That's why there were all the
                comments on server behavior. (In truth, the browser may *also* close
                the connection at will, but that's a less common case.)

                >
                > So this causes me to be curious about how BROWSERS will typically behave in
                > this context.
                >
                > Assuming one writes one's own HTTP/1.1-speaking gizmo (according to RFC2616),
                > then I have these questions...
                >
                >

                I am not a browser implementer, so my answers are speculative, but....

                > Q1. Do the popular BROWSERS typically take the platform's OS's TCP defaults
                > for the keepalive (IF such capability is provided by the TCP/IP stack, and IF
                > it is actually used by the browser), or do they typically set this value to
                > something in particular?

                They take the OS's defaults.

                >
                >
                > Q2. What typical assumptions are made on the BROWSERS' parts about an
                > established connection to a HTTP/1.1-speaking server in the absence of user
                > actions? If a browser opened a HTTP/1.1 connection and such a server is
                > behaving as-specified by RFC2616, then it is up to the browser to close the
                > connection. What do browsers typically do?

                Browsers probably open connections with the hope of keeping them open as
                long as possible. But as I said earlier, that doesn't mean they will stay
                open. I believe the only way the client discovers the connection has been
                closed is by getting an error response when it tries to write to a (now-closed)
                connection.

                [Rest deleted -- I believe the questions were answered earlier.]

                Dave Kristol
              • Marc Slemko
                ... It is fair to say that the client has no idea of what sort of resource limits the server may have, so it should leave the connection open unless it has
                Message 7 of 25 , Nov 2, 2000
                  On Thu, 2 Nov 2000, James Lacey wrote:

                  > Jeff.Hodges@... wrote:
                  >
                  > > I'm curious about how HTTP/1.1 [RFC2616] persistent connections typically work
                  > > with respect to the typical browsers out in the wild today (Netscape &
                  > > Microsoft being the two I'm particularly interested in). If I cause a browser
                  > > to send a GET request for a given URL (using HTTP/1.1) to a server, and the
                  > > server doesn't encounter any errors in processing it and responding, and then
                  > > I (say) don't touch the browser for hours, what *typically* happens to the
                  > > established HTTP/1.1 (-over-TCP) connection?
                  >
                  > The web server will close the connection due to inactivity.
                  >
                  > You have to realize that web servers are trying to service literally
                  > thousands of clients with only a relatively few TCP/IP connections.
                  > If you don't use it you lose it.

                  It is fair to say that the client has no idea of what sort of resource
                  limits the server may have, so it should leave the connection open
                  unless it has some particular reason (eg. wanting to limit the total
                  number of connections it has open to all servers) to close it. In
                  reality, common browsers probably put some fixed time limit on how long
                  they keep a connection open; if anyone wants to know exactly what
                  whatever browsers they are concerned with do, try it.

                  On the same token, the server has no great desire to arbitrarily close
                  connections, so it will leave it open unless it has some "reason" to
                  close it. Currently, many servers use a very simple metric for this,
                  which is a fixed timeout of x seconds after the last response was
                  sent. This is arguably a very poor metric to use, there has been some
                  small amount of research on adaptive keepalive timeouts based on load.
                  The server, however, has a lot more information on which to base a decision
                  to close a connection.

                  It isn't that this sort of behaviour doesn't follow the spec, but simply
                  that there is no need to embed this sort of runtime information in a
                  protocol spec. If you read the spec carefully, you will note it goes to
                  great lengths to ensure that the server may close the connection whenever
                  it is idle, which requires things like a half-duplex close, etc.

                  The bottom line, however, is that if you open a connection and make
                  one request every few hours, I really don't see why you should care
                  if it is persistent or not, since persistent connections are just an
                  optimization.

                  > Also, most web servers actively look for reasons to close a connection
                  > to a client. For example, if the web server generates any dynamic content
                  > for the client, then it will usually close the connection after the response it
                  > sent back. Regardless of whether or not the client supplied a Connection:
                  > Keep-Alive header or not. The reasoning behind this is that if the sever
                  > had to generate dynamic content on your behalf, then you've had your share
                  > and its time to give some other poor slob a turn.

                  Huh? That would be a webserver with some very... odd ideas.

                  If you are talking about "Connection: Keep-Alive" then you appear to be
                  talking about HTTP/1.0, in which there is no chunked encoding so
                  unless the server puts a content-length on its dynamic content
                  (which is perfectly possible for it to do, but many don't for
                  reasons that are also perfectly legitimate) then there is no way
                  to use a persistent connection since in that case the only
                  end-of-reponse marker you have is the close of connection.

                  >
                  > >
                  > >
                  > > I note that RFC2616 says (in part)..
                  > >
                  > > :
                  > > 8 Connections
                  > >
                  > > 8.1 Persistent Connections
                  > > :
                  > > HTTP implementations SHOULD implement persistent connections.
                  > > :
                  > > A significant difference between HTTP/1.1 and earlier versions of
                  > > HTTP is that persistent connections are the default behavior of any
                  > > HTTP connection. That is, unless otherwise indicated, the client
                  > > SHOULD assume that the server will maintain a persistent connection,
                  > > even after error responses from the server.
                  > > :
                  > >
                  > > As it is written, this effectively puts the responsibility for closing the
                  > > HTTP/1.1-cum-TCP connection on the client.
                  >
                  > Nope. See my comments above.
                  >
                  > Also, you have to realize that just cuz the client sends a
                  > Connection: Keep-Alive header it is in no way a guarantee
                  > that the server will not close the connection after the
                  > response is sent back.
                  >
                  > There is a HUGE difference between the way the HTTP spec
                  > is written and the way that web server's are actually designed
                  > to work on the internet.

                  The HTTP spec also doesn't describe what sort of config files you
                  should use for your server, since that isn't part of the protocol either.

                  >
                  > I'm not saying that the web server designers violated the
                  > HTTP protocol. Rather they have simply done what they
                  > have to do in order to protect their web server from
                  > attacks, deadlocks, and starving clients.
                  >
                  > Read further in the spec and you will see that the HTTP
                  > spec says that unsafe methods should not be pipelined.
                  >
                  > An unsafe method is a method that in some sense changes
                  > the state of the server and will not necessarily generate the
                  > same response every time it is executed.
                  >
                  > On the internet unsafe methods are typically used to
                  > represent a clients actions on the internet (i.e. I've just
                  > sent a request to buy product X with my credit card
                  > number aaaa-bbbb-cccc-dddd). Before submitting
                  > another unsafe method I should be allowed to get
                  > feedback about the current unsafe method and determine
                  > if I wish to proceed with the next unsafe method or not.
                  > So, when receiving an unsafe method (POST) most
                  > web servers will close the connection after the response
                  > is generated. Even if more unsafe methods have been sent
                  > into the pipe. They are simply discarded.

                  Umh... again, your reasoning here is a little confused. Closing the
                  connection after sending the response to a POST by no means ensures that
                  this problem is avoided and, at that state of the game, is pointless.

                  It is legitimate to send a non-idempotent request with several idempotent
                  requests pipelined after it.

                  I think you are again getting confused by the requirement for a
                  content-length or chunking or the lack of a response body in order to
                  do persistent connections.
                • James Lacey
                  ... As a concrete example the iPlanet Enterprise v4.1 server always closes the connection after responding to a POST request or anytime that dynamic content is
                  Message 8 of 25 , Nov 2, 2000
                    Marc Slemko wrote:

                    > On Thu, 2 Nov 2000, James Lacey wrote:
                    >
                    > > Jeff.Hodges@... wrote:
                    > >
                    > > > I'm curious about how HTTP/1.1 [RFC2616] persistent connections typically work
                    > > > with respect to the typical browsers out in the wild today (Netscape &
                    > > > Microsoft being the two I'm particularly interested in). If I cause a browser
                    > > > to send a GET request for a given URL (using HTTP/1.1) to a server, and the
                    > > > server doesn't encounter any errors in processing it and responding, and then
                    > > > I (say) don't touch the browser for hours, what *typically* happens to the
                    > > > established HTTP/1.1 (-over-TCP) connection?
                    > >
                    > > The web server will close the connection due to inactivity.
                    > >
                    > > You have to realize that web servers are trying to service literally
                    > > thousands of clients with only a relatively few TCP/IP connections.
                    > > If you don't use it you lose it.
                    >
                    > It is fair to say that the client has no idea of what sort of resource
                    > limits the server may have, so it should leave the connection open
                    > unless it has some particular reason (eg. wanting to limit the total
                    > number of connections it has open to all servers) to close it. In
                    > reality, common browsers probably put some fixed time limit on how long
                    > they keep a connection open; if anyone wants to know exactly what
                    > whatever browsers they are concerned with do, try it.
                    >
                    > On the same token, the server has no great desire to arbitrarily close
                    > connections, so it will leave it open unless it has some "reason" to
                    > close it. Currently, many servers use a very simple metric for this,
                    > which is a fixed timeout of x seconds after the last response was
                    > sent. This is arguably a very poor metric to use, there has been some
                    > small amount of research on adaptive keepalive timeouts based on load.
                    > The server, however, has a lot more information on which to base a decision
                    > to close a connection.
                    >
                    > It isn't that this sort of behaviour doesn't follow the spec, but simply
                    > that there is no need to embed this sort of runtime information in a
                    > protocol spec. If you read the spec carefully, you will note it goes to
                    > great lengths to ensure that the server may close the connection whenever
                    > it is idle, which requires things like a half-duplex close, etc.
                    >
                    > The bottom line, however, is that if you open a connection and make
                    > one request every few hours, I really don't see why you should care
                    > if it is persistent or not, since persistent connections are just an
                    > optimization.
                    >
                    > > Also, most web servers actively look for reasons to close a connection
                    > > to a client. For example, if the web server generates any dynamic content
                    > > for the client, then it will usually close the connection after the response it
                    > > sent back. Regardless of whether or not the client supplied a Connection:
                    > > Keep-Alive header or not. The reasoning behind this is that if the sever
                    > > had to generate dynamic content on your behalf, then you've had your share
                    > > and its time to give some other poor slob a turn.
                    >
                    > Huh? That would be a webserver with some very... odd ideas.
                    >
                    > If you are talking about "Connection: Keep-Alive" then you appear to be
                    > talking about HTTP/1.0, in which there is no chunked encoding so
                    > unless the server puts a content-length on its dynamic content
                    > (which is perfectly possible for it to do, but many don't for
                    > reasons that are also perfectly legitimate) then there is no way
                    > to use a persistent connection since in that case the only
                    > end-of-reponse marker you have is the close of connection.

                    As a concrete example the iPlanet Enterprise v4.1 server always
                    closes the connection after responding to a POST request or
                    anytime that dynamic content is generated (possibly because
                    the response does not have a Content-Length: header field
                    and the response is not chunked).

                    I have verified this and it is clearly documented in their
                    literature.


                    >
                    >
                    > >
                    > > >
                    > > >
                    > > > I note that RFC2616 says (in part)..
                    > > >
                    > > > :
                    > > > 8 Connections
                    > > >
                    > > > 8.1 Persistent Connections
                    > > > :
                    > > > HTTP implementations SHOULD implement persistent connections.
                    > > > :
                    > > > A significant difference between HTTP/1.1 and earlier versions of
                    > > > HTTP is that persistent connections are the default behavior of any
                    > > > HTTP connection. That is, unless otherwise indicated, the client
                    > > > SHOULD assume that the server will maintain a persistent connection,
                    > > > even after error responses from the server.
                    > > > :
                    > > >
                    > > > As it is written, this effectively puts the responsibility for closing the
                    > > > HTTP/1.1-cum-TCP connection on the client.
                    > >
                    > > Nope. See my comments above.
                    > >
                    > > Also, you have to realize that just cuz the client sends a
                    > > Connection: Keep-Alive header it is in no way a guarantee
                    > > that the server will not close the connection after the
                    > > response is sent back.
                    > >
                    > > There is a HUGE difference between the way the HTTP spec
                    > > is written and the way that web server's are actually designed
                    > > to work on the internet.
                    >
                    > The HTTP spec also doesn't describe what sort of config files you
                    > should use for your server, since that isn't part of the protocol either.
                    >
                    > >
                    > > I'm not saying that the web server designers violated the
                    > > HTTP protocol. Rather they have simply done what they
                    > > have to do in order to protect their web server from
                    > > attacks, deadlocks, and starving clients.
                    > >
                    > > Read further in the spec and you will see that the HTTP
                    > > spec says that unsafe methods should not be pipelined.
                    > >
                    > > An unsafe method is a method that in some sense changes
                    > > the state of the server and will not necessarily generate the
                    > > same response every time it is executed.
                    > >
                    > > On the internet unsafe methods are typically used to
                    > > represent a clients actions on the internet (i.e. I've just
                    > > sent a request to buy product X with my credit card
                    > > number aaaa-bbbb-cccc-dddd). Before submitting
                    > > another unsafe method I should be allowed to get
                    > > feedback about the current unsafe method and determine
                    > > if I wish to proceed with the next unsafe method or not.
                    > > So, when receiving an unsafe method (POST) most
                    > > web servers will close the connection after the response
                    > > is generated. Even if more unsafe methods have been sent
                    > > into the pipe. They are simply discarded.
                    >
                    > Umh... again, your reasoning here is a little confused. Closing the
                    > connection after sending the response to a POST by no means ensures that
                    > this problem is avoided and, at that state of the game, is pointless.
                    >
                    > It is legitimate to send a non-idempotent request with several idempotent
                    > requests pipelined after it.
                    >
                    > I think you are again getting confused by the requirement for a
                    > content-length or chunking or the lack of a response body in order to
                    > do persistent connections.

                    Nope. I'm clear on that. All I was trying to say was that some
                    web servers always close the connection after they have processed
                    what they believe to be a non-idempotent request; iPlanet is
                    a case in point.
                  • Carl Kugler/Boulder/IBM
                    ... This is not good. If this is the server s normal behavior, the client has no way to distinguish a dropped connection from end of file. So the client can
                    Message 9 of 25 , Nov 2, 2000
                      ...
                      >
                      >For example, suppose the server sends back an HTTP response to the client
                      >that does not have a Content-Length: header field and that it is not
                      >chunked.
                      >
                      >Then the only way the client knows that it has read the
                      >entire response off of the pipe is when the server closes the connection.
                      >When the server closes the connection the client will receive a
                      >zero-byte read which is socket layer's indication that the pipe
                      >is broken.
                      >
                      This is not good. If this is the server's normal behavior, the client has
                      no way to distinguish a dropped connection from end of file. So the client
                      can never be sure it received an entire message.

                      If this is the server's behavior for error conditions, this is still not
                      good unless the server either waits for the entire request (could be a
                      humongous POST and/or a very slow connection), or only closes one half of
                      the connection (which, BTW, is impossible in Java, except maybe in the
                      latest releases). Otherwise, the client might get a RST while transmitting
                      the request, and will then never see the error response.

                      ...
                      >
                      >I've done some snoops on browsers that are GETting HTML pages with
                      >lots of embedded links that are in the same realm as the original HTML
                      >page.
                      >
                      >Despite this obvious opportunity to take advantage of persistent connections
                      >the browser opens a connection for each subsequent GET.
                      >
                      I was looking at IE's (version 5 somthing) traffic yesterday, and it seems to send two requests per connection.


                      -Carl
                    • Fielding, Roy
                      The decision on when to close is left to either side. A server will close the connection based on its resource-consumption requirements which may vary
                      Message 10 of 25 , Nov 2, 2000
                        The decision on when to close is left to either side. A server will
                        close the connection based on its resource-consumption requirements
                        which may vary substantially based on the type of server and the
                        number of clients it is intended to serve. A client will close the
                        connection if it is connection-limited and needs to open many other
                        connections, or if it just believes in being network friendly.

                        Unfortunately, none of the major browsers are network friendly,
                        so they typically ignore the connection (not even recognizing FIN
                        as an event) until they later attempt to use it again. Most
                        general-purpose servers have a short activity time-out on
                        connections and will close the connection after that time-out
                        (typically under 10 seconds, though a high-activity server will
                        set this to one second or turn off persistent connections altogether).

                        Cheers,

                        Roy T. Fielding, Chief Scientist, eBuilt, Inc. (www.ebuilt.com)
                        Chairman, The Apache Software Foundation (www.apache.org)
                      • Marc Slemko
                        ... You are missing the point; it doesn t do it out of some odd desire to share since a client has had it s fill of dynamic content for now , like you
                        Message 11 of 25 , Nov 2, 2000
                          On Thu, 2 Nov 2000, James Lacey wrote:

                          > > > to a client. For example, if the web server generates any dynamic content
                          > > > for the client, then it will usually close the connection after the response it
                          > > > sent back. Regardless of whether or not the client supplied a Connection:
                          > > > Keep-Alive header or not. The reasoning behind this is that if the sever
                          > > > had to generate dynamic content on your behalf, then you've had your share
                          > > > and its time to give some other poor slob a turn.
                          > >
                          > > Huh? That would be a webserver with some very... odd ideas.
                          > >
                          > > If you are talking about "Connection: Keep-Alive" then you appear to be
                          > > talking about HTTP/1.0, in which there is no chunked encoding so
                          > > unless the server puts a content-length on its dynamic content
                          > > (which is perfectly possible for it to do, but many don't for
                          > > reasons that are also perfectly legitimate) then there is no way
                          > > to use a persistent connection since in that case the only
                          > > end-of-reponse marker you have is the close of connection.
                          >
                          > As a concrete example the iPlanet Enterprise v4.1 server always
                          > closes the connection after responding to a POST request or
                          > anytime that dynamic content is generated (possibly because
                          > the response does not have a Content-Length: header field
                          > and the response is not chunked).
                          >
                          > I have verified this and it is clearly documented in their
                          > literature.

                          You are missing the point; it doesn't do it out of some odd desire
                          to "share" since a client has "had it's fill of dynamic content
                          for now", like you suggested it does.
                        • Wallace, William
                          What experimentation, if any, has been done with adaptive server timeouts? A project I m working on reduces timeouts based on current load. When a certain
                          Message 12 of 25 , Nov 2, 2000
                            What experimentation, if any, has been done with adaptive server timeouts? A
                            project I'm working on reduces timeouts based on current load. When a
                            certain point is reached connections are closed immediately until the load
                            drops down.

                            I haven't got as far as testing this on a live site so I'm curious whether
                            anyone has any real world experience.

                            > -----Original Message-----
                            > From: Fielding, Roy [mailto:fielding@...]
                            > Sent: Thursday, November 02, 2000 1:08 PM
                            > To: 'Jeff.Hodges@...'; http-wg@...
                            > Subject: RE: Of HTTP/1.1 persistent connections and TCP
                            > Keepalive timers
                            >
                            >
                            > The decision on when to close is left to either side. A server will
                            > close the connection based on its resource-consumption requirements
                            > which may vary substantially based on the type of server and the
                            > number of clients it is intended to serve. A client will close the
                            > connection if it is connection-limited and needs to open many other
                            > connections, or if it just believes in being network friendly.
                            >
                            > Unfortunately, none of the major browsers are network friendly,
                            > so they typically ignore the connection (not even recognizing FIN
                            > as an event) until they later attempt to use it again. Most
                            > general-purpose servers have a short activity time-out on
                            > connections and will close the connection after that time-out
                            > (typically under 10 seconds, though a high-activity server will
                            > set this to one second or turn off persistent connections altogether).
                            >
                            > Cheers,
                            >
                            > Roy T. Fielding, Chief Scientist, eBuilt, Inc.
                            > (www.ebuilt.com)
                            > Chairman, The Apache Software Foundation
                            (www.apache.org)
                          • Fred Bohle
                            Fred Bohle@NEON 11/02/2000 12:27 PM We seem to be diverging into TCP coding. A read will return zero length when the other end has issued a normal close (and
                            Message 13 of 25 , Nov 2, 2000
                              Fred Bohle@NEON
                              11/02/2000 12:27 PM


                              We seem to be diverging into TCP coding. A read will return zero
                              length
                              when the other end has issued a normal close (and all the data has been
                              read).
                              A read will return -1 when the connection is ReSeT, or there is a
                              connection time-out
                              of any sort. So the server can too tell the difference between end of data
                              and
                              a connection failure.

                              Fred




                              From: Carl Kugler/Boulder/IBM <kugler@...> on 11/02/2000 11:59 AM

                              To: James Lacey <James.Lacey@...>
                              cc: http-wg <http-wg@...>

                              Subject: Re: Of HTTP/1.1 persistent connections and TCP Keepalive timers


                              ...
                              >
                              >For example, suppose the server sends back an HTTP response to the client
                              >that does not have a Content-Length: header field and that it is not
                              >chunked.
                              >
                              >Then the only way the client knows that it has read the
                              >entire response off of the pipe is when the server closes the connection.
                              >When the server closes the connection the client will receive a
                              >zero-byte read which is socket layer's indication that the pipe
                              >is broken.
                              >
                              This is not good. If this is the server's normal behavior, the client has
                              no way to distinguish a dropped connection from end of file. So the client
                              can never be sure it received an entire message.

                              If this is the server's behavior for error conditions, this is still not
                              good unless the server either waits for the entire request (could be a
                              humongous POST and/or a very slow connection), or only closes one half of
                              the connection (which, BTW, is impossible in Java, except maybe in the
                              latest releases). Otherwise, the client might get a RST while transmitting
                              the request, and will then never see the error response.

                              ...
                              >
                              >I've done some snoops on browsers that are GETting HTML pages with
                              >lots of embedded links that are in the same realm as the original HTML
                              >page.
                              >
                              >Despite this obvious opportunity to take advantage of persistent
                              connections
                              >the browser opens a connection for each subsequent GET.
                              >
                              I was looking at IE's (version 5 somthing) traffic yesterday, and it seems
                              to send two requests per connection.


                              -Carl
                            • Carl Kugler/Boulder/IBM
                              I stand corrected. But then, why was Content-Length added in HTTP/1.0 and Tranfer-Encoding: chunked in HTTP/1.1? -Carl Fred Bohle on
                              Message 14 of 25 , Nov 2, 2000
                                I stand corrected. But then, why was Content-Length added in HTTP/1.0 and
                                Tranfer-Encoding: chunked in HTTP/1.1?

                                -Carl



                                "Fred Bohle" <fbohle@...> on 11/02/2000 11:27:24 AM

                                To: Carl Kugler/Boulder/IBM@IBMUS
                                cc: James Lacey <James.Lacey@...>, http-wg
                                <http-wg@...>
                                Subject: Re: Of HTTP/1.1 persistent connections and TCP Keepalive timers







                                Fred Bohle@NEON
                                11/02/2000 12:27 PM


                                We seem to be diverging into TCP coding. A read will return zero
                                length
                                when the other end has issued a normal close (and all the data has been
                                read).
                                A read will return -1 when the connection is ReSeT, or there is a
                                connection time-out
                                of any sort. So the server can too tell the difference between end of data
                                and
                                a connection failure.

                                Fred




                                From: Carl Kugler/Boulder/IBM <kugler@...> on 11/02/2000 11:59 AM

                                To: James Lacey <James.Lacey@...>
                                cc: http-wg <http-wg@...>

                                Subject: Re: Of HTTP/1.1 persistent connections and TCP Keepalive timers


                                ...
                                >
                                >For example, suppose the server sends back an HTTP response to the client
                                >that does not have a Content-Length: header field and that it is not
                                >chunked.
                                >
                                >Then the only way the client knows that it has read the
                                >entire response off of the pipe is when the server closes the connection.
                                >When the server closes the connection the client will receive a
                                >zero-byte read which is socket layer's indication that the pipe
                                >is broken.
                                >
                                This is not good. If this is the server's normal behavior, the client has
                                no way to distinguish a dropped connection from end of file. So the client
                                can never be sure it received an entire message.

                                If this is the server's behavior for error conditions, this is still not
                                good unless the server either waits for the entire request (could be a
                                humongous POST and/or a very slow connection), or only closes one half of
                                the connection (which, BTW, is impossible in Java, except maybe in the
                                latest releases). Otherwise, the client might get a RST while transmitting
                                the request, and will then never see the error response.

                                ...
                                >
                                >I've done some snoops on browsers that are GETting HTML pages with
                                >lots of embedded links that are in the same realm as the original HTML
                                >page.
                                >
                                >Despite this obvious opportunity to take advantage of persistent
                                connections
                                >the browser opens a connection for each subsequent GET.
                                >
                                I was looking at IE's (version 5 somthing) traffic yesterday, and it seems
                                to send two requests per connection.


                                -Carl
                              • James Lacey
                                ... Content-Length was added so that persistent connections would be possible. Without a content length, neither the client or the server would know where the
                                Message 15 of 25 , Nov 2, 2000
                                  Carl Kugler/Boulder/IBM wrote:

                                  > I stand corrected. But then, why was Content-Length added in HTTP/1.0 and
                                  > Tranfer-Encoding: chunked in HTTP/1.1?

                                  Content-Length was added so that persistent connections would be possible.

                                  Without a content length, neither the client or the server would know
                                  where the message ended. You have to realize that TCP is a stream and
                                  it is up to the users of the stream to impose some kind of framing protocol
                                  onto the stream.

                                  Transfer-Encoding: chunked was added so that servers could return
                                  possibly large dynamic content in chunks, as it was generated, so that
                                  the server did not have to buffer up all of the dynamic content before
                                  sending it back. With chunking you can also have persistent connections
                                  w/o the need of a Content-Length header field.

                                  One curious think about chunking that I do not understand is that
                                  HTTP proxies are required to move the chunked (any?) encoding before
                                  forwarding the response.

                                  >
                                  >
                                  > -Carl
                                  >
                                  > "Fred Bohle" <fbohle@...> on 11/02/2000 11:27:24 AM
                                  >
                                  > To: Carl Kugler/Boulder/IBM@IBMUS
                                  > cc: James Lacey <James.Lacey@...>, http-wg
                                  > <http-wg@...>
                                  > Subject: Re: Of HTTP/1.1 persistent connections and TCP Keepalive timers
                                  >
                                  > Fred Bohle@NEON
                                  > 11/02/2000 12:27 PM
                                  >
                                  > We seem to be diverging into TCP coding. A read will return zero
                                  > length
                                  > when the other end has issued a normal close (and all the data has been
                                  > read).
                                  > A read will return -1 when the connection is ReSeT, or there is a
                                  > connection time-out
                                  > of any sort. So the server can too tell the difference between end of data
                                  > and
                                  > a connection failure.
                                  >
                                  > Fred
                                  >
                                  > From: Carl Kugler/Boulder/IBM <kugler@...> on 11/02/2000 11:59 AM
                                  >
                                  > To: James Lacey <James.Lacey@...>
                                  > cc: http-wg <http-wg@...>
                                  >
                                  > Subject: Re: Of HTTP/1.1 persistent connections and TCP Keepalive timers
                                  >
                                  > ...
                                  > >
                                  > >For example, suppose the server sends back an HTTP response to the client
                                  > >that does not have a Content-Length: header field and that it is not
                                  > >chunked.
                                  > >
                                  > >Then the only way the client knows that it has read the
                                  > >entire response off of the pipe is when the server closes the connection.
                                  > >When the server closes the connection the client will receive a
                                  > >zero-byte read which is socket layer's indication that the pipe
                                  > >is broken.
                                  > >
                                  > This is not good. If this is the server's normal behavior, the client has
                                  > no way to distinguish a dropped connection from end of file. So the client
                                  > can never be sure it received an entire message.
                                  >
                                  > If this is the server's behavior for error conditions, this is still not
                                  > good unless the server either waits for the entire request (could be a
                                  > humongous POST and/or a very slow connection), or only closes one half of
                                  > the connection (which, BTW, is impossible in Java, except maybe in the
                                  > latest releases). Otherwise, the client might get a RST while transmitting
                                  > the request, and will then never see the error response.
                                  >
                                  > ...
                                  > >
                                  > >I've done some snoops on browsers that are GETting HTML pages with
                                  > >lots of embedded links that are in the same realm as the original HTML
                                  > >page.
                                  > >
                                  > >Despite this obvious opportunity to take advantage of persistent
                                  > connections
                                  > >the browser opens a connection for each subsequent GET.
                                  > >
                                  > I was looking at IE's (version 5 somthing) traffic yesterday, and it seems
                                  > to send two requests per connection.
                                  >
                                  > -Carl
                                • James Lacey
                                  ... OK. OK. I get your point. It may not be out of some odd desire to share . But it is certainly based on the idea that many clients need to be serviced with
                                  Message 16 of 25 , Nov 2, 2000
                                    Marc Slemko wrote:

                                    > On Thu, 2 Nov 2000, James Lacey wrote:
                                    >
                                    > > > > to a client. For example, if the web server generates any dynamic content
                                    > > > > for the client, then it will usually close the connection after the response it
                                    > > > > sent back. Regardless of whether or not the client supplied a Connection:
                                    > > > > Keep-Alive header or not. The reasoning behind this is that if the sever
                                    > > > > had to generate dynamic content on your behalf, then you've had your share
                                    > > > > and its time to give some other poor slob a turn.
                                    > > >
                                    > > > Huh? That would be a webserver with some very... odd ideas.
                                    > > >
                                    > > > If you are talking about "Connection: Keep-Alive" then you appear to be
                                    > > > talking about HTTP/1.0, in which there is no chunked encoding so
                                    > > > unless the server puts a content-length on its dynamic content
                                    > > > (which is perfectly possible for it to do, but many don't for
                                    > > > reasons that are also perfectly legitimate) then there is no way
                                    > > > to use a persistent connection since in that case the only
                                    > > > end-of-reponse marker you have is the close of connection.
                                    > >
                                    > > As a concrete example the iPlanet Enterprise v4.1 server always
                                    > > closes the connection after responding to a POST request or
                                    > > anytime that dynamic content is generated (possibly because
                                    > > the response does not have a Content-Length: header field
                                    > > and the response is not chunked).
                                    > >
                                    > > I have verified this and it is clearly documented in their
                                    > > literature.
                                    >
                                    > You are missing the point; it doesn't do it out of some odd desire
                                    > to "share" since a client has "had it's fill of dynamic content
                                    > for now", like you suggested it does.

                                    OK. OK. I get your point.

                                    It may not be out of some odd desire to "share".

                                    But it is certainly based on the idea that many
                                    clients need to be serviced with relatively few
                                    connections.
                                  • Carl Kugler/Boulder/IBM
                                    ... and ... protocol ... Ah yes, make perfect sense now. ... That s news to me. Doesn t that apply only to MIME gateways or something? Wouldn t that require a
                                    Message 17 of 25 , Nov 2, 2000
                                      >Sent by: jlacey@...
                                      >Carl Kugler/Boulder/IBM wrote:
                                      >
                                      >> I stand corrected. But then, why was Content-Length added in HTTP/1.0
                                      and
                                      >> Tranfer-Encoding: chunked in HTTP/1.1?
                                      >
                                      >Content-Length was added so that persistent connections would be possible.
                                      >
                                      >Without a content length, neither the client or the server would know
                                      >where the message ended. You have to realize that TCP is a stream and
                                      >it is up to the users of the stream to impose some kind of framing
                                      protocol
                                      >onto the stream.
                                      >
                                      >Transfer-Encoding: chunked was added so that servers could return
                                      >possibly large dynamic content in chunks, as it was generated, so that
                                      >the server did not have to buffer up all of the dynamic content before
                                      >sending it back. With chunking you can also have persistent connections
                                      >w/o the need of a Content-Length header field.
                                      >
                                      Ah yes, make perfect sense now.

                                      >One curious think about chunking that I do not understand is that
                                      >HTTP proxies are required to move the chunked (any?) encoding before
                                      >forwarding the response.
                                      >
                                      That's news to me. Doesn't that apply only to MIME gateways or something?
                                      Wouldn't that require a potentially infinite buffer in the proxy? Wouldn't
                                      it have a horrible impact on response time?

                                      -Carl

                                      >>
                                      >>
                                      >> -Carl
                                      >>
                                      >> "Fred Bohle" <fbohle@...> on 11/02/2000 11:27:24 AM
                                      >>
                                      >> To: Carl Kugler/Boulder/IBM@IBMUS
                                      >> cc: James Lacey <James.Lacey@...>, http-wg
                                      >> <http-wg@...>
                                      >> Subject: Re: Of HTTP/1.1 persistent connections and TCP Keepalive timers
                                      >>
                                      >> Fred Bohle@NEON
                                      >> 11/02/2000 12:27 PM
                                      >>
                                      >> We seem to be diverging into TCP coding. A read will return zero
                                      >> length
                                      >> when the other end has issued a normal close (and all the data has been
                                      >> read).
                                      >> A read will return -1 when the connection is ReSeT, or there is a
                                      >> connection time-out
                                      >> of any sort. So the server can too tell the difference between end of data
                                      >> and
                                      >> a connection failure.
                                      >>
                                      >> Fred
                                      >>
                                      >> From: Carl Kugler/Boulder/IBM <kugler@...> on 11/02/2000 11:59 AM
                                      >>
                                      >> To: James Lacey <James.Lacey@...>
                                      >> cc: http-wg <http-wg@...>
                                      >>
                                      >> Subject: Re: Of HTTP/1.1 persistent connections and TCP Keepalive timers
                                      >>
                                      >> ...
                                      >> >
                                      >> >For example, suppose the server sends back an HTTP response to the client
                                      >> >that does not have a Content-Length: header field and that it is not
                                      >> >chunked.
                                      >> >
                                      >> >Then the only way the client knows that it has read the
                                      >> >entire response off of the pipe is when the server closes the connection.
                                      >> >When the server closes the connection the client will receive a
                                      >> >zero-byte read which is socket layer's indication that the pipe
                                      >> >is broken.
                                      >> >
                                      >> This is not good. If this is the server's normal behavior, the client has
                                      >> no way to distinguish a dropped connection from end of file. So the client
                                      >> can never be sure it received an entire message.
                                      >>
                                      >> If this is the server's behavior for error conditions, this is still not
                                      >> good unless the server either waits for the entire request (could be a
                                      >> humongous POST and/or a very slow connection), or only closes one half of
                                      >> the connection (which, BTW, is impossible in Java, except maybe in the
                                      >> latest releases). Otherwise, the client might get a RST while transmitting
                                      >> the request, and will then never see the error response.
                                      >>
                                      >> ...
                                      >> >
                                      >> >I've done some snoops on browsers that are GETting HTML pages with
                                      >> >lots of embedded links that are in the same realm as the original HTML
                                      >> >page.
                                      >> >
                                      >> >Despite this obvious opportunity to take advantage of persistent
                                      >> connections
                                      >> >the browser opens a connection for each subsequent GET.
                                      >> >
                                      >> I was looking at IE's (version 5 somthing) traffic yesterday, and it seems
                                      >> to send two requests per connection.
                                      >>
                                      >> -Carl
                                      >
                                    • James Lacey
                                      ... I didn t say that it wasn t a controversial requirement ...
                                      Message 18 of 25 , Nov 2, 2000
                                        Carl Kugler/Boulder/IBM wrote:

                                        > >Sent by: jlacey@...
                                        > >Carl Kugler/Boulder/IBM wrote:
                                        > >
                                        > >> I stand corrected. But then, why was Content-Length added in HTTP/1.0
                                        > and
                                        > >> Tranfer-Encoding: chunked in HTTP/1.1?
                                        > >
                                        > >Content-Length was added so that persistent connections would be possible.
                                        > >
                                        > >Without a content length, neither the client or the server would know
                                        > >where the message ended. You have to realize that TCP is a stream and
                                        > >it is up to the users of the stream to impose some kind of framing
                                        > protocol
                                        > >onto the stream.
                                        > >
                                        > >Transfer-Encoding: chunked was added so that servers could return
                                        > >possibly large dynamic content in chunks, as it was generated, so that
                                        > >the server did not have to buffer up all of the dynamic content before
                                        > >sending it back. With chunking you can also have persistent connections
                                        > >w/o the need of a Content-Length header field.
                                        > >
                                        > Ah yes, make perfect sense now.
                                        >
                                        > >One curious think about chunking that I do not understand is that
                                        > >HTTP proxies are required to move the chunked (any?) encoding before
                                        > >forwarding the response.
                                        > >
                                        > That's news to me. Doesn't that apply only to MIME gateways or something?
                                        > Wouldn't that require a potentially infinite buffer in the proxy? Wouldn't
                                        > it have a horrible impact on response time?
                                        >
                                        > -Carl

                                        I didn't say that it wasn't a controversial requirement ...

                                        >
                                        >
                                        > >>
                                        > >>
                                        > >> -Carl
                                        > >>
                                        > >> "Fred Bohle" <fbohle@...> on 11/02/2000 11:27:24 AM
                                        > >>
                                        > >> To: Carl Kugler/Boulder/IBM@IBMUS
                                        > >> cc: James Lacey <James.Lacey@...>, http-wg
                                        > >> <http-wg@...>
                                        > >> Subject: Re: Of HTTP/1.1 persistent connections and TCP Keepalive timers
                                        > >>
                                        > >> Fred Bohle@NEON
                                        > >> 11/02/2000 12:27 PM
                                        > >>
                                        > >> We seem to be diverging into TCP coding. A read will return zero
                                        > >> length
                                        > >> when the other end has issued a normal close (and all the data has been
                                        > >> read).
                                        > >> A read will return -1 when the connection is ReSeT, or there is a
                                        > >> connection time-out
                                        > >> of any sort. So the server can too tell the difference between end of data
                                        > >> and
                                        > >> a connection failure.
                                        > >>
                                        > >> Fred
                                        > >>
                                        > >> From: Carl Kugler/Boulder/IBM <kugler@...> on 11/02/2000 11:59 AM
                                        > >>
                                        > >> To: James Lacey <James.Lacey@...>
                                        > >> cc: http-wg <http-wg@...>
                                        > >>
                                        > >> Subject: Re: Of HTTP/1.1 persistent connections and TCP Keepalive timers
                                        > >>
                                        > >> ...
                                        > >> >
                                        > >> >For example, suppose the server sends back an HTTP response to the client
                                        > >> >that does not have a Content-Length: header field and that it is not
                                        > >> >chunked.
                                        > >> >
                                        > >> >Then the only way the client knows that it has read the
                                        > >> >entire response off of the pipe is when the server closes the connection.
                                        > >> >When the server closes the connection the client will receive a
                                        > >> >zero-byte read which is socket layer's indication that the pipe
                                        > >> >is broken.
                                        > >> >
                                        > >> This is not good. If this is the server's normal behavior, the client has
                                        > >> no way to distinguish a dropped connection from end of file. So the client
                                        > >> can never be sure it received an entire message.
                                        > >>
                                        > >> If this is the server's behavior for error conditions, this is still not
                                        > >> good unless the server either waits for the entire request (could be a
                                        > >> humongous POST and/or a very slow connection), or only closes one half of
                                        > >> the connection (which, BTW, is impossible in Java, except maybe in the
                                        > >> latest releases). Otherwise, the client might get a RST while transmitting
                                        > >> the request, and will then never see the error response.
                                        > >>
                                        > >> ...
                                        > >> >
                                        > >> >I've done some snoops on browsers that are GETting HTML pages with
                                        > >> >lots of embedded links that are in the same realm as the original HTML
                                        > >> >page.
                                        > >> >
                                        > >> >Despite this obvious opportunity to take advantage of persistent
                                        > >> connections
                                        > >> >the browser opens a connection for each subsequent GET.
                                        > >> >
                                        > >> I was looking at IE's (version 5 somthing) traffic yesterday, and it seems
                                        > >> to send two requests per connection.
                                        > >>
                                        > >> -Carl
                                        > >
                                      • Patrick McManus
                                        ... transfer encodings (of which chunk is one) are hop to hop.. so they have to be theoretically removed.. but if the outgoing encoding is the same as the
                                        Message 19 of 25 , Nov 2, 2000
                                          > One curious think about chunking that I do not understand is that
                                          > HTTP proxies are required to move the chunked (any?) encoding before
                                          > forwarding the response.

                                          transfer encodings (of which chunk is one) are hop to hop.. so they
                                          have to be theoretically removed.. but if the outgoing encoding is the
                                          same as the incoming encoding that phsyical operation can certainly be
                                          optimized away. The requirement is that the outgoing encoding meets
                                          the requirements of the request received by the proxy.. so if the
                                          proxy gets a 1.0 request it will need to remove 'chunked' and
                                          terminate the message in some other fashion.

                                          That other way is probably to buffer it and add a CL, or more likely
                                          terminate it by closing the connection.

                                          which leads me to a pet peeve of mine.. 2616 prohibits the sending of
                                          CL when a non identity transfer encoding is used.. consider the case I
                                          just used of a 1.1 proxy with a 1.1 origin-server and a 1.0
                                          user-agent.. the proxy receives a 1.1 chunked (and something else non
                                          identity) response but it's only choice to send it downstream is to
                                          buffer it in a store and fowrard fashion to figure out the content
                                          length or close the connection w/ EOF which would rob of it of the
                                          chance to use 1.0 style keepalives...

                                          admittedly some folks think 1.0 keepalives ought to be burned.. but by
                                          that argument all responses to 1.0 reqs should lack CL's and be TCP
                                          terminated, right?

                                          if the content length is known by the original sender it seems a
                                          little odd to lose this information if for no reason other than use in
                                          resource allocation or UI thermometer widgets. Certainly with dynamic
                                          content the server probably doesn't know the CL anyhow at header time,
                                          but what if it's just sending say trasnfer-encoding: deflate, chunked
                                          ??

                                          -P
                                        • Carl Kugler/Boulder/IBM
                                          ... infinite ... ... How does an idle connection consume bandwidth? -Carl
                                          Message 20 of 25 , Nov 2, 2000
                                            > -----Original Message-----
                                            > From: Joris Dobbelsteen [mailto:joris.dobbelsteen@m...]
                                            ...
                                            >
                                            > The browser can leave the connection open for an infinite ammount of time
                                            > (if not restricted by lower-level protocols, like TCP).
                                            >
                                            > However, idle time-outs when the connection is CONNECTED are, in the real
                                            > world, handled by the server instead of the client. This is because the
                                            > server is not interrested in HTTP connections that are idle for an
                                            infinite
                                            > ammount of time and just consuming bandwidth (the lower-level connection
                                            > must be maintained)...
                                            >
                                            ...

                                            How does an idle connection consume bandwidth?

                                            -Carl
                                          • Julien Pierre
                                            Hi, ... Transfer-Encoding chunked also works on the client side, and would allow a client to send dynamic content in its request. If both the client and the
                                            Message 21 of 25 , Nov 3, 2000
                                              Hi,

                                              James Lacey wrote:

                                              > Carl Kugler/Boulder/IBM wrote:
                                              >
                                              > > I stand corrected. But then, why was Content-Length added in HTTP/1.0 and
                                              > > Tranfer-Encoding: chunked in HTTP/1.1?
                                              >
                                              > Content-Length was added so that persistent connections would be possible.
                                              >
                                              > Without a content length, neither the client or the server would know
                                              > where the message ended. You have to realize that TCP is a stream and
                                              > it is up to the users of the stream to impose some kind of framing protocol
                                              > onto the stream.
                                              >
                                              > Transfer-Encoding: chunked was added so that servers could return
                                              > possibly large dynamic content in chunks, as it was generated, so that
                                              > the server did not have to buffer up all of the dynamic content before
                                              > sending it back. With chunking you can also have persistent connections
                                              > w/o the need of a Content-Length header field.

                                              "Transfer-Encoding chunked" also works on the client side, and would allow a
                                              client to send dynamic content in its request. If both the client and the server
                                              are doing chunking, it makes streaming applications possible over a standard HTTP
                                              connection (eg: VPN, multimedia, etc).

                                              I'd like to point out that with iPlanet Web Server 4.1, you can easily write such
                                              applications in NSAPI and all the chunking issues are transparently taken care of
                                              for the application developer. The only requirement is of course that the client
                                              must be HTTP/1.1 compliant and support chunking. The client counterpart of such a
                                              server-side streaming application could be a Java applet or browser plugin.

                                              > One curious think about chunking that I do not understand is that
                                              > HTTP proxies are required to move the chunked (any?) encoding before
                                              > forwarding the response.

                                              It seems that such a proxy would break the type of application mentioned above
                                              ... Full duplex would be required for such an application. This is something that
                                              is not very well addressed in current HTTP specs (hint: this is just a trick to
                                              get immediate attention from the RFC2616 owners :)).

                                              Traditionally, with the content-length indicated in requests and responses, many
                                              browsers and servers have used a synchronous model, where clients try to submit
                                              the whole request, then only after it's complete, begin read the server response.
                                              Many servers operate similarly and read the entire request before starting the
                                              reponse. This is generally true for small GET requests, which the server can
                                              afford to buffer. However, for POST, the server will not buffer since that would
                                              make the memory footprint of the server very high when multiplied by the number
                                              of connections. So the HTTP server will just act as a sort of intermediate and
                                              use a relatively small buffer for the POST data, then pass it gradually to the
                                              consumer of the request - whether it's a CGI, servlet, or NSAPI application ...

                                              I have found that HTTP clients typically break if the server tries to respond too
                                              early to a request, before it is complete. For example, this can happen with a
                                              large POST request to a non-existing object. The server detects early on that the
                                              URI doesn't point to a valid object on the server, and will try to respond
                                              immediately with a 404 response. This is done before it reads the entire amount
                                              of POST data ; since that can be very large and slow to read.

                                              At that point however, the client is still submitting the large POST data ; and
                                              the connection ends up being reset since both sides are sending and the client
                                              isn't trying to read until its write loop is over. In iWS, we have some
                                              workarounds that I consider ugly, including trying to read more POST data on such
                                              erroneous large requests and just discarding it, as well as doing shutdown before
                                              close, just to make sure the client doesn't see a "connection reset" error. This
                                              doesn't work 100% of the time since we set some limits, but obviously we aren't
                                              going to read megabytes of data if the request is too large, or wait a long time
                                              on that socket, since that could be a denial of service attack - eating up socket
                                              descriptors and in some cases also holding up a thread in the server. It would be
                                              really good if the HTTP spec would address such cases and require both client and
                                              servers to try both sides of the socket before closing/shutting the connection
                                              down ... Such workarounds would not be necessary and the servers could just
                                              peacefully shut down the connection after reporting an error, and assume that the
                                              client in fact was really able to receive that error - not causing a "connection
                                              reset".

                                              --
                                              The network is the computer, and it's down
                                            • Ruslan Belkin
                                              ... that s cuz the right thing to do would be to send expect continue along with POST then wait for continue then send the data. But many servers are broken 2
                                              Message 22 of 25 , Nov 3, 2000
                                                Julien Pierre wrote:

                                                > Hi,
                                                >
                                                > James Lacey wrote:
                                                >
                                                > > Carl Kugler/Boulder/IBM wrote:
                                                > >
                                                > > > I stand corrected. But then, why was Content-Length added in HTTP/1.0 and
                                                > > > Tranfer-Encoding: chunked in HTTP/1.1?
                                                > >
                                                > I have found that HTTP clients typically break if the server tries to respond too
                                                > early to a request, before it is complete. For example, this can happen with a

                                                that's cuz the right thing to do would be to send expect continue along with POST
                                                then wait for continue then send the data. But many servers are broken 2 with respect
                                                to this,

                                                Ruslan
                                              • Joris Dobbelsteen
                                                Content-length in HTTP/1.0 is not directly needed, except when using the addional RFC purposing persistent connections. It gave the client a indication of the
                                                Message 23 of 25 , Nov 5, 2000
                                                  Content-length in HTTP/1.0 is not directly needed, except when using the addional RFC purposing persistent connections. It gave the client a indication of the length of the file. This is nice when downloading a large file, you know that you have all the data received (in case of line failures)...

                                                  HTTP/1.1 added chunked encoding in order to let HTTP implementations benefit when sending content where the length is not known (like dynamic content or an real-time video/measurement) and benefit from the persistent connections. The next request (e.g. a picture) can be requested without waisting time to open a new connection.


                                                  That looks like the most logic explaination for me....

                                                  - Joris

                                                  > -----Original Message-----
                                                  > From: Carl Kugler/Boulder/IBM [mailto:kugler@...]
                                                  > Sent: Thursday, 02 November 2000 20:07
                                                  > To: Fred Bohle
                                                  > Cc: James Lacey; http-wg
                                                  > Subject: Re: Of HTTP/1.1 persistent connections and TCP
                                                  > Keepalive timers
                                                  >
                                                  >
                                                  >
                                                  > I stand corrected. But then, why was Content-Length added in
                                                  > HTTP/1.0 and
                                                  > Tranfer-Encoding: chunked in HTTP/1.1?
                                                  >
                                                  > -Carl
                                                  >
                                                  >
                                                  >
                                                  > "Fred Bohle" <fbohle@...> on 11/02/2000 11:27:24 AM
                                                  >
                                                  > To: Carl Kugler/Boulder/IBM@IBMUS
                                                  > cc: James Lacey <James.Lacey@...>, http-wg
                                                  > <http-wg@...>
                                                  > Subject: Re: Of HTTP/1.1 persistent connections and TCP
                                                  > Keepalive timers
                                                  >
                                                  >
                                                  >
                                                  >
                                                  >
                                                  >
                                                  >
                                                  > Fred Bohle@NEON
                                                  > 11/02/2000 12:27 PM
                                                  >
                                                  >
                                                  > We seem to be diverging into TCP coding. A read will return zero
                                                  > length
                                                  > when the other end has issued a normal close (and all the
                                                  > data has been
                                                  > read).
                                                  > A read will return -1 when the connection is ReSeT, or there is a
                                                  > connection time-out
                                                  > of any sort. So the server can too tell the difference
                                                  > between end of data
                                                  > and
                                                  > a connection failure.
                                                  >
                                                  > Fred
                                                  >
                                                  >
                                                  >
                                                  >
                                                  > From: Carl Kugler/Boulder/IBM <kugler@...> on
                                                  > 11/02/2000 11:59 AM
                                                  >
                                                  > To: James Lacey <James.Lacey@...>
                                                  > cc: http-wg <http-wg@...>
                                                  >
                                                  > Subject: Re: Of HTTP/1.1 persistent connections and TCP
                                                  > Keepalive timers
                                                  >
                                                  >
                                                  > ...
                                                  > >
                                                  > >For example, suppose the server sends back an HTTP response
                                                  > to the client
                                                  > >that does not have a Content-Length: header field and that it is not
                                                  > >chunked.
                                                  > >
                                                  > >Then the only way the client knows that it has read the
                                                  > >entire response off of the pipe is when the server closes
                                                  > the connection.
                                                  > >When the server closes the connection the client will receive a
                                                  > >zero-byte read which is socket layer's indication that the pipe
                                                  > >is broken.
                                                  > >
                                                  > This is not good. If this is the server's normal behavior,
                                                  > the client has
                                                  > no way to distinguish a dropped connection from end of file.
                                                  > So the client
                                                  > can never be sure it received an entire message.
                                                  >
                                                  > If this is the server's behavior for error conditions, this
                                                  > is still not
                                                  > good unless the server either waits for the entire request (could be a
                                                  > humongous POST and/or a very slow connection), or only closes
                                                  > one half of
                                                  > the connection (which, BTW, is impossible in Java, except maybe in the
                                                  > latest releases). Otherwise, the client might get a RST
                                                  > while transmitting
                                                  > the request, and will then never see the error response.
                                                  >
                                                  > ...
                                                  > >
                                                  > >I've done some snoops on browsers that are GETting HTML pages with
                                                  > >lots of embedded links that are in the same realm as the
                                                  > original HTML
                                                  > >page.
                                                  > >
                                                  > >Despite this obvious opportunity to take advantage of persistent
                                                  > connections
                                                  > >the browser opens a connection for each subsequent GET.
                                                  > >
                                                  > I was looking at IE's (version 5 somthing) traffic yesterday,
                                                  > and it seems
                                                  > to send two requests per connection.
                                                  >
                                                  >
                                                  > -Carl
                                                  >
                                                  >
                                                  >
                                                  >
                                                  >
                                                  >
                                                  >
                                                  >
                                                  >
                                                  >
                                                • Joris Dobbelsteen
                                                  As for removing chunked encoding. Why is it actually needed??? The only reason I see is when you make a HTTP/1.0 request (no chunked support) and the browser a
                                                  Message 24 of 25 , Nov 5, 2000
                                                    As for removing chunked encoding. Why is it actually needed???

                                                    The only reason I see is when you make a HTTP/1.0 request (no chunked support) and the browser a subsequent HTTP/1.1 request and the server returns a chunked response back.
                                                    I would think that it might be to ensure the client can always understand the request. Also it takes less time to make a subsequent request to a proxy than to the web server (most of the time).
                                                    Also as Carl indicated this requires the buffer of the proxy to be of infinite length if the connection must be kept open and this would even dramatically downgrade response time.

                                                    I think you shouldn't do this over HTTP/1.1 connections, and over HTTP/1.0 connections just by sending NO content-length...



                                                    About MSIE sending only two requests at a time: Microsoft has documented this and it is possibility to have the lowest response times for the web page. If one requests takes a lot of time, to the requests after this, the server will respond later. As a browser you cannot know the size of a request.
                                                    There was once a discussion about HOL (Head-of-Line) blocking that went into this problem. Microsoft avoided the problem this way...

                                                    - Joris


                                                    > -----Original Message-----
                                                    > From: Carl Kugler/Boulder/IBM [mailto:kugler@...]
                                                    > Sent: Thursday, 02 November 2000 20:46
                                                    > To: James Lacey
                                                    > Cc: http-wg
                                                    > Subject: Re: Of HTTP/1.1 persistent connections and TCP
                                                    > Keepalive timers
                                                    >
                                                    >
                                                    >
                                                    > >Sent by: jlacey@...
                                                    > >Carl Kugler/Boulder/IBM wrote:
                                                    > >
                                                    > >> I stand corrected. But then, why was Content-Length added
                                                    > in HTTP/1.0
                                                    > and
                                                    > >> Tranfer-Encoding: chunked in HTTP/1.1?
                                                    > >
                                                    > >Content-Length was added so that persistent connections
                                                    > would be possible.
                                                    > >
                                                    > >Without a content length, neither the client or the server would know
                                                    > >where the message ended. You have to realize that TCP is a stream and
                                                    > >it is up to the users of the stream to impose some kind of framing
                                                    > protocol
                                                    > >onto the stream.
                                                    > >
                                                    > >Transfer-Encoding: chunked was added so that servers could return
                                                    > >possibly large dynamic content in chunks, as it was
                                                    > generated, so that
                                                    > >the server did not have to buffer up all of the dynamic
                                                    > content before
                                                    > >sending it back. With chunking you can also have persistent
                                                    > connections
                                                    > >w/o the need of a Content-Length header field.
                                                    > >
                                                    > Ah yes, make perfect sense now.
                                                    >
                                                    > >One curious think about chunking that I do not understand is that
                                                    > >HTTP proxies are required to move the chunked (any?) encoding before
                                                    > >forwarding the response.
                                                    > >
                                                    > That's news to me. Doesn't that apply only to MIME gateways
                                                    > or something?
                                                    > Wouldn't that require a potentially infinite buffer in the
                                                    > proxy? Wouldn't
                                                    > it have a horrible impact on response time?
                                                    >
                                                    > -Carl
                                                    >
                                                    > >>
                                                    > >>
                                                    > >> -Carl
                                                    > >>
                                                    > >> "Fred Bohle" <fbohle@...> on 11/02/2000 11:27:24 AM
                                                    > >>
                                                    > >> To: Carl Kugler/Boulder/IBM@IBMUS
                                                    > >> cc: James Lacey <James.Lacey@...>, http-wg
                                                    > >> <http-wg@...>
                                                    > >> Subject: Re: Of HTTP/1.1 persistent connections and TCP
                                                    > Keepalive timers
                                                    > >>
                                                    > >> Fred Bohle@NEON
                                                    > >> 11/02/2000 12:27 PM
                                                    > >>
                                                    > >> We seem to be diverging into TCP coding. A read will
                                                    > return zero
                                                    > >> length
                                                    > >> when the other end has issued a normal close (and all the
                                                    > data has been
                                                    > >> read).
                                                    > >> A read will return -1 when the connection is ReSeT, or there is a
                                                    > >> connection time-out
                                                    > >> of any sort. So the server can too tell the difference
                                                    > between end of data
                                                    > >> and
                                                    > >> a connection failure.
                                                    > >>
                                                    > >> Fred
                                                    > >>
                                                    > >> From: Carl Kugler/Boulder/IBM <kugler@...> on
                                                    > 11/02/2000 11:59 AM
                                                    > >>
                                                    > >> To: James Lacey <James.Lacey@...>
                                                    > >> cc: http-wg <http-wg@...>
                                                    > >>
                                                    > >> Subject: Re: Of HTTP/1.1 persistent connections and TCP
                                                    > Keepalive timers
                                                    > >>
                                                    > >> ...
                                                    > >> >
                                                    > >> >For example, suppose the server sends back an HTTP
                                                    > response to the client
                                                    > >> >that does not have a Content-Length: header field and
                                                    > that it is not
                                                    > >> >chunked.
                                                    > >> >
                                                    > >> >Then the only way the client knows that it has read the
                                                    > >> >entire response off of the pipe is when the server closes
                                                    > the connection.
                                                    > >> >When the server closes the connection the client will receive a
                                                    > >> >zero-byte read which is socket layer's indication that the pipe
                                                    > >> >is broken.
                                                    > >> >
                                                    > >> This is not good. If this is the server's normal
                                                    > behavior, the client has
                                                    > >> no way to distinguish a dropped connection from end of
                                                    > file. So the client
                                                    > >> can never be sure it received an entire message.
                                                    > >>
                                                    > >> If this is the server's behavior for error conditions,
                                                    > this is still not
                                                    > >> good unless the server either waits for the entire request
                                                    > (could be a
                                                    > >> humongous POST and/or a very slow connection), or only
                                                    > closes one half of
                                                    > >> the connection (which, BTW, is impossible in Java, except
                                                    > maybe in the
                                                    > >> latest releases). Otherwise, the client might get a RST
                                                    > while transmitting
                                                    > >> the request, and will then never see the error response.
                                                    > >>
                                                    > >> ...
                                                    > >> >
                                                    > >> >I've done some snoops on browsers that are GETting HTML pages with
                                                    > >> >lots of embedded links that are in the same realm as the
                                                    > original HTML
                                                    > >> >page.
                                                    > >> >
                                                    > >> >Despite this obvious opportunity to take advantage of persistent
                                                    > >> connections
                                                    > >> >the browser opens a connection for each subsequent GET.
                                                    > >> >
                                                    > >> I was looking at IE's (version 5 somthing) traffic
                                                    > yesterday, and it seems
                                                    > >> to send two requests per connection.
                                                    > >>
                                                    > >> -Carl
                                                    > >
                                                    >
                                                    >
                                                  • Joris Dobbelsteen
                                                    ... Some servers can close a connection after sending dynamic content because of security resons. (or because they are bad implementations.) As for POST, if a
                                                    Message 25 of 25 , Nov 5, 2000
                                                      > -----Original Message-----
                                                      > From: jlacey@...
                                                      > [mailto:jlacey@...]On Behalf Of
                                                      > James Lacey
                                                      > Sent: Thursday, 02 November 2000 20:22
                                                      > To: Marc Slemko
                                                      > Cc: http-wg
                                                      > Subject: Re: Of HTTP/1.1 persistent connections and TCP
                                                      > Keepalive timers
                                                      >
                                                      >
                                                      > Marc Slemko wrote:
                                                      >
                                                      > > On Thu, 2 Nov 2000, James Lacey wrote:
                                                      > >
                                                      > > > > > to a client. For example, if the web server generates
                                                      > any dynamic content
                                                      > > > > > for the client, then it will usually close the
                                                      > connection after the response it
                                                      > > > > > sent back. Regardless of whether or not the client
                                                      > supplied a Connection:
                                                      > > > > > Keep-Alive header or not. The reasoning behind this
                                                      > is that if the sever
                                                      > > > > > had to generate dynamic content on your behalf, then
                                                      > you've had your share
                                                      > > > > > and its time to give some other poor slob a turn.
                                                      > > > >
                                                      > > > > Huh? That would be a webserver with some very... odd ideas.
                                                      > > > >
                                                      > > > > If you are talking about "Connection: Keep-Alive" then
                                                      > you appear to be
                                                      > > > > talking about HTTP/1.0, in which there is no chunked encoding so
                                                      > > > > unless the server puts a content-length on its dynamic content
                                                      > > > > (which is perfectly possible for it to do, but many don't for
                                                      > > > > reasons that are also perfectly legitimate) then there is no way
                                                      > > > > to use a persistent connection since in that case the only
                                                      > > > > end-of-reponse marker you have is the close of connection.
                                                      > > >
                                                      > > > As a concrete example the iPlanet Enterprise v4.1 server always
                                                      > > > closes the connection after responding to a POST request or
                                                      > > > anytime that dynamic content is generated (possibly because
                                                      > > > the response does not have a Content-Length: header field
                                                      > > > and the response is not chunked).
                                                      > > >
                                                      > > > I have verified this and it is clearly documented in their
                                                      > > > literature.
                                                      > >
                                                      > > You are missing the point; it doesn't do it out of some odd desire
                                                      > > to "share" since a client has "had it's fill of dynamic content
                                                      > > for now", like you suggested it does.
                                                      >
                                                      > OK. OK. I get your point.
                                                      >
                                                      > It may not be out of some odd desire to "share".
                                                      >
                                                      > But it is certainly based on the idea that many
                                                      > clients need to be serviced with relatively few
                                                      > connections.
                                                      >
                                                      Some servers can close a connection after sending dynamic content because of security resons. (or because they are bad implementations.)

                                                      As for POST, if a server responds (except the 1xx status codes), the request must have ended. At least, this is the most logic thing for me. Also I don't know how to send a POST request without using content-length or chunked encoding and having the server knowning absolutely sure that request has ended.

                                                      - Joris
                                                    Your message has been successfully submitted and would be delivered to recipients shortly.