Chunked transfer encoding on responses.

[Graham Dumpleton <gr...@gmail.com>]

Have a question about past design decisions regarding Apache.

My question has come about due to a slightly heated discussion over
chunked transfer encoding and its application to response content. In
the mod_wsgi module I am implementing for Apache I preserve the
default behaviour of Apache which is not to switch on chunked transfer
encoding for response content. Because the Python WSGI specification
that the module is implementing an adapter for doesn't itself provide
a way for an application to indicate that it wants chunked transfer
encoding to be used, I have as a convenience provided a directive to
control it. The default for the directive is Off (preserving default
behaviour of Apache) with other values being On, which is always
enable it, ie., set r->chunked to 1 always, and Auto, which will only
set r->chunked to 1 if there is no content length header provided for
the response by the application.

On the WSGI list though, one person has labeled the directive as
pointless. Their belief is that a quality HTTP server supporting
HTTP/1.1 should always behave in the Auto mode above. Ie., if content
length header has not been specified, that the web server should by
itself decide to enable chunked transfer encoding. Obviously then he
would have to feel that the default behaviour of Apache is wrong.

Thus my question is, why when Apache was updated to support HTTP/1.1
did it just preserve the HTTP/1.0 type behaviour and not in cases
where it could automatically apply chunked transfer encoding to the
response, apply it?

The person on the WSGI list is more or less claiming that there would
be no harm in a web server always applying chunked transfer encoding
to a response which doesn't specify a content length, of course
providing it is a HTTP/1.1 client, but something tells me that this
doesn't sound quite right and there must be good reasons why Apache
doesn't do it. Is this persons claim reasonable, or are there good
reasons why one wouldn't want to use chunked transfer encoding on
responses with no content length header and thus why Apache requires
r->chunked to be set explicitly by an application to enable it.

Thanks in advance for any feedback and hopefully this question about
how Apache is designed isn't seen as being off topic for the list. :-)

Graham

Re: Chunked transfer encoding on responses.

[André Malo <nd...@perlig.de>]

* Henrik Nordstrom wrote:

> lör 2007-04-07 klockan 09:18 +0200 skrev André Malo:
> > Hmm, you may get something wrong here. The httpd does apply chunked
> > encoding automatically when it needs to. That is in keep-alive
> > situations without given or determineable Content-Length.
> >
> > Why doesn't it do it in all other cases? My answer is: because it would
> > be useless (as in: not of any use :-).
>
> I don't agree fully here. chunked is not useless in the non-keepalive
> case. What it adds there compared to the HTTP/1.0 method of just closing
> the connection is error detection. A receiver seeing the connection
> closed before the final "eof" chunk knows something went wrong and the
> response is not complete. If chunked is not used the receiver usually
> can not tell that there was a problem.

Ah, hmm. True :)

nd
-- 
"Solides und umfangreiches Buch"
                                          -- aus einer Rezension

<http://pub.perlig.de/books.html#apache2>

Re: Chunked transfer encoding on responses.

[Henrik Nordstrom <he...@henriknordstrom.net>]

lör 2007-04-07 klockan 09:18 +0200 skrev André Malo:

> Hmm, you may get something wrong here. The httpd does apply chunked encoding 
> automatically when it needs to. That is in keep-alive situations without 
> given or determineable Content-Length.
> 
> Why doesn't it do it in all other cases? My answer is: because it would be 
> useless (as in: not of any use :-).

I don't agree fully here. chunked is not useless in the non-keepalive
case. What it adds there compared to the HTTP/1.0 method of just closing
the connection is error detection. A receiver seeing the connection
closed before the final "eof" chunk knows something went wrong and the
response is not complete. If chunked is not used the receiver usually
can not tell that there was a problem.

Regards
Henrik

Re: Chunked transfer encoding on responses.

[Graham Dumpleton <gr...@gmail.com>]

On 07/04/07, André Malo <nd...@perlig.de> wrote:
> * Graham Dumpleton wrote:
>
> > Thus my question is, why when Apache was updated to support HTTP/1.1
> > did it just preserve the HTTP/1.0 type behaviour and not in cases
> > where it could automatically apply chunked transfer encoding to the
> > response, apply it?
>
> Hmm, you may get something wrong here. The httpd does apply chunked encoding
> automatically when it needs to. That is in keep-alive situations without
> given or determineable Content-Length.

So it seems. :-(

I dug into the code again and from that worked out that I had managed
to inadvertently disable it in my Apache configuration files by
commenting out the inclusion of extra/httpd-default.conf when I was
trying to debug some issue. Ie., the KeepAlive directive wasn't being
set.

Thanks for pointing out my mistake.

Graham

Re: Chunked transfer encoding on responses.

[André Malo <nd...@perlig.de>]

* Graham Dumpleton wrote:

> Thus my question is, why when Apache was updated to support HTTP/1.1
> did it just preserve the HTTP/1.0 type behaviour and not in cases
> where it could automatically apply chunked transfer encoding to the
> response, apply it?

Hmm, you may get something wrong here. The httpd does apply chunked encoding 
automatically when it needs to. That is in keep-alive situations without 
given or determineable Content-Length.

Why doesn't it do it in all other cases? My answer is: because it would be 
useless (as in: not of any use :-).

nd
-- 
> [...] weiß jemand zufällig, was der Tag DIV ausgeschrieben bedeutet?
DIVerses. Benannt nach all dem unstrukturierten Zeug, was die Leute da
so reinpacken und dann absolut positionieren ...
                           -- Florian Hartig und Lars Kasper in dciwam

Re: Chunked transfer encoding on responses.

[Justin Erenkrantz <ju...@erenkrantz.com>]

On 4/8/07, Henrik Nordstrom <he...@henriknordstrom.net> wrote:
> So why is there a dependency on keep-alive being enabled?

If keep-alive is disabled for the connection, then Connection: Close
tends to be more efficient anyway...  -- justin

Re: Chunked transfer encoding on responses.

[Henrik Nordstrom <he...@henriknordstrom.net>]

lör 2007-04-07 klockan 04:00 -0500 skrev William A. Rowe, Jr.:

> Of course this person is entirely wrong if the client doesn't
> Accept-Encoding: chunked
> 
> which is exactly the logic we test.

So why is there a dependency on keep-alive being enabled?

Regards
Henrik

Re: Chunked transfer encoding on responses.

["William A. Rowe, Jr." <wr...@rowe-clan.net>]

Justin Erenkrantz wrote:
> 
> Chunking support on a response is implicit if you claim HTTP/1.1
> support.  You don't need to signal it with Accept-Encoding (you can, I
> guess).  IOW, an HTTP/1.1 client should always a expect a server may
> give back chunking...  -- justin

Of course, my bad.  (Contrawise, if server is HTTP/1.1 client can expect
that an HTTP/1.1 request body will also be decoded if chunked.)

Re: Chunked transfer encoding on responses.

[Justin Erenkrantz <ju...@erenkrantz.com>]

On 4/7/07, William A. Rowe, Jr. <wr...@rowe-clan.net> wrote:
> Graham Dumpleton wrote:
> >
> > The person on the WSGI list is more or less claiming that there would
> > be no harm in a web server always applying chunked transfer encoding
> > to a response which doesn't specify a content length
>
> Of course this person is entirely wrong if the client doesn't
> Accept-Encoding: chunked
>
> which is exactly the logic we test.

Chunking support on a response is implicit if you claim HTTP/1.1
support.  You don't need to signal it with Accept-Encoding (you can, I
guess).  IOW, an HTTP/1.1 client should always a expect a server may
give back chunking...  -- justin

Re: Chunked transfer encoding on responses.

["William A. Rowe, Jr." <wr...@rowe-clan.net>]

Graham Dumpleton wrote:
> 
> The person on the WSGI list is more or less claiming that there would
> be no harm in a web server always applying chunked transfer encoding
> to a response which doesn't specify a content length

Of course this person is entirely wrong if the client doesn't
Accept-Encoding: chunked

which is exactly the logic we test.