Re: chunked transfer-encoding ( and YAHOO )

[TO...@aol.com]

In a message dated 00-06-23 06:34:33 EDT, Greg Stein writes...

>  The chunked transfer-encoding allows the server to signal EOD. What's
>  wrong with that? If the client is not HTTP/1.1, then yah: close the
>  connection. But when that client tells the server "hey! I'm an HTTP/1.1
>  client" then the server simply uses chunked transfer-encoding.

Sometimes the client is lying and sometimes it can't even do
what it says it can. That's the 'reality' these days. See other post.

BTW: What if a filter has already USED the 'Transfer-encoding:' field?

Then what? If you think you can just 'tack' 'chunked' onto the end of
the existing transfer encoding value with a comma separator like the
RFC says you can and have that work... think again. See below.
  
>  Forget the Content-length header.

Ok, it's forgotten. Lock and load and let's see what happens.
  
>  I think the real take-away from this, is that the content-length header
>  should be axed if a filter is present.

That's a simple rule... go for it... but I would make sure that Ken Coar has
an auto-response function key set up for 'Your problem has already
been reported'.
  
>  What about the chunked transfer-encoding? Browsers can use that to 
determine
>  the transfer size. Or are you stating that a compression mixed with chunked
>  encoding does not work on these browsers?

That's EXACTLY what I am saying.

http://sports.yahoo.com is sending compressed data all the time and
it is doing it in exactly the way you imagine. ( See other post ).

Here is the request that goes out from MSIE 5.00.2314
for http://sports.yahoo.com...

GET / HTTP/1.1
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*
Accept-Language: en-us
Accept-Encoding: gzip,deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)
Host: sports.yahoo.com
Connection: Keep-Alive

Here is what http://sports.yahoo.com sends back...

HTTP/1.1 200 OK
Date: Fri, 23 Jun 2000 15:20:55 GMT
Cache-Control: private
Vary: Accept-Encoding
Connection: close
Transfer Encoding: chunked
Content-Type: text/html
Content-Encoding: gzip
(blank line)
GZIP data is here.

Hmm... notice something curious?

Even though the client is HTTP 1.1 compliant and even
though the Server has decided it's OK to send back
no content length it has STILL decided that it better
close the connection even though the Client says
it wants 'Keep-Alive' which should normally be respected.

Why do they do it that way?... Because they know that
even if they send 'chunked' data they can't rely totally
on the browser seeing EOD unless they close the connnection.

You won't ever get sports.yahoo.com to do Keep-Alive.
It is hard-coded to never even try because it knows it
can't depend on 'chunked' at all times.

Notice something else...

Why bother using BOTH Transfer-encoding: chunked 
AND Content-encoding: gzip?

According to the RFC's you should be able to just
kill 2 birds with one stone and say...

'Transfer-Encoding: gzip'

Right?

Wrong. 

There's no way that dog will hunt and YAHOO
now knows it. Believe me... they tried it the other way 
and found out for themselves how broken that all is.

So there's another 'gotcha'... if you have to rely on 
'Transfer-encoding: chunked' because you can't get 
a valid EOD going any other way then now there's
no possibility of using any other type of Transfer-Encoding
along with it and if a filter in a multi-layered filtering
scheme doesn't KNOW that the 'Transfer-Encoding'
field is 'off limits' because the filtering scheme itself
is going to force a 'chunked' type then... well, you 
have confusion and limited functionality.

The specs might say you are 'supposed' to be able to
'gang-up' Transfer-Encoding: 'types' if multiple ones
have been applied but believe me I have tried it every
which way from Sunday and it is totally broken in 
all versions of MSIE and Netscape I have tried.

I have yet to find one single user-agent that can handle 
'Transfer-Encoding: chunked,gzip' even though the 
HTTP 1.1 specs say it's 'supposed' to be able to handle
any number of comma-separated applications of different
encoding types.

As a matter of fact... trying to send multiply Transfer-Encoded
data is a a sure way to see the 'blue screen of death' with
any version of any browser, as far as I have seen.

So when is a spec not a spec? When it won't work, that's when.

>  The chunked transfer-encoding provides a way for the client to detect an
>  abnormal termination. 

The spec does 'Provides a way'... Does it really work? Not all the time.
Again... see above and see other post.

> Sorry, but it DOES exist. 
> It is the chunked transfer-encoding. End of story.

Existing and working seem to be 2 very different things 
these days in the world of Internet protocols.

End of story? Perhaps. I don't know.

See other post.

Yours...

Kevin Kiley
CTO, Remote Communications, Inc.
http://www.RemoteCommunications.com
http://www.rctp.com - Online Internet Content Compression Server