chunked write discussion

[Dean Gaudet <dg...@arctic.org>]

While bcwrite() with my patch is more correct it's certainly not
efficient.  It causes 3 segments to be plopped on the net for each chunk. 
The simple fix for this is to allocate a buffer in bcwrite() and copy the
stuff to be written into it and then do a single write().  But this
involves an extra copy... something that'd be nice to avoid. 

My first idea was to fix it by using writev(), but writev() on linux (but
not IRIX or Solaris) drops a segment on the wire for each vector element.
(BOO HISS!) 

Then my next idea was to fix it by overallocating the buffer so that it
has room at the beginning for the chunk header and at the end for the
chunk tail.  Then I read the HTTP/1.1 standard and found out that chunk
sizes cannot start with extra 0 digits (BOO HISS!).  It's still not hard
to do it this way. 

Then I finally realised that if we're doing stacked disciplines for 2.0
then we're going to have that extra buffer copy anyhow.  So, should I
submit a patch that implements the first method (buffer copy) or the third
method (overallocated buffer)?

I'm not trying to get this in for 1.2b5 or even 1.2 for that matter.   I
don't even know how to test chunked encodings, is there a client that
supports it?

Dean

Re: chunked write discussion

[Dean Gaudet <dg...@arctic.org>]

On Sat, 25 Jan 1997, Alexei Kosut wrote: 
> The other problem, btw, though less of one, is that the current
> chunking code calls bflush() in a number of places that it
> shouldn't.

Oh right I forgot about that.  Here's a kludge that solves that and the
bcwrite efficiency problem.  Add an element to buff_struct:  char
*chunk_header;.  If non null then it points to a (sufficiently large
enough) chunk header in the current buffer.  When we switch to chunking we
flush only if we can't fit the minimal header, bufsiz/4 data (pulled that
one out of my butt, needs to be at least 1 byte) and the trailer into the
buffer.  We also decrement the bufsiz by 2 so we can fit the trailer into
the buffer. 

So essentially this is like my overallocating plan except we don't over
allocate.  We just ensure that we put the chunk header and trailer into
the buffer and clean the rest of the code appropriately.

This is a kludge.  But given the existance of the apache code working in
this manner it would essentially force browser writers to do things
correctly.  i.e. it'd really suck if someone writes a browser that somehow
depends on the chunk starting at the beginning of a segment... 

Dean

Re: chunked write discussion

[Alexei Kosut <ak...@nueva.pvt.k12.ca.us>]

On Sat, 25 Jan 1997, Dean Gaudet wrote:

> While bcwrite() with my patch is more correct it's certainly not
> efficient.  It causes 3 segments to be plopped on the net for each chunk. 
> The simple fix for this is to allocate a buffer in bcwrite() and copy the
> stuff to be written into it and then do a single write().  But this
> involves an extra copy... something that'd be nice to avoid. 

[...]

> Then my next idea was to fix it by overallocating the buffer so that it
> has room at the beginning for the chunk header and at the end for the
> chunk tail.  Then I read the HTTP/1.1 standard and found out that chunk
> sizes cannot start with extra 0 digits (BOO HISS!).  It's still not hard
> to do it this way. 

Actually, what you can do (and what some servers do do) is to pad it
with extra spaces atthe end of the chunk size. This is legal.

The other problem, btw, though less of one, is that the current
chunking code calls bflush() in a number of places that it
shouldn't. Actually, it's not the chunking code's fault, but the fault
of the code I wrote for BUFF to implement chunking. Namely, that
bsetflag always calls bflush(). It has to, because chunking is
implemented as part of bwrite(), and if there anything left in the
buffer, it'd get the wrong chunk property (on vs. off).

So it flushes before it starts chunking (to get all the headers -
which shouldn't be chunked - out), and right after the last chunk (it
then adds the 0 chunk in finalize_request_protocol() and waits for a
new request).

I couldn't figure out how to get around this easily when I did it
originally. But as long as we're trying to get rid of
flushes... anyone have any bright ideas?

-- 
________________________________________________________________________
Alexei Kosut <ak...@nueva.pvt.k12.ca.us>      The Apache HTTP Server
URL: http://www.nueva.pvt.k12.ca.us/~akosut/   http://www.apache.org/