Re: filtered I/O - flow control

["Roy T. Fielding" <fi...@kiwi.ICS.UCI.EDU>]

Bucket brigades.  I don't believe there is any simpler way of doing
it without taking robustness away from the server or functionality away
from filtering.

>>>>           If at any point there is too much data to write to
>>>> the network, it will be up to Apache's internal buffering to catch it and
>>>> hold it until the socket is ready.  
>>> 
>>> Seems like that could potentially cause a large increase in storage use
>>> for huge files.  
>> 
>> There is no more potential for this than there is currently.  Remember,
>> that in both approaches, the maximum that can be sent to the filter is the
>> current maximum that can be sent to the network.  In both approaches the
>> amount of data returned from the filters is likely to grow very large, but
>
>The link-based approach does not "return" data. It sends it down through
>the layers to network as it is being generated.
>
>> in both approaches, the filter is going to end up being limited by the
>> same limits that Apache has.  I believe this is a red herring.

Whether it is true in both approaches doesn't matter -- whatever filter
mechanism is used, it cannot assume that it can hold the entire response
data in memory and it cannot assume that it can completely write through
the stack of filters before it runs out of internal buffer space.  That
is just the nature of filtering -- sometimes you need to know the end
of an input record before you can filter the beginning.  The result is
that you either need to specifically limit filtering to something lame
(like pre- and end-content or 8kB max frames) or implement a special form
of memory handling that can drop itself into VM space when needed.
Hence sfio.  Hence bucket brigades.  There are probably other examples.

The other thing I think is unavoidable is the need to treat metadata
(header fields) as part of the filtered stream and not as a separate
table.  But that can wait until the next version of HTTP.

....Roy