Re: should input filter return the exact amount of bytes asked for?

["William A. Rowe, Jr." <wr...@apache.org>]

At 03:31 AM 11/11/2003, Justin Erenkrantz wrote:
>On Tue, Nov 04, 2003 at 01:41:46AM -0800, Stas Bekman wrote:
>> filter. What happens if the filter returns less bytes (while there is still 
>> more data coming?) What happens if the filter returns more bytes than 
>> requested (e.g. because it uncompressed some data). After all the incoming 
>
>Less bytes = OK.

But not great if there is more incoming data available (consider that one
can call with NONBLOCK and dig up some more.  There is a balance to be
found here, one doesn't want to slurp 15mb of a file at onces, but one doesn't
want bytes to trickle up one at a time.

>Same bytes = OK.

Of course

>More bytes = Not OK.  (Theoretically possible though with bad filters.)

Wrong.  This is OK across the board, please consider;

module requests 1000 arbitrary bytes;

  codepage module requests 1000

    http reads one 'chunk' available, 8000 bytes
    and will return that page

  codepage can translate 7998 bytes and comes to
  a screeching halt for a 3 byte sequence.  returns
  our Now Translated 4000 bytes

module sees a 4000 byte heap bucket.

What can you do?  Instead of treating that bucket as a singleton
when you want 1000 bytes, consume the first 1000 bytes from that
bucket (or the brigade.)

Please review the archives for this discussion (the brigades on the
apr list, the filter api on httpd.)  This was a very long thread, but the
net result of filters is that you get what is available/handy, not any
specific number of bytes.

BIll 

Re: should input filter return the exact amount of bytes asked for?

[Justin Erenkrantz <ju...@erenkrantz.com>]

--On Tuesday, November 11, 2003 11:24 AM -0600 "William A. Rowe, Jr." 
<wr...@apache.org> wrote:

>> More bytes = Not OK.  (Theoretically possible though with bad filters.)
>
> Wrong.  This is OK across the board, please consider;

Uh, no.  We changed the filter semantics some time ago to stop this insanity. 
It was inefficient to call AP_MODE_READBYTES and have it return more than 
asked for.  Check out the CVS log for util_filter.h, specifically around 
revision 1.62.

> module requests 1000 arbitrary bytes;
>
>   codepage module requests 1000
>
>     http reads one 'chunk' available, 8000 bytes
>     and will return that page
>
>   codepage can translate 7998 bytes and comes to
>   a screeching halt for a 3 byte sequence.  returns
>   our Now Translated 4000 bytes
>
> module sees a 4000 byte heap bucket.
>
> What can you do?  Instead of treating that bucket as a singleton
> when you want 1000 bytes, consume the first 1000 bytes from that
> bucket (or the brigade.)

No.  That means you have 3k more bytes you have to consume that you didn't ask 
for.  The filter wouldn't return it again.  Writing code that used input 
filters and having to deal with that it could get more than asked for was just 
confusing and led to lots of error-prone code.

If it asks for 1k in AP_MODE_READBYTES, it gets at most 1k.  Anything else is 
broken.  (util_filter.h AP_MODE_READBYTES says as much, but that's not fair, 
because I wrote that comment.)

> Please review the archives for this discussion (the brigades on the
> apr list, the filter api on httpd.)  This was a very long thread, but the
> net result of filters is that you get what is available/handy, not any
> specific number of bytes.

That *was* indeed the position at one time, but when I redid the input filters 
(which was about rewrite #14 of input filters), we corrected this because it 
was causing lots of problems to return more than asked for - this is when we 
added the mode argument to ap_get_brigade.  mod_ssl's input filtering code was 
just broken under that old API.

And, the big boys even reviewed the code and semantic changes before it went 
in.  So, it was definitely RTC.  ;-)  -- justin