A few questions on Input Filters

[Martin Townsend <ma...@power-oasis.com>]

Hi,

I have an input filter for processing POST requests and it was working 
well until recently.  After debugging I found that it was due to a large 
POST request (around 2680 bytes) which most of the time was handled as a 
single bucket brigade with one heap bucket.  The problem occured when 
the POST request was split into two brigades which are passed 
independently to my filter. So my first question is this expected?  I 
assume it is so I have to alter my filter to handle partial bucket 
brigades.
If so, I take it I can infer a partial brigade by the fact that the EOS 
bucket is not present?
Whilst looking through other input filters I notice they handle FLUSH 
buckets, for my input filter I take it I can ignore these buckets as all 
I'm trying to do is extract the POST data to a buffer and then process 
it without altering it.
I noticed that one module's input filter ignored sub requests, does 
anyone know when sub requests occur within the input filter phase and 
whether I can ignore these too.

Many Thanks,
Martin.

Re: A few questions on Input Filters

[Joe Lewis <jo...@joe-lewis.com>]

On 01/13/2012 09:24 AM, Martin Townsend wrote:
> Thanks Joe for the info, my input filter is now behaving itself again.
> I've not seen any FLUSH buckets yet so I doubt I will as we are 
> running the bare minimum of modules.  I'll let you know if I do though.
> One last question, when I don't see the EOS bucket should I return a 
> certain APR_ error code to say the POST request hasn't finished yet. 
> I'm currently return APR_OK and this seems to work.
>
> Cheers,
> Martin.

Leave it as OK.  If you return a different APR_ error, it could cause 
the entire request to hang.

By returning OK, you are simply stating that your function has finished 
what it was passed, and all is well.

Joe
--
www.silverhawk.net

Re: A few questions on Input Filters

[Martin Townsend <ma...@power-oasis.com>]

Thanks Joe for the info, my input filter is now behaving itself again.
I've not seen any FLUSH buckets yet so I doubt I will as we are running 
the bare minimum of modules.  I'll let you know if I do though.
One last question, when I don't see the EOS bucket should I return a 
certain APR_ error code to say the POST request hasn't finished yet. I'm 
currently return APR_OK and this seems to work.

Cheers,
Martin.

On 11/01/2012 17:12, Joe Lewis wrote:
> On 01/11/2012 04:17 AM, Martin Townsend wrote:
>> The problem occured when the POST request was split into two brigades 
>> which are passed independently to my filter. So my first question is 
>> this expected?
>
> You should definitely expect that.  Don't assume that the entire 
> content will always come in the same way.  In this kind of development 
> architecture (where anyone can build a module), we should expect the 
> unexpected.
>
>>   I assume it is so I have to alter my filter to handle partial 
>> bucket brigades.
>> If so, I take it I can infer a partial brigade by the fact that the 
>> EOS bucket is not present?
>> Whilst looking through other input filters I notice they handle FLUSH 
>> buckets, for my input filter I take it I can ignore these buckets as 
>> all I'm trying to do is extract the POST data to a buffer and then 
>> process it without altering it.
>
> If the brigade doesn't have that EOS, there is more to the stream to 
> be read.  When you see the FLUSH bucket, you should really be passing 
> the brigade on to the next chain (FLUSH buckets are created when the 
> brigade needs to be split).
>
> I had originally thought that FLUSH buckets were output buckets to 
> prevent the client from waiting too long.  Are you seeing these on an 
> input chain?  If so, what other modules are involved?  I'm curious for 
> my own understanding of how other modules might effect some of the 
> stuff I have written.
>
>> I noticed that one module's input filter ignored sub requests, does 
>> anyone know when sub requests occur within the input filter phase and 
>> whether I can ignore these too.
>
> The input's have already been done when a sub request is created.  
> Usually, a sub request is happening when an output filter or a content 
> generator are being called, so I'm not sure a sub-request will see the 
> input from the parent filter.
>
>>
>> Many Thanks,
>> Martin.
>
> That is what the list is for.  Hope you can get things straightened out!
> Joe Lewis
> -- 
> www.silverhawk.net

Re: A few questions on Input Filters

[Joe Lewis <jo...@joe-lewis.com>]

On 01/11/2012 04:17 AM, Martin Townsend wrote:
> The problem occured when the POST request was split into two brigades 
> which are passed independently to my filter. So my first question is 
> this expected?

You should definitely expect that.  Don't assume that the entire content 
will always come in the same way.  In this kind of development 
architecture (where anyone can build a module), we should expect the 
unexpected.

>   I assume it is so I have to alter my filter to handle partial bucket 
> brigades.
> If so, I take it I can infer a partial brigade by the fact that the 
> EOS bucket is not present?
> Whilst looking through other input filters I notice they handle FLUSH 
> buckets, for my input filter I take it I can ignore these buckets as 
> all I'm trying to do is extract the POST data to a buffer and then 
> process it without altering it.

If the brigade doesn't have that EOS, there is more to the stream to be 
read.  When you see the FLUSH bucket, you should really be passing the 
brigade on to the next chain (FLUSH buckets are created when the brigade 
needs to be split).

I had originally thought that FLUSH buckets were output buckets to 
prevent the client from waiting too long.  Are you seeing these on an 
input chain?  If so, what other modules are involved?  I'm curious for 
my own understanding of how other modules might effect some of the stuff 
I have written.

> I noticed that one module's input filter ignored sub requests, does 
> anyone know when sub requests occur within the input filter phase and 
> whether I can ignore these too.

The input's have already been done when a sub request is created.  
Usually, a sub request is happening when an output filter or a content 
generator are being called, so I'm not sure a sub-request will see the 
input from the parent filter.

>
> Many Thanks,
> Martin.

That is what the list is for.  Hope you can get things straightened out!
Joe Lewis
--
www.silverhawk.net

Re: A few questions on Input Filters

[Nick Kew <ni...@apache.org>]

On Wed, 11 Jan 2012 11:17:30 +0000
Martin Townsend <ma...@power-oasis.com> wrote:

> Hi,
> [chop]

One point that Joe omitted to mention is that since input
filtering is a PULL API, it's entirely up to your filter
how much data to pull before returning to its upstream.
If your filter can't deal with partial data, it can block for
as long as necessary to read more.  With just a couple of Kb 
to handle it's not exactly going to be a performance hit!

Regarding the flush buckets, conceptually I'd expect they
might feature if you were handling streaming data, or
maybe multipart uploads, or some extension to the HTTP
protocol.  Maybe there's some such application- or ptotocol-
oriented filter in your chain?

-- 
Nick Kew