You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Jeff Trawick <tr...@bellsouth.net> on 2000/09/21 17:56:01 UTC
subrequest filtering (yet again)
I gather that most folks want a subrequest to inherit the content
filters associated with the original request. Why is this? If the
set of filters isn't appropriate, you can't get rid of them. If the
set of filters is appropriate, then the configuration associated with
the subrequest URI would have caused the right filters to be added
anyway (unless the URI of the subrequest is only requested via a
subrequest).
What am I missing?
--
Jeff Trawick | trawick@ibm.net | PGP public key at web site:
http://www.geocities.com/SiliconValley/Park/9289/
Born in Roswell... married an alien...
Re: subrequest filtering (yet again)
Posted by Tony Finch <do...@dotat.at>.
rbb@covalent.net wrote:
>On Thu, 21 Sep 2000, Jim Winstead wrote:
>>
>> i don't see how including the results of a subrequest should be
>> any different than including a file. or the results of an external
>> request. or any other bit of data that a content-generator generates
>> (whether it be the first thing in the filter-chain or something in
>> the middle that is inserting content).
>>
>> it almost seems like it should be its own bucket-type.
>
>This is actually one of the designs that Tony and I considered, but it is
>a bit non-orthogonal. The problem is that sub-requests basically convert
>to some other bucket, be it a file or heap, whatever. It is not possible
>to read from a sub-request bucket until it has been converted. This
>conversion could be hidden behind the read function, but it isn't
>completely clean.
>
>Imagine a sub-request that resovles to a CGI script that takes a long time
>to output all of it's data. The sub-request bucket has to stick around
>throughout all of that processing.
It's a bit nastier than that.
At the moment the way we get data hold of the data in a bucket is via
the read() method, which may have the side-effect of reading the data
into memory if it isn't already there. The idea is that we delay this
operation as long as possible so that if the filter chain is really
simple then buckets can get right down to the core filter without
mutating so that the core can handle them in the most efficient manner
it knows (e.g. sendfile).
URI buckets can use this API, but that would compromise efficiency --
sendfile would then be unusable because file buckets would mutate into
mmap or heap buckets in order to bring the file data into memory.
The alternative is to add a new API that unwraps the URI bucket into a
bucket brigade representing its contents in the most efficient manner
possible. This is a disaster if the sub-request is for a CGI
outputting SSI, because there is no efficient way to represent the
output: the entire output must be buffered in memory. (The
non-orthogonality arises here because this operation makes very little
sense at all for the other types of bucket. Yes, the same is true for
setaside() and I think it is bogus too.)
So what is required is some compromise between these two designs that
maintains the efficiency advantages of buckets whilst still being able
to process the output of the sub-request as it is produced if this
happens to be a long slow job. We already have such a design for
handling the response data from the main request, so it makes sense to
use that.
Tony.
--
en oeccget g mtcaa f.a.n.finch
v spdlkishrhtewe y dot@dotat.at
eatp o v eiti i d. fanf@covalent.net
Re: subrequest filtering (yet again)
Posted by Jeff Trawick <tr...@bellsouth.net>.
Jim Winstead <ji...@trainedmonkey.com> writes:
> On Sep 21, Jeff Trawick wrote:
> > Jim Winstead <ji...@trainedmonkey.com> writes:
> >
> > > On Sep 21, Jeff Trawick wrote:
> > > > I gather that most folks want a subrequest to inherit the content
> > > > filters associated with the original request. Why is this? If the
> > > > set of filters isn't appropriate, you can't get rid of them. If the
> > > > set of filters is appropriate, then the configuration associated with
> > > > the subrequest URI would have caused the right filters to be added
> > > > anyway (unless the URI of the subrequest is only requested via a
> > > > subrequest).
> > > >
> > > > What am I missing?
> > >
> > > i don't see how including the results of a subrequest should be
> > > any different than including a file. or the results of an external
> > > request. or any other bit of data that a content-generator generates
> > > (whether it be the first thing in the filter-chain or something in
> > > the middle that is inserting content).
> >
> > A subrequest is different in that it will cause its own configured
> > filters to be added to the chain. I think I'm getting hung up on
> > cases where filters for the original request clash with filters added
> > for the subrequest.
>
> why are they added to the chain? only the contents generated by
> that subrequest should go through any filters that subrequest
> uses.
>
> jim
True... I meant "added to" in a different sense; the contents
generated by the subrequest will go through any filters for the
subrequest in addition to remaining filters for the request. (Today,
unfortunately, subrequest output goes through all filters for the
request instead of just the remaining ones.)
If the request filters are
FILTER1 FILTER2 FILTER3
FILTER2 creates a subrequest, whose filters are
FILTER4 FILTER 5 FILTER6
Once a current problem is fixed:
The subrequest output will go through FILTER4 FILTER5 FILTER6
FILTER3. In other words, the subrequest filters were added on to the
remaining request filters.
--
Jeff Trawick | trawick@ibm.net | PGP public key at web site:
http://www.geocities.com/SiliconValley/Park/9289/
Born in Roswell... married an alien...
Re: subrequest filtering (yet again)
Posted by Jim Winstead <ji...@trainedmonkey.com>.
On Sep 21, Jeff Trawick wrote:
> Jim Winstead <ji...@trainedmonkey.com> writes:
>
> > On Sep 21, Jeff Trawick wrote:
> > > I gather that most folks want a subrequest to inherit the content
> > > filters associated with the original request. Why is this? If the
> > > set of filters isn't appropriate, you can't get rid of them. If the
> > > set of filters is appropriate, then the configuration associated with
> > > the subrequest URI would have caused the right filters to be added
> > > anyway (unless the URI of the subrequest is only requested via a
> > > subrequest).
> > >
> > > What am I missing?
> >
> > i don't see how including the results of a subrequest should be
> > any different than including a file. or the results of an external
> > request. or any other bit of data that a content-generator generates
> > (whether it be the first thing in the filter-chain or something in
> > the middle that is inserting content).
>
> A subrequest is different in that it will cause its own configured
> filters to be added to the chain. I think I'm getting hung up on
> cases where filters for the original request clash with filters added
> for the subrequest.
why are they added to the chain? only the contents generated by
that subrequest should go through any filters that subrequest
uses.
jim
Re: subrequest filtering (yet again)
Posted by Jeff Trawick <tr...@bellsouth.net>.
Jim Winstead <ji...@trainedmonkey.com> writes:
> On Sep 21, Jeff Trawick wrote:
> > I gather that most folks want a subrequest to inherit the content
> > filters associated with the original request. Why is this? If the
> > set of filters isn't appropriate, you can't get rid of them. If the
> > set of filters is appropriate, then the configuration associated with
> > the subrequest URI would have caused the right filters to be added
> > anyway (unless the URI of the subrequest is only requested via a
> > subrequest).
> >
> > What am I missing?
>
> i don't see how including the results of a subrequest should be
> any different than including a file. or the results of an external
> request. or any other bit of data that a content-generator generates
> (whether it be the first thing in the filter-chain or something in
> the middle that is inserting content).
A subrequest is different in that it will cause its own configured
filters to be added to the chain. I think I'm getting hung up on
cases where filters for the original request clash with filters added
for the subrequest.
--
Jeff Trawick | trawick@ibm.net | PGP public key at web site:
http://www.geocities.com/SiliconValley/Park/9289/
Born in Roswell... married an alien...
Re: subrequest filtering (yet again)
Posted by rb...@covalent.net.
On Thu, 21 Sep 2000, Jim Winstead wrote:
> On Sep 21, Jeff Trawick wrote:
> > I gather that most folks want a subrequest to inherit the content
> > filters associated with the original request. Why is this? If the
> > set of filters isn't appropriate, you can't get rid of them. If the
> > set of filters is appropriate, then the configuration associated with
> > the subrequest URI would have caused the right filters to be added
> > anyway (unless the URI of the subrequest is only requested via a
> > subrequest).
> >
> > What am I missing?
>
> i don't see how including the results of a subrequest should be
> any different than including a file. or the results of an external
> request. or any other bit of data that a content-generator generates
> (whether it be the first thing in the filter-chain or something in
> the middle that is inserting content).
>
> it almost seems like it should be its own bucket-type. so the
> "output" of mod_include for this file:
>
> <!--#include virtual="header.html"-->
> <p>this is the real page content
> <!--#include virtual="footer.html"-->
This is actually one of the designs that Tony and I considered, but it is
a bit non-orthogonal. The problem is that sub-requests basically convert
to some other bucket, be it a file or heap, whatever. It is not possible
to read from a sub-request bucket until it has been converted. This
conversion could be hidden behind the read function, but it isn't
completely clean.
Imagine a sub-request that resovles to a CGI script that takes a long time
to output all of it's data. The sub-request bucket has to stick around
throughout all of that processing.
> to it, lets say it is the gzip filter, it may decide that it actually
> needs the raw bits of the buckets, so it will call bucket->read
> (or whatever), which will actually trigger the subrequest to get
> the raw bits that result from that. (it may be an important
> distinction to think of it as the filter asking for the results of
> this subrequest, *not* the browser making the original request. so
> this subrequest shouldn't trigger any filters based on the user
> agent of the browser (unless the subrequest handler specifically
> looks to the original request), for example.)
The problem is that our output is usually handled as a push, and
sub-requests would have to be a pull. It might work, but it might
not. Again, we really need code to determine the "best" way to handle
this. Until we have code, I fear we are thrashing. :-)
Ryan
>
> does that make any more sense? does it help? maybe i'm
> talking about a different filtering than what's going on in
> apache 2.0. unfortunately, i haven't had the time to follow
> the code very closely.
>
> jim
>
_______________________________________________________________________________
Ryan Bloom rbb@apache.org
406 29th St.
San Francisco, CA 94131
-------------------------------------------------------------------------------
Re: subrequest filtering (yet again)
Posted by Jim Winstead <ji...@trainedmonkey.com>.
On Sep 21, Jeff Trawick wrote:
> I gather that most folks want a subrequest to inherit the content
> filters associated with the original request. Why is this? If the
> set of filters isn't appropriate, you can't get rid of them. If the
> set of filters is appropriate, then the configuration associated with
> the subrequest URI would have caused the right filters to be added
> anyway (unless the URI of the subrequest is only requested via a
> subrequest).
>
> What am I missing?
i don't see how including the results of a subrequest should be
any different than including a file. or the results of an external
request. or any other bit of data that a content-generator generates
(whether it be the first thing in the filter-chain or something in
the middle that is inserting content).
it almost seems like it should be its own bucket-type. so the
"output" of mod_include for this file:
<!--#include virtual="header.html"-->
<p>this is the real page content
<!--#include virtual="footer.html"-->
would be a stream ("brigade"?) that looks like:
* bucket of type "headers" (headers output by mod_include)
* bucket of type "subrequest" (for header.html)
* bucket of type "data" (the real page content)
* bucket of type "subrequest" (for footer.html)
* bucket of type "end-of-stream"
now, when the next filter in the chain gets those buckets handed
to it, lets say it is the gzip filter, it may decide that it actually
needs the raw bits of the buckets, so it will call bucket->read
(or whatever), which will actually trigger the subrequest to get
the raw bits that result from that. (it may be an important
distinction to think of it as the filter asking for the results of
this subrequest, *not* the browser making the original request. so
this subrequest shouldn't trigger any filters based on the user
agent of the browser (unless the subrequest handler specifically
looks to the original request), for example.)
does that make any more sense? does it help? maybe i'm
talking about a different filtering than what's going on in
apache 2.0. unfortunately, i haven't had the time to follow
the code very closely.
jim
Re: subrequest filtering (yet again)
Posted by Jeff Trawick <tr...@bellsouth.net>.
Tony Finch <do...@dotat.at> writes:
> I think that modules should examine the existing filter stack before
> inserting their filters to ensure that the result makes sense.
When do modules get a chance to examine the filter stack? Not until
the filter really runs. At that point, a filter would need to run
r->output_filters and no-op itself or a later instance of itself. You
can't no-op an earlier instance of course because that instance passed
a brigade down to you.
--
Jeff Trawick | trawick@ibm.net | PGP public key at web site:
http://www.geocities.com/SiliconValley/Park/9289/
Born in Roswell... married an alien...
Re: subrequest filtering (yet again)
Posted by rb...@covalent.net.
On Thu, 21 Sep 2000, Tony Finch wrote:
> Jeff Trawick <tr...@bellsouth.net> wrote:
> >
> >Now sometimes the filters defined for the subrequest will clash with
> >the filters defined for the original request. What if both the
> >original request and the subrequest have a gzip filter defined?
> >Garbage comes out. What if both have some character set translation
> >defined? Garbage comes out.
>
> I think that modules should examine the existing filter stack before
> inserting their filters to ensure that the result makes sense. E.g.1
> gzip only occurs in the main request filter chain. E.g.2 the output of
> the sub-request's character set translation must be the input of the
> main request's character set translation. This is rather gnarly and
> easily leads to absurdly inefficient setups but I think we can
> consider any that turn up in real life to be the webmaster's fault.
>
> The alternative is to add loads of cleverness to the core to match
> filter input and output types, but I don't think we know how to do
> that :-)
While I agree in theory, in practice, this just won't work. The problem
is that the core may insert the filters, and the core doesn't know if a
filter makes sense to be in the stack twice. This is why I would really
like to punt this to the filter. Basically, the filter gets called once,
sets a flag, and if it is called a second time for the same data, the
second call is removed from the stack. This has the nasty problem of not
working if the second call is the one that is desired, but I'm not sure
how often that will happen.
Ryan
_______________________________________________________________________________
Ryan Bloom rbb@apache.org
406 29th St.
San Francisco, CA 94131
-------------------------------------------------------------------------------
Re: subrequest filtering (yet again)
Posted by Tony Finch <do...@dotat.at>.
Jeff Trawick <tr...@bellsouth.net> wrote:
>
>Now sometimes the filters defined for the subrequest will clash with
>the filters defined for the original request. What if both the
>original request and the subrequest have a gzip filter defined?
>Garbage comes out. What if both have some character set translation
>defined? Garbage comes out.
I think that modules should examine the existing filter stack before
inserting their filters to ensure that the result makes sense. E.g.1
gzip only occurs in the main request filter chain. E.g.2 the output of
the sub-request's character set translation must be the input of the
main request's character set translation. This is rather gnarly and
easily leads to absurdly inefficient setups but I think we can
consider any that turn up in real life to be the webmaster's fault.
The alternative is to add loads of cleverness to the core to match
filter input and output types, but I don't think we know how to do
that :-)
>Assume the hopefully-normal case where the filters for the request and
>the filters for the subrequest don't clash:
>
> When we create a subrequest, it inherits the full set of filters from
> the request. This is appropriate (I think) if the subrequest was
> initiated by a handler (non-filter).
Yes, and it turns out to be a special case of the following...
> If the subrequest was initiated by a filter, perhaps the filter could
> set subrequest->output_filters to f->next after creating the
> subrequest but before running it. I think that this would cause
> subsequent filters for the original request to be executed at the
> right time.
I agree.
Tony.
--
en oeccget g mtcaa f.a.n.finch
v spdlkishrhtewe y dot@dotat.at
eatp o v eiti i d. fanf@covalent.net
Re: subrequest filtering (yet again)
Posted by rb...@covalent.net.
> Now sometimes the filters defined for the subrequest will clash with
> the filters defined for the original request. What if both the
> original request and the subrequest have a gzip filter defined?
> Garbage comes out. What if both have some character set translation
> defined? Garbage comes out.
Yes it does. Luckily both of those filters are pretty low in the filter
chain, or at least they should be. They also aren't generally filters
that would be added for a sub-request. When a filter is added, the thing
doing the adding should probably check if it is a sub-request, assuming it
know if the filter makes sense for sub-requests.
> Assume the hopefully-normal case where the filters for the request and
> the filters for the subrequest don't clash:
>
> When we create a subrequest, it inherits the full set of filters from
> the request. This is appropriate (I think) if the subrequest was
> initiated by a handler (non-filter).
Yes.
>
> If the subrequest was initiated by a filter, perhaps the filter could
> set subrequest->output_filters to f->next after creating the
> subrequest but before running it. I think that this would cause
> subsequent filters for the original request to be executed at the
> right time. We haven't gotten to the insert filters hook (sorry!) for
> the subrequest yet so we still have time to add the filters
> appropriate to the subrequest.
Yep, that was the thought.
> As for the hopefully-rare case where the filters for the request and
> the filters for the subrequest clash:
>
> I guess your comment about the need to remove filters is intended to
> cover this?
Mostly. Some filters will need to know that they make no sense if called
from sub-requests. They will either have to ensure that they are not
added, or that they are removed before they do any work.
Ryan
_______________________________________________________________________________
Ryan Bloom rbb@apache.org
406 29th St.
San Francisco, CA 94131
-------------------------------------------------------------------------------
Re: subrequest filtering (yet again)
Posted by Jeff Trawick <tr...@bellsouth.net>.
rbb@covalent.net writes:
> A couple of things. Tony and I have had multiple discussions of sub
> request filtering, and we still haven't come up with a great decision,
> which is why we haven't posted anything before. I will try to summarize
> what we keep going back and forth about.
>
> Our original hypothesis was that sub requests should just inherit the
> connection based filters, and all content filters should be re-generated
> based on the actual content.
>
> But, A sub request is essentially a part of this current request, so the
> output of the sub request really has to go through the same filters as the
> main-line request.
>
> Take for example mod_include. If we include a file through a #include
> virtual directive, we use a sub-request. Well, all of the data in that
> sub-request is actually a part of the original request. What if we have a
> gzip filter setup for the original request, the sub request has to go
> through the same gzip filter in the correct order.
O.k... I see.
> The best we have really come up with so far, is that we the sub-request
> output should go to the next filter in the stack. At least I am pretty
> sure that was what we decided.
That makes a lot of sense.
Now sometimes the filters defined for the subrequest will clash with
the filters defined for the original request. What if both the
original request and the subrequest have a gzip filter defined?
Garbage comes out. What if both have some character set translation
defined? Garbage comes out.
> As far as removing filters. We need that ability. I am really hoping to
> get back into the Apache coding soon. I know I keep saying that. One of
> my first projects will be to just allow people to remove a filter from the
> filter stack.
>
> I think the big thing that is missing, is that a sub-request does want to
> inherit some of the filters. I really think we need to code this one way
> and see if it works, because right now, we are really just flailing
> blindly at what we think the design should be, based on logic, not
> experience. As we all know, sometimes the best design just doesn't take
> the smallest detail into account.
Assume the hopefully-normal case where the filters for the request and
the filters for the subrequest don't clash:
When we create a subrequest, it inherits the full set of filters from
the request. This is appropriate (I think) if the subrequest was
initiated by a handler (non-filter).
If the subrequest was initiated by a filter, perhaps the filter could
set subrequest->output_filters to f->next after creating the
subrequest but before running it. I think that this would cause
subsequent filters for the original request to be executed at the
right time. We haven't gotten to the insert filters hook (sorry!) for
the subrequest yet so we still have time to add the filters
appropriate to the subrequest.
As for the hopefully-rare case where the filters for the request and
the filters for the subrequest clash:
I guess your comment about the need to remove filters is intended to
cover this?
--
Jeff Trawick | trawick@ibm.net | PGP public key at web site:
http://www.geocities.com/SiliconValley/Park/9289/
Born in Roswell... married an alien...
Re: subrequest filtering (yet again)
Posted by rb...@covalent.net.
A couple of things. Tony and I have had multiple discussions of sub
request filtering, and we still haven't come up with a great decision,
which is why we haven't posted anything before. I will try to summarize
what we keep going back and forth about.
Our original hypothesis was that sub requests should just inherit the
connection based filters, and all content filters should be re-generated
based on the actual content.
But, A sub request is essentially a part of this current request, so the
output of the sub request really has to go through the same filters as the
main-line request.
Take for example mod_include. If we include a file through a #include
virtual directive, we use a sub-request. Well, all of the data in that
sub-request is actually a part of the original request. What if we have a
gzip filter setup for the original request, the sub request has to go
through the same gzip filter in the correct order.
The best we have really come up with so far, is that we the sub-request
output should go to the next filter in the stack. At least I am pretty
sure that was what we decided.
As far as removing filters. We need that ability. I am really hoping to
get back into the Apache coding soon. I know I keep saying that. One of
my first projects will be to just allow people to remove a filter from the
filter stack.
I think the big thing that is missing, is that a sub-request does want to
inherit some of the filters. I really think we need to code this one way
and see if it works, because right now, we are really just flailing
blindly at what we think the design should be, based on logic, not
experience. As we all know, sometimes the best design just doesn't take
the smallest detail into account.
Ryan
On Thu, 21 Sep 2000, Jeff Trawick wrote:
> I gather that most folks want a subrequest to inherit the content
> filters associated with the original request. Why is this? If the
> set of filters isn't appropriate, you can't get rid of them. If the
> set of filters is appropriate, then the configuration associated with
> the subrequest URI would have caused the right filters to be added
> anyway (unless the URI of the subrequest is only requested via a
> subrequest).
>
> What am I missing?
>
> --
> Jeff Trawick | trawick@ibm.net | PGP public key at web site:
> http://www.geocities.com/SiliconValley/Park/9289/
> Born in Roswell... married an alien...
>
_______________________________________________________________________________
Ryan Bloom rbb@apache.org
406 29th St.
San Francisco, CA 94131
-------------------------------------------------------------------------------