You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Greg Stein <gs...@lyra.org> on 2001/08/26 11:04:57 UTC
Re: cvs commit: httpd-2.0/server core.c request.c
On Sat, Aug 25, 2001 at 11:43:19PM -0000, wrowe@apache.org wrote:
> wrowe 01/08/25 16:43:19
>
> Modified: . CHANGES
> include http_request.h
> modules/http http_core.c http_protocol.c http_request.c
> mod_core.h
> server core.c request.c
> Log:
> Introduce the map_to_storage hook, which allows modules to bypass
> the directory_walk and file_walk for non-file requests. TRACE
How is this different from the translate_name hook? Your new hook and the
translate_name are both about translating URLs into the backend storage.
Why was this new one added? (did I miss some discussion somewhere?) And if
so, then why does translate_name stick around?
--
Greg Stein, http://www.lyra.org/
Re: cvs commit: httpd-2.0/server core.c request.c
Posted by Bill Stoddard <bi...@wstoddard.com>.
> > >
> > > It does? Then you need to find a clue. quick_handler should have been
> > > axed from the outset, and now that this handler allows a non-filesystem
> > > request to be handled properly, it really should be gone.
> >
> > map_to_storage is definitely not the same as quick_handler. But
> > quick_handler does have a use so it stays in.
>
> Actually, they are closer than you might think. quick_handler's purpose is to allow
> requests to be served as quickly as possible. It skips a lot of phases of the
> request to accomplish that. If map_to_storage is tweaked a bit, it can accomplish
> the same thing.
>
> Basically, the map_to_storage function that succeeds would tell the server how many
> of the hooks that follow map_to_storage that it should skip before the handler phase.
>
> This has a couple of advantages over quick_handler, the first one being that it
> is more general, and the second that it means we go back to having one location that
> data is sent to the network.
>
> If that change is made to map_to_storage, then I would be with OtherBill, and
> suggest that quick_handler should be removed.
>
Give me a while to experiment with getting mod_cache working with map_to_storage. I've
been considering moving quick_handler to after the first Location walk, which brings the
functionality I am looking for very close to map_to_storage. We are meeting in the middle
:-)
Bill
Re: cvs commit: httpd-2.0/server core.c request.c
Posted by "William A. Rowe, Jr." <wr...@rowe-clan.net>.
From: "Roy T. Fielding" <fi...@ebuilt.com>
Sent: Monday, August 27, 2001 4:45 PM
> A quick_handler operates the way that a Web server is supposed to operate.
> I should be able to add a hook that detects a Code Red request and immediately
> close the connection with as little server overhead as possible. Normally
> I would do that with a custom change to the HTTP parser, but it makes more
> sense to do it as a hook with quick_handler.
I'd really like to see a quick_handler patch that has the socket drop that
connection entirely, waiting in SYN for the ACK response :)
Bill
Re: cvs commit: httpd-2.0/server core.c request.c
Posted by "Roy T. Fielding" <fi...@ebuilt.com>.
> Quick_handler bypasses everything. We ignore the client's headers, the
> server's configuration, everything. Frankly, quick_handler is not Apache,
> and anyone who writes a quick_handler hack and calls it a module for
> 'Apache' is full of sh*t. Powered by the Apr/Apache MPM/load manager is
> more like it (that isn't badness, that's just reality.)
That's a goofy way of looking at it. You might as well say that Redirect
is not Apache because it ignores Directory sections. And let's not
even get into what can be done by mod_rewrite.
A quick_handler operates the way that a Web server is supposed to operate.
I should be able to add a hook that detects a Code Red request and immediately
close the connection with as little server overhead as possible. Normally
I would do that with a custom change to the HTTP parser, but it makes more
sense to do it as a hook with quick_handler.
It isn't a security hole -- it is just another config directive.
....Roy
Re: cvs commit: httpd-2.0/server core.c request.c
Posted by "William A. Rowe, Jr." <wr...@rowe-clan.net>.
From: "Bill Stoddard" <bi...@wstoddard.com>
Sent: Monday, August 27, 2001 10:54 AM
> > > > map_to_storage is definitely not the same as quick_handler. But
> > > > quick_handler does have a use so it stays in.
> > >
> > > Actually, they are closer than you might think. quick_handler's purpose is to allow
> > > requests to be served as quickly as possible. It skips a lot of phases of the
> > > request to accomplish that. If map_to_storage is tweaked a bit, it can accomplish
> > > the same thing.
> >
> > Quick_handler bypasses everything. We ignore the client's headers, the server's
> > configuration, everything. Frankly, quick_handler is not Apache, and anyone who writes
> > a quick_handler hack and calls it a module for 'Apache' is full of sh*t. Powered by the
> > Apr/Apache MPM/load manager is more like it (that isn't badness, that's just reality.)
>
> So tell us how you -really- feel :-)
Let me fine-tune my statement. Modules that use quick_handler bypass the _user's_ expectations,
and our documented behavior. If this is ok for a given module, and is well documented, then fine.
I feel that nothing frustrates me more, personally, than to follow docs and observe a different
result than was doumented. That's why I turned Greg to the <Location >/<Limit > combinations
for DAV, since they have odd side-effects.
> I am looking at the map_to_storage hook to see how it works. Stay tuned. The
> quick_handler hook was named after (and is used by) Mike Abbott's quick cache, an -Apache-
> module. It is a specialized hook to serve a specialized purpose: serving web pages as fast
> as possible from a user space resident cache. And it can be used without compromising the
> security of the server.
The best example is loading the mod_proxy/proxy_http combination. I have to say that the patch
to mod_proxy was about as clear as I've ever written. It will give you a terrific feel for what
can be done :)
>From an optimization perspective, see the <Location > patch from last night, it shows what you
can do with pre-cached per_dir_config's, you would just need to use config wide pools and voila,
mod_file_cache can have preconstructed per_dir_configs too. I'll be modifying <Directory > to
also observe any preconstruction, so all internal_internal_redirect and subreq entities can go
back through directory_walk without consuming significant CPU.
Bill
Re: cvs commit: httpd-2.0/server core.c request.c
Posted by Bill Stoddard <bi...@wstoddard.com>.
> > > map_to_storage is definitely not the same as quick_handler. But
> > > quick_handler does have a use so it stays in.
> >
> > Actually, they are closer than you might think. quick_handler's purpose is to allow
> > requests to be served as quickly as possible. It skips a lot of phases of the
> > request to accomplish that. If map_to_storage is tweaked a bit, it can accomplish
> > the same thing.
>
> It's not going to be tweaked (you security sloths, you :) We need to look at each of
> the hook phases, and figure out how a module can selectively bypass that hook, when
> it should, and why. I'll probably start a paper detailing how the current Apache system
> exposes hooks, and what's the harm in using/bypassing those hooks.
>
> Quick_handler bypasses everything. We ignore the client's headers, the server's
> configuration, everything. Frankly, quick_handler is not Apache, and anyone who writes
> a quick_handler hack and calls it a module for 'Apache' is full of sh*t. Powered by the
> Apr/Apache MPM/load manager is more like it (that isn't badness, that's just reality.)
So tell us how you -really- feel :-)
I am looking at the map_to_storage hook to see how it works. Stay tuned. The
quick_handler hook was named after (and is used by) Mike Abbott's quick cache, an -Apache-
module. It is a specialized hook to serve a specialized purpose: serving web pages as fast
as possible from a user space resident cache. And it can be used without compromising the
security of the server.
Bill
Re: cvs commit: httpd-2.0/server core.c request.c
Posted by "William A. Rowe, Jr." <wr...@rowe-clan.net>.
From: "Ryan Bloom" <rb...@covalent.net>
Sent: Monday, August 27, 2001 8:41 AM
> > map_to_storage is definitely not the same as quick_handler. But
> > quick_handler does have a use so it stays in.
>
> Actually, they are closer than you might think. quick_handler's purpose is to allow
> requests to be served as quickly as possible. It skips a lot of phases of the
> request to accomplish that. If map_to_storage is tweaked a bit, it can accomplish
> the same thing.
It's not going to be tweaked (you security sloths, you :) We need to look at each of
the hook phases, and figure out how a module can selectively bypass that hook, when
it should, and why. I'll probably start a paper detailing how the current Apache system
exposes hooks, and what's the harm in using/bypassing those hooks.
Quick_handler bypasses everything. We ignore the client's headers, the server's
configuration, everything. Frankly, quick_handler is not Apache, and anyone who writes
a quick_handler hack and calls it a module for 'Apache' is full of sh*t. Powered by the
Apr/Apache MPM/load manager is more like it (that isn't badness, that's just reality.)
> Basically, the map_to_storage function that succeeds would tell the server how many
> of the hooks that follow map_to_storage that it should skip before the handler phase.
No... I thought long and hard about that, and I dislike it. It's full of opportunities
to mess up security and bypass things you didn't expect (in v. 2.1 or whenever.)
hook_to_storage (named wrong, we agree) is the trap to walk a set of dir_config's that are
customized for the module. 50% of the modules serve files, so they _don't_ hook this call.
40% of the modules invent content in a "yes or no" sort of way. They are generally driven
by the <Location > block, so that block can contain their access restrictions. They don't
want us to hook up any extra dir_config's, so they may hook this call and just return OK.
10% of the modules (e.g. proxy, perhaps a database-driven content store) don't have 'files'
in the traditional sense, but they want a tree mapping of restrictions, and it won't
correspond to our <Directory ><Files > sections in a 1:1 manner. For those modules, they
can hook and replace dir_walk/file_walk with their own (e.g. proxy_walk.)
> This has a couple of advantages over quick_handler, the first one being that it
> is more general, and the second that it means we go back to having one location that
> data is sent to the network.
Yes - it's definately a design decision that an author needs to be aware of. We aren't
working well enough with too many modules as it is (take mod_autoindex/negotation/mime
security reports, or tomcat security reports.) We don't want module authors to suffer
from a slow model. But we don't want them wandering too far out of the fold, until they
fall in the pit that so many modules have landed in.
> If that change is made to map_to_storage, then I would be with OtherBill, and
> suggest that quick_handler should be removed.
I'm about to suggest that as well, but I don't think we can do so until we optimize our
whole server.
That is, I just optimized the two-pass <Location > parsing. Unless the URI is remapped
(a significant context change that requires rewalking the location), or we have been mapped
to a virtual server with a different set of locations (they probably wern't inherited from
the base server), then we will do zero or one directory merges. Zero merges if no module
modified the per_dir_config since we left it (no dir/file/proxy walk), or one merge if we
see someone reset per_dir_config, so we will overlay it.
Almost every hook can be a bit more optimized this way. I'm most concerned about subreq
and internal redirect mechansims, and will continue to work to assure 1. they do the same
thing, even in different contexts, and 2. they rely on the original request as much as
possible to drop cpu/filesystem load.
An example ... dir_walk can do the same thing as location_walk does. If we already walked
the first 3 directory elements, and come back with a subreq for a subdir, why duplicate the
work? We have some great opportunities here for cleaner, smaller and faster code. Then
I'll suggest dropping quick_handler, and have a document to back up my position :)
Bill
Re: cvs commit: httpd-2.0/server core.c request.c
Posted by Ryan Bloom <rb...@covalent.net>.
On Monday 27 August 2001 06:22, Bill Stoddard wrote:
> > From: "Bill Stoddard" <bi...@wstoddard.com>
> > Sent: Sunday, August 26, 2001 11:48 AM
> >
> > > > On Sat, Aug 25, 2001 at 11:43:19PM -0000, wrowe@apache.org wrote:
> > > > > wrowe 01/08/25 16:43:19
> > > > >
> > > > > Modified: . CHANGES
> > > > > include http_request.h
> > > > > modules/http http_core.c http_protocol.c
> > > > > http_request.c mod_core.h
> > > > > server core.c request.c
> > > > > Log:
> > > > > Introduce the map_to_storage hook, which allows modules to
> > > > > bypass the directory_walk and file_walk for non-file requests.
> > > > > TRACE
> > > >
> > > > How is this different from the translate_name hook? Your new hook and
> > > > the translate_name are both about translating URLs into the backend
> > > > storage.
> > > >
> > > > Why was this new one added? (did I miss some discussion somewhere?)
> > > > And if so, then why does translate_name stick around?
> > >
> > > Sounds similar to the quick_handler hook as well.
> >
> > It does? Then you need to find a clue. quick_handler should have been
> > axed from the outset, and now that this handler allows a non-filesystem
> > request to be handled properly, it really should be gone.
>
> map_to_storage is definitely not the same as quick_handler. But
> quick_handler does have a use so it stays in.
Actually, they are closer than you might think. quick_handler's purpose is to allow
requests to be served as quickly as possible. It skips a lot of phases of the
request to accomplish that. If map_to_storage is tweaked a bit, it can accomplish
the same thing.
Basically, the map_to_storage function that succeeds would tell the server how many
of the hooks that follow map_to_storage that it should skip before the handler phase.
This has a couple of advantages over quick_handler, the first one being that it
is more general, and the second that it means we go back to having one location that
data is sent to the network.
If that change is made to map_to_storage, then I would be with OtherBill, and
suggest that quick_handler should be removed.
Ryan
______________________________________________________________
Ryan Bloom rbb@apache.org
Covalent Technologies rbb@covalent.net
--------------------------------------------------------------
Re: cvs commit: httpd-2.0/server core.c request.c
Posted by Bill Stoddard <bi...@wstoddard.com>.
> From: "Bill Stoddard" <bi...@wstoddard.com>
> Sent: Sunday, August 26, 2001 11:48 AM
>
>
> > > On Sat, Aug 25, 2001 at 11:43:19PM -0000, wrowe@apache.org wrote:
> > > > wrowe 01/08/25 16:43:19
> > > >
> > > > Modified: . CHANGES
> > > > include http_request.h
> > > > modules/http http_core.c http_protocol.c http_request.c
> > > > mod_core.h
> > > > server core.c request.c
> > > > Log:
> > > > Introduce the map_to_storage hook, which allows modules to bypass
> > > > the directory_walk and file_walk for non-file requests. TRACE
> > >
> > > How is this different from the translate_name hook? Your new hook and the
> > > translate_name are both about translating URLs into the backend storage.
> > >
> > > Why was this new one added? (did I miss some discussion somewhere?) And if
> > > so, then why does translate_name stick around?
> >
> > Sounds similar to the quick_handler hook as well.
>
> It does? Then you need to find a clue. quick_handler should have been axed
> from the outset, and now that this handler allows a non-filesystem request to
> be handled properly, it really should be gone.
map_to_storage is definitely not the same as quick_handler. But quick_handler does have a
use so it stays in.
Bill
Re: cvs commit: httpd-2.0/server core.c request.c
Posted by "William A. Rowe, Jr." <wr...@rowe-clan.net>.
From: "Bill Stoddard" <bi...@wstoddard.com>
Sent: Sunday, August 26, 2001 11:48 AM
> > On Sat, Aug 25, 2001 at 11:43:19PM -0000, wrowe@apache.org wrote:
> > > wrowe 01/08/25 16:43:19
> > >
> > > Modified: . CHANGES
> > > include http_request.h
> > > modules/http http_core.c http_protocol.c http_request.c
> > > mod_core.h
> > > server core.c request.c
> > > Log:
> > > Introduce the map_to_storage hook, which allows modules to bypass
> > > the directory_walk and file_walk for non-file requests. TRACE
> >
> > How is this different from the translate_name hook? Your new hook and the
> > translate_name are both about translating URLs into the backend storage.
> >
> > Why was this new one added? (did I miss some discussion somewhere?) And if
> > so, then why does translate_name stick around?
>
> Sounds similar to the quick_handler hook as well.
It does? Then you need to find a clue. quick_handler should have been axed
from the outset, and now that this handler allows a non-filesystem request to
be handled properly, it really should be gone.
Re: cvs commit: httpd-2.0/server core.c request.c
Posted by Bill Stoddard <bi...@wstoddard.com>.
> On Sat, Aug 25, 2001 at 11:43:19PM -0000, wrowe@apache.org wrote:
> > wrowe 01/08/25 16:43:19
> >
> > Modified: . CHANGES
> > include http_request.h
> > modules/http http_core.c http_protocol.c http_request.c
> > mod_core.h
> > server core.c request.c
> > Log:
> > Introduce the map_to_storage hook, which allows modules to bypass
> > the directory_walk and file_walk for non-file requests. TRACE
>
> How is this different from the translate_name hook? Your new hook and the
> translate_name are both about translating URLs into the backend storage.
>
> Why was this new one added? (did I miss some discussion somewhere?) And if
> so, then why does translate_name stick around?
Sounds similar to the quick_handler hook as well.
Bill
Re: cvs commit: httpd-2.0/server core.c request.c
Posted by Bill Stoddard <bi...@wstoddard.com>.
> On Sat, Aug 25, 2001 at 11:43:19PM -0000, wrowe@apache.org wrote:
> > wrowe 01/08/25 16:43:19
> >
> > Modified: . CHANGES
> > include http_request.h
> > modules/http http_core.c http_protocol.c http_request.c
> > mod_core.h
> > server core.c request.c
> > Log:
> > Introduce the map_to_storage hook, which allows modules to bypass
> > the directory_walk and file_walk for non-file requests. TRACE
>
> How is this different from the translate_name hook? Your new hook and the
> translate_name are both about translating URLs into the backend storage.
>
> Why was this new one added? (did I miss some discussion somewhere?) And if
> so, then why does translate_name stick around?
>
I have the same question. Check out mod_jk and how it uses the translate_name hook.
Bill
Re: cvs commit: httpd-2.0/server core.c request.c
Posted by "William A. Rowe, Jr." <wr...@rowe-clan.net>.
From: "Greg Stein" <gs...@lyra.org>
Sent: Sunday, August 26, 2001 4:04 AM
> On Sat, Aug 25, 2001 at 11:43:19PM -0000, wrowe@apache.org wrote:
> > wrowe 01/08/25 16:43:19
> >
> > Modified: . CHANGES
> > include http_request.h
> > modules/http http_core.c http_protocol.c http_request.c
> > mod_core.h
> > server core.c request.c
> > Log:
> > Introduce the map_to_storage hook, which allows modules to bypass
> > the directory_walk and file_walk for non-file requests. TRACE
>
> How is this different from the translate_name hook? Your new hook and the
> translate_name are both about translating URLs into the backend storage.
>
> Why was this new one added? (did I miss some discussion somewhere?) And if
> so, then why does translate_name stick around?
The original idea for the hook name was walkers (as in ap_run_walkers()).
map_to_storage actually validates that the translated name is serveable,
and sets up any storage specific per_dir_config values. It was also going
to pick up the downgrade http protocol, but that doesn't make sense in this
hook. I'll move that aspect to the bottom of the translate_name phase.
Perhaps this should be get_storage_dir_config or something like that.
Note it is run first, instead of run all (as the translate_name hook is),
so that _one_ module can assume ownership and pick back up the request
after the authz/authn phases.
This hook has enormous reprecussions on avoiding directory walks and file walks
that we have all complained so long about. The patches today to mod_proxy
(adding a proxy_walk phase, and skipping the old cruft) illustrate this. Now
that proxy and directory sections are listed seperately, we save a ton of tests
in both given a complex server configuration. dir walks <Dir > sections, and
proxy walks <Proxy > sections, period.
The last remaining hurdle outside of dir/file walking is the duplicate <Location >
walk we perform. I'm preparing a patch to do this walk once, and as long as the
r->uri is unchanged, we continue to merge the same 'preconstructed' location walk
every time we repeat that step.
I want to lock down the default handler as well, to error 500 if we didn't call
dir_walk/file_walk because a badly written module circumvented the core
map_to_storage hook fn. After that, I believe we will be as good as can be, with
some major cleanup needed to subreq/internal_internal_redirect handling.
Bill