You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Greg Ames <gr...@remulak.net> on 2002/04/02 00:39:20 UTC
2.0.34 - erratic behavior with autoindexes
I noticed something weird on daedalus on the test build. It looks like we're
not autoindexing directories with HEADER.html or README.html files correctly.
You see the contents of HEADER.html and/or README.html, but not the other files
or subdirs in the directory. Often, the Mozilla animation keeps going as if the
browser thinks more info is coming.
Examples of the failure:
http://www.apache.org:8092/dist/
http://www.apache.org:8092/dist/httpd/
http://jakarta.apache.org:8092/builds/
The second one gives me different results from IE5 when I hit refresh -
sometimes the end of the README.html appears to be missing; sometimes it
displays as plain text with the first part of the HEADER.html missing, the http
headers visable in the middle, followed by duplicate html.
But this one, which doesn't have a HEADER.html or README.html, works fine:
http://jakarta.apache.org:8092/builds/jakarta-tomcat/
(get rid of the :8092 to see how the production server behaves.)
I have band practice tonight so I'll be offline shortly. I'll leave 2.0.34 up
on daedalus in case anyone wants to check it out.
Greg
Re: 2.0.34 - erratic behavior with autoindexes
Posted by Justin Erenkrantz <je...@apache.org>.
On Tue, Apr 02, 2002 at 04:28:31PM -0500, Jeff Trawick wrote:
> maybe this helps... haven't tested yet
I believe we took that out specifically to ensure that any
requests that may be promoted to a real request do not have
the subreq filter. IIRC, if the subreq is being created with
a NULL parent request, then it may be "promoted" and will not
be served as a subreq. HTH. -- justin
RE: 2.0.34 - erratic behavior with autoindexes
Posted by Ryan Bloom <rb...@covalent.net>.
> >The core can't do anything about it. If you are creating a
sub-request,
> >you must flush the data. We could have the core always flush, but
that
> >would cause us to send flush buckets before any data has been
generated,
> >or in cases where a flush is not required. The module MUST be the
thing
> >that does the flush.
>
> Of course it can... why not simply have ap_run_sub_req automatically
flush
> the rwrite'n contents of the main request upon invocation, and
> automatically
> flush the rwritten contents of the subrequest upon completion?
>
> Then all is transparent
That is what I meant by "We could have the core always flush....". As I
said, that would mean that we are flushing when we don't need too, and
we are potentially sending small chunks of data to the socket when we
don't need to. It is and should be up to the module to ensure that data
is sent to the client when it is supposed to be.
Ryan
RE: 2.0.34 - erratic behavior with autoindexes
Posted by "William A. Rowe, Jr." <wr...@rowe-clan.net>.
At 04:27 PM 4/2/2002, you wrote:
>The core can't do anything about it. If you are creating a sub-request,
>you must flush the data. We could have the core always flush, but that
>would cause us to send flush buckets before any data has been generated,
>or in cases where a flush is not required. The module MUST be the thing
>that does the flush.
Of course it can... why not simply have ap_run_sub_req automatically flush
the rwrite'n contents of the main request upon invocation, and automatically
flush the rwritten contents of the subrequest upon completion?
Then all is transparent
RE: 2.0.34 - erratic behavior with autoindexes
Posted by Ryan Bloom <rb...@covalent.net>.
Okay, I have had more time to investigate the problem. This is kind of
strange. First of all, I have included a patch with this post that is
absolutely 100% wrong. The only purpose of the patch is to show the
problem, and hopefully prove that I have isolated the problem correctly.
I wasn't able to reproduce the exact problem, but I traced the problem,
and I think I saw the cause, even if I didn't see the symptoms.
The problem is that we have a main request that does NOT have the
OLD_WRITE filter. Then we create the sub-request, which means that the
SUB_REQ_EOS filter is added without the OLD_WRITE filter. Once we have
created the sub-request (notice we haven't run it yet), we write to the
next filter using ap_r* functions. This will insert the OLD_WRITE
filter, and the data will be buffered in the filter. Finally, we
actually run the sub-request.
The important thing here, is that we have never actually put the
OLD_WRITE filter into the sub-request, because it was added after we
created the sub-request. This means that all of the data from the
sub-request is written BEFORE the pre-amble. Because of that, we get
VERY invalid HTML.
The patch below forces mod_autoindex to insert the OLD_WRITE filter
before the sub-request is created. As I have said, this is the WRONG
solution. I have tested it at home, and it does force things to be
written in the correct order. My hope is that this patch can be tested
on daedalus to ensure that it will work to solve the problem we have
seen.
_IF_ this does work, then I can think of three possible solutions to fix
the problem for real.
1) Make sure we write the pre-amble before we create the sub-request.
The logic is possible, but it is kind of ugly. I definitely won't have
time to write that patch until Thursday night, so I hope somebody else
can beat me to it.
2) The second option is to modify ap_run_sub_req(), so that it detects
this case and inserts the OLD_WRITE filter for the sub-request. This
will be exceedingly ugly, and I would prefer to not use this method.
3) Modify mod_autoindex to not use ap_r*. This doesn't solve the
underlying problem, but it does remove it from the release.
Ryan
Index: modules/generators/mod_autoindex.c
===================================================================
RCS file: /home/cvs/httpd-2.0/modules/generators/mod_autoindex.c,v
retrieving revision 1.102
diff -u -d -b -w -u -r1.102 mod_autoindex.c
--- modules/generators/mod_autoindex.c 20 Mar 2002 17:41:54 -0000
1.102
+++ modules/generators/mod_autoindex.c 3 Apr 2002 07:12:53 -0000
@@ -980,6 +980,9 @@
* instead of a text document, meaning nothing will be displayed, but
* oh well.
*/
+typedef struct {
+ apr_bucket_brigade *bb;
+} old_write_filter_ctx;
static void emit_head(request_rec *r, char *header_fname, int
suppress_amble,
char *title)
{
@@ -990,6 +993,7 @@
int emit_H1 = 1;
const char *r_accept;
const char *r_accept_enc;
+ old_write_filter_ctx *ctx;
/*
* If there's a header file, send a subrequest to look for it. If
it's
@@ -1003,6 +1007,9 @@
if ((header_fname != NULL) && r->args) {
header_fname = apr_pstrcat(r->pool, header_fname, "?", r->args,
NULL);
}
+
+ ctx = apr_pcalloc(r->pool, sizeof(*ctx));
+ ap_add_output_filter("OLD_WRITE", ctx, r, r->connection);
if ((header_fname != NULL)
&& (rr = ap_sub_req_lookup_uri(header_fname, r,
r->output_filters))
----------------------------------------------
Ryan Bloom rbb@covalent.net
645 Howard St. rbb@apache.org
San Francisco, CA
RE: 2.0.34 - erratic behavior with autoindexes ( I believe the bug is analyzed in this message)
Posted by Ryan Bloom <rb...@covalent.net>.
> > > > We are talking about the main request flushing its
> > > > buffers before it runs the request.
> > >
> > > Um. How'd the buffers get filled before running the request?
> >
> > The problem is that the main request generates content, and that
goes
> > into the OLD_WRITE filter buffer. Then it creates a sub-request.
If
> > the sub-request doesn't use the same OLD_WRITE instance, you have a
> > major bug.
>
> Huh? No...
>
> Let's say the main req created an OLD_WRITE and put some content in
there.
> Next, the sub-request inserts its own OLD_WRITE instance (which it
won't,
> but let's run with this). The subreq then starts jamming content into
its
> own OLD_WRITE. When the subreq goes away, an EOS will hit the
OLD_WRITE in
> the sub_req, which will flush the content on up to the main request.
The
> main request's OLD_WRITE will prepend its data, and continue flushing
the
> data down the pipe.
>
> Now, the SUBREQ filter actually intercepts that EOS, but it does pass
the
> data along. The OLD_WRITEs are emptying their buffers properly.
>
> Now, the real situation is that a second OLD_WRITE won't be added.
When
> you
> call ap_r*(), it will scan the entire chain looking for an existing
> OLD_WRITE to be present. If it is there, then another won't be added.
If
> the
> filter is not at the top, it won't be used.
>
> And lastly, remember the ordering of the filters. Even if the subreq
> inserts
> stuff, the OLD_WRITE should always remain at the "top".
No. The filters are ordered on a per-request basis, so even if the main
request has the OLD_WRITE filter, sub-request filters won't be in filter
chain below that OLD_WRITE filter.
A graphic might help:
If I have a request with the filter chain:
OLD_WRITE -> INCLUDES -> byterange -> C-L -> CORE
There are two ways to create the sub-request.
1) Point the sub-request at the filter stack, which should produce:
(THIS IS WHERE THE BUG IS HAPPENING!!!!!!!)
SUB_REQ_EOS -> OLD_WRITE -> INCLUDES -> byterange -> C-L -> CORE
Any other filters for the sub-request will be inserted before the
SUB_REQ_EOS filter.
However, what is happening instead, is that filters are being added
like:
OLD_WRITE -> SUB_REQ_EOS -> INCLUDES -> byterange -> C-L -> CORE
The problem with this, is that the sub-request->output_filters pointer
is pointing to SUB_REQ_EOS, not to OLD_WRITE, which means that the
OLD_WRITE filter is being missed entirely for the sub-request.
2) Don't point the sub-request at the filter stack: (All RESOURCE and
CONTENT_SET filters are removed from the sub-request)
SUB_REQ_EOS -> byterange -> C-L -> CORE
I hope that explains it. I am hoping to look at this in a lot more
detail later tonight.
Ryan
Re: 2.0.34 - erratic behavior with autoindexes
Posted by Greg Stein <gs...@lyra.org>.
On Tue, Apr 02, 2002 at 03:30:07PM -0800, Ryan Bloom wrote:
>...
> > > We are talking about the main request flushing its
> > > buffers before it runs the request.
> >
> > Um. How'd the buffers get filled before running the request?
>
> The problem is that the main request generates content, and that goes
> into the OLD_WRITE filter buffer. Then it creates a sub-request. If
> the sub-request doesn't use the same OLD_WRITE instance, you have a
> major bug.
Huh? No...
Let's say the main req created an OLD_WRITE and put some content in there.
Next, the sub-request inserts its own OLD_WRITE instance (which it won't,
but let's run with this). The subreq then starts jamming content into its
own OLD_WRITE. When the subreq goes away, an EOS will hit the OLD_WRITE in
the sub_req, which will flush the content on up to the main request. The
main request's OLD_WRITE will prepend its data, and continue flushing the
data down the pipe.
Now, the SUBREQ filter actually intercepts that EOS, but it does pass the
data along. The OLD_WRITEs are emptying their buffers properly.
Now, the real situation is that a second OLD_WRITE won't be added. When you
call ap_r*(), it will scan the entire chain looking for an existing
OLD_WRITE to be present. If it is there, then another won't be added. If the
filter is not at the top, it won't be used.
And lastly, remember the ordering of the filters. Even if the subreq inserts
stuff, the OLD_WRITE should always remain at the "top".
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
RE: 2.0.34 - erratic behavior with autoindexes
Posted by Ryan Bloom <rb...@covalent.net>.
> > > > The core can't do anything about it. If you are creating a
> > sub-request,
> > > > you must flush the data.
> > >
> > > Bing!
> > >
> > > ANY filter inserted by the subreq could be holding content. The
> > subrequest
> > > definitely MUST flush out its filter stack before going away.
> > >
> > > Yes, old_write buffers, and if that is "attached" to the
subrequest,
> > then
> > > it
> > > must be flushed. But it really isn't about old_write.
> >
> > We aren't talking about the sub-request flushing its buffers at the
end
> > of the request. That happens automatically, because the sub-request
> > must send an EOS.
>
> Right.
>
> And ignore my comment about the new bucket type for a subreq ending. I
> forgot about the EOS thing.
>
> > We are talking about the main request flushing its
> > buffers before it runs the request.
>
> Um. How'd the buffers get filled before running the request?
The problem is that the main request generates content, and that goes
into the OLD_WRITE filter buffer. Then it creates a sub-request. If
the sub-request doesn't use the same OLD_WRITE instance, you have a
major bug.
Ryan
Re: 2.0.34 - erratic behavior with autoindexes
Posted by Greg Stein <gs...@lyra.org>.
On Tue, Apr 02, 2002 at 02:58:11PM -0800, Ryan Bloom wrote:
> > > The core can't do anything about it. If you are creating a
> sub-request,
> > > you must flush the data.
> >
> > Bing!
> >
> > ANY filter inserted by the subreq could be holding content. The
> subrequest
> > definitely MUST flush out its filter stack before going away.
> >
> > Yes, old_write buffers, and if that is "attached" to the subrequest,
> then
> > it
> > must be flushed. But it really isn't about old_write.
>
> We aren't talking about the sub-request flushing its buffers at the end
> of the request. That happens automatically, because the sub-request
> must send an EOS.
Right.
And ignore my comment about the new bucket type for a subreq ending. I
forgot about the EOS thing.
> We are talking about the main request flushing its
> buffers before it runs the request.
Um. How'd the buffers get filled before running the request?
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
RE: 2.0.34 - erratic behavior with autoindexes
Posted by Ryan Bloom <rb...@covalent.net>.
> > The core can't do anything about it. If you are creating a
sub-request,
> > you must flush the data.
>
> Bing!
>
> ANY filter inserted by the subreq could be holding content. The
subrequest
> definitely MUST flush out its filter stack before going away.
>
> Yes, old_write buffers, and if that is "attached" to the subrequest,
then
> it
> must be flushed. But it really isn't about old_write.
We aren't talking about the sub-request flushing its buffers at the end
of the request. That happens automatically, because the sub-request
must send an EOS. We are talking about the main request flushing its
buffers before it runs the request.
Ryan
Re: 2.0.34 - erratic behavior with autoindexes
Posted by Greg Stein <gs...@lyra.org>.
On Tue, Apr 02, 2002 at 02:27:17PM -0800, Ryan Bloom wrote:
>...
> > So it looks like we have to do something about flushing the data that
> > old_write is hanging on to. I'd rather see the core figure out how to
> > do this, rather than mod_autoindex. Otherwise we are going to impact
> > 1.3 modules that use ap_r* more than we need to.
Per my previous email, I seriously doubt that old_write is posing a problem
here. Yes, it might have buffered content, but per Ryan's point:
> The core can't do anything about it. If you are creating a sub-request,
> you must flush the data.
Bing!
ANY filter inserted by the subreq could be holding content. The subrequest
definitely MUST flush out its filter stack before going away.
Yes, old_write buffers, and if that is "attached" to the subrequest, then it
must be flushed. But it really isn't about old_write.
> We could have the core always flush, but that
> would cause us to send flush buckets before any data has been generated,
> or in cases where a flush is not required. The module MUST be the thing
> that does the flush.
I think the subrequest would need to flush its filter chain before going
away. Has nothing to do with the modules, since they don't know they are
operating as part of a subreq filter chain.
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
Re: 2.0.34 - erratic behavior with autoindexes
Posted by Greg Ames <gr...@remulak.net>.
Ryan Bloom wrote:
> Did you try just http://httpd.apache.org:8092?
not until now. Thanks for the extra eyeballs.
> It looks like that patch completely broke DirectoryIndex pages.
It's definitely hosed. But you and Justin both said something about subrequests
which get promoted. That's a new concept to me. I'm not surprised that we
would need different logic for promoted and non-promoted subreqs.
Greg
RE: 2.0.34 - erratic behavior with autoindexes
Posted by Ryan Bloom <rb...@covalent.net>.
> -----Original Message-----
> From: gregames@Mail.MeepZor.Com [mailto:gregames@Mail.MeepZor.Com] On
> Behalf Of Greg Ames
> Sent: Tuesday, April 02, 2002 2:35 PM
> To: dev@httpd.apache.org
> Subject: Re: 2.0.34 - erratic behavior with autoindexes
>
> Greg Ames wrote:
> >
> > Jeff Trawick wrote:
> >
> > > > Here's a backtrace while processing /dist/httpd/. Where is the
> > > > subrequest filter?
> > >
> > > maybe this helps... haven't tested yet
> > >
> > > Index: request.c
> >
> > yep, this solves about half of the problem. With this on, I now see
all
> the
> > data, it's just in the wrong order.
>
> I put Jeff's patch on the 2.0.34 test server on daedalus and bounced
it.
> http://www.apache.org:8092/dist/ looks perfect, since it only has a
> HEADER.html. You can see the out-of-order data at
> http://www.apache.org:8092/dist/httpd/ (that dir has both a
HEADER.html
> and
> README.html).
Did you try just http://httpd.apache.org:8092? It looks like that patch
completely broke DirectoryIndex pages.
Ryan
Re: 2.0.34 - erratic behavior with autoindexes
Posted by Greg Ames <gr...@remulak.net>.
Greg Ames wrote:
>
> Jeff Trawick wrote:
>
> > > Here's a backtrace while processing /dist/httpd/. Where is the
> > > subrequest filter?
> >
> > maybe this helps... haven't tested yet
> >
> > Index: request.c
>
> yep, this solves about half of the problem. With this on, I now see all the
> data, it's just in the wrong order.
I put Jeff's patch on the 2.0.34 test server on daedalus and bounced it.
http://www.apache.org:8092/dist/ looks perfect, since it only has a
HEADER.html. You can see the out-of-order data at
http://www.apache.org:8092/dist/httpd/ (that dir has both a HEADER.html and
README.html).
Greg
RE: 2.0.34 - erratic behavior with autoindexes
Posted by Ryan Bloom <rb...@covalent.net>.
> Jeff Trawick wrote:
>
> > > Here's a backtrace while processing /dist/httpd/. Where is the
> > > subrequest filter?
> >
> > maybe this helps... haven't tested yet
> >
> > Index: request.c
>
> yep, this solves about half of the problem. With this on, I now see
all
> the
> data, it's just in the wrong order. I see the contents of
HEADER.html,
> followed
> by the contents of README.html, followed by the dirent stuff that
> handle_autoindex generates dynamically. The dynamic stuff should be
in
> the
> middle.
>
> So it looks like we have to do something about flushing the data that
> old_write
> is hanging on to. I'd rather see the core figure out how to do this,
> rather
> than mod_autoindex. Otherwise we are going to impact 1.3 modules that
use
> ap_r*
> more than we need to.
The core can't do anything about it. If you are creating a sub-request,
you must flush the data. We could have the core always flush, but that
would cause us to send flush buckets before any data has been generated,
or in cases where a flush is not required. The module MUST be the thing
that does the flush.
As for the patch that did solve the first part of the problem, this is
likely to cause problems if the sub-request is promoted to the main
request, as is the case with DirectoryIndex. We need to add logic to
handle that case.
Ryan
Re: 2.0.34 - erratic behavior with autoindexes
Posted by Greg Ames <gr...@remulak.net>.
Jeff Trawick wrote:
> > Here's a backtrace while processing /dist/httpd/. Where is the
> > subrequest filter?
>
> maybe this helps... haven't tested yet
>
> Index: request.c
yep, this solves about half of the problem. With this on, I now see all the
data, it's just in the wrong order. I see the contents of HEADER.html, followed
by the contents of README.html, followed by the dirent stuff that
handle_autoindex generates dynamically. The dynamic stuff should be in the
middle.
So it looks like we have to do something about flushing the data that old_write
is hanging on to. I'd rather see the core figure out how to do this, rather
than mod_autoindex. Otherwise we are going to impact 1.3 modules that use ap_r*
more than we need to.
Greg
Re: 2.0.34 - erratic behavior with autoindexes
Posted by Jeff Trawick <tr...@attglobal.net>.
Jeff Trawick <tr...@attglobal.net> writes:
> Greg Ames <gr...@remulak.net> writes:
>
> > I noticed something weird on daedalus on the test build. It looks like we're
> > not autoindexing directories with HEADER.html or README.html files correctly.
> > You see the contents of HEADER.html and/or README.html, but not the other files
> > or subdirs in the directory. Often, the Mozilla animation keeps going as if the
> > browser thinks more info is coming.
>
> FYI... ripping out INCLUDES doesn't seem to make a difference
> (simplification of the problem :) ).
>
> Here's a backtrace while processing /dist/httpd/. Where is the
> subrequest filter?
maybe this helps... haven't tested yet
Index: request.c
===================================================================
RCS file: /home/cvs/httpd-2.0/server/request.c,v
retrieving revision 1.108
diff -u -r1.108 request.c
--- request.c 27 Mar 2002 22:42:16 -0000 1.108
+++ request.c 2 Apr 2002 21:30:33 -0000
@@ -1535,8 +1535,6 @@
rnew->proto_input_filters = r->proto_input_filters;
rnew->output_filters = next_filter;
rnew->proto_output_filters = r->proto_output_filters;
- ap_add_output_filter_handle(ap_subreq_core_filter_handle,
- NULL, rnew, rnew->connection);
}
else {
/* If NULL - we are expecting to be internal_fast_redirect'ed
@@ -1550,6 +1548,9 @@
rnew->input_filters = r->proto_input_filters;
rnew->output_filters = r->proto_output_filters;
}
+
+ ap_add_output_filter_handle(ap_subreq_core_filter_handle,
+ NULL, rnew, rnew->connection);
/* no input filters for a subrequest */
--
Jeff Trawick | trawick@attglobal.net
Born in Roswell... married an alien...
Re: 2.0.34 - erratic behavior with autoindexes
Posted by Jeff Trawick <tr...@attglobal.net>.
Greg Ames <gr...@remulak.net> writes:
> I noticed something weird on daedalus on the test build. It looks like we're
> not autoindexing directories with HEADER.html or README.html files correctly.
> You see the contents of HEADER.html and/or README.html, but not the other files
> or subdirs in the directory. Often, the Mozilla animation keeps going as if the
> browser thinks more info is coming.
FYI... ripping out INCLUDES doesn't seem to make a difference
(simplification of the problem :) ).
Here's a backtrace while processing /dist/httpd/. Where is the
subrequest filter? Why have we made it past
ap_content_length_filter()? One result is that the content length is
busted. I think the EOS for the subrequest isn't getting eaten and is
fouling up things too.
(or maybe I forgot *everything* about how filters work :) )
(gdb) where
#0 core_output_filter (f=0x812b3d8, b=0x812b5d8) at core.c:3579
#1 0x8070be9 in ap_pass_brigade (next=0x812b3d8, bb=0x81406d0) at util_filter.c:534
#2 0x8060d3e in ap_http_header_filter (f=0x8133808, b=0x81406d0) at http_protocol.c:1472
#3 0x8070be9 in ap_pass_brigade (next=0x8133808, bb=0x81406d0) at util_filter.c:534
#4 0x80731b3 in ap_content_length_filter (f=0x81337f0, b=0x81406d0) at protocol.c:1263
#5 0x8070be9 in ap_pass_brigade (next=0x81337f0, bb=0x81406d0) at util_filter.c:534
#6 0x8062e25 in ap_byterange_filter (f=0x81337d8, bb=0x81406d0) at http_protocol.c:2758
#7 0x8070be9 in ap_pass_brigade (next=0x81337d8, bb=0x81406d0) at util_filter.c:534
#8 0x8077eda in default_handler (r=0x8141048) at core.c:3247
#9 0x8065e37 in ap_run_handler (r=0x8141048) at config.c:193
#10 0x80663d7 in ap_invoke_handler (r=0x8141048) at config.c:373
#11 0x807be70 in ap_run_sub_req (r=0x8141048) at request.c:1867
#12 0x281ebca0 in emit_head (r=0x8133048, header_fname=0x80e4b28 "HEADER", suppress_amble=0,
title=0x813eb08 "/dist/httpd") at mod_autoindex.c:1032
#13 0x281edcc4 in index_directory (r=0x8133048, autoindex_conf=0x813d4c0) at mod_autoindex.c:2014
#14 0x281ee006 in handle_autoindex (r=0x8133048) at mod_autoindex.c:2113
#15 0x8065e37 in ap_run_handler (r=0x8133048) at config.c:193
#16 0x80663d7 in ap_invoke_handler (r=0x8133048) at config.c:373
#17 0x80633f3 in ap_process_request (r=0x8133048) at http_request.c:261
#18 0x805e9db in ap_process_http_connection (c=0x812b120) at http_core.c:291
#19 0x806ec3b in ap_run_process_connection (c=0x812b120) at connection.c:85
#20 0x806ef50 in ap_process_connection (c=0x812b120, csd=0x812b048) at connection.c:207
#21 0x8064854 in child_main (child_num_arg=0) at prefork.c:671
#22 0x80649be in make_child (s=0x809dd60, slot=0) at prefork.c:765
#23 0x8064a2b in startup_children (number_to_start=1) at prefork.c:783
#24 0x8064d9a in ap_mpm_run (_pconf=0x809c010, plog=0x80c6010, s=0x809dd60) at prefork.c:998
#25 0x806a825 in main (argc=3, argv=0xbfbffbac) at main.c:618
--
Jeff Trawick | trawick@attglobal.net
Born in Roswell... married an alien...
Re: OLD_WRITE stuff (was: Re: 2.0.34 - erratic behavior with autoindexes)
Posted by Greg Stein <gs...@lyra.org>.
On Tue, Apr 02, 2002 at 02:56:33PM -0800, Ryan Bloom wrote:
>
> Everything below is true, assuming you are not talking about
> sub-requests. According to the stack that Greg posted, the OLD_WRITE
> filter was not found in the sub-request filter stack. That is to be
> expected, because the OLD_WRITE filter is a RESOURCE filter. Since
> RESOURCE filters are not kept for sub-requests, the data must be
> flushed.
Right.
> Hmmmmm.... Actually it seems like Resource filters should be kept for
> sub-requests that are created with a filter list. If that is the case,
> then it looks like the OLD_WRITE filter should be in the filter list.
If the sub-request goes, then its resource filters should go. Those filters
are associated with that particular sub-request. That implies that the
resource filters should be flushed before they die.
[ but we should probably have a SUBREQ_ENDING_FLUSH bucket so the SUBREQ
filter can chew it up; we don't want to flush all the way to the network
just cuz the subreq is disappearing. ]
> Unless, the OLD_WRITE filter is being added after the sub-request is
> created......
The filter is added at the first call to ap_r*(). We decided on the lazy
approach so that pure-brigade systems would not suffer the overhead.
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
RE: OLD_WRITE stuff (was: Re: 2.0.34 - erratic behavior with autoindexes)
Posted by Ryan Bloom <rb...@covalent.net>.
Everything below is true, assuming you are not talking about
sub-requests. According to the stack that Greg posted, the OLD_WRITE
filter was not found in the sub-request filter stack. That is to be
expected, because the OLD_WRITE filter is a RESOURCE filter. Since
RESOURCE filters are not kept for sub-requests, the data must be
flushed.
Hmmmmm.... Actually it seems like Resource filters should be kept for
sub-requests that are created with a filter list. If that is the case,
then it looks like the OLD_WRITE filter should be in the filter list.
Unless, the OLD_WRITE filter is being added after the sub-request is
created......
Lots to look at......
Ryan
> On Tue, Apr 02, 2002 at 01:50:59PM -0800, Ryan Bloom wrote:
> >...
> > > chain. Bill S. stuck his head in here and said something about a
rule
> > > that if old_write is ever used, it has to always be used.
> >
> > If you read through mod_include, you will see that whenever we
create a
> > sub-request, we send the output from the current filter to the next
> > filter. This is required to make sure that things are output in the
> > correct order. If you are a handler and are generating data through
> > ap_r* functions, you must flush the filter stack so that you are
sure
> > that OLD_WRITE isn't buffering for you.
>
> That's not true. OLD_WRITE was designed to allow a mix/match of
writing to
> the stack *and* using the ap_r* functions.
>
> The design is that ap_r* implies a write into the filter stack. That
write
> is optimized to use a brigade for buffering (so every ap_rputc()
doesn't
> traverse the filter stack). If OLD_WRITE isn't at the top, then it
doesn't
> do the buffering (because there are filters between the top and the
> OLD_WRITE buffers).
>
> If somebody writes directly into the filter stack, then it prepends
any
> buffered content to whatever is written, and then passes it all down
the
> stack.
>
> [ just reviewed the code; it seems to still match the original design
]
>
> It all looks pretty good, but there might be a problem where:
>
> *) ap_r* is used, so OLD_WRITE is inserted, and the content is stored
into
> its buffer.
>
> *) another filter is inserted "above" OLD_WRITE, so it is not the top.
> ap_r*
> content should now go into this new filter.
>
> *) ap_r* is called again, but it thinks it cannot buffer. it passes
the
> new
> data directly into the filter stack.
>
> BUG: the previously-buffered content is not prepended.
>
>
> So... the bug might appear because of the increased dynamicity of the
> filter
> chain nowadays. Back when OLD_WRITE was written, it was perceived that
the
> chain would be quite static by the time the first write occurred.
>
> Note that OLD_WRITE tries quite hard to keep its filter on "top" (its
> filter
> type is RESOURCE-10). If something broke that, then it is possible to
hit
> the above bug.
>
> The fix to the (potential) bug is to change line 1370 in
server/protocol.c
> to concat the brigades, similar to ap_old_write_filter() (defined just
> above, at line 1316).
>
> Another question is whether anybody defines their filter type as less
than
> RESOURCE-10 and ends up ahead of OLD_WRITE.
>
> And lastly: I just realized that if a filter gets in front of
OLD_WRITE,
> then the branch at line 1366 will get called for each ap_r* call. If
that
> is
> a series of 1000's of ap_rputc() calls, then you'll end up creating
> thousands of brigades for those transient buckets(!).
>
> [ a similar brigade over-creation can occur if you alternate ap_r* and
> writing to the filter stack; ap_old_write_filter() will forget about
the
> brigade it created, so the next ap_r* needs to recreate one ]
>
> The answer might be to test for brigade-empty rather than ctx->bb ==
NULL.
> Then the branch at 1366 can store and reuse a brigade in ctx->bb.
>
> Cheers,
> -g
OLD_WRITE stuff (was: Re: 2.0.34 - erratic behavior with autoindexes)
Posted by Greg Stein <gs...@lyra.org>.
On Tue, Apr 02, 2002 at 01:50:59PM -0800, Ryan Bloom wrote:
>...
> > chain. Bill S. stuck his head in here and said something about a rule
> > that if old_write is ever used, it has to always be used.
>
> If you read through mod_include, you will see that whenever we create a
> sub-request, we send the output from the current filter to the next
> filter. This is required to make sure that things are output in the
> correct order. If you are a handler and are generating data through
> ap_r* functions, you must flush the filter stack so that you are sure
> that OLD_WRITE isn't buffering for you.
That's not true. OLD_WRITE was designed to allow a mix/match of writing to
the stack *and* using the ap_r* functions.
The design is that ap_r* implies a write into the filter stack. That write
is optimized to use a brigade for buffering (so every ap_rputc() doesn't
traverse the filter stack). If OLD_WRITE isn't at the top, then it doesn't
do the buffering (because there are filters between the top and the
OLD_WRITE buffers).
If somebody writes directly into the filter stack, then it prepends any
buffered content to whatever is written, and then passes it all down the
stack.
[ just reviewed the code; it seems to still match the original design ]
It all looks pretty good, but there might be a problem where:
*) ap_r* is used, so OLD_WRITE is inserted, and the content is stored into
its buffer.
*) another filter is inserted "above" OLD_WRITE, so it is not the top. ap_r*
content should now go into this new filter.
*) ap_r* is called again, but it thinks it cannot buffer. it passes the new
data directly into the filter stack.
BUG: the previously-buffered content is not prepended.
So... the bug might appear because of the increased dynamicity of the filter
chain nowadays. Back when OLD_WRITE was written, it was perceived that the
chain would be quite static by the time the first write occurred.
Note that OLD_WRITE tries quite hard to keep its filter on "top" (its filter
type is RESOURCE-10). If something broke that, then it is possible to hit
the above bug.
The fix to the (potential) bug is to change line 1370 in server/protocol.c
to concat the brigades, similar to ap_old_write_filter() (defined just
above, at line 1316).
Another question is whether anybody defines their filter type as less than
RESOURCE-10 and ends up ahead of OLD_WRITE.
And lastly: I just realized that if a filter gets in front of OLD_WRITE,
then the branch at line 1366 will get called for each ap_r* call. If that is
a series of 1000's of ap_rputc() calls, then you'll end up creating
thousands of brigades for those transient buckets(!).
[ a similar brigade over-creation can occur if you alternate ap_r* and
writing to the filter stack; ap_old_write_filter() will forget about the
brigade it created, so the next ap_r* needs to recreate one ]
The answer might be to test for brigade-empty rather than ctx->bb == NULL.
Then the branch at 1366 can store and reuse a brigade in ctx->bb.
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
RE: 2.0.34 - erratic behavior with autoindexes
Posted by Ryan Bloom <rb...@covalent.net>.
> In handle_autoindex, we are working on the main request upon entry.
> r->output_filters looks like:
>
> old_write
> byterange
> content_length
> http_header
> core
>
> When it sees that HEADER.html exists, it apparently fires off a
subrequest
> (can't say for sure, I hit "n" in gdb rather than "s" at the wrong
time :-
> ( ).
> That finishes, and index_directory starts reading the directory
entries
> and
> creating the html fragments for them. This all seems fine. It writes
> them with
> ap_rputs, which should buffer the little pieces of data in old_write.
>
> Once it gets done with the normal files and subdirs, it finds
README.html,
> and
> definitely calls
> ap_run_sub_rec for that (I hit "s" this time :-). When we get to
> default_handler, r->output_filters looks like:
>
> includes
> byterange
> content_length
> http_header
> core
>
> The includes filter is using the subrequest's r; byterange, c/l, and
> http_header
> are using the main request's r; core has a null r.
>
> If you've been following this list for a while, you probably know that
> things
> are going to get funky real quick. AFAIK old_write is still buffering
all
> the
> data generated by the main request when the trailer file is sent down
the
> filter
> chain. Bill S. stuck his head in here and said something about a rule
> that if
> old_write is ever used, it has to always be used.
If you read through mod_include, you will see that whenever we create a
sub-request, we send the output from the current filter to the next
filter. This is required to make sure that things are output in the
correct order. If you are a handler and are generating data through
ap_r* functions, you must flush the filter stack so that you are sure
that OLD_WRITE isn't buffering for you.
This will solve SOME of the problem.
> * stepping thru the http_header filter on the README/trailer
subrequest
> was
> interesting. It sure looks like that bad boy is generating headers
during
> the
> subrequest. It has a debug assert(!r->main). It doesn't hit because
the
> r came
> from the filter structure, and it's the main request's r. It does
take
> itself
> out of the filter chain after it is done with the trailer subrequest.
The http_header filter is never called for the sub-request, because it
is only added for the main request.
> * what happened to the subrequest filter? what is going to stop
> subrequest EOS
> buckets from really flushing data to the network prematurely?
This is the thing that is most worrying. Without that filter,
everything is going to hell in a hand-basket quickly.
Ryan
Re: 2.0.34 - erratic behavior with autoindexes
Posted by Greg Ames <gr...@remulak.net>.
Greg Ames wrote:
>
> I noticed something weird on daedalus on the test build. It looks like we're
> not autoindexing directories with HEADER.html or README.html files correctly.
> You see the contents of HEADER.html and/or README.html, but not the other files
> or subdirs in the directory. Often, the Mozilla animation keeps going as if the
> browser thinks more info is coming.
>
> Examples of the failure:
>
> http://www.apache.org:8092/dist/
> http://www.apache.org:8092/dist/httpd/
> http://jakarta.apache.org:8092/builds/
OK, the same bad behavior happens on my ThinkPad with HEAD, so I've been
slogging thru it with gdb. I suspect the cause is the change the re-classified
the filters and changed when they are added etc.
In handle_autoindex, we are working on the main request upon entry.
r->output_filters looks like:
old_write
byterange
content_length
http_header
core
When it sees that HEADER.html exists, it apparently fires off a subrequest
(can't say for sure, I hit "n" in gdb rather than "s" at the wrong time :-( ).
That finishes, and index_directory starts reading the directory entries and
creating the html fragments for them. This all seems fine. It writes them with
ap_rputs, which should buffer the little pieces of data in old_write.
Once it gets done with the normal files and subdirs, it finds README.html, and
definitely calls
ap_run_sub_rec for that (I hit "s" this time :-). When we get to
default_handler, r->output_filters looks like:
includes
byterange
content_length
http_header
core
The includes filter is using the subrequest's r; byterange, c/l, and http_header
are using the main request's r; core has a null r.
If you've been following this list for a while, you probably know that things
are going to get funky real quick. AFAIK old_write is still buffering all the
data generated by the main request when the trailer file is sent down the filter
chain. Bill S. stuck his head in here and said something about a rule that if
old_write is ever used, it has to always be used.
Other things that may or may not have a bearing:
* the includes filter parses the HEADER and README files, so the subrequest
FILE buckets get morphed into MMAP buckets.
* stepping thru the http_header filter on the README/trailer subrequest was
interesting. It sure looks like that bad boy is generating headers during the
subrequest. It has a debug assert(!r->main). It doesn't hit because the r came
from the filter structure, and it's the main request's r. It does take itself
out of the filter chain after it is done with the trailer subrequest.
* what happened to the subrequest filter? what is going to stop subrequest EOS
buckets from really flushing data to the network prematurely?
Greg
Re: 2.0.34 - erratic behavior with autoindexes
Posted by "William A. Rowe, Jr." <wr...@rowe-clan.net>.
At 04:39 PM 4/1/2002, you wrote:
>I noticed something weird on daedalus on the test build. It looks like we're
>not autoindexing directories with HEADER.html or README.html files correctly.
>You see the contents of HEADER.html and/or README.html, but not the other
>files
>or subdirs in the directory. Often, the Mozilla animation keeps going as
>if the
>browser thinks more info is coming.
>
>Examples of the failure:
>
>http://www.apache.org:8092/dist/
>http://www.apache.org:8092/dist/httpd/
>http://jakarta.apache.org:8092/builds/
I consider this to be a showstopper to a responsible release of 2.0.34...
that release is on -hold-.
Let's all try [I'll take a whack in the a.m.] to figure out this pickle.
Bill