Wierd problem with apache/post test

[Gary Benson <gb...@redhat.com>]

Hi all,

I came across a wierd problem with the apache/post test which seems to
occur on certain OSes. The symptoms are that the test locks up on one of
the subtests, LWP eventually times out on the request, the test fails and
then all subsequent tests that are run fail.

The problem seems to be this:
 - LWP writes the entire request in one go.
 - mod_echo_post send its response as it processes the request.
 - only when it has finished sending the request does LWP attempt to read
   the response.

I think what is happening is that, with large requests, a large amount of
data is simultanously being fed into the socket from both directions, and
some buffer somewhere in the OSes socket handler is being filled. My
suspicions are backed up by the fact that I modified mod_echo_post (see
attached) to read the entire request and only then send the response, and
this solves the problem.

If this is the case then it really should be fixed by making LWP read and
write simultanously; you may have guessed from my hazy terminology that I
don't really know a lot about how sockets _really_ work, but do any of
you guys think that this sounds about right? If so, I'll email the LWP
maintainers and see what they say.

Later,
Gary

[ gbenson@redhat.com ][ GnuPG 85A8F78B ][ http://inauspicious.org/ ]

Re: Wierd problem with apache/post test

[Gary Benson <gb...@redhat.com>]

On Fri, 16 Nov 2001, Cliff Woolley wrote:

> On Fri, 16 Nov 2001, Gary Benson wrote:
>
> > Anyone have any objections to my rewriting mod_echo_post to dump
> > everything to a temporary file and then read it back?
>
> I wish I understood exactly why this was happening... it seems almost
> indicative of some problem in Apache or APR.  The module shouldn't have to
> buffer all this stuff... it should be able to do a read from the input
> filter chain, process it, and send some output down the output filter
> chain, and that ought to be it.  Buffering the whole thing either in RAM
> or on disk in a production environment would open a huge DoS hole.

The problem isn't in Apache: what is happening is that the way mod_echo_post
works is exposing a weakness in the client code, in this case libwwwperl.
The clients should really read and write simultaneously, but this isn't
normally a problem since what happens with the vast majority of requests
is that the handler reads the request in its entirity, chews it over a bit
and then returns something. This behaviour doesn't saturate the buffers.

Of course, I say it is the clients' fault, but perhaps it is the OS: the
problem seems to affect only some OSes and not others -- perhaps these
OSes have fixed size socket buffers? A 100k mod_echo_post request is
enough to cause problems and none of the machines we use are _that_ short
on memory ;)

Gary

[ gbenson@redhat.com ][ GnuPG 85A8F78B ][ http://inauspicious.org/ ]

Re: Wierd problem with apache/post test

[Cliff Woolley <cl...@yahoo.com>]

On Fri, 16 Nov 2001, Gary Benson wrote:

> Anyone have any objections to my rewriting mod_echo_post to dump
> everything to a temporary file and then read it back?

I wish I understood exactly why this was happening... it seems almost
indicative of some problem in Apache or APR.  The module shouldn't have to
buffer all this stuff... it should be able to do a read from the input
filter chain, process it, and send some output down the output filter
chain, and that ought to be it.  Buffering the whole thing either in RAM
or on disk in a production environment would open a huge DoS hole.

--Cliff

--------------------------------------------------------------
   Cliff Woolley
   cliffwoolley@yahoo.com
   Charlottesville, VA

Re: Wierd problem with apache/post test

[Gary Benson <gb...@redhat.com>]

On Thu, 15 Nov 2001, Doug MacEachern wrote:

> yeah, the socket gets saturated, saw the same problem with ssl, see the
> comment in apache/post.t:
> #300, 500, 2000, 4000, 6000, 10_000);
> #XXX: ssl currently falls over here

I wondered if this was the same problem :)

> i think your solution to modify mod_echo_post.c is the way to go, just
> make sure you free() the malloced buffer.  either that or write the post
> data to a tmp file then send that back out.

Write to a file was my favourite option (some of our machines are low on
RAM), but I did the memory thing as a quick hack to check if it worked.

Anyone have any objections to my rewriting mod_echo_post to dump
everything to a temporary file and then read it back?

Gary

[ gbenson@redhat.com ][ GnuPG 85A8F78B ][ http://inauspicious.org/ ]

Re: Wierd problem with apache/post test

[Doug MacEachern <do...@covalent.net>]

yeah, the socket gets saturated, saw the same problem with ssl, see the
comment in apache/post.t:
#300, 500, 2000, 4000, 6000, 10_000);
#XXX: ssl currently falls over here

also happens if you were to post the same size data using netscape.

i think your solution to modify mod_echo_post.c is the way to go, just
make sure you free() the malloced buffer.  either that or write the post
data to a tmp file then send that back out.