Possible infinite loop in mod_proxy

[Chris Monson <ch...@orangatango.net>]

I apologize if this is the wrong forum for this information.  I started 
with the mod_proxy dev list, and they suggested that the problem was 
probably with Apache's filter code, and that I should post the issue here.

I have been working with mod_proxy under Apache 2.0.44 and have been 
having some problems.

The basic setup is using the worker MPM and is as follows:

Listen 8008
<VirtualHost *:8008>
   ServerName myproxy.mydomain.mytld
   DocumentRoot "/unnecessary/path/to/files"

   ProxyRequests On
   ProxyPreserveHost on
   ProxyReceiveBufferSize 4096
   ProxyTimeout 20
   ProxyVia off
</VirtualHost>

In my browser, I set the proxy to be myproxy.mydomain.mytld at port 
8008.  Things seem to work all right for simple requests.  However, 
requesting a really messy page like "www.cnn.com" causes some 
interesting problems.  (For bug duplication purposes, I am currently 
using Mozilla 1.3a, but this has been duplicated on various versions of 
IE, as well).

After making the request to www.cnn.com, Apache must be forcefully 
restarted in order serve any other requests reliably.  Request threads 
do not appear to want to give up their connection to the internet once 
the request is completed.  An ethereal dump indicates that Apache is 
still sending TCP traffic to the destination server.  An endless stream 
of ACK packets is sent to the destination server until Apache is 
forcefully restarted.  I have verified that it is not the browser that 
is doing this.

Additionally, the problem is greatly alleviated (though not eradicated) 
by turning off keepalive in the proxy code or in the conf file.  I can 
make several requests reliably with keepalive turned off, but eventually 
the server gets hung up.  It doesn't send an endless stream of packets 
to the destination server without keepalive, however.

When the server hangs like that, a restart indicates that the children 
must receive a SIGKILL, but it never finishes restarting.  If I run

$ apachectl stop

then it wakes up and starts sending data again, strangely.  This appears 
to me to be some kind of a deadly embrace in the worker code, but that 
is a very preliminary assumption and has not yet been tested.  It just 
feels like that kind of a problem since attempting to kill the process 
seems to wake things up (mostly) again.

I checked the bug database for this kind of problem, and it appears that 
in October of last year a bug was submitted that may be related, dealing 
with a keepalive-related memory leak in Apache.  I believe that bug to 
be somewhat related, though not quite the same thing I am experiencing.  
What I get is a completely frozen server.

C

Re: Possible infinite loop in mod_proxy

[Chris Monson <ch...@orangatango.net>]

I have a little more information now, though I am still not feeling 
confident enough to go about attaching gdb at the moment. ;-)

I swapped out the worker MPM for the prefork MPM and the problem 
disappeared.  It did seem like a threading issue of some kind because of 
the strange restart behavior.  I'll continue trying to track it down, 
but right now I'm under a lot of time pressure to just get something 
working.

Anyway, if anyone is interested, there you are.

:)

C

Ian Holsman wrote:

> Hi Chris.
> can you do me a favor, and put a TCPdump of the interaction somewhere.
> and if you could either to a truss/strace and/or a GDB br so we know 
> what part of the code is looping hanging.
>
> also, can you try disabling proxyreceivebuffersize & proxypreservehost
> and see if that makes a difference?
>
> TIA
> ian.
>
> ps.. raising a bug is also a good choice on a next-action too ;-)
>
> Chris Monson wrote:
>
>> I apologize if this is the wrong forum for this information.  I 
>> started with the mod_proxy dev list, and they suggested that the 
>> problem was probably with Apache's filter code, and that I should 
>> post the issue here.
>>
>> I have been working with mod_proxy under Apache 2.0.44 and have been 
>> having some problems.
>>
>> The basic setup is using the worker MPM and is as follows:
>>
>> Listen 8008
>> <VirtualHost *:8008>
>>   ServerName myproxy.mydomain.mytld
>>   DocumentRoot "/unnecessary/path/to/files"
>>
>>   ProxyRequests On
>>   ProxyPreserveHost on
>>   ProxyReceiveBufferSize 4096
>>   ProxyTimeout 20
>>   ProxyVia off
>> </VirtualHost>
>>
>> In my browser, I set the proxy to be myproxy.mydomain.mytld at port 
>> 8008.  Things seem to work all right for simple requests.  However, 
>> requesting a really messy page like "www.cnn.com" causes some 
>> interesting problems.  (For bug duplication purposes, I am currently 
>> using Mozilla 1.3a, but this has been duplicated on various versions 
>> of IE, as well).
>>
>> After making the request to www.cnn.com, Apache must be forcefully 
>> restarted in order serve any other requests reliably.  Request 
>> threads do not appear to want to give up their connection to the 
>> internet once the request is completed.  An ethereal dump indicates 
>> that Apache is still sending TCP traffic to the destination server.  
>> An endless stream of ACK packets is sent to the destination server 
>> until Apache is forcefully restarted.  I have verified that it is not 
>> the browser that is doing this.
>>
>> Additionally, the problem is greatly alleviated (though not 
>> eradicated) by turning off keepalive in the proxy code or in the conf 
>> file.  I can make several requests reliably with keepalive turned 
>> off, but eventually the server gets hung up.  It doesn't send an 
>> endless stream of packets to the destination server without 
>> keepalive, however.
>>
>> When the server hangs like that, a restart indicates that the 
>> children must receive a SIGKILL, but it never finishes restarting.  
>> If I run
>>
>> $ apachectl stop
>>
>> then it wakes up and starts sending data again, strangely.  This 
>> appears to me to be some kind of a deadly embrace in the worker code, 
>> but that is a very preliminary assumption and has not yet been 
>> tested.  It just feels like that kind of a problem since attempting 
>> to kill the process seems to wake things up (mostly) again.
>>
>> I checked the bug database for this kind of problem, and it appears 
>> that in October of last year a bug was submitted that may be related, 
>> dealing with a keepalive-related memory leak in Apache.  I believe 
>> that bug to be somewhat related, though not quite the same thing I am 
>> experiencing.  What I get is a completely frozen server.
>>
>> C
>>
>
>

Re: Possible infinite loop in mod_proxy

[Chris Monson <ch...@orangatango.net>]

Ian Holsman wrote:

> Hi Chris.
> can you do me a favor, and put a TCPdump of the interaction somewhere.
> and if you could either to a truss/strace and/or a GDB br so we know 
> what part of the code is looping hanging.
>
> also, can you try disabling proxyreceivebuffersize & proxypreservehost
> and see if that makes a difference?

Disabling the options mentioned above has no effect.  I just had them on 
for kicks.  It didn't seem to make a difference.

Where would you like me to put a TCP dump?  It may be just as easy to 
use a stock Apache 2.0.44 with mod_proxy enabled using the worker MPM as 
indicated and just try to load CNN.  It's pretty consistent over here.  
I would be happy to attach a debugger to it, too, when I get a moment.  
I haven't used gdb in simply ages :)

Anyway, if you still want the dump, let me know where you would like me 
to put it.  I am sure you probably don't want it on this list.

C

Re: Possible infinite loop in mod_proxy

[Ian Holsman <li...@holsman.net>]

Hi Chris.
can you do me a favor, and put a TCPdump of the interaction somewhere.
and if you could either to a truss/strace and/or a GDB br so we know 
what part of the code is looping hanging.

also, can you try disabling proxyreceivebuffersize & proxypreservehost
and see if that makes a difference?

TIA
ian.

ps.. raising a bug is also a good choice on a next-action too ;-)

Chris Monson wrote:
> I apologize if this is the wrong forum for this information.  I started 
> with the mod_proxy dev list, and they suggested that the problem was 
> probably with Apache's filter code, and that I should post the issue here.
> 
> I have been working with mod_proxy under Apache 2.0.44 and have been 
> having some problems.
> 
> The basic setup is using the worker MPM and is as follows:
> 
> Listen 8008
> <VirtualHost *:8008>
>   ServerName myproxy.mydomain.mytld
>   DocumentRoot "/unnecessary/path/to/files"
> 
>   ProxyRequests On
>   ProxyPreserveHost on
>   ProxyReceiveBufferSize 4096
>   ProxyTimeout 20
>   ProxyVia off
> </VirtualHost>
> 
> In my browser, I set the proxy to be myproxy.mydomain.mytld at port 
> 8008.  Things seem to work all right for simple requests.  However, 
> requesting a really messy page like "www.cnn.com" causes some 
> interesting problems.  (For bug duplication purposes, I am currently 
> using Mozilla 1.3a, but this has been duplicated on various versions of 
> IE, as well).
> 
> After making the request to www.cnn.com, Apache must be forcefully 
> restarted in order serve any other requests reliably.  Request threads 
> do not appear to want to give up their connection to the internet once 
> the request is completed.  An ethereal dump indicates that Apache is 
> still sending TCP traffic to the destination server.  An endless stream 
> of ACK packets is sent to the destination server until Apache is 
> forcefully restarted.  I have verified that it is not the browser that 
> is doing this.
> 
> Additionally, the problem is greatly alleviated (though not eradicated) 
> by turning off keepalive in the proxy code or in the conf file.  I can 
> make several requests reliably with keepalive turned off, but eventually 
> the server gets hung up.  It doesn't send an endless stream of packets 
> to the destination server without keepalive, however.
> 
> When the server hangs like that, a restart indicates that the children 
> must receive a SIGKILL, but it never finishes restarting.  If I run
> 
> $ apachectl stop
> 
> then it wakes up and starts sending data again, strangely.  This appears 
> to me to be some kind of a deadly embrace in the worker code, but that 
> is a very preliminary assumption and has not yet been tested.  It just 
> feels like that kind of a problem since attempting to kill the process 
> seems to wake things up (mostly) again.
> 
> I checked the bug database for this kind of problem, and it appears that 
> in October of last year a bug was submitted that may be related, dealing 
> with a keepalive-related memory leak in Apache.  I believe that bug to 
> be somewhat related, though not quite the same thing I am experiencing.  
> What I get is a completely frozen server.
> 
> C
>