Possible infinite loop in mod_proxy

[Chris Monson <ch...@orangatango.net>]

I apologize if this is not the correct place to ask questions about 
potential mod_proxy bugs.  It seemed the best choice, so please correct 
me (gently) if I am wrong.

I have been working with mod_proxy under Apache 2.0.44 and have been 
having some problems.

The basic setup is as follows:

Listen 8008
<VirtualHost *:8008>
    ServerName myproxy.mydomain.mytld
    DocumentRoot "/unnecessary/path/to/files"

    ProxyRequests On
    ProxyPreserveHost on
    ProxyReceiveBufferSize 4096
    ProxyTimeout 20
    ProxyVia off
</VirtualHost>

In my browser, I set the proxy to be myproxy.mydomain.mytld at port 
8008.  Things seem to work all right for simple requests.  However, 
requesting a really messy page like "www.cnn.com" causes some 
interesting problems.  (For bug duplication purposes, I am currently 
using Mozilla 1.3a, but this has been duplicated on various versions of 
IE, as well).

After making the request to www.cnn.com, Apache must be forcefully 
restarted in order serve any other requests reliably.  Request threads 
do not appear to want to give up their connection to the internet once 
the request is completed.  An ethereal dump indicates that Apache is 
still sending TCP traffic to the destination server.  An endless stream 
of ACK packets is sent to the destination server until Apache is 
forcefully restarted.  I have verified that it is not the browser that 
is doing this.

Of course, since the backend continues to think it is speaking with a 
server, the frontend socket gets tied up and requests to other pages 
block until the server is restarted, causing a host of other symptoms.  
I believe that the rest of the problems I am experiencing are entirely 
due to this seeming infinite loop.

Apache 2.0 is a complex piece of software, so I apologize if this is in 
fact a bug in the server core rather than in mod_proxy.  The server 
seems to work all right without serving as a proxy, so it seemed 
reasonable to hit this list.

C

Re: Possible infinite loop in mod_proxy

[Chris Monson <ch...@orangatango.net>]

>
> Please repost this to the main dev list - this looks like it could be 
> a problem within the filters code, which the proxy makes extensive use 
> of. 


Thanks, I will do that, along with attaching a debugger as you suggested.

C

Re: Possible infinite loop in mod_proxy

[Graham Leggett <mi...@sharp.fm>]

Chris Monson wrote:

> I apologize if this is not the correct place to ask questions about 
> potential mod_proxy bugs.  It seemed the best choice, so please correct 
> me (gently) if I am wrong.

Please repost this to the main dev list - this looks like it could be a 
problem within the filters code, which the proxy makes extensive use of.

Regards,
Graham
-- 
-----------------------------------------
minfrin@sharp.fm		"There's a moon
					over Bourbon Street
						tonight..."

Re: Possible infinite loop in mod_proxy

[Graham Leggett <mi...@sharp.fm>]

Chris Monson wrote:

> So, here is my take on this.  I got mod_proxy to at least stop hanging 
> and constantly hitting the origin server by adding the following two 
> lines of code to mod_proxy.c:
> 
> origin->keep_alive = 0;
> apr_table_set( r->headers_in, "Connection", "Close );

What you have effectively done is turn keepalives off.

> This is, of course, not the right thing to do, but I wanted to see if it 
> would fix the problem, and it did.  I can now correctly proxy, even if 
> it is slow because of the lack of keep-alive.  I believe what is 
> happening is mod_proxy_http is deciding to keep the connection alive on 
> the backend, which is correct, but it is failing to inform the browser 
> that it has done so.

Keepalive is implied by the HTTP/1.1 protocol, there is no need for 
proxy to inform the browser.

It would help if you attach a debugger to the stuck process and see 
where it is getting stuck in the infinite loop.

Regards,
Graham
-- 
-----------------------------------------
minfrin@sharp.fm		"There's a moon
					over Bourbon Street
						tonight..."

Re: Possible infinite loop in mod_proxy

[Chris Monson <ch...@orangatango.net>]

I have more information on this apparent bug.

As I watch the requests move around, I noticed that when making the 
request through Squid, the connection: keep-alive and other relevant 
headers are set correctly.  When making the request through mod_proxy, 
those headers disappear.

So, here is my take on this.  I got mod_proxy to at least stop hanging 
and constantly hitting the origin server by adding the following two 
lines of code to mod_proxy.c:

origin->keep_alive = 0;
apr_table_set( r->headers_in, "Connection", "Close );

This was done in the middle of the ap_proxy_http_request function in 
proxy_http.c.

This is, of course, not the right thing to do, but I wanted to see if it 
would fix the problem, and it did.  I can now correctly proxy, even if 
it is slow because of the lack of keep-alive.  I believe what is 
happening is mod_proxy_http is deciding to keep the connection alive on 
the backend, which is correct, but it is failing to inform the browser 
that it has done so.  In the absence of information about keep-alive, 
the browser does not make any more requests on that socket, but also 
fails to open more of them, since it has a limit as to how many it can 
open, and the socket is not yet closed.

I am no expert on the nuances of when to send keep-alive headers and 
when not to, but it is clear that something is missing here.

I hope this helps.  Please let me know if more details are needed.

C

> In my browser, I set the proxy to be myproxy.mydomain.mytld at port 
> 8008.  Things seem to work all right for simple requests.  However, 
> requesting a really messy page like "www.cnn.com" causes some 
> interesting problems.  (For bug duplication purposes, I am currently 
> using Mozilla 1.3a, but this has been duplicated on various versions 
> of IE, as well).
>
> After making the request to www.cnn.com, Apache must be forcefully 
> restarted in order serve any other requests reliably.  Request threads 
> do not appear to want to give up their connection to the internet once 
> the request is completed.  An ethereal dump indicates that Apache is 
> still sending TCP traffic to the destination server.  An endless 
> stream of ACK packets is sent to the destination server until Apache 
> is forcefully restarted.  I have verified that it is not the browser 
> that is doing this.