Issue with persistent http proxy backend connection

[Ruediger Pluem <rp...@apache.org>]

There is an issue with the proxy code that if a request is sent over
a persistent backend connection (currently only looking at the http case, not
sure if the same thing can happen for other backends like ajp and fastcgi)
it could happen that this connection gets closed by the backend for timeout reasons
after the is_connected check and before / while sending the request.
For reference see e.g:

http://mail-archives.apache.org/mod_mbox/httpd-dev/200602.mbox/%3c20060220135107.GA4335@redhat.com%3e
http://mail-archives.apache.org/mod_mbox/httpd-dev/200602.mbox/%3c43F0FBFD.6040403@apache.org%3e

I had some virtual hackathon with Joe Orton via IRC on this topic and try to summarize our
findings:

1. RFC 2616 does not allow clients to resend a non idempotent request in those cases without
   user interaction (8.1.4). It is concluded that the same applies to a proxy in this case.

2. Although GET is mentioned to be idempotent in RFC 2616 (9.1.2) along with some other methods
   it is not a good idea to regard a GET / HEAD with query parameters to be idempotent.

3. Sometimes servers (including httpd) include a keep-alive header in their response with the parameter
   timeout set. This can give a hint when the backend will close its persistent connection due to a
   timeout.

If there is a non idempotent request to the proxy and there is not enough timeout time left on the
backend connection, then close the pooled backend connection and forward the request to the
backend via a new connection. The size of "enough timeout time left" is not defined yet and
besides a good default value for it, there should be the possibility to change that via configuration.
If the server does not sent the timeout parameter back in its reponse, it is regarded as 0.

If there is a idempotent request, resent it if it failed because of a read / write failure to the
backend *before* the first successful read from the backend. It is important that this is really
a socket error. Simply checking for a failure of ap_proxy_http_request is not *enough*, because
it should not be resent if it failed due to some other problem (e.g. parsing error).


Regards

Rüdiger

Re: Issue with persistent http proxy backend connection

[Jeff Trawick <tr...@gmail.com>]

On 10/12/06, Ruediger Pluem <rp...@apache.org> wrote:
>
>
> On 10/11/2006 11:09 PM, Jeff Trawick wrote:
> > On 10/11/06, Ruediger Pluem <rp...@apache.org> wrote:
>
> >> 2. Although GET is mentioned to be idempotent in RFC 2616 (9.1.2)
> >> along with some other methods
> >>    it is not a good idea to regard a GET / HEAD with query parameters
> >> to be idempotent.
> >
> >
> > <Location /proxy/all_gets_are_idempotent>
> > SetEnv foomatic=1
> > </Location>
>
> Sorry I don't get the point here. Could you please explain?

Allow the administrator to declare that GETs/HEADs are idempotent even
if they have query args, with a few lines of code to check some
envvar.

Re: Issue with persistent http proxy backend connection

[Ruediger Pluem <rp...@apache.org>]

On 10/11/2006 11:09 PM, Jeff Trawick wrote:
> On 10/11/06, Ruediger Pluem <rp...@apache.org> wrote:

>> 2. Although GET is mentioned to be idempotent in RFC 2616 (9.1.2)
>> along with some other methods
>>    it is not a good idea to regard a GET / HEAD with query parameters
>> to be idempotent.
> 
> 
> <Location /proxy/all_gets_are_idempotent>
> SetEnv foomatic=1
> </Location>

Sorry I don't get the point here. Could you please explain?

> 
> (In case it isn't stated explicitly already)  No such retry would be
> performed on the initial request sent on a connection, right?

Yes, this should be only done if we get an existing connection from the pool.

Regards

Rüdiger

Re: Issue with persistent http proxy backend connection

[Henrik Nordstrom <hn...@squid-cache.org>]

tor 2006-10-12 klockan 13:19 +0200 skrev Ruediger Pluem:

> I do not think that this matters all too much, because the backend closes
> the connection *immediately* after sending out the response.

To help this, perhaps there should be a check just before sending the
response as well, and send "Connection: close" if it's likely this
thread should terminate after this response?

MaxRequestsPerChild certainly can be evaluated before the response is
sent.

Regards
Henrik

Re: Issue with persistent http proxy backend connection

[Jeff Trawick <tr...@gmail.com>]

On 10/12/06, Ruediger Pluem <rp...@apache.org> wrote:
>
>
> On 10/11/2006 11:09 PM, Jeff Trawick wrote:
> > On 10/11/06, Ruediger Pluem <rp...@apache.org> wrote:
>
> >
> >> 3. Sometimes servers (including httpd) include a keep-alive header in
> >> their response with the parameter
> >>    timeout set. This can give a hint when the backend will close its
> >> persistent connection due to a
> >>    timeout.
> >
> >
> > Unfortunately, our own hint in the response is not useful when sent by
> > a child process that is being terminated due to excessive idle
> > processes/threads or MaxRequestsPerChild, since we can break out of
> > ap_process_connection right after telling the client they have N
> > seconds to send the next request.  This will happen enough to be
>
> I do not think that this matters all too much, because the backend closes
> the connection *immediately* after sending out the response. So the socket
> connection check on proxy side that is executed before reusing a connection
> will detect that this socket has been closed by the remote side.
> But yes, in theory their remains a race here if the proxy reuses the connection
> faster then the backend needs time to close the socket after sending the response.

Agreed.  response sent, then FIN (lingering close), so client/proxy
should be able to detect terminated connection immediately; it would
only break with pipelining, which proxy doesn't support

Re: Issue with persistent http proxy backend connection

[Ruediger Pluem <rp...@apache.org>]

On 12.10.2006 13:26, Mladen Turk wrote:
> Ruediger Pluem wrote:
> 
>>
>> But yes, in theory their remains a race here if the proxy reuses the
>> connection
>> faster then the backend needs time to close the socket after sending
>> the response.
>>
> 
> Does the problem exists with the current trunk as well?
> The way how trunk detects the closed backend has changed
> by not using read any more.

Yes, because of a race condition. The remote side could close the connection
just after the check happened. The closing could even happen while the first
data is on its way to the backend. So the first write(s) can succeed, because
the FIN from the backend is on its way during this time.
Nevertheless it is very good to have a reliable socket connection check (as on
trunk) as this reduces the number of cases where this happens a lot and makes
handling of such cases a lot easier.


Regards

Rüdiger

Re: Issue with persistent http proxy backend connection

[Mladen Turk <mt...@apache.org>]

Ruediger Pluem wrote:
> 
> But yes, in theory their remains a race here if the proxy reuses the connection
> faster then the backend needs time to close the socket after sending the response.
>

Does the problem exists with the current trunk as well?
The way how trunk detects the closed backend has changed
by not using read any more.

Regards,
Mladen.

Re: Issue with persistent http proxy backend connection

[Ruediger Pluem <rp...@apache.org>]

On 10/11/2006 11:09 PM, Jeff Trawick wrote:
> On 10/11/06, Ruediger Pluem <rp...@apache.org> wrote:

> 
>> 3. Sometimes servers (including httpd) include a keep-alive header in
>> their response with the parameter
>>    timeout set. This can give a hint when the backend will close its
>> persistent connection due to a
>>    timeout.
> 
> 
> Unfortunately, our own hint in the response is not useful when sent by
> a child process that is being terminated due to excessive idle
> processes/threads or MaxRequestsPerChild, since we can break out of
> ap_process_connection right after telling the client they have N
> seconds to send the next request.  This will happen enough to be

I do not think that this matters all too much, because the backend closes
the connection *immediately* after sending out the response. So the socket
connection check on proxy side that is executed before reusing a connection
will detect that this socket has been closed by the remote side.
But yes, in theory their remains a race here if the proxy reuses the connection
faster then the backend needs time to close the socket after sending the response.


Regards

Rüdiger

Re: Issue with persistent http proxy backend connection

[Jeff Trawick <tr...@gmail.com>]

On 10/11/06, Ruediger Pluem <rp...@apache.org> wrote:
> There is an issue with the proxy code that if a request is sent over
> a persistent backend connection (currently only looking at the http case, not
> sure if the same thing can happen for other backends like ajp and fastcgi)
> it could happen that this connection gets closed by the backend for timeout reasons
> after the is_connected check and before / while sending the request.

> 1. RFC 2616 does not allow clients to resend a non idempotent request in those cases without
>    user interaction (8.1.4). It is concluded that the same applies to a proxy in this case.

A non-compliant behavior (i.e., not default Apache behavior) which
would be useful would be to wait for 100-Continue before sending POST
bodies.  If connection is dropped before receiving 100-Continue, try
on a new connection.

> 2. Although GET is mentioned to be idempotent in RFC 2616 (9.1.2) along with some other methods
>    it is not a good idea to regard a GET / HEAD with query parameters to be idempotent.

<Location /proxy/all_gets_are_idempotent>
SetEnv foomatic=1
</Location>

> 3. Sometimes servers (including httpd) include a keep-alive header in their response with the parameter
>    timeout set. This can give a hint when the backend will close its persistent connection due to a
>    timeout.

Unfortunately, our own hint in the response is not useful when sent by
a child process that is being terminated due to excessive idle
processes/threads or MaxRequestsPerChild, since we can break out of
ap_process_connection right after telling the client they have N
seconds to send the next request.  This will happen enough to be
disturbing, though perhaps not often enough to quickly diagnose it.
("Gosh, try turning off proxy keepalive and see if it gets better.  If
so, that's your permanent solution.")

> If there is a non idempotent request to the proxy and there is not enough timeout time left on the
> backend connection, then close the pooled backend connection and forward the request to the
> backend via a new connection. The size of "enough timeout time left" is not defined yet and
> besides a good default value for it, there should be the possibility to change that via configuration.
> If the server does not sent the timeout parameter back in its reponse, it is regarded as 0.
>
> If there is a idempotent request, resent it if it failed because of a read / write failure to the
> backend *before* the first successful read from the backend. It is important that this is really
> a socket error. Simply checking for a failure of ap_proxy_http_request is not *enough*, because
> it should not be resent if it failed due to some other problem (e.g. parsing error).

(In case it isn't stated explicitly already)  No such retry would be
performed on the initial request sent on a connection, right?