You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Jan Kaluža <jk...@redhat.com> on 2013/12/05 13:51:49 UTC

Re: mod_proxy, oooled backend connections and the keep-alive race condition

Hi,

should this be fixed in trunk already? I see some commits in proxy code 
based on your ideas Yann, but I'm not sure if they address this 
particular problem too.

Jan Kaluza

On 10/17/2013 04:52 PM, Yann Ylavic wrote:
>
> On Thu, Oct 17, 2013 at 11:36 AM, Thomas Eckert
> <thomas.r.w.eckert@gmail.com <ma...@gmail.com>> wrote:
>
>      > Hence,
>     why cannot mod_proxy_http prefetch the client's body *before* trying
>     anything with the backend connection, and only if it's all good,
>     connect (or reuse a connection to) the
>      > backend and then start forwarding the data immediatly (flushing
>     the prefetched ones) ?
>
>     Now I'm confused. Are we talking about flushing everything
>     immediately from client to backend (that's what I understood from
>     your first patch proposal in this thread) or are we talking about
>     pre fetching the whole request body from the client and then passing
>     it on to the backend as a whole ?
>
>
> Sorry, I really should not have proposed 2 patches in the same thread
> without being clear about their different purposes.
>
> I first thought flushing everything received from the client to the
> backend immediatly, and disabling *retries* in the 16K-prefetch, would
> address the issue (my first patch,
> "httpd-trunk-mod_proxy_http_flushall.patch"). But as you said in a
> previous message, that did not address the case where the client and/or
> an input filter retain the very first body bytes, and mod_proxy_http
> blocks, and the backend timeouts...
>
> Hence the second patch ("httpd-trunk-mod_proxy_http_prefetch.patch"),
> which does prefetch *before* establishing/checking the backend's
> connection, so to minimize (to the minimum) the future delay between
> that and the first bytes sent to the backend (hence the chances for it
> to timeout). The prefetch is still up to 16K (same as existing), and not
> the "whole" body of course.
>
> Then, when it's time to forward (what is available from) the client's
> request, the patched mod_proxy_http will establish or check+reuse a
> backend connection and always flush immediatly what it has  already
> (headers + prefetch, see stream_reqbody_* functions).
>
> For the remaining request's body chunks (if any), I intentionally did
> not change the current behaviour of letting the output filters choose
> when to flush, this is not the keep-alive timeout anymore, so it becomes
> out of the issue's scope. But I'm definitively +1 to flush-all here too
> (optionaly), since mod_proxy may retain small chunks and can then face
> another backend timeout later! That's where patch1 meets patch2, and
> comes patch3 ("httpd-trunk-mod_proxy_http_prefetch+flushall.patch",
> attached here).
>
> There is still the potential (minimal) race condition, the backend could
> have closed in the (short) meantime, that's indeed *not* addressed by my
> patch(es).
>
>
>     The problem I mentioned in the first message is about treating the
>     backend connection in an error prone way. By not ensuring the
>     connection is still valid - e.g. as seen by the peer - we risk
>     running into this problem no matter what we do elsewhere. So I
>     really think the proper way to handle this is to reopen the
>     connection if necessary - be this with a filter or integrated into
>     the connection handling itself - and only fail after we tried at
>     least once.
>
>
> IMHO this is hardly feasible for non-idempotent (and no body) requests
> (without disabling connection reuse, or using "proxy-initial-not-pooled"
> to let the client choose), even if mod_proxy takes care of not sending
> the full request in one time to be able to detect connection problems
> before it's too late (to retry), there is still a possibility that it
> won't know before sending the last chunk, and the issue remain.
> mod_proxy_http addresses retries for requests *with body* by the "ping"
> parameter, maybe it could also retry idempotent requests (like
> mod_proxy_ajp does), but this is still not an overall solution, is there
> one?
>
>
>     Re-thinking the proposed flush mechanism, I don't think using the
>     flushing will really solve the problem, albeit it probably narrowing
>     down the window of opportunity for the problem significantly in most
>     scenarios. I mention this because I'm getting the feeling two
>     different discussions are being mixed up: Solving the
>     "keep-alive time out" problem and introducing the
>     "flush-all-immediately" option. While the latter might improve the
>     situation with the first, it is no complete solution and there are a
>     lot of scenarios where it does not apply due to other factors like
>     filters (as discussed previously).
>
>
> See above, the flushall does not resolve the issue (filters...), but the
> early prefetch + flushall reduces it considerably, IMHO.
>
>
>     The flush option itself sounds like a good idea, so I see no reason
>     why not to put it in. I just don't want to loose focus on the actual
>     problem ;-)
>
>
> I don't see either...
>
> Please consider the new (yet) patch attached, can you still encounter
> some "proxy: error reading status line from remote server" with it?
> Even without "proxy-flushall" set, you shouldn't, whatever the time
> taken to read the request first 16K bytes...
>
> Regards,
> Yann.


Re: mod_proxy, oooled backend connections and the keep-alive race condition

Posted by Yann Ylavic <yl...@gmail.com>.
I don't want to overdo this thread (apologies if it is already), last
comment yet...

On Thu, Dec 5, 2013 at 10:03 PM, Yann Ylavic <yl...@gmail.com> wrote:

> Is sending a partial request first what you'd like to (be) implement(ed),
> and only if that succeeds send the remaining data?
>

The issue would still be there for the remaining data since nothing
guarantees the partial (initial) sending has reached the backend, which
might still have closed the connection in the meantime ("error reading
status line from remote server" is the proof, the failure occurs on the
recv(), not the send()).

Maybe one (other than me) facing the race condition can give my patch a
chance (no more "error reading status line..." here with TTL < KA-timeout,
pretetch, and slow clients).

No noticeable slow-down either, although statements like this are worthless
without measure...
I plan to do that soon (and keep you informed, for those who still care :)

Regards,
Yann.

Re: mod_proxy, oooled backend connections and the keep-alive race condition

Posted by Micha Lenk <mi...@lenk.info>.
Hi Yann,

Am 09.12.2013 00:30, schrieb Yann Ylavic:
>     Now, if trying to send the first bytes of the request immediately fails
>     because the socket isn't connected anymore (e.g. EPIPE), you *know* that
>     exactly *none* bits of your current request reached the server. For this
>     reason it should be safe to retry by establishing a new connection to
>     the server and to try to send the request again over the newly
>     established connection.
> 
> 
> The send() will never fail at this point if the peer FIN-closed the
> connection (EPIPE or whatever on non-unixes), since it is only half-closed.
> You can't read anymore, but you still can write, and this is precisely
> why the error log is "error *reading* status line" and not "error
> writing ...".
> A second send() could fail however, though possibly with ECONNRESET, and
> you are no better here.
> 
> The point is that mod_proxy already checks whether the connection is
> alive before trying to send a (successive) request on the kept alive
> connection, by using a recv(MSG_PEEK) (see ap_proxy_is_socket_connected).
> If that fails, a new connection is established and the request is sent
> on it, this is already the current behaviour.
> The race condition is when this check succeeds, but the peer closes the
> connection before the request reaches it (in the meantime).

Okay, got that. If it really works like it should then I don't know how
to improve the code. So the remaining question is, whether the checks
whether the connection is alive are sufficient. Unfortunately I have no
resources to investigate this any further.

Regards,
Micha

Re: mod_proxy, oooled backend connections and the keep-alive race condition

Posted by Yann Ylavic <yl...@gmail.com>.
On Sun, Dec 8, 2013 at 11:51 AM, Micha Lenk <mi...@lenk.info> wrote:

>
> Am 05.12.2013 22:03, schrieb Yann Ylavic:
> > When mod_proxy encounters a closed (reused) connection while forwarding
> > the client's request, it has just sent the request once, right?
>
> That depends on the actual state. What Thomas is talking about (I know
> because I worked with him on the same product he is referring to here)
> is the state where you have everything in place to send out the next
> request headers on a connection that was established by a previous
> request and that was kept open using HTTP Keep-Alive. Let me stress the
> fact that in this state you haven't sent a single bit of the current
> request over the wire yet.
>

Yes, here we are.



>
> Now, if trying to send the first bytes of the request immediately fails
> because the socket isn't connected anymore (e.g. EPIPE), you *know* that
> exactly *none* bits of your current request reached the server. For this
> reason it should be safe to retry by establishing a new connection to
> the server and to try to send the request again over the newly
> 
> established connection.


The send() will never fail at this point if the peer FIN-closed the
connection (EPIPE or whatever on non-unixes), since it is only half-closed.
You can't read anymore, but you still can write, and this is precisely why
the error log is "error *reading* status line" and not "error writing ...".
A second send() could fail however, though possibly with ECONNRESET, and
you are no better here.

The point is that mod_proxy already checks whether the connection is alive
before trying to send a (successive) request on the kept alive connection,
by using a recv(MSG_PEEK) (see ap_proxy_is_socket_connected).
If that fails, a new connection is established and the request is sent on
it, this is already the current behaviour.
The race condition is when this check succeeds, but the peer closes the
connection before the request reaches it (in the meantime).


>
> In the sense of getting the reverse proxy request correctly processed by
> the backend server, this approach *does* fix the keep-alive race
> condition issue where the server closes the backend connection because
> it sees no (new) client request for too long time -- IIRC this is what
> this discussion started with.
>

It does *not* fix the race, nothing can, and the proxy can't know whether
its request as been processed by the peer based on the (second) send()
error.

IMHO, the best that can be done is HTTP ping for POST requests (using
"Expect: 100-continue", already implemented), retry idempotent requests
(TODO), reduce the race time (my proposal/patch), and let client client
decide for others ("proxy-initial-not-pooled" can help it).
 

>
> I hope I could shed some more some light on this issue and on the idea
> how to possibly tackle it in Apache httpd.
>

Thanks, I really wish it could be, and Thomas prove me wrong!

Regards,
Yann.

Re: mod_proxy, oooled backend connections and the keep-alive race condition

Posted by Micha Lenk <mi...@lenk.info>.
Hi all,

Am 05.12.2013 22:03, schrieb Yann Ylavic:
>     I'm not talking about retrying requests but retrying writing on the
>     socket after trying to "re-"open a connection. When mod_proxy tries
>     to forward the client request to the backends and encounters a
>     closed connection due to the described race condition it will fail
>     back to the client. Instead, I suggest trying to re-open the
>     connection once and then either send the client request over that
>     connection or go back to the client with an error. No double-sending
>     requests anywhere.
> 
> I must be missing something.
> 
> When mod_proxy encounters a closed (reused) connection while forwarding
> the client's request, it has just sent the request once, right?

That depends on the actual state. What Thomas is talking about (I know
because I worked with him on the same product he is referring to here)
is the state where you have everything in place to send out the next
request headers on a connection that was established by a previous
request and that was kept open using HTTP Keep-Alive. Let me stress the
fact that in this state you haven't sent a single bit of the current
request over the wire yet.

Now, if trying to send the first bytes of the request immediately fails
because the socket isn't connected anymore (e.g. EPIPE), you *know* that
exactly *none* bits of your current request reached the server. For this
reason it should be safe to retry by establishing a new connection to
the server and to try to send the request again over the newly
established connection. To avoid an endless retry loop, you certainly
should do this retry approach only once per request.

In the sense of getting the reverse proxy request correctly processed by
the backend server, this approach *does* fix the keep-alive race
condition issue where the server closes the backend connection because
it sees no (new) client request for too long time -- IIRC this is what
this discussion started with.

I hope I could shed some more some light on this issue and on the idea
how to possibly tackle it in Apache httpd.

Regards,
Micha

Re: mod_proxy, oooled backend connections and the keep-alive race condition

Posted by Yann Ylavic <yl...@gmail.com>.
On Thu, Dec 5, 2013 at 7:03 PM, Thomas Eckert
<th...@gmail.com>wrote:

> I'm not talking about retrying requests but retrying writing on the socket
> after trying to "re-"open a connection. When mod_proxy tries to forward the
> client request to the backends and encounters a closed connection due to
> the described race condition it will fail back to the client. Instead, I
> suggest trying to re-open the connection once and then either send the
> client request over that connection or go back to the client with an error.
> No double-sending requests anywhere.
>

I must be missing something.

When mod_proxy encounters a closed (reused) connection while forwarding the
client's request, it has just sent the request once, right?
If, in that case, it opens a new connection and send that same request on
it, this is the second time, still right?

Unless it didn't send the complete request the first time, I mean
deliberately by sending only part of the request (incomplete body if any,
or incomplete header otherwise), how can the proxy be sure the first
request has not been interpreted by the backend already?
This is not because the send() failed on the proxy side the backend has not
recv()ed all the bytes (there might be a problem with the ack of the last
bytes for example), hence the request could be valid from the backend side.

Is sending a partial request first what you'd like to (be) implement(ed),
and only if that succeeds send the remaining data?


> > Same as above, this can't be done (even less at the connection level).
> > There is really no overall solution, IMHO.
>
> I doubt that statement is correct. As I said before, I've seen it work
> with another proxy so the basic approach is no problem. You might be
> correct in that it's not possible with apache but I cannot confirm or
> disconfirm this until I have tried adding it.
>

A proxy that re-sends the same non-idempotent request twice is simply not
HTTP compliant (
http://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-25, 6.3.1.
Retrying Requests).

However, the RFC does not talk about incomplete requests.

Regards,
Yann.

Re: mod_proxy, oooled backend connections and the keep-alive race condition

Posted by Thomas Eckert <th...@gmail.com>.
> You can't retry all the requests (particularly non idempotent ones), not
even once.
> Suppose it is a charged $100 order, you wouldn't like any proxy to double
that because of network problems...

I'm not talking about retrying requests but retrying writing on the socket
after trying to "re-"open a connection. When mod_proxy tries to forward the
client request to the backends and encounters a closed connection due to
the described race condition it will fail back to the client. Instead, I
suggest trying to re-open the connection once and then either send the
client request over that connection or go back to the client with an error.
No double-sending requests anywhere.

> Same as above, this can't be done (even less at the connection level).
> There is really no overall solution, IMHO.

I doubt that statement is correct. As I said before, I've seen it work with
another proxy so the basic approach is no problem. You might be correct in
that it's not possible with apache but I cannot confirm or disconfirm this
until I have tried adding it.


Anyway, this is getting us nowhere until I (or anyone) come(s) up with a
corresponding patch. I hope I'll get around for it sometime until January.
Until then I'll just see this as declined.


On Thu, Dec 5, 2013 at 5:26 PM, Yann Ylavic <yl...@gmail.com> wrote:

> On Thu, Dec 5, 2013 at 5:04 PM, Thomas Eckert <thomas.r.w.eckert@gmail.com
> > wrote:
>
>> > It also seems that it adds more cycles to Apache on the front to
>> > reduce a race condition that can't really be removed.
>>
>> While it's true that the race condition itself cannot be avoided we can
>> definitely work around the resulting problem situation, e.g. by trying to
>> open the connection again once and then either send the data or fail back
>> to the client as we do right now. The approach itself is not theoretical: I
>> know that's exactly how certain forward proxies solve the problem.
>>
>>
> You can't retry all the requests (particularly non idempotent ones), not
> even once.
> Suppose it is a charged $100 order, you wouldn't like any proxy to double
> that because of network problems...
>
>
>
>>
>>
>> > Plus, if we do this here, shouldn't we do it for
>> > all proxy protocol submodules (fcgi, etc...)?
>>
>> I go with my original suggestion: Have the above mentioned retry-once
>> mechanism apply on connection level so that request modules don't even
>> notice. Otherwise I would agree in having it added to the appropriate
>> modules. I would like to come up with the appropriate patch but I doubt
>> I'll have the time for it anytime soon :-(
>>
>
> Same as above, this can't be done (even less at the connection level).
> There is really no overall solution, IMHO.
>
>
>>
>> As it stands I think Yann's approach in "flushing through the reverse
>> proxy" is better then doing nothing. Though I expect that to be useless for
>> a lot of deployments where stuff like AV scanning or mod_security will need
>> to buffer the request bodies - then you will have the original problem
>> again.
>>
>>
> My first approach was to flush only, but the latest patch take also care
> about buffered request, the backend won't be connected/reused until the
> bytes are in mod_proxy.
>
>

Re: mod_proxy, oooled backend connections and the keep-alive race condition

Posted by Yann Ylavic <yl...@gmail.com>.
On Thu, Dec 5, 2013 at 5:04 PM, Thomas Eckert
<th...@gmail.com>wrote:

> > It also seems that it adds more cycles to Apache on the front to
> > reduce a race condition that can't really be removed.
>
> While it's true that the race condition itself cannot be avoided we can
> definitely work around the resulting problem situation, e.g. by trying to
> open the connection again once and then either send the data or fail back
> to the client as we do right now. The approach itself is not theoretical: I
> know that's exactly how certain forward proxies solve the problem.
>
>
You can't retry all the requests (particularly non idempotent ones), not
even once.
Suppose it is a charged $100 order, you wouldn't like any proxy to double
that because of network problems...



>
>
> > Plus, if we do this here, shouldn't we do it for
> > all proxy protocol submodules (fcgi, etc...)?
>
> I go with my original suggestion: Have the above mentioned retry-once
> mechanism apply on connection level so that request modules don't even
> notice. Otherwise I would agree in having it added to the appropriate
> modules. I would like to come up with the appropriate patch but I doubt
> I'll have the time for it anytime soon :-(
>

Same as above, this can't be done (even less at the connection level).
There is really no overall solution, IMHO.


>
> As it stands I think Yann's approach in "flushing through the reverse
> proxy" is better then doing nothing. Though I expect that to be useless for
> a lot of deployments where stuff like AV scanning or mod_security will need
> to buffer the request bodies - then you will have the original problem
> again.
>
>
My first approach was to flush only, but the latest patch take also care
about buffered request, the backend won't be connected/reused until the
bytes are in mod_proxy.

Re: mod_proxy, oooled backend connections and the keep-alive race condition

Posted by Thomas Eckert <th...@gmail.com>.
> It also seems that it adds more cycles to Apache on the front to
> reduce a race condition that can't really be removed.

While it's true that the race condition itself cannot be avoided we can
definitely work around the resulting problem situation, e.g. by trying to
open the connection again once and then either send the data or fail back
to the client as we do right now. The approach itself is not theoretical: I
know that's exactly how certain forward proxies solve the problem.


> Plus, if we do this here, shouldn't we do it for
> all proxy protocol submodules (fcgi, etc...)?

I go with my original suggestion: Have the above mentioned retry-once
mechanism apply on connection level so that request modules don't even
notice. Otherwise I would agree in having it added to the appropriate
modules. I would like to come up with the appropriate patch but I doubt
I'll have the time for it anytime soon :-(


As it stands I think Yann's approach in "flushing through the reverse
proxy" is better then doing nothing. Though I expect that to be useless for
a lot of deployments where stuff like AV scanning or mod_security will need
to buffer the request bodies - then you will have the original problem
again.


On Thu, Dec 5, 2013 at 4:05 PM, Jim Jagielski <ji...@jagunet.com> wrote:

> There hardly seemed any consensus on the patch... It also
> seems that it adds more cycles to Apache on the front to
> reduce a race condition that can't really be removed.
> IMO, a reverse proxy should get out of the way as
> quickly as possible.
>
> Plus, if we do this here, shouldn't we do it for
> all proxy protocol submodules (fcgi, etc...)?
>
> Plus, as mentioned, this single patch includes 2 "fixes"
> which can be/should be independent.
>
> On Dec 5, 2013, at 8:06 AM, Yann Ylavic <yl...@gmail.com> wrote:
>
> > Hi Jan,
> >
> > I don't think it is fixed in trunk, but I may have missed the commits.
> > Which ones are you talking about?
> >
> > Regards,
> > Yann.
> >
> >
> > On Thu, Dec 5, 2013 at 1:51 PM, Jan Kaluža <jk...@redhat.com> wrote:
> > Hi,
> >
> > should this be fixed in trunk already? I see some commits in proxy code
> based on your ideas Yann, but I'm not sure if they address this particular
> problem too.
> >
> > Jan Kaluza
> >
> >
> > On 10/17/2013 04:52 PM, Yann Ylavic wrote:
> >
> > On Thu, Oct 17, 2013 at 11:36 AM, Thomas Eckert
> > <thomas.r.w.eckert@gmail.com <ma...@gmail.com>>
> wrote:
> >
> >      > Hence,
> >     why cannot mod_proxy_http prefetch the client's body *before* trying
> >     anything with the backend connection, and only if it's all good,
> >     connect (or reuse a connection to) the
> >      > backend and then start forwarding the data immediatly (flushing
> >     the prefetched ones) ?
> >
> >     Now I'm confused. Are we talking about flushing everything
> >     immediately from client to backend (that's what I understood from
> >     your first patch proposal in this thread) or are we talking about
> >     pre fetching the whole request body from the client and then passing
> >     it on to the backend as a whole ?
> >
> >
> > Sorry, I really should not have proposed 2 patches in the same thread
> > without being clear about their different purposes.
> >
> > I first thought flushing everything received from the client to the
> > backend immediatly, and disabling *retries* in the 16K-prefetch, would
> > address the issue (my first patch,
> > "httpd-trunk-mod_proxy_http_flushall.patch"). But as you said in a
> > previous message, that did not address the case where the client and/or
> > an input filter retain the very first body bytes, and mod_proxy_http
> > blocks, and the backend timeouts...
> >
> > Hence the second patch ("httpd-trunk-mod_proxy_http_prefetch.patch"),
> > which does prefetch *before* establishing/checking the backend's
> > connection, so to minimize (to the minimum) the future delay between
> > that and the first bytes sent to the backend (hence the chances for it
> > to timeout). The prefetch is still up to 16K (same as existing), and not
> > the "whole" body of course.
> >
> > Then, when it's time to forward (what is available from) the client's
> > request, the patched mod_proxy_http will establish or check+reuse a
> > backend connection and always flush immediatly what it has  already
> > (headers + prefetch, see stream_reqbody_* functions).
> >
> > For the remaining request's body chunks (if any), I intentionally did
> > not change the current behaviour of letting the output filters choose
> > when to flush, this is not the keep-alive timeout anymore, so it becomes
> > out of the issue's scope. But I'm definitively +1 to flush-all here too
> > (optionaly), since mod_proxy may retain small chunks and can then face
> > another backend timeout later! That's where patch1 meets patch2, and
> > comes patch3 ("httpd-trunk-mod_proxy_http_prefetch+flushall.patch",
> > attached here).
> >
> > There is still the potential (minimal) race condition, the backend could
> > have closed in the (short) meantime, that's indeed *not* addressed by my
> > patch(es).
> >
> >
> >     The problem I mentioned in the first message is about treating the
> >     backend connection in an error prone way. By not ensuring the
> >     connection is still valid - e.g. as seen by the peer - we risk
> >     running into this problem no matter what we do elsewhere. So I
> >     really think the proper way to handle this is to reopen the
> >     connection if necessary - be this with a filter or integrated into
> >     the connection handling itself - and only fail after we tried at
> >     least once.
> >
> >
> > IMHO this is hardly feasible for non-idempotent (and no body) requests
> > (without disabling connection reuse, or using "proxy-initial-not-pooled"
> > to let the client choose), even if mod_proxy takes care of not sending
> > the full request in one time to be able to detect connection problems
> > before it's too late (to retry), there is still a possibility that it
> > won't know before sending the last chunk, and the issue remain.
> > mod_proxy_http addresses retries for requests *with body* by the "ping"
> > parameter, maybe it could also retry idempotent requests (like
> > mod_proxy_ajp does), but this is still not an overall solution, is there
> > one?
> >
> >
> >     Re-thinking the proposed flush mechanism, I don't think using the
> >     flushing will really solve the problem, albeit it probably narrowing
> >     down the window of opportunity for the problem significantly in most
> >     scenarios. I mention this because I'm getting the feeling two
> >     different discussions are being mixed up: Solving the
> >     "keep-alive time out" problem and introducing the
> >     "flush-all-immediately" option. While the latter might improve the
> >     situation with the first, it is no complete solution and there are a
> >     lot of scenarios where it does not apply due to other factors like
> >     filters (as discussed previously).
> >
> >
> > See above, the flushall does not resolve the issue (filters...), but the
> > early prefetch + flushall reduces it considerably, IMHO.
> >
> >
> >     The flush option itself sounds like a good idea, so I see no reason
> >     why not to put it in. I just don't want to loose focus on the actual
> >     problem ;-)
> >
> >
> > I don't see either...
> >
> > Please consider the new (yet) patch attached, can you still encounter
> > some "proxy: error reading status line from remote server" with it?
> > Even without "proxy-flushall" set, you shouldn't, whatever the time
> > taken to read the request first 16K bytes...
> >
> > Regards,
> > Yann.
> >
> >
>
>

Re: mod_proxy, oooled backend connections and the keep-alive race condition

Posted by Yann Ylavic <yl...@gmail.com>.
On Thu, Dec 5, 2013 at 5:45 PM, Yann Ylavic <yl...@gmail.com> wrote:

> On Thu, Dec 5, 2013 at 5:07 PM, Yann Ylavic <yl...@gmail.com> wrote:
>
>> On Thu, Dec 5, 2013 at 4:05 PM, Jim Jagielski <ji...@jagunet.com> wrote:
>>
>>> There hardly seemed any consensus on the patch... It also
>>> seems that it adds more cycles to Apache on the front to
>>> reduce a race condition that can't really be removed.
>>>
>>
>> I don't think more cycles are added by this patch.
>> What is done is that ap_proxy_http_request() is split in two functions,
>> the prefetch part is now in ap_proxy_http_prefetch(), and
>> ap_proxy_http_prefetch()+ap_proxy_http_request() has no more cycles than
>> the previous ap_proxy_http_request().
>> Unless you consider dereferencing pointer arguments a cycles-overhead
>> compared to using local variables...
>>
>
> That's not totally true, maybe you refer to this cycles :
>
>
> +        /* Preserve the header/input brigades since they may be retried.
> */
> +        input_bb = apr_brigade_create(p, backconn->bucket_alloc);
> +        header_bb = apr_brigade_create(p, backconn->bucket_alloc);
>
> +        proxy_buckets_lifetime_transform(r, input_brigade, input_bb);
> +        proxy_buckets_lifetime_transform(r, header_brigade, header_bb);
>
> mod_proxy may need to setaside these buckets.
>

Oups, I meant, since input_brigade has already been setaside 
during prefetch, reading its buckets in
proxy_buckets_lifetime_transform() shouldn't take much time (though a few
more cycles).

Re: mod_proxy, oooled backend connections and the keep-alive race condition

Posted by Yann Ylavic <yl...@gmail.com>.
On Thu, Dec 5, 2013 at 5:07 PM, Yann Ylavic <yl...@gmail.com> wrote:

> On Thu, Dec 5, 2013 at 4:05 PM, Jim Jagielski <ji...@jagunet.com> wrote:
>
>> There hardly seemed any consensus on the patch... It also
>> seems that it adds more cycles to Apache on the front to
>> reduce a race condition that can't really be removed.
>>
>
> I don't think more cycles are added by this patch.
> What is done is that ap_proxy_http_request() is split in two functions,
> the prefetch part is now in ap_proxy_http_prefetch(), and
> ap_proxy_http_prefetch()+ap_proxy_http_request() has no more cycles than
> the previous ap_proxy_http_request().
> Unless you consider dereferencing pointer arguments a cycles-overhead
> compared to using local variables...
>

That's not totally true, maybe you refer to this cycles :

+        /* Preserve the header/input brigades since they may be retried. */
+        input_bb = apr_brigade_create(p, backconn->bucket_alloc);
+        header_bb = apr_brigade_create(p, backconn->bucket_alloc);
+        proxy_buckets_lifetime_transform(r, input_brigade, input_bb);
+        proxy_buckets_lifetime_transform(r, header_brigade, header_bb);

mod_proxy may need to setaside these buckets.

Re: mod_proxy, oooled backend connections and the keep-alive race condition

Posted by Yann Ylavic <yl...@gmail.com>.
On Thu, Dec 5, 2013 at 4:05 PM, Jim Jagielski <ji...@jagunet.com> wrote:

> There hardly seemed any consensus on the patch... It also
> seems that it adds more cycles to Apache on the front to
> reduce a race condition that can't really be removed.
>

I don't think more cycles are added by this patch.
What is done is that ap_proxy_http_request() is split in two functions,
the prefetch part is now in ap_proxy_http_prefetch(), and
ap_proxy_http_prefetch()+ap_proxy_http_request() has no more cycles than
the previous ap_proxy_http_request().
Unless you consider dereferencing pointer arguments a cycles-overhead
compared to using local variables...

I agree the race condition can't really be removed, mod_proxy can (must?)
strive for limiting it.
With this patch and the reslist's TTL below the backend's KeepAliveTimeout,
the race should not occur unless the backend closed before the timeout
(which may indeed be the case when, say, mpm_event is busy).



> IMO, a reverse proxy should get out of the way as
> quickly as possible.
>

I'm not sure to understand this but, IMHO, a reverse proxy should not
"reserve" a backend connection

(possibly) far before using it, and presume it will still be alive when
it's time to use it.



>
> Plus, if we do this here, shouldn't we do it for
> all proxy protocol submodules (fcgi, etc...)?
>

Agreed.




>
> Plus, as mentioned, this single patch includes 2 "fixes"
> which can be/should be independent.
>

This was just to show that "early prefetch" and "flushall" can work
together, but they are indeed 2 different things.
I can split them into different patches, should there be an interest in one
or/and the other.

Regards,
Yann.

Re: mod_proxy, oooled backend connections and the keep-alive race condition

Posted by Jim Jagielski <ji...@jaguNET.com>.
There hardly seemed any consensus on the patch... It also
seems that it adds more cycles to Apache on the front to
reduce a race condition that can't really be removed.
IMO, a reverse proxy should get out of the way as
quickly as possible.

Plus, if we do this here, shouldn't we do it for
all proxy protocol submodules (fcgi, etc...)?

Plus, as mentioned, this single patch includes 2 "fixes"
which can be/should be independent.

On Dec 5, 2013, at 8:06 AM, Yann Ylavic <yl...@gmail.com> wrote:

> Hi Jan,
> 
> I don't think it is fixed in trunk, but I may have missed the commits.
> Which ones are you talking about?
> 
> Regards,
> Yann.
> 
> 
> On Thu, Dec 5, 2013 at 1:51 PM, Jan Kaluža <jk...@redhat.com> wrote:
> Hi,
> 
> should this be fixed in trunk already? I see some commits in proxy code based on your ideas Yann, but I'm not sure if they address this particular problem too.
> 
> Jan Kaluza
> 
> 
> On 10/17/2013 04:52 PM, Yann Ylavic wrote:
> 
> On Thu, Oct 17, 2013 at 11:36 AM, Thomas Eckert
> <thomas.r.w.eckert@gmail.com <ma...@gmail.com>> wrote:
> 
>      > Hence,
>     why cannot mod_proxy_http prefetch the client's body *before* trying
>     anything with the backend connection, and only if it's all good,
>     connect (or reuse a connection to) the
>      > backend and then start forwarding the data immediatly (flushing
>     the prefetched ones) ?
> 
>     Now I'm confused. Are we talking about flushing everything
>     immediately from client to backend (that's what I understood from
>     your first patch proposal in this thread) or are we talking about
>     pre fetching the whole request body from the client and then passing
>     it on to the backend as a whole ?
> 
> 
> Sorry, I really should not have proposed 2 patches in the same thread
> without being clear about their different purposes.
> 
> I first thought flushing everything received from the client to the
> backend immediatly, and disabling *retries* in the 16K-prefetch, would
> address the issue (my first patch,
> "httpd-trunk-mod_proxy_http_flushall.patch"). But as you said in a
> previous message, that did not address the case where the client and/or
> an input filter retain the very first body bytes, and mod_proxy_http
> blocks, and the backend timeouts...
> 
> Hence the second patch ("httpd-trunk-mod_proxy_http_prefetch.patch"),
> which does prefetch *before* establishing/checking the backend's
> connection, so to minimize (to the minimum) the future delay between
> that and the first bytes sent to the backend (hence the chances for it
> to timeout). The prefetch is still up to 16K (same as existing), and not
> the "whole" body of course.
> 
> Then, when it's time to forward (what is available from) the client's
> request, the patched mod_proxy_http will establish or check+reuse a
> backend connection and always flush immediatly what it has  already
> (headers + prefetch, see stream_reqbody_* functions).
> 
> For the remaining request's body chunks (if any), I intentionally did
> not change the current behaviour of letting the output filters choose
> when to flush, this is not the keep-alive timeout anymore, so it becomes
> out of the issue's scope. But I'm definitively +1 to flush-all here too
> (optionaly), since mod_proxy may retain small chunks and can then face
> another backend timeout later! That's where patch1 meets patch2, and
> comes patch3 ("httpd-trunk-mod_proxy_http_prefetch+flushall.patch",
> attached here).
> 
> There is still the potential (minimal) race condition, the backend could
> have closed in the (short) meantime, that's indeed *not* addressed by my
> patch(es).
> 
> 
>     The problem I mentioned in the first message is about treating the
>     backend connection in an error prone way. By not ensuring the
>     connection is still valid - e.g. as seen by the peer - we risk
>     running into this problem no matter what we do elsewhere. So I
>     really think the proper way to handle this is to reopen the
>     connection if necessary - be this with a filter or integrated into
>     the connection handling itself - and only fail after we tried at
>     least once.
> 
> 
> IMHO this is hardly feasible for non-idempotent (and no body) requests
> (without disabling connection reuse, or using "proxy-initial-not-pooled"
> to let the client choose), even if mod_proxy takes care of not sending
> the full request in one time to be able to detect connection problems
> before it's too late (to retry), there is still a possibility that it
> won't know before sending the last chunk, and the issue remain.
> mod_proxy_http addresses retries for requests *with body* by the "ping"
> parameter, maybe it could also retry idempotent requests (like
> mod_proxy_ajp does), but this is still not an overall solution, is there
> one?
> 
> 
>     Re-thinking the proposed flush mechanism, I don't think using the
>     flushing will really solve the problem, albeit it probably narrowing
>     down the window of opportunity for the problem significantly in most
>     scenarios. I mention this because I'm getting the feeling two
>     different discussions are being mixed up: Solving the
>     "keep-alive time out" problem and introducing the
>     "flush-all-immediately" option. While the latter might improve the
>     situation with the first, it is no complete solution and there are a
>     lot of scenarios where it does not apply due to other factors like
>     filters (as discussed previously).
> 
> 
> See above, the flushall does not resolve the issue (filters...), but the
> early prefetch + flushall reduces it considerably, IMHO.
> 
> 
>     The flush option itself sounds like a good idea, so I see no reason
>     why not to put it in. I just don't want to loose focus on the actual
>     problem ;-)
> 
> 
> I don't see either...
> 
> Please consider the new (yet) patch attached, can you still encounter
> some "proxy: error reading status line from remote server" with it?
> Even without "proxy-flushall" set, you shouldn't, whatever the time
> taken to read the request first 16K bytes...
> 
> Regards,
> Yann.
> 
> 


Re: mod_proxy, oooled backend connections and the keep-alive race condition

Posted by Yann Ylavic <yl...@gmail.com>.
Hi Jan,

I don't think it is fixed in trunk, but I may have missed the commits.
Which ones are you talking about?

Regards,
Yann.


On Thu, Dec 5, 2013 at 1:51 PM, Jan Kaluža <jk...@redhat.com> wrote:

> Hi,
>
> should this be fixed in trunk already? I see some commits in proxy code
> based on your ideas Yann, but I'm not sure if they address this particular
> problem too.
>
> Jan Kaluza
>
>
> On 10/17/2013 04:52 PM, Yann Ylavic wrote:
>
>>
>> On Thu, Oct 17, 2013 at 11:36 AM, Thomas Eckert
>> <thomas.r.w.eckert@gmail.com <ma...@gmail.com>> wrote:
>>
>>      > Hence,
>>     why cannot mod_proxy_http prefetch the client's body *before* trying
>>     anything with the backend connection, and only if it's all good,
>>     connect (or reuse a connection to) the
>>      > backend and then start forwarding the data immediatly (flushing
>>     the prefetched ones) ?
>>
>>     Now I'm confused. Are we talking about flushing everything
>>     immediately from client to backend (that's what I understood from
>>     your first patch proposal in this thread) or are we talking about
>>     pre fetching the whole request body from the client and then passing
>>     it on to the backend as a whole ?
>>
>>
>> Sorry, I really should not have proposed 2 patches in the same thread
>> without being clear about their different purposes.
>>
>> I first thought flushing everything received from the client to the
>> backend immediatly, and disabling *retries* in the 16K-prefetch, would
>> address the issue (my first patch,
>> "httpd-trunk-mod_proxy_http_flushall.patch"). But as you said in a
>> previous message, that did not address the case where the client and/or
>> an input filter retain the very first body bytes, and mod_proxy_http
>> blocks, and the backend timeouts...
>>
>> Hence the second patch ("httpd-trunk-mod_proxy_http_prefetch.patch"),
>> which does prefetch *before* establishing/checking the backend's
>> connection, so to minimize (to the minimum) the future delay between
>> that and the first bytes sent to the backend (hence the chances for it
>> to timeout). The prefetch is still up to 16K (same as existing), and not
>> the "whole" body of course.
>>
>> Then, when it's time to forward (what is available from) the client's
>> request, the patched mod_proxy_http will establish or check+reuse a
>> backend connection and always flush immediatly what it has  already
>> (headers + prefetch, see stream_reqbody_* functions).
>>
>> For the remaining request's body chunks (if any), I intentionally did
>> not change the current behaviour of letting the output filters choose
>> when to flush, this is not the keep-alive timeout anymore, so it becomes
>> out of the issue's scope. But I'm definitively +1 to flush-all here too
>> (optionaly), since mod_proxy may retain small chunks and can then face
>> another backend timeout later! That's where patch1 meets patch2, and
>> comes patch3 ("httpd-trunk-mod_proxy_http_prefetch+flushall.patch",
>> attached here).
>>
>> There is still the potential (minimal) race condition, the backend could
>> have closed in the (short) meantime, that's indeed *not* addressed by my
>> patch(es).
>>
>>
>>     The problem I mentioned in the first message is about treating the
>>     backend connection in an error prone way. By not ensuring the
>>     connection is still valid - e.g. as seen by the peer - we risk
>>     running into this problem no matter what we do elsewhere. So I
>>     really think the proper way to handle this is to reopen the
>>     connection if necessary - be this with a filter or integrated into
>>     the connection handling itself - and only fail after we tried at
>>     least once.
>>
>>
>> IMHO this is hardly feasible for non-idempotent (and no body) requests
>> (without disabling connection reuse, or using "proxy-initial-not-pooled"
>> to let the client choose), even if mod_proxy takes care of not sending
>> the full request in one time to be able to detect connection problems
>> before it's too late (to retry), there is still a possibility that it
>> won't know before sending the last chunk, and the issue remain.
>> mod_proxy_http addresses retries for requests *with body* by the "ping"
>> parameter, maybe it could also retry idempotent requests (like
>> mod_proxy_ajp does), but this is still not an overall solution, is there
>> one?
>>
>>
>>     Re-thinking the proposed flush mechanism, I don't think using the
>>     flushing will really solve the problem, albeit it probably narrowing
>>     down the window of opportunity for the problem significantly in most
>>     scenarios. I mention this because I'm getting the feeling two
>>     different discussions are being mixed up: Solving the
>>     "keep-alive time out" problem and introducing the
>>     "flush-all-immediately" option. While the latter might improve the
>>     situation with the first, it is no complete solution and there are a
>>     lot of scenarios where it does not apply due to other factors like
>>     filters (as discussed previously).
>>
>>
>> See above, the flushall does not resolve the issue (filters...), but the
>> early prefetch + flushall reduces it considerably, IMHO.
>>
>>
>>     The flush option itself sounds like a good idea, so I see no reason
>>     why not to put it in. I just don't want to loose focus on the actual
>>     problem ;-)
>>
>>
>> I don't see either...
>>
>> Please consider the new (yet) patch attached, can you still encounter
>> some "proxy: error reading status line from remote server" with it?
>> Even without "proxy-flushall" set, you shouldn't, whatever the time
>> taken to read the request first 16K bytes...
>>
>> Regards,
>> Yann.
>>
>
>