You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Plüm,
Rüdiger,
Vodafone Group <ru...@vodafone.com> on 2019/05/13 12:42:38 UTC
Questions with mod_proxy_http2
I recently started using mod_proxy_http2 and some questions popped up for me:
1. Why do we retry 5 times hardcoded (leading to AH10023 in case we fail)?
The other protocols only try to retry once if the ping failed (additional Expect 100 header in the HTTP case, PING packet in the AJP case).
2. Could we try to leverage the ping configuration parameter for workers used by HTTP / AJP by sending a PING frame on the HTTP/2 backend
connection and wait for the reply for the configured seconds in ping, retry once if failed with a new connection and return 503 if failed again
or continue processing if things were fine?
Regards
Rüdiger
C2 General
Re: Questions with mod_proxy_http2
Posted by Ruediger Pluem <rp...@apache.org>.
On 05/14/2019 09:38 AM, Stefan Eissing wrote:
>
>
>> Am 14.05.2019 um 09:30 schrieb Ruediger Pluem <rp...@apache.org>:
>>
>>
>>
>> On 05/13/2019 03:30 PM, Stefan Eissing wrote:
>>>
>>>
>>>> Am 13.05.2019 um 14:42 schrieb Plüm, Rüdiger, Vodafone Group <ru...@vodafone.com>:
>>>>
>>>> I recently started using mod_proxy_http2 and some questions popped up for me:
>>>>
>>>> 1. Why do we retry 5 times hardcoded (leading to AH10023 in case we fail)?
>>>> The other protocols only try to retry once if the ping failed (additional Expect 100 header in the HTTP case, PING packet in the AJP case).
>>>
>>> I think that is buried in history. I cannot think of a good reason for it.
>>
>> Thanks for confirmation. For trunk my immediate idea would be to change '5' to '2' and thus be somewhat in line with the
>> other schemas. But I guess this is not possible for 2.4.x due to compatibility concerns. I guess we need to make it
>> configurable there with a default of 5. Opinions?
>
> mod_proxy_http2 is still experimental. Feel free to change anything that improves in your setup!
Done on trunk in r1859213. So anybody objections if I backport this straight to 2.4.x?
>
>>>
>>>> 2. Could we try to leverage the ping configuration parameter for workers used by HTTP / AJP by sending a PING frame on the HTTP/2 backend
>>>> connection and wait for the reply for the configured seconds in ping, retry once if failed with a new connection and return 503 if failed again
>>>> or continue processing if things were fine?
>>>
>>> Is that something the module can provide directly or is this for the proxy/balancer infrastructure?
>>
>> This is something the module provides. It just needs to reply with a 503 if the worker is seen as faulty. This causes
>> a possibly configured loadbalancer that has this worker as member to fail over to a different member.
>>
>> I understand the modules current behavior as follows:
>>
>> 1. If the TCP connection is reused and no frame was received within the last second a PING frame is added before
>> the stream for the request is set up.
>> 2. The request body if any is not sent until the PING reply has arrived.
>> 3. If the ping does not arrive reply with 503.
>
> Yes, that was my intention. I added it to prevent a race with a GOAWAY frame from the backend (e.g. its keepalive timed out).
>
>> The issue I see with the above is:
>>
>> 1. Once the request is sent only idempotent requests could be sent to another worker in case we have a loadbalancer
>> setup. Once a non idempotent request is sent we cannot be sure if it was not processed somehow by the backend.
>> 2. Even if for new connections a TCP connection can be established it is not clear if the backend is ready to process it
>> due to things like backlogs, deferred accept setups or separate accept threads. Using a different short ping
>> timeout removes these uncertainties. With regards to the ping overhead this should only be done if the admin
>> configured the ping and hence is aware of the additional overhead with respect to traffic and latency.
>>
>> A similar behavior to the other modules would be
>>
>> 1. Sent the PING frame no matter if it is a new or existing connection in case the ping option is configured on
>> the worker.
>> 2. Wait for the time configured in the ping option for the PING reply to arrive. If it does not arrive reply with a 503,
>> if it does continue with the request.
>
> I agree to your observation in regard to idempotency. Holding back the body alone is not enough. Always sending a PING simplifies things, although for a new connection, the arrival of the remote SETTINGS frames should be indiction enough that the HTTP/2 is sound. But since latency to backend usually is low, this may not be an issue.
>
I guess seeing any frame returning from the backend within the timeout set by ping before sending the request should be
enough, but PING seems to be the correct thing to use for these kind of tests.
The important thing is that if configured we wait for ping time to see something from the backend until we send the
request.
Regards
Rüdiger
Re: Questions with mod_proxy_http2
Posted by Stefan Eissing <st...@greenbytes.de>.
> Am 14.05.2019 um 09:30 schrieb Ruediger Pluem <rp...@apache.org>:
>
>
>
> On 05/13/2019 03:30 PM, Stefan Eissing wrote:
>>
>>
>>> Am 13.05.2019 um 14:42 schrieb Plüm, Rüdiger, Vodafone Group <ru...@vodafone.com>:
>>>
>>> I recently started using mod_proxy_http2 and some questions popped up for me:
>>>
>>> 1. Why do we retry 5 times hardcoded (leading to AH10023 in case we fail)?
>>> The other protocols only try to retry once if the ping failed (additional Expect 100 header in the HTTP case, PING packet in the AJP case).
>>
>> I think that is buried in history. I cannot think of a good reason for it.
>
> Thanks for confirmation. For trunk my immediate idea would be to change '5' to '2' and thus be somewhat in line with the
> other schemas. But I guess this is not possible for 2.4.x due to compatibility concerns. I guess we need to make it
> configurable there with a default of 5. Opinions?
mod_proxy_http2 is still experimental. Feel free to change anything that improves in your setup!
>>
>>> 2. Could we try to leverage the ping configuration parameter for workers used by HTTP / AJP by sending a PING frame on the HTTP/2 backend
>>> connection and wait for the reply for the configured seconds in ping, retry once if failed with a new connection and return 503 if failed again
>>> or continue processing if things were fine?
>>
>> Is that something the module can provide directly or is this for the proxy/balancer infrastructure?
>
> This is something the module provides. It just needs to reply with a 503 if the worker is seen as faulty. This causes
> a possibly configured loadbalancer that has this worker as member to fail over to a different member.
>
> I understand the modules current behavior as follows:
>
> 1. If the TCP connection is reused and no frame was received within the last second a PING frame is added before
> the stream for the request is set up.
> 2. The request body if any is not sent until the PING reply has arrived.
> 3. If the ping does not arrive reply with 503.
Yes, that was my intention. I added it to prevent a race with a GOAWAY frame from the backend (e.g. its keepalive timed out).
> The issue I see with the above is:
>
> 1. Once the request is sent only idempotent requests could be sent to another worker in case we have a loadbalancer
> setup. Once a non idempotent request is sent we cannot be sure if it was not processed somehow by the backend.
> 2. Even if for new connections a TCP connection can be established it is not clear if the backend is ready to process it
> due to things like backlogs, deferred accept setups or separate accept threads. Using a different short ping
> timeout removes these uncertainties. With regards to the ping overhead this should only be done if the admin
> configured the ping and hence is aware of the additional overhead with respect to traffic and latency.
>
> A similar behavior to the other modules would be
>
> 1. Sent the PING frame no matter if it is a new or existing connection in case the ping option is configured on
> the worker.
> 2. Wait for the time configured in the ping option for the PING reply to arrive. If it does not arrive reply with a 503,
> if it does continue with the request.
I agree to your observation in regard to idempotency. Holding back the body alone is not enough. Always sending a PING simplifies things, although for a new connection, the arrival of the remote SETTINGS frames should be indiction enough that the HTTP/2 is sound. But since latency to backend usually is low, this may not be an issue.
- Stefan
>
> Regards
>
> Rüdiger
Re: Questions with mod_proxy_http2
Posted by Ruediger Pluem <rp...@apache.org>.
On 05/13/2019 03:30 PM, Stefan Eissing wrote:
>
>
>> Am 13.05.2019 um 14:42 schrieb Plüm, Rüdiger, Vodafone Group <ru...@vodafone.com>:
>>
>> I recently started using mod_proxy_http2 and some questions popped up for me:
>>
>> 1. Why do we retry 5 times hardcoded (leading to AH10023 in case we fail)?
>> The other protocols only try to retry once if the ping failed (additional Expect 100 header in the HTTP case, PING packet in the AJP case).
>
> I think that is buried in history. I cannot think of a good reason for it.
Thanks for confirmation. For trunk my immediate idea would be to change '5' to '2' and thus be somewhat in line with the
other schemas. But I guess this is not possible for 2.4.x due to compatibility concerns. I guess we need to make it
configurable there with a default of 5. Opinions?
>
>> 2. Could we try to leverage the ping configuration parameter for workers used by HTTP / AJP by sending a PING frame on the HTTP/2 backend
>> connection and wait for the reply for the configured seconds in ping, retry once if failed with a new connection and return 503 if failed again
>> or continue processing if things were fine?
>
> Is that something the module can provide directly or is this for the proxy/balancer infrastructure?
This is something the module provides. It just needs to reply with a 503 if the worker is seen as faulty. This causes
a possibly configured loadbalancer that has this worker as member to fail over to a different member.
I understand the modules current behavior as follows:
1. If the TCP connection is reused and no frame was received within the last second a PING frame is added before
the stream for the request is set up.
2. The request body if any is not sent until the PING reply has arrived.
3. If the ping does not arrive reply with 503.
The issue I see with the above is:
1. Once the request is sent only idempotent requests could be sent to another worker in case we have a loadbalancer
setup. Once a non idempotent request is sent we cannot be sure if it was not processed somehow by the backend.
2. Even if for new connections a TCP connection can be established it is not clear if the backend is ready to process it
due to things like backlogs, deferred accept setups or separate accept threads. Using a different short ping
timeout removes these uncertainties. With regards to the ping overhead this should only be done if the admin
configured the ping and hence is aware of the additional overhead with respect to traffic and latency.
A similar behavior to the other modules would be
1. Sent the PING frame no matter if it is a new or existing connection in case the ping option is configured on
the worker.
2. Wait for the time configured in the ping option for the PING reply to arrive. If it does not arrive reply with a 503,
if it does continue with the request.
Regards
Rüdiger
Re: Questions with mod_proxy_http2
Posted by Stefan Eissing <st...@greenbytes.de>.
> Am 13.05.2019 um 14:42 schrieb Plüm, Rüdiger, Vodafone Group <ru...@vodafone.com>:
>
> I recently started using mod_proxy_http2 and some questions popped up for me:
>
> 1. Why do we retry 5 times hardcoded (leading to AH10023 in case we fail)?
> The other protocols only try to retry once if the ping failed (additional Expect 100 header in the HTTP case, PING packet in the AJP case).
I think that is buried in history. I cannot think of a good reason for it.
> 2. Could we try to leverage the ping configuration parameter for workers used by HTTP / AJP by sending a PING frame on the HTTP/2 backend
> connection and wait for the reply for the configured seconds in ping, retry once if failed with a new connection and return 503 if failed again
> or continue processing if things were fine?
Is that something the module can provide directly or is this for the proxy/balancer infrastructure?
> Regards
>
> Rüdiger
>
> C2 General
-Stefan