You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by dE <de...@gmail.com> on 2016/07/17 07:41:36 UTC
[users@httpd] mod_proxy: When does a backend be considered as failed?
Hi,
It appears that mod_proxy considers a backend as failed only when
the transport layer connection to that backend fails. Is this expected?
The backends are VMs and only when I SIGSTOP the VMs, the backend is
considered in an error state and the retry= parameter has an affect.
If I've set ping=10, the client has to wait for a full minute before a
503 occurs. On subsequent requests, the requests keep coming to this
failed server as if it's healthy effectively making the ping= parameter
pointless (it does nothing).
If a timeout occurs (as set in ProxyTimeout), then too the backend is
not considered as failed and subsequent requests keep coming to it.
When I've set status=E, requests still come to the origin and get served.
failonstatus= has no effect regardless of the value. Requires will
always get served.
Re: [users@httpd] mod_proxy: When does a backend be considered as
failed?
Posted by dE <de...@gmail.com>.
On 07/30/2016 04:37 PM, Yann Ylavic wrote:
> On Sat, Jul 30, 2016 at 8:36 AM, dE <de...@gmail.com> wrote:
>> Yes, as you said I did try that before --
> Your previous configuration had timeout=10 on the ProxyPass line, not
> the BalancerMember one as expected.
>
>> BalancerMember balancer://localbalance/ http://[fc00::1:4]/ timeout=10
>> ProxyPass / balancer://localbalance/ failontimeout=on
>>
>> I even tried the retry (set to 600) and forcerecovery parameter.
> Really, without the timeout=10 on the ProxyPass (like the
> configuration you mentioned in your previous message)?
>
>> Have you ever tried this in the tests that I'm doing (SIGSTOP apache)?
> Actually yes, with that exact configuration:
>
> <VirtualHost *:8080>
> ServerName localhost:8080
> ProxyPass / balancer://localbalance/ failontimeout=on
> failonstatus=502 forcerecovery=off
> BalancerMember balancer://localbalance/ http://127.0.0.1:80/
> timeout=10 retry=30
> </VirtualHost>
>
>
> 1. The backend is initially not started:
>
> $ time nc localhost 8080 <<\EOF
> GET / HTTP/1.1
> Host: localhost:8080
>
> EOF
> HTTP/1.1 503 Service Unavailable
> Date: Sat, 30 Jul 2016 10:33:29 GMT
> Server: Apache/2.4.24-dev (Unix) OpenSSL/1.0.2g
> [...]
> real 0m0.005s
> user 0m0.000s
> sys 0m0.000s
>
> $ tail -3 error_log
> [Sat Jul 30 12:33:29.725314 2016] [proxy:error] [pid 14463:tid
> 139636299003648] (111)Connection refused: AH00957: HTTP: attempt to
> connect to 127.0.0.1:80 (127.0.0.1) failed
> [Sat Jul 30 12:33:29.725392 2016] [proxy:error] [pid 14463:tid
> 139636299003648] AH00959: ap_proxy_connect_backend disabling worker
> for (127.0.0.1) for 30s
> [Sat Jul 30 12:33:29.725415 2016] [proxy_http:error] [pid 14463:tid
> 139636299003648] [client ::1:50006] AH01114: HTTP: failed to make
> connection to backend: 127.0.0.1
>
>
> 2. The backend is still not started, ~6 seconds later (i.e. < 30s):
> GET / HTTP/1.1
> Host: localhost:8080
>
> EOF
> HTTP/1.1 503 Service Unavailable
> Date: Sat, 30 Jul 2016 10:33:35 GMT
> Server: Apache/2.4.24-dev (Unix) OpenSSL/1.0.2g
> [..]
> real 0m0.004s
> user 0m0.000s
> sys 0m0.000s
>
> $ tail -1 error_log
> [Sat Jul 30 12:33:35.574164 2016] [proxy_balancer:error] [pid
> 14463:tid 139636381173504] [client ::1:50010] AH01170:
> balancer://localbalance: All workers are in error state
>
>
> 3. The backend is now started, ~15s later (i.e. still < 30s):
>
> $ time nc localhost 8080 <<\EOF
> GET / HTTP/1.1
> Host: localhost:8080
>
> EOF
> HTTP/1.1 503 Service Unavailable
> Date: Sat, 30 Jul 2016 10:33:44 GMT
> Server: Apache/2.4.24-dev (Unix) OpenSSL/1.0.2g
> [...]
> real 0m0.004s
> user 0m0.000s
> sys 0m0.000s
>
> $ tail -1 error_log
> [Sat Jul 30 12:33:44.327913 2016] [proxy_balancer:error] [pid
> 14463:tid 139636282218240] [client ::1:50012] AH01170:
> balancer://localbalance: All workers are in error state
>
>
> 4. The backend is still running normally, ~1mn later (i.e. > 30s):
> $ time nc localhost 8080 <<\EOF
> GET / HTTP/1.1
> Host: localhost:8080
>
> EOF
> HTTP/1.1 200 OK
> Date: Sat, 30 Jul 2016 10:34:40 GMT
> Server: Apache
> [...]
> real 0m0.006s
> user 0m0.004s
> sys 0m0.000s
>
>
> 5. The backend is SIGSTOP-ed (no VM used, so "ServerLimit 1" and kill
> -STOP the only child):
> $ time nc localhost 8080 <<\EOF
> GET / HTTP/1.1
> Host: localhost:8080
>
> EOF
> HTTP/1.1 502 Proxy Error
> Date: Sat, 30 Jul 2016 10:34:51 GMT
> Server: Apache/2.4.24-dev (Unix) OpenSSL/1.0.2g
> [...]
> real 0m10.015s
> user 0m0.000s
> sys 0m0.004s
>
> $ tail -4 error_log
> [Sat Jul 30 12:35:01.355768 2016] [proxy_http:error] [pid 14463:tid
> 139636273825536] (70007)The timeout specified has expired: [client
> ::1:50018] AH01102: error reading status line from remote server
> 127.0.0.1:80
> [Sat Jul 30 12:35:01.355849 2016] [proxy:error] [pid 14463:tid
> 139636273825536] [client ::1:50018] AH00898: Error reading from remote
> server returned by /
> [Sat Jul 30 12:35:01.355942 2016] [proxy_balancer:error] [pid
> 14463:tid 139636273825536] [client ::1:50018] AH01174:
> balancer://localbalance: Forcing worker (http://127.0.0.1/) into error
> state due to status code 502 matching 'failonstatus' balancer
> parameter
> [Sat Jul 30 12:35:01.355964 2016] [proxy_balancer:error] [pid
> 14463:tid 139636273825536] [client ::1:50018] AH02460:
> balancer://localbalance: Forcing worker (http://127.0.0.1/) into error
> state due to timeout and 'failontimeout' parameter being set
>
>
> 6. The backend is still SIGSTOP-ed, ~25s later (i.e. < 30s):
> $ time nc localhost 8080 <<\EOF
> GET / HTTP/1.1
> Host: localhost:8080
>
> EOF
> HTTP/1.1 503 Service Unavailable
> Date: Sat, 30 Jul 2016 10:35:26 GMT
> Server: Apache/2.4.24-dev (Unix) OpenSSL/1.0.2g
> [...]
> real 0m0.004s
> user 0m0.000s
> sys 0m0.000s
>
> $ tail -1 error_log
> [Sat Jul 30 12:35:26.797383 2016] [proxy_balancer:error] [pid
> 14463:tid 139636265432832] [client ::1:50022] AH01170:
> balancer://localbalance: All workers are in error state
>
>
> So everything works as expected for me, timeouts are always 10s max,
> and the worker never wakes up before the retry= period...
>
> Regards,
> Yann.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
Got it! It works now. Thank you so much!
I just forgot a few parameters.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] mod_proxy: When does a backend be considered as failed?
Posted by Yann Ylavic <yl...@gmail.com>.
On Sat, Jul 30, 2016 at 8:36 AM, dE <de...@gmail.com> wrote:
>
> Yes, as you said I did try that before --
Your previous configuration had timeout=10 on the ProxyPass line, not
the BalancerMember one as expected.
>
> BalancerMember balancer://localbalance/ http://[fc00::1:4]/ timeout=10
> ProxyPass / balancer://localbalance/ failontimeout=on
>
> I even tried the retry (set to 600) and forcerecovery parameter.
Really, without the timeout=10 on the ProxyPass (like the
configuration you mentioned in your previous message)?
>
> Have you ever tried this in the tests that I'm doing (SIGSTOP apache)?
Actually yes, with that exact configuration:
<VirtualHost *:8080>
ServerName localhost:8080
ProxyPass / balancer://localbalance/ failontimeout=on
failonstatus=502 forcerecovery=off
BalancerMember balancer://localbalance/ http://127.0.0.1:80/
timeout=10 retry=30
</VirtualHost>
1. The backend is initially not started:
$ time nc localhost 8080 <<\EOF
GET / HTTP/1.1
Host: localhost:8080
EOF
HTTP/1.1 503 Service Unavailable
Date: Sat, 30 Jul 2016 10:33:29 GMT
Server: Apache/2.4.24-dev (Unix) OpenSSL/1.0.2g
[...]
real 0m0.005s
user 0m0.000s
sys 0m0.000s
$ tail -3 error_log
[Sat Jul 30 12:33:29.725314 2016] [proxy:error] [pid 14463:tid
139636299003648] (111)Connection refused: AH00957: HTTP: attempt to
connect to 127.0.0.1:80 (127.0.0.1) failed
[Sat Jul 30 12:33:29.725392 2016] [proxy:error] [pid 14463:tid
139636299003648] AH00959: ap_proxy_connect_backend disabling worker
for (127.0.0.1) for 30s
[Sat Jul 30 12:33:29.725415 2016] [proxy_http:error] [pid 14463:tid
139636299003648] [client ::1:50006] AH01114: HTTP: failed to make
connection to backend: 127.0.0.1
2. The backend is still not started, ~6 seconds later (i.e. < 30s):
GET / HTTP/1.1
Host: localhost:8080
EOF
HTTP/1.1 503 Service Unavailable
Date: Sat, 30 Jul 2016 10:33:35 GMT
Server: Apache/2.4.24-dev (Unix) OpenSSL/1.0.2g
[..]
real 0m0.004s
user 0m0.000s
sys 0m0.000s
$ tail -1 error_log
[Sat Jul 30 12:33:35.574164 2016] [proxy_balancer:error] [pid
14463:tid 139636381173504] [client ::1:50010] AH01170:
balancer://localbalance: All workers are in error state
3. The backend is now started, ~15s later (i.e. still < 30s):
$ time nc localhost 8080 <<\EOF
GET / HTTP/1.1
Host: localhost:8080
EOF
HTTP/1.1 503 Service Unavailable
Date: Sat, 30 Jul 2016 10:33:44 GMT
Server: Apache/2.4.24-dev (Unix) OpenSSL/1.0.2g
[...]
real 0m0.004s
user 0m0.000s
sys 0m0.000s
$ tail -1 error_log
[Sat Jul 30 12:33:44.327913 2016] [proxy_balancer:error] [pid
14463:tid 139636282218240] [client ::1:50012] AH01170:
balancer://localbalance: All workers are in error state
4. The backend is still running normally, ~1mn later (i.e. > 30s):
$ time nc localhost 8080 <<\EOF
GET / HTTP/1.1
Host: localhost:8080
EOF
HTTP/1.1 200 OK
Date: Sat, 30 Jul 2016 10:34:40 GMT
Server: Apache
[...]
real 0m0.006s
user 0m0.004s
sys 0m0.000s
5. The backend is SIGSTOP-ed (no VM used, so "ServerLimit 1" and kill
-STOP the only child):
$ time nc localhost 8080 <<\EOF
GET / HTTP/1.1
Host: localhost:8080
EOF
HTTP/1.1 502 Proxy Error
Date: Sat, 30 Jul 2016 10:34:51 GMT
Server: Apache/2.4.24-dev (Unix) OpenSSL/1.0.2g
[...]
real 0m10.015s
user 0m0.000s
sys 0m0.004s
$ tail -4 error_log
[Sat Jul 30 12:35:01.355768 2016] [proxy_http:error] [pid 14463:tid
139636273825536] (70007)The timeout specified has expired: [client
::1:50018] AH01102: error reading status line from remote server
127.0.0.1:80
[Sat Jul 30 12:35:01.355849 2016] [proxy:error] [pid 14463:tid
139636273825536] [client ::1:50018] AH00898: Error reading from remote
server returned by /
[Sat Jul 30 12:35:01.355942 2016] [proxy_balancer:error] [pid
14463:tid 139636273825536] [client ::1:50018] AH01174:
balancer://localbalance: Forcing worker (http://127.0.0.1/) into error
state due to status code 502 matching 'failonstatus' balancer
parameter
[Sat Jul 30 12:35:01.355964 2016] [proxy_balancer:error] [pid
14463:tid 139636273825536] [client ::1:50018] AH02460:
balancer://localbalance: Forcing worker (http://127.0.0.1/) into error
state due to timeout and 'failontimeout' parameter being set
6. The backend is still SIGSTOP-ed, ~25s later (i.e. < 30s):
$ time nc localhost 8080 <<\EOF
GET / HTTP/1.1
Host: localhost:8080
EOF
HTTP/1.1 503 Service Unavailable
Date: Sat, 30 Jul 2016 10:35:26 GMT
Server: Apache/2.4.24-dev (Unix) OpenSSL/1.0.2g
[...]
real 0m0.004s
user 0m0.000s
sys 0m0.000s
$ tail -1 error_log
[Sat Jul 30 12:35:26.797383 2016] [proxy_balancer:error] [pid
14463:tid 139636265432832] [client ::1:50022] AH01170:
balancer://localbalance: All workers are in error state
So everything works as expected for me, timeouts are always 10s max,
and the worker never wakes up before the retry= period...
Regards,
Yann.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] mod_proxy: When does a backend be considered as
failed?
Posted by dE <de...@gmail.com>.
On 07/25/16 17:47, Yann Ylavic wrote:
> On Mon, Jul 25, 2016 at 5:18 AM, dE <de...@gmail.com> wrote:
>> Config --
> []
>> ProxyPass / balancer://localbalance/ failontimeout=on timeout=10 failonstatus=502
> You are defining a balancer here, so the timeout= parameter relates to
> the balancer (like the two others)
> However for a balancer, the timeout= parameter is not a connection
> timeout (like the one used with BalancerMember), but "the maximum time
> to wait for a free worker/member" (e.g. when all the workers are busy
> *or* in error state, the default being to not wait).
> Thus, your are hitting this timeout for each connection in your test.
>
> You probably should use something like this instead:
> ProxyPass / balancer://localbalance/ failontimeout=on
> failonstatus=502 forcerecovery=off
>
> and leave the (connection) timeout setting alone in the BalancerMember
> (as you did already):
> BalancerMember balancer://localbalance/ http://[fc00::1:4]/
> timeout=10 retry=600
>
> Regards,
> Yann.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
Yes, as you said I did try that before --
|BalancerMember balancer://localbalance/ http://[fc00::1:4]/ timeout=10||
||ProxyPass / balancer://localbalance/ failontimeout=on|
I even tried the retry (set to 600) and forcerecovery parameter.
Have you ever tried this in the tests that I'm doing (SIGSTOP apache)?
Re: [users@httpd] mod_proxy: When does a backend be considered as failed?
Posted by Yann Ylavic <yl...@gmail.com>.
On Mon, Jul 25, 2016 at 5:18 AM, dE <de...@gmail.com> wrote:
>
> Config --
[]
>
> ProxyPass / balancer://localbalance/ failontimeout=on timeout=10 failonstatus=502
You are defining a balancer here, so the timeout= parameter relates to
the balancer (like the two others)
However for a balancer, the timeout= parameter is not a connection
timeout (like the one used with BalancerMember), but "the maximum time
to wait for a free worker/member" (e.g. when all the workers are busy
*or* in error state, the default being to not wait).
Thus, your are hitting this timeout for each connection in your test.
You probably should use something like this instead:
ProxyPass / balancer://localbalance/ failontimeout=on
failonstatus=502 forcerecovery=off
and leave the (connection) timeout setting alone in the BalancerMember
(as you did already):
BalancerMember balancer://localbalance/ http://[fc00::1:4]/
timeout=10 retry=600
Regards,
Yann.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] mod_proxy: When does a backend be considered as
failed?
Posted by dE <de...@gmail.com>.
On 07/19/16 17:54, Yann Ylavic wrote:
> Hello,
>
> On Sun, Jul 17, 2016 at 9:41 AM, dE <de...@gmail.com> wrote:
>> It appears that mod_proxy considers a backend as failed only when the
>> transport layer connection to that backend fails. Is this expected?
> Unless failonstatus/failontimeout is used, usually.
>
> Which httpd version are you using?
> Could you please share some minimal configuration to reproduce?
I did try failontimeout with no affect.
Config --
BalancerMember balancer://localbalance/ http://[fc00::1:4]/ timeout=10
ProxyPass / balancer://localbalance/ failontimeout=on
With this config, for every successive request made, waits for 10
seconds before a 502 is returned, when, in the backend, the webserver
process (not the VM) is stopped.
>
>> The backends are VMs and only when I SIGSTOP the VMs, the backend is
>> considered in an error state and the retry= parameter has an affect.
>>
>> If I've set ping=10, the client has to wait for a full minute before a 503
>> occurs. On subsequent requests, the requests keep coming to this failed
>> server as if it's healthy effectively making the ping= parameter pointless
>> (it does nothing).
> The ping parameter is only relevant for requests with a body (e.g.
> POST method), since it uses the HTTP 100-continue mechanism.
> The goal is to retry sending the request immediatly if the first
> 100-continue attempt failed (both tries using the given timeout), and
> only if both attempts failed the backend is elligible for error state,
> according to configured connect/failonstatus/failontimeout rules.
>
>> If a timeout occurs (as set in ProxyTimeout), then too the backend is not
>> considered as failed and subsequent requests keep coming to it.
> A timeout at connect time is turned to a 503 error (Service
> Unavailable), which should trigger an error state for the
> BalancerMember (for "retry" seconds).
> A timeout at response read time is handled by failontimeout (and
> possible failonstatus=502) only to trigger an error state.
I set failonstatus=502 too
ProxyPass / balancer://localbalance/ failontimeout=on timeout=10
failonstatus=502
BalancerMember balancer://localbalance/ http://[fc00::1:4]/ timeout=10
retry=600
> Did you configure multiple BalancerMember(s) and one (at least) is
> still available?
> Otherwise, they may be retried before the end of the pediod if there
> is no backend available at all in the cluster?
> No ProxyErrorOverride configured either?
>
> Regards,
> Yann.
ProxyErrorOverride is off by default. Besides when the origin server has
SIGSTOPed, there is no from it.
forcerecovery maybe causing issues, but setting it to off has no affect.
Thanks for helping out with this!
Re: [users@httpd] mod_proxy: When does a backend be considered as failed?
Posted by Yann Ylavic <yl...@gmail.com>.
Hello,
On Sun, Jul 17, 2016 at 9:41 AM, dE <de...@gmail.com> wrote:
> It appears that mod_proxy considers a backend as failed only when the
> transport layer connection to that backend fails. Is this expected?
Unless failonstatus/failontimeout is used, usually.
Which httpd version are you using?
Could you please share some minimal configuration to reproduce?
>
> The backends are VMs and only when I SIGSTOP the VMs, the backend is
> considered in an error state and the retry= parameter has an affect.
>
> If I've set ping=10, the client has to wait for a full minute before a 503
> occurs. On subsequent requests, the requests keep coming to this failed
> server as if it's healthy effectively making the ping= parameter pointless
> (it does nothing).
The ping parameter is only relevant for requests with a body (e.g.
POST method), since it uses the HTTP 100-continue mechanism.
The goal is to retry sending the request immediatly if the first
100-continue attempt failed (both tries using the given timeout), and
only if both attempts failed the backend is elligible for error state,
according to configured connect/failonstatus/failontimeout rules.
>
> If a timeout occurs (as set in ProxyTimeout), then too the backend is not
> considered as failed and subsequent requests keep coming to it.
A timeout at connect time is turned to a 503 error (Service
Unavailable), which should trigger an error state for the
BalancerMember (for "retry" seconds).
A timeout at response read time is handled by failontimeout (and
possible failonstatus=502) only to trigger an error state.
Did you configure multiple BalancerMember(s) and one (at least) is
still available?
Otherwise, they may be retried before the end of the pediod if there
is no backend available at all in the cluster?
No ProxyErrorOverride configured either?
Regards,
Yann.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
[users@httpd] Re: mod_proxy: When does a backend be considered as failed?
Posted by dE <de...@gmail.com>.
Looks like this module is deprecated.
On 07/17/16 13:11, dE wrote:
> Hi,
> It appears that mod_proxy considers a backend as failed only when
> the transport layer connection to that backend fails. Is this expected?
>
> The backends are VMs and only when I SIGSTOP the VMs, the backend is
> considered in an error state and the retry= parameter has an affect.
>
> If I've set ping=10, the client has to wait for a full minute before a
> 503 occurs. On subsequent requests, the requests keep coming to this
> failed server as if it's healthy effectively making the ping=
> parameter pointless (it does nothing).
>
> If a timeout occurs (as set in ProxyTimeout), then too the backend is
> not considered as failed and subsequent requests keep coming to it.
>
> When I've set status=E, requests still come to the origin and get served.
>
> failonstatus= has no effect regardless of the value. Requires will
> always get served.