You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by pratyoosh sharma <pr...@gmail.com> on 2010/07/16 06:10:34 UTC

[users@httpd] Apache 2.2.11 mod_proxy returns 503 (service unavailable) under load

We are running a high volume portal that uses Apache 2.2.11 mod_proxy
to reverse proxy content from downstream apache-> application servers,
we recently started observing 503 errors on the Apache which happen
sporadically throughout the day during activity on random requests.

The requests returned with 503 don't appear on the backend apache
access logs at all, there are no 503 failures on backend
apache.Network traces of connectivity between frontend apache & the
backend don't reveal anything either.

An interesting pattern is the failed requests always take ~3-4 seconds

Frontend Apache:
Error log:
[Wed Jul 14 15:59:46 2010] [error] (110)Connection timed out: proxy:
HTTP: attempt to connect to 1xx.7x.xxx.xxx:80 (web.mycompany.net)
failed
[Wed Jul 14 15:59:46 2010] [error] ap_proxy_connect_backend disabling
worker for (web.mycompany.net)
Access log:
"USER13" xx.xx.xxx.xxx - [14/Jul/2010:15:59:43 -0400] "GET
/myapp/web/css/ext-all.css HTTP/1.1" 503 1030  3041412

Backend:
This request never reached the backend server & doesn't shows up in
the access logs of backend server


All the failing requests like the above take ~3 seconds so i guess its
like a timeout but i can't find any such configuration.

Is there some known Apache bug for this behavior? We have retry=0 &
disablereuse on for all proxy pass directives

What kind of starvation should i look for in Apache we have mod_status
enabled, file descriptor limit set a 64K, CPU is almost always idle &
25 Gig memory free on the server.

Thanks

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Re: [users@httpd] Apache 2.2.11 mod_proxy returns 503 (service unavailable) under load

Posted by pratyoosh sharma <pr...@gmail.com>.
We finally figured out the reason for these 503 errors was an actual
connectivity issue to the backend. net.ipv4.tcp_syn_retries kernel
parameter on Linux was set as 0 maybe to avoid DDoS attacks this was
causing connect to fail on the first try, setting it to the OS default
5 causes OS to send SYN packets 5 times to initiate 3 way handshake
before it gives up on connecting & resolves the 503 issues.

Not that this actually undermines underlying network issues, but this
stops the bleeding immediately therefore interesting value to verify
when looking into similar issues.

On Fri, Jul 16, 2010 at 4:01 AM, alin vasile <al...@yahoo.com> wrote:
> Hi,
>
>  Since in  your apache logs you can see a  timeout while connecting to the
> backend :
>
>   "[Wed Jul 14 15:59:46 2010] [error] (110)Connection timed out: proxy:
> HTTP: attempt to connect to 1xx.7x.xxx.xxx:80 (web.mycompany.net)
> failed"
>
>   it is normal that there not to have any logs at that time.
>
>   The problem here is that your backend may not support a spike load or can
> hove some capacity problems.
>
> ________________________________
> From: pratyoosh sharma <pr...@gmail.com>
> To: users@httpd.apache.org
> Sent: Fri, July 16, 2010 7:10:34 AM
> Subject: [users@httpd] Apache 2.2.11 mod_proxy returns 503 (service
> unavailable) under load
>
> We are running a high volume portal that uses Apache 2.2.11 mod_proxy
> to reverse proxy content from downstream apache-> application servers,
> we recently started observing 503 errors on the Apache which happen
> sporadically throughout the day during activity on random requests.
>
> The requests returned with 503 don't appear on the backend apache
> access logs at all, there are no 503 failures on backend
> apache.Network traces of connectivity between frontend apache & the
> backend don't reveal anything either.
>
> An interesting pattern is the failed requests always take ~3-4 seconds
>
> Frontend Apache:
> Error log:
> [Wed Jul 14 15:59:46 2010] [error] (110)Connection timed out: proxy:
> HTTP: attempt to connect to 1xx.7x.xxx.xxx:80 (web.mycompany.net)
> failed
> [Wed Jul 14 15:59:46 2010] [error] ap_proxy_connect_backend disabling
> worker for (web.mycompany.net)
> Access log:
> "USER13" xx.xx.xxx.xxx - [14/Jul/2010:15:59:43 -0400] "GET
> /myapp/web/css/ext-all.css HTTP/1.1" 503 1030  3041412
>
> Backend:
> This request never reached the backend server & doesn't shows up in
> the access logs of backend server
>
>
> All the failing requests like the above take ~3 seconds so i guess its
> like a timeout but i can't find any such configuration.
>
> Is there some known Apache bug for this behavior? We have retry=0 &
> disablereuse on for all proxy pass directives
>
> What kind of starvation should i look for in Apache we have mod_status
> enabled, file descriptor limit set a 64K, CPU is almost always idle &
> 25 Gig memory free on the server.
>
> Thanks
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>   "  from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
>



-- 
Thanks & Regards,
Pratyoosh Sharma

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Re: [users@httpd] Apache 2.2.11 mod_proxy returns 503 (service unavailable) under load

Posted by alin vasile <al...@yahoo.com>.
Hi,

 Since in  your apache logs you can see a  timeout while connecting to the 
backend :

  "[Wed Jul 14 15:59:46 2010] [error] (110)Connection timed out: proxy:
HTTP:  attempt to connect to 1xx.7x.xxx.xxx:80 (web.mycompany.net)
failed"

  it is normal that there not to have any logs at that time. 

  The problem here is that your backend may not support a spike load or can hove 
some capacity problems.




________________________________
From: pratyoosh sharma <pr...@gmail.com>
To: users@httpd.apache.org
Sent: Fri, July 16, 2010 7:10:34 AM
Subject: [users@httpd] Apache 2.2.11 mod_proxy returns 503 (service unavailable) 
under load

We are running a high volume portal that uses Apache 2.2.11 mod_proxy
to reverse proxy content from downstream apache-> application servers,
we recently started observing 503 errors on the Apache which happen
sporadically throughout the day during activity on random requests.

The requests returned with 503 don't appear on the backend apache
access logs at all, there are no 503 failures on backend
apache.Network traces of connectivity between frontend apache & the
backend don't reveal anything either.

An interesting pattern is the failed requests always take ~3-4 seconds

Frontend Apache:
Error log:
[Wed Jul 14 15:59:46 2010] [error] (110)Connection timed out: proxy:
HTTP: attempt to connect to 1xx.7x.xxx.xxx:80 (web.mycompany.net)
failed
[Wed Jul 14 15:59:46 2010] [error] ap_proxy_connect_backend disabling
worker for (web.mycompany.net)
Access log:
"USER13" xx.xx.xxx.xxx - [14/Jul/2010:15:59:43 -0400] "GET
/myapp/web/css/ext-all.css HTTP/1.1" 503 1030  3041412

Backend:
This request never reached the backend server & doesn't shows up in
the access logs of backend server


All the failing requests like the above take ~3 seconds so i guess its
like a timeout but i can't find any such configuration.

Is there some known Apache bug for this behavior? We have retry=0 &
disablereuse on for all proxy pass directives

What kind of starvation should i look for in Apache we have mod_status
enabled, file descriptor limit set a 64K, CPU is almost always idle &
25 Gig memory free on the server.

Thanks

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org