You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Ahmed Bakir <ab...@gmail.com> on 2010/11/17 20:18:57 UTC
[users@httpd] httpd choking (503 errors) when stressing mod_proxy
Hi all,
For the application I am working on right now, I need to send large files
between two servers via Apache. I am using reverse proxying to accomplish
this. When I hammer my host with requests from the client (several requests
in short succession), error.log starts to fill up with "OS Error 10055"
messages.
I notice this problem on httpd 2.2.17; I do not see it on httpd 2.0.64.
Other than the standard conversion of some module names (ex, mod_access) for
compatibility with 2.0.64, my httpd.conf did not change.
Could anyone provide some insight on the different behavior between 2.0.x
and 2.2.x?
Thanks in advance!
Ahmed
Re: [users@httpd] httpd choking (503 errors) when stressing mod_proxy
Posted by Jeroen Geilman <je...@adaptr.nl>.
On 11/17/2010 08:18 PM, Ahmed Bakir wrote:
> Hi all,
>
> For the application I am working on right now, I need to send large
> files between two servers via Apache. I am using reverse proxying to
> accomplish this. When I hammer my host with requests from the client
> (several requests in short succession), error.log starts to fill up
> with "OS Error 10055" messages.
OS error 10055 = No buffer space available.
You need to increase your TCP receive buffers, because apache cannot
properly buffer the data between the client and the backend, and has to
tell the client it cannot serve the request (which is a 503).
Is there a significant bandwidth difference between the connections ?
>
> I notice this problem on httpd 2.2.17; I do not see it on httpd
> 2.0.64. Other than the standard conversion of some module names (ex,
> mod_access) for compatibility with 2.0.64, my httpd.conf did not change.
However, the behaviour of settings may have changed.
We're going to need a lot more detailed information, such as: MPM used ,
MPM settings, maxclients, maxrequestsperchild, keepalive and timeout
settings, proxypass options set, etc.
--
J.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] httpd choking (503 errors) when stressing mod_proxy
Posted by "William A. Rowe Jr." <wr...@rowe-clan.net>.
On 11/18/2010 3:03 PM, Ahmed Bakir wrote:
> Thanks for the tip, Igor.
>
> What do you think might be the discrepancy between 2.0 and 2.2? Does 2.2 just serve
> requests faster?
mod_proxy_balancer after 2.2.9 is an entirely unrelated beast to the earlier
monster. By conserving/recycling the connections, it is far more efficient.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] httpd choking (503 errors) when stressing mod_proxy
Posted by Ahmed Bakir <ab...@gmail.com>.
Thanks for the tip, Igor.
What do you think might be the discrepancy between 2.0 and 2.2? Does 2.2
just serve requests faster?
Regards,
Ahmed
2010/11/18 Igor Galić <i....@brainsware.org>
>
> ----- "Ahmed Bakir" <ab...@gmail.com> wrote:
>
> > Ugh, it reared it's ugly head again.
> >
> > When i try to use ProxyPassMatch, i get the nasty 503 errors again
>
> 503 means Service Unavailable
>
> > Are mod_proxy/mod_rewrite just totally broken on 2.2 for regular
> > expressions under load?
>
> That doesn't mean your proxy can't take the load, it means your
> back-end can't take the load.
>
> Consider running multiple back-ends with mod_proxy_balancer.
>
> > Thanks,
> > Ahmed
>
> i
>
> --
> Igor Galić
>
> Tel: +43 (0) 664 886 22 883
> Mail: i.galic@brainsware.org
> URL: http://brainsware.org/
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
Re: [users@httpd] httpd choking (503 errors) when stressing
mod_proxy
Posted by Igor Galić <i....@brainsware.org>.
----- "Ahmed Bakir" <ab...@gmail.com> wrote:
> Ugh, it reared it's ugly head again.
>
> When i try to use ProxyPassMatch, i get the nasty 503 errors again
503 means Service Unavailable
> Are mod_proxy/mod_rewrite just totally broken on 2.2 for regular
> expressions under load?
That doesn't mean your proxy can't take the load, it means your
back-end can't take the load.
Consider running multiple back-ends with mod_proxy_balancer.
> Thanks,
> Ahmed
i
--
Igor Galić
Tel: +43 (0) 664 886 22 883
Mail: i.galic@brainsware.org
URL: http://brainsware.org/
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] httpd choking (503 errors) when stressing mod_proxy
Posted by Ahmed Bakir <ab...@gmail.com>.
Ugh, it reared it's ugly head again.
When i try to use ProxyPassMatch, i get the nasty 503 errors again
Are mod_proxy/mod_rewrite just totally broken on 2.2 for regular expressions
under load?
Thanks,
Ahmed
On Thu, Nov 18, 2010 at 9:20 AM, Ahmed Bakir <ab...@gmail.com> wrote:
> Thanks, Jeroen.
>
> I used a proxy block with a proxypass option and that fixed my problem. It
> actually worked perfectly at the base configuration, the proxy pool settings
> etc helped alleviate the problem but did not remove it completely. The same
> was observed for KeepAlive Off.
>
> I still suspect that the 2.2 module for mod_rewrite is broken.
>
> Thanks again,
> Ahmed
>
>
> On Wed, Nov 17, 2010 at 1:35 PM, Jeroen Geilman <je...@adaptr.nl> wrote:
>
>> On 11/17/2010 10:23 PM, Ahmed Bakir wrote:
>>
>>> Thanks for the responses!
>>>
>>> @jeroen:
>>>
>>> - There is no bandwidth difference between the connections. Both the
>>> client and the host are either running on the same machine or on the same
>>> LAN (I have tested on both and see the problem in both cases)
>>> - I have increased TCP receiver buffers, and that has alleviated the
>>> problem but it's not a feasible solution for my application.
>>>
>>
>> Because ?
>>
>>
>> - I am running httpd on Windows (I have tried both Windows 7 and Windows
>>> XP)
>>>
>>
>> Ugh.
>> Fugly TCP stack.
>>
>>
>> - These are my MPM details:
>>>
>>> Timeout 300
>>> MaxKeepAliveRequests 100
>>> KeepAliveTimeout 15
>>>
>>>
>> For a reverse proxy with heavy load, DISABLE keepalives. COMPLETELY.
>> They don't serve any useful purpose, and may in fact slow things down.
>>
>> You won't observe the issues when testing from one client, because that
>> one client will re-use its connections.
>>
>> However, this does not work in the real world - proxy connections are
>> fire-and-forget as far as the server is concerned.
>>
>> Keepalive under heavy load contributes to what you are seeing - service
>> unavailable.
>>
>>
>> <IfModule mpm_winnt.c>
>>> ThreadsPerChild 250
>>> MaxRequestsPerChild 0
>>> </IfModule>
>>>
>>> - I have configured my reverse proxy using rewrite rules with the [P]
>>> directive
>>>
>>> This is an example:
>>>
>>> RewriteRule ^files/(.*)$ http://localhost:16/$1 [P]
>>>
>>
>> Yeah... that means you can't regulate the proxy pool size and other
>> settings.
>> I'd reconsider using rewriterules when you don't really know why you're
>> using them.
>> A proper FilesMatch or Proxy block will do just fine:
>>
>> <Proxy /files/*>
>> ProxyPass http://localhost:16/ min=100 max=250 smax=100 acquire=1000ms
>> </Proxy>
>>
>> For example; I am by no means an expert on proxy configuration, but this
>> I could infer from the documentation in a few minutes.
>>
>> You need to match the number of backend threads to your expected or -
>> better - observed connections.
>>
>>
>> - I did not set any ProxyPass options in my config.
>>>
>>
>> Perhaps you should!
>>
>> --
>> J.
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> The official User-To-User support forum of the Apache HTTP Server Project.
>> See <URL:http://httpd.apache.org/userslist.html> for more info.
>> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>> " from the digest: users-digest-unsubscribe@httpd.apache.org
>> For additional commands, e-mail: users-help@httpd.apache.org
>>
>>
>
Re: [users@httpd] httpd choking (503 errors) when stressing mod_proxy
Posted by Ahmed Bakir <ab...@gmail.com>.
Thanks, Jeroen.
I used a proxy block with a proxypass option and that fixed my problem. It
actually worked perfectly at the base configuration, the proxy pool settings
etc helped alleviate the problem but did not remove it completely. The same
was observed for KeepAlive Off.
I still suspect that the 2.2 module for mod_rewrite is broken.
Thanks again,
Ahmed
On Wed, Nov 17, 2010 at 1:35 PM, Jeroen Geilman <je...@adaptr.nl> wrote:
> On 11/17/2010 10:23 PM, Ahmed Bakir wrote:
>
>> Thanks for the responses!
>>
>> @jeroen:
>>
>> - There is no bandwidth difference between the connections. Both the
>> client and the host are either running on the same machine or on the same
>> LAN (I have tested on both and see the problem in both cases)
>> - I have increased TCP receiver buffers, and that has alleviated the
>> problem but it's not a feasible solution for my application.
>>
>
> Because ?
>
>
> - I am running httpd on Windows (I have tried both Windows 7 and Windows
>> XP)
>>
>
> Ugh.
> Fugly TCP stack.
>
>
> - These are my MPM details:
>>
>> Timeout 300
>> MaxKeepAliveRequests 100
>> KeepAliveTimeout 15
>>
>>
> For a reverse proxy with heavy load, DISABLE keepalives. COMPLETELY.
> They don't serve any useful purpose, and may in fact slow things down.
>
> You won't observe the issues when testing from one client, because that one
> client will re-use its connections.
>
> However, this does not work in the real world - proxy connections are
> fire-and-forget as far as the server is concerned.
>
> Keepalive under heavy load contributes to what you are seeing - service
> unavailable.
>
>
> <IfModule mpm_winnt.c>
>> ThreadsPerChild 250
>> MaxRequestsPerChild 0
>> </IfModule>
>>
>> - I have configured my reverse proxy using rewrite rules with the [P]
>> directive
>>
>> This is an example:
>>
>> RewriteRule ^files/(.*)$ http://localhost:16/$1 [P]
>>
>
> Yeah... that means you can't regulate the proxy pool size and other
> settings.
> I'd reconsider using rewriterules when you don't really know why you're
> using them.
> A proper FilesMatch or Proxy block will do just fine:
>
> <Proxy /files/*>
> ProxyPass http://localhost:16/ min=100 max=250 smax=100 acquire=1000ms
> </Proxy>
>
> For example; I am by no means an expert on proxy configuration, but this I
> could infer from the documentation in a few minutes.
>
> You need to match the number of backend threads to your expected or -
> better - observed connections.
>
>
> - I did not set any ProxyPass options in my config.
>>
>
> Perhaps you should!
>
> --
> J.
>
>
>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
Re: [users@httpd] httpd choking (503 errors) when stressing mod_proxy
Posted by Jeroen Geilman <je...@adaptr.nl>.
On 11/17/2010 10:23 PM, Ahmed Bakir wrote:
> Thanks for the responses!
>
> @jeroen:
>
> - There is no bandwidth difference between the connections. Both the
> client and the host are either running on the same machine or on the
> same LAN (I have tested on both and see the problem in both cases)
> - I have increased TCP receiver buffers, and that has alleviated the
> problem but it's not a feasible solution for my application.
Because ?
> - I am running httpd on Windows (I have tried both Windows 7 and
> Windows XP)
Ugh.
Fugly TCP stack.
> - These are my MPM details:
>
> Timeout 300
> MaxKeepAliveRequests 100
> KeepAliveTimeout 15
>
For a reverse proxy with heavy load, DISABLE keepalives. COMPLETELY.
They don't serve any useful purpose, and may in fact slow things down.
You won't observe the issues when testing from one client, because that
one client will re-use its connections.
However, this does not work in the real world - proxy connections are
fire-and-forget as far as the server is concerned.
Keepalive under heavy load contributes to what you are seeing - service
unavailable.
> <IfModule mpm_winnt.c>
> ThreadsPerChild 250
> MaxRequestsPerChild 0
> </IfModule>
>
> - I have configured my reverse proxy using rewrite rules with the [P]
> directive
>
> This is an example:
>
> RewriteRule ^files/(.*)$ http://localhost:16/$1 [P]
Yeah... that means you can't regulate the proxy pool size and other
settings.
I'd reconsider using rewriterules when you don't really know why you're
using them.
A proper FilesMatch or Proxy block will do just fine:
<Proxy /files/*>
ProxyPass http://localhost:16/ min=100 max=250 smax=100 acquire=1000ms
</Proxy>
For example; I am by no means an expert on proxy configuration, but
this I could infer from the documentation in a few minutes.
You need to match the number of backend threads to your expected or -
better - observed connections.
> - I did not set any ProxyPass options in my config.
Perhaps you should!
--
J.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] httpd choking (503 errors) when stressing mod_proxy
Posted by Ahmed Bakir <ab...@gmail.com>.
Thanks for the responses!
@jeroen:
- There is no bandwidth difference between the connections. Both the client
and the host are either running on the same machine or on the same LAN (I
have tested on both and see the problem in both cases)
- I have increased TCP receiver buffers, and that has alleviated the problem
but it's not a feasible solution for my application.
- I am running httpd on Windows (I have tried both Windows 7 and Windows XP)
- These are my MPM details:
Timeout 300
MaxKeepAliveRequests 100
KeepAliveTimeout 15
<IfModule mpm_winnt.c>
ThreadsPerChild 250
MaxRequestsPerChild 0
</IfModule>
- I have configured my reverse proxy using rewrite rules with the [P]
directive
This is an example:
RewriteRule ^files/(.*)$ http://localhost:16/$1 [P]
- I did not set any ProxyPass options in my config.
@nick:
- The transfer never kicks off because the server is returning 503 errors, I
am interpreting this as both uploads and downloads failing.
- I am not using AcceptFilter; I can research it and see if I can use it in
this application.
- I am not using the proxy balancer
Thanks again,
Ahmed
On Wed, Nov 17, 2010 at 12:43 PM, Nick Kew <ni...@webthing.com> wrote:
> On Wed, 17 Nov 2010 11:18:57 -0800
> Ahmed Bakir <ab...@gmail.com> wrote:
>
> > Could anyone provide some insight on the different behavior between 2.0.x
> > and 2.2.x?
>
> Apart from what adaptr said ...
>
> Is it uploads or downloads that are failing?
> Does the setting of AcceptFilter in 2.2 make any difference to this
> problem?
> Are you using the proxy balancer?
>
> --
> Nick Kew
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
Re: [users@httpd] httpd choking (503 errors) when stressing
mod_proxy
Posted by Nick Kew <ni...@webthing.com>.
On Wed, 17 Nov 2010 11:18:57 -0800
Ahmed Bakir <ab...@gmail.com> wrote:
> Could anyone provide some insight on the different behavior between 2.0.x
> and 2.2.x?
Apart from what adaptr said ...
Is it uploads or downloads that are failing?
Does the setting of AcceptFilter in 2.2 make any difference to this problem?
Are you using the proxy balancer?
--
Nick Kew
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org