You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Skye Poier Nott <sk...@F4.ca> on 2008/03/19 01:51:16 UTC
[users@httpd] mod_proxy performance with apache 2.2
Hi,
I am doing some load testing on a reverse proxy with apache 2.2 +
mod_proxy and I'm getting terrible throughput, I was wondering if
anyone had a suggestion.
I have the following config, all on gigabit ethernet:
1 origin server -> 2 mod_proxy servers -> 4 client simulators (flood)
The origin and proxy servers are configured with 100 name-based
virtual hosts, the proxy servers with config like this:
<VirtualHost *:80>
ServerName w0099.example.com
ProxyPass / http://10.100.10.XX:80/w0099/
ProxyPassReverse / http://10.100.10.XX:80/w0099/
</VirtualHost>
When I start up flood on the client simulators (250 clients each), I
only get about 4 Mbit/sec out of the origin server. I've tried
twiddling all the settings in mod_proxy to no avail (like smax=64
max=512 ttl=120 min=8 acquire=1)
If I set up lighttpd on the proxy servers, I fully saturate the gigE
on the origin server (~500 Mbit/sec)
What could explain this 125x difference in performance? Is there some
sort of resource or lock or something contention in mod_proxy that I
should know about?
Thanks,
Skye
Re: [users@httpd] mod_proxy performance with apache 2.2
Posted by Joshua Slive <jo...@slive.ca>.
On Thu, Mar 20, 2008 at 2:13 PM, Skye Poier Nott <sk...@f4.ca> wrote:
> I originally discovered the problem because the initial "fill" of
> mod_disk_cache from the origin server was taking forever. The stats I
> list below are with mod_proxy only (mod_cache is loaded, but not
> enabled for any host)
I'd recommend trying the dev@httpd list to see if one of the
developers has advice.
Joshua.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] mod_proxy performance with apache 2.2
Posted by Skye Poier Nott <sk...@F4.ca>.
I originally discovered the problem because the initial "fill" of
mod_disk_cache from the origin server was taking forever. The stats I
list below are with mod_proxy only (mod_cache is loaded, but not
enabled for any host)
Skye
On 19-Mar-08, at 3:07 PM, Robert Anderson wrote:
> I had a similar issue that was caused by caching. Do you have
> mod_cache or mod_disk_cache active? are they clearing properly?
>
> Skye Poier Nott wrote:
>>
>> Hi Joshua,
>>
>> I've tried changing MaxClients from 256 to 512 to 1024, no effect.
>> I've tried changing from MPM prefork to worker, and twiddled
>> threads per child, no effect.
>>
>> Just can't seem to get more than about 4 Mbit/sec out of the
>> origin. I'll give dev@ a ping.
>>
>> Thanks,
>> Skye
>>
>>
>> On 19-Mar-08, at 12:28 PM, Joshua Slive wrote:
>>
>>> On Tue, Mar 18, 2008 at 8:51 PM, Skye Poier Nott <sk...@f4.ca> wrote:
>>>> Hi,
>>>>
>>>> I am doing some load testing on a reverse proxy with apache 2.2 +
>>>> mod_proxy and I'm getting terrible throughput, I was wondering if
>>>> anyone had a suggestion.
>>>>
>>>> I have the following config, all on gigabit ethernet:
>>>>
>>>> 1 origin server -> 2 mod_proxy servers -> 4 client simulators
>>>> (flood)
>>>>
>>>> The origin and proxy servers are configured with 100 name-based
>>>> virtual hosts, the proxy servers with config like this:
>>>>
>>>> <VirtualHost *:80>
>>>> ServerName w0099.example.com
>>>> ProxyPass / http://10.100.10.XX:80/w0099/
>>>> ProxyPassReverse / http://10.100.10.XX:80/w0099/
>>>> </VirtualHost>
>>>>
>>>>
>>>> When I start up flood on the client simulators (250 clients
>>>> each), I
>>>> only get about 4 Mbit/sec out of the origin server. I've tried
>>>> twiddling all the settings in mod_proxy to no avail (like smax=64
>>>> max=512 ttl=120 min=8 acquire=1)
>>>>
>>>> If I set up lighttpd on the proxy servers, I fully saturate the
>>>> gigE
>>>> on the origin server (~500 Mbit/sec)
>>>>
>>>> What could explain this 125x difference in performance? Is there
>>>> some
>>>> sort of resource or lock or something contention in mod_proxy
>>>> that I
>>>> should know about?
>>>
>>> I'm not a mod_proxy performance expert (and you might need to ask on
>>> the dev@httpd list to find one), but you should start with the
>>> basics:
>>> have you looked at the mod_status server-status output? Are you
>>> exceeding MaxClients? Is there anything interesting in the error
>>> log?
>>>
>>> Joshua.
>>>
>>> ---------------------------------------------------------------------
>>> The official User-To-User support forum of the Apache HTTP Server
>>> Project.
>>> See <URL:http://httpd.apache.org/userslist.html> for more info.
>>> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>>> " from the digest: users-digest-unsubscribe@httpd.apache.org
>>> For additional commands, e-mail: users-help@httpd.apache.org
>>>
>>
Re: [users@httpd] mod_proxy performance with apache 2.2
Posted by Robert Anderson <ro...@ecsglobal.com>.
I had a similar issue that was caused by caching. Do you have mod_cache
or mod_disk_cache active? are they clearing properly?
Regards,
Robert Anderson
<http://www.ecsglobal.com>
B2B ECS TORONTO
17 Gormley Rd. W.
Richmond Hill, ON, L4E 1A2
TF: 866-927-1824
PH: 905-927-1824
FX: 905-927-0712
CL: 416-270-6255
EMERGENCY: 416-270-6255
EM: sales@ecsglobal.com
B2B ECS SASKATOON
105M-111 Research Drive
Saskatoon, SK, S7N 3R2
TF: 866-927-1824
PH: 306-978-6255
FX: 306-975-0244
EMERGENCY: 306-270-5982
EM: sales@ecsglobal.com
Skye Poier Nott wrote:
> Hi Joshua,
>
> I've tried changing MaxClients from 256 to 512 to 1024, no effect.
> I've tried changing from MPM prefork to worker, and twiddled threads
> per child, no effect.
>
> Just can't seem to get more than about 4 Mbit/sec out of the origin.
> I'll give dev@ a ping.
>
> Thanks,
> Skye
>
>
> On 19-Mar-08, at 12:28 PM, Joshua Slive wrote:
>
>> On Tue, Mar 18, 2008 at 8:51 PM, Skye Poier Nott <sk...@f4.ca> wrote:
>>> Hi,
>>>
>>> I am doing some load testing on a reverse proxy with apache 2.2 +
>>> mod_proxy and I'm getting terrible throughput, I was wondering if
>>> anyone had a suggestion.
>>>
>>> I have the following config, all on gigabit ethernet:
>>>
>>> 1 origin server -> 2 mod_proxy servers -> 4 client simulators (flood)
>>>
>>> The origin and proxy servers are configured with 100 name-based
>>> virtual hosts, the proxy servers with config like this:
>>>
>>> <VirtualHost *:80>
>>> ServerName w0099.example.com
>>> ProxyPass / http://10.100.10.XX:80/w0099/
>>> ProxyPassReverse / http://10.100.10.XX:80/w0099/
>>> </VirtualHost>
>>>
>>>
>>> When I start up flood on the client simulators (250 clients each), I
>>> only get about 4 Mbit/sec out of the origin server. I've tried
>>> twiddling all the settings in mod_proxy to no avail (like smax=64
>>> max=512 ttl=120 min=8 acquire=1)
>>>
>>> If I set up lighttpd on the proxy servers, I fully saturate the gigE
>>> on the origin server (~500 Mbit/sec)
>>>
>>> What could explain this 125x difference in performance? Is there some
>>> sort of resource or lock or something contention in mod_proxy that I
>>> should know about?
>>
>> I'm not a mod_proxy performance expert (and you might need to ask on
>> the dev@httpd list to find one), but you should start with the basics:
>> have you looked at the mod_status server-status output? Are you
>> exceeding MaxClients? Is there anything interesting in the error log?
>>
>> Joshua.
>>
>> ---------------------------------------------------------------------
>> The official User-To-User support forum of the Apache HTTP Server
>> Project.
>> See <URL:http://httpd.apache.org/userslist.html> for more info.
>> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>> " from the digest: users-digest-unsubscribe@httpd.apache.org
>> For additional commands, e-mail: users-help@httpd.apache.org
>>
>
Re: [users@httpd] mod_proxy performance with apache 2.2
Posted by Skye Poier Nott <sk...@F4.ca>.
Hi Joshua,
I've tried changing MaxClients from 256 to 512 to 1024, no effect.
I've tried changing from MPM prefork to worker, and twiddled threads
per child, no effect.
Just can't seem to get more than about 4 Mbit/sec out of the origin.
I'll give dev@ a ping.
Thanks,
Skye
On 19-Mar-08, at 12:28 PM, Joshua Slive wrote:
> On Tue, Mar 18, 2008 at 8:51 PM, Skye Poier Nott <sk...@f4.ca> wrote:
>> Hi,
>>
>> I am doing some load testing on a reverse proxy with apache 2.2 +
>> mod_proxy and I'm getting terrible throughput, I was wondering if
>> anyone had a suggestion.
>>
>> I have the following config, all on gigabit ethernet:
>>
>> 1 origin server -> 2 mod_proxy servers -> 4 client simulators (flood)
>>
>> The origin and proxy servers are configured with 100 name-based
>> virtual hosts, the proxy servers with config like this:
>>
>> <VirtualHost *:80>
>> ServerName w0099.example.com
>> ProxyPass / http://10.100.10.XX:80/w0099/
>> ProxyPassReverse / http://10.100.10.XX:80/w0099/
>> </VirtualHost>
>>
>>
>> When I start up flood on the client simulators (250 clients each), I
>> only get about 4 Mbit/sec out of the origin server. I've tried
>> twiddling all the settings in mod_proxy to no avail (like smax=64
>> max=512 ttl=120 min=8 acquire=1)
>>
>> If I set up lighttpd on the proxy servers, I fully saturate the gigE
>> on the origin server (~500 Mbit/sec)
>>
>> What could explain this 125x difference in performance? Is there
>> some
>> sort of resource or lock or something contention in mod_proxy that I
>> should know about?
>
> I'm not a mod_proxy performance expert (and you might need to ask on
> the dev@httpd list to find one), but you should start with the basics:
> have you looked at the mod_status server-status output? Are you
> exceeding MaxClients? Is there anything interesting in the error log?
>
> Joshua.
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server
> Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
Re: [users@httpd] mod_proxy performance with apache 2.2
Posted by Joshua Slive <jo...@slive.ca>.
On Tue, Mar 18, 2008 at 8:51 PM, Skye Poier Nott <sk...@f4.ca> wrote:
> Hi,
>
> I am doing some load testing on a reverse proxy with apache 2.2 +
> mod_proxy and I'm getting terrible throughput, I was wondering if
> anyone had a suggestion.
>
> I have the following config, all on gigabit ethernet:
>
> 1 origin server -> 2 mod_proxy servers -> 4 client simulators (flood)
>
> The origin and proxy servers are configured with 100 name-based
> virtual hosts, the proxy servers with config like this:
>
> <VirtualHost *:80>
> ServerName w0099.example.com
> ProxyPass / http://10.100.10.XX:80/w0099/
> ProxyPassReverse / http://10.100.10.XX:80/w0099/
> </VirtualHost>
>
>
> When I start up flood on the client simulators (250 clients each), I
> only get about 4 Mbit/sec out of the origin server. I've tried
> twiddling all the settings in mod_proxy to no avail (like smax=64
> max=512 ttl=120 min=8 acquire=1)
>
> If I set up lighttpd on the proxy servers, I fully saturate the gigE
> on the origin server (~500 Mbit/sec)
>
> What could explain this 125x difference in performance? Is there some
> sort of resource or lock or something contention in mod_proxy that I
> should know about?
I'm not a mod_proxy performance expert (and you might need to ask on
the dev@httpd list to find one), but you should start with the basics:
have you looked at the mod_status server-status output? Are you
exceeding MaxClients? Is there anything interesting in the error log?
Joshua.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: mod_proxy performance with apache 2.2
Posted by Skye Poier Nott <sk...@F4.ca>.
Well it appears the bottleneck was related to configuration on either
the proxy or origin.
Changing the origin (10.100.10.65) vhost config to:
UseCanonicalName Off
NameVirtualHost *:80
<VirtualHost *:80>
ServerName none.example.com
VirtualDocumentRoot /path/to/docroot/$0
</VirtualHost>
and the proxy config to:
ProxyPreserveHost On
ProxyPass / http://10.100.10.65/
ProxyPassReverse / http://10.100.10.65/
Now I'm getting 500Mbps instead of 4Mbps. I guess the proxy servers
really didn't like having a couple of hundred VirtualHost blocks with
individual ProxyPass directives.
Case closed!
Skye
Re: mod_proxy performance with apache 2.2
Posted by Skye Poier Nott <sk...@F4.ca>.
Uh, ignore the last configuration question... I just found
"ProxyPreserveHost" *blush*
Skye
Re: mod_proxy performance with apache 2.2
Posted by Skye Poier Nott <sk...@F4.ca>.
Interesting, well maybe a reworked proxy config would help.
How do I proxy requests without rewriting the URL?
Say my client makes a request for http://w0000.example.com/index.html,
which resolves to the IP of my proxy on the client. I want the proxy
to get the file from 10.100.10.65, the origin, and do the same with
virtual hosts w0001.example.com, w0002.example.com, etc.
All the examples in the Apache docs seem to assume the origin
(backend) is only serving one virtual host.
If I configure the proxy with either of
ProxyPass / http://10.100.10.65/
RewriteRule (.*) http://10.100.10.65$1 [P]
Then all the requests on the origin are for the IP of the origin, not
the original named virtual host, so the wrong files get served from
the origin.
The only way I've been able to do this is to do:
RewriteRule (.*) http://%{HTTP_HOST}$1 [P]
and have the proxy server's DNS configured to resolve
wXXXX.example.com to 10.100.10.65, but that's not very nice if I'm
going to have 10,000+ vhosts. I just want to pass the original
request to 10.100.10.65, is this possible???
I did this rather easily with lighttpd with the directive:
proxy.server = ( "/" => ( ( "host" => "10.100.10.65", "port" => 80 )))
Thanks
Skye
RE: mod_proxy performance with apache 2.2
Posted by Jesse Nelson <je...@military-inc.com>.
What size file what are you using to test ?
I just did this on 2.2.8 about 5 days ago 3 clients running ~600workers
each hitting proxy server that was server files between 15k and 600k on
3 backends. was able to top it out over 110Mb/sec. and something like
1800-2k req/sec without really hurting the proxy.
This config also was using a mix of rewrites/proxypass rules. Over 200
or so rules.
This was on one virt however, and you could be hitting on something
there.
-----Original Message-----
From: skye@seattle.f4.ca [mailto:skye@seattle.f4.ca] On Behalf Of Skye
Poier Nott
Sent: Thursday, March 20, 2008 12:56 PM
To: dev@httpd.apache.org
Subject: mod_proxy performance with apache 2.2
(Moved from users@ list as recommended)
Hi,
I am doing some load testing on a reverse proxy with apache 2.2 +
mod_proxy (no caching) and I'm getting terrible throughput, I was
wondering if anyone had a suggestion.
I have the following test network, all on gigabit ethernet:
1 origin server -> 2 mod_proxy servers -> 4 client simulators (flood)
doing round-robin on the proxies
The origin and proxy servers are configured with name-based virtual
hosts, the proxy servers with config containing 100 of these:
<VirtualHost *:80>
ServerName w0099.example.com
ProxyPass / http://10.100.10.XX:80/w0099/
ProxyPassReverse / http://10.100.10.XX:80/w0099/
</VirtualHost>
When I start up flood on the client simulators (250 clients each), I
only get about 4 Mbit/sec out of the origin server! I've tried
twiddling all the settings in mod_proxy to no avail (like smax=64
max=512 ttl=120 min=8 acquire=1)
If I set up lighttpd+proxy on the proxy servers instead of apache, I
fully saturate the gigE on the origin server (~500 Mbit/sec) as
expected.
I've tried changing MaxClients from 256 to 512 to 1024, no effect.
I've tried changing from MPM prefork to worker, and twiddled threads
per child, no effect.
What could explain this 125x difference in performance? Is there some
sort of resource or lock or something contention in mod_proxy that I
should know about?
Are the virtual host lookups taking too long? Eventually I'd like to
support up to 10,000 virtual hosts through this proxy. Is there
something like mod_vhost_alias for proxy for large configurations?
Is using subdirectories in ProxyPass slowing things down? Really, I'd
just like to forward all requests for anything to the origin server IP
and let the origin worry about serving the right vhost's files.
Thanks,
Skye
Re: mod_proxy performance with apache 2.2
Posted by Guy Ferraiolo <gu...@cnet.com>.
Definitely check the service time of the requests. If your server isn't
closing the connection, flood won't either. This is the correct
behavior but it's non-intuitive. If all your times are equal to the
timeout, this could be the problem. As for why httpd might not be
closing the connection, that can be tricky and also compliant.
Guy
On Thu, 2008-03-20 at 12:56 -0700, Skye Poier Nott wrote:
> (Moved from users@ list as recommended)
>
> Hi,
>
> I am doing some load testing on a reverse proxy with apache 2.2 +
> mod_proxy (no caching) and I'm getting terrible throughput, I was
> wondering if anyone had a suggestion.
>
> I have the following test network, all on gigabit ethernet:
>
> 1 origin server -> 2 mod_proxy servers -> 4 client simulators (flood)
> doing round-robin on the proxies
>
> The origin and proxy servers are configured with name-based virtual
> hosts, the proxy servers with config containing 100 of these:
>
> <VirtualHost *:80>
> ServerName w0099.example.com
> ProxyPass / http://10.100.10.XX:80/w0099/
> ProxyPassReverse / http://10.100.10.XX:80/w0099/
> </VirtualHost>
>
> When I start up flood on the client simulators (250 clients each), I
> only get about 4 Mbit/sec out of the origin server! I've tried
> twiddling all the settings in mod_proxy to no avail (like smax=64
> max=512 ttl=120 min=8 acquire=1)
>
> If I set up lighttpd+proxy on the proxy servers instead of apache, I
> fully saturate the gigE on the origin server (~500 Mbit/sec) as
> expected.
>
> I've tried changing MaxClients from 256 to 512 to 1024, no effect.
> I've tried changing from MPM prefork to worker, and twiddled threads
> per child, no effect.
>
> What could explain this 125x difference in performance? Is there some
> sort of resource or lock or something contention in mod_proxy that I
> should know about?
>
> Are the virtual host lookups taking too long? Eventually I'd like to
> support up to 10,000 virtual hosts through this proxy. Is there
> something like mod_vhost_alias for proxy for large configurations?
>
> Is using subdirectories in ProxyPass slowing things down? Really, I'd
> just like to forward all requests for anything to the origin server IP
> and let the origin worry about serving the right vhost's files.
>
> Thanks,
> Skye
>
--
Guy Ferraiolo mailto:guyf@CNET.com
Performance Measurement & Analysis http://CNET.com
CNET tel: 1.908.541.3739
1200 Route 22 East fax: 1.908.575.7474
Bridgewater, NJ 08807 cel: 1.732.618.0250
Re: mod_proxy performance with apache 2.2
Posted by Guy Ferraiolo <gu...@cnet.com>.
Skye
What are the actual service times for the requests? Also, consider
trying less than 1000 clients. flood clients don't really map 1-to-1 to
human users.
Guy
On Thu, 2008-03-20 at 12:56 -0700, Skye Poier Nott wrote:
> (Moved from users@ list as recommended)
>
> Hi,
>
> I am doing some load testing on a reverse proxy with apache 2.2 +
> mod_proxy (no caching) and I'm getting terrible throughput, I was
> wondering if anyone had a suggestion.
>
> I have the following test network, all on gigabit ethernet:
>
> 1 origin server -> 2 mod_proxy servers -> 4 client simulators (flood)
> doing round-robin on the proxies
>
> The origin and proxy servers are configured with name-based virtual
> hosts, the proxy servers with config containing 100 of these:
>
> <VirtualHost *:80>
> ServerName w0099.example.com
> ProxyPass / http://10.100.10.XX:80/w0099/
> ProxyPassReverse / http://10.100.10.XX:80/w0099/
> </VirtualHost>
>
> When I start up flood on the client simulators (250 clients each), I
> only get about 4 Mbit/sec out of the origin server! I've tried
> twiddling all the settings in mod_proxy to no avail (like smax=64
> max=512 ttl=120 min=8 acquire=1)
>
> If I set up lighttpd+proxy on the proxy servers instead of apache, I
> fully saturate the gigE on the origin server (~500 Mbit/sec) as
> expected.
>
> I've tried changing MaxClients from 256 to 512 to 1024, no effect.
> I've tried changing from MPM prefork to worker, and twiddled threads
> per child, no effect.
>
> What could explain this 125x difference in performance? Is there some
> sort of resource or lock or something contention in mod_proxy that I
> should know about?
>
> Are the virtual host lookups taking too long? Eventually I'd like to
> support up to 10,000 virtual hosts through this proxy. Is there
> something like mod_vhost_alias for proxy for large configurations?
>
> Is using subdirectories in ProxyPass slowing things down? Really, I'd
> just like to forward all requests for anything to the origin server IP
> and let the origin worry about serving the right vhost's files.
>
> Thanks,
> Skye
>
--
Guy Ferraiolo mailto:guyf@CNET.com
Performance Measurement & Analysis http://CNET.com
CNET tel: 1.908.541.3739
1200 Route 22 East fax: 1.908.575.7474
Bridgewater, NJ 08807 cel: 1.732.618.0250
mod_proxy performance with apache 2.2
Posted by Skye Poier Nott <sk...@F4.ca>.
(Moved from users@ list as recommended)
Hi,
I am doing some load testing on a reverse proxy with apache 2.2 +
mod_proxy (no caching) and I'm getting terrible throughput, I was
wondering if anyone had a suggestion.
I have the following test network, all on gigabit ethernet:
1 origin server -> 2 mod_proxy servers -> 4 client simulators (flood)
doing round-robin on the proxies
The origin and proxy servers are configured with name-based virtual
hosts, the proxy servers with config containing 100 of these:
<VirtualHost *:80>
ServerName w0099.example.com
ProxyPass / http://10.100.10.XX:80/w0099/
ProxyPassReverse / http://10.100.10.XX:80/w0099/
</VirtualHost>
When I start up flood on the client simulators (250 clients each), I
only get about 4 Mbit/sec out of the origin server! I've tried
twiddling all the settings in mod_proxy to no avail (like smax=64
max=512 ttl=120 min=8 acquire=1)
If I set up lighttpd+proxy on the proxy servers instead of apache, I
fully saturate the gigE on the origin server (~500 Mbit/sec) as
expected.
I've tried changing MaxClients from 256 to 512 to 1024, no effect.
I've tried changing from MPM prefork to worker, and twiddled threads
per child, no effect.
What could explain this 125x difference in performance? Is there some
sort of resource or lock or something contention in mod_proxy that I
should know about?
Are the virtual host lookups taking too long? Eventually I'd like to
support up to 10,000 virtual hosts through this proxy. Is there
something like mod_vhost_alias for proxy for large configurations?
Is using subdirectories in ProxyPass slowing things down? Really, I'd
just like to forward all requests for anything to the origin server IP
and let the origin worry about serving the right vhost's files.
Thanks,
Skye