You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Rob Morin <ro...@ilabsinc.com> on 2011/05/25 16:27:19 UTC
[users@httpd] Apache 2.x configuration for high load servers
Hello all my first time posting , so please be gentile. :)
We have 5 Apache 2.2.3 web servers behind a HAproxy load balancer. Each web
server is a Quad core Xeon E5405@2.00GHz with 16 gigs RAM and a hardware
RAID 1 of 300 gigs
Apache is configure in prefork mode, as per Centos 5.5 install
We have recently had a 30% increase in traffic and will be expecting more. I
would like to know the correct way to calculate the proper settings for
prefork.c my settings, here is what I have now...
<IfModule prefork.c>
StartServers 150
MinSpareServers 125
MaxSpareServers 250
ServerLimit 600
MaxClients 600
MaxRequestsPerChild 1500
</IfModule>
I use to have these setting even higher, but when i calculated how much the
server limit and max clients are giving that i have 16 gigs minus our
memcache settings and some OS stuff it leaves me with about 9 gigs of RAM
left that would calculate to about 382 as the max clients setting...
I am going to add more RAM today to make a total of 24 gigs on each server,
and i am going to add an additional web server in the next week or so...
Each day we get about 5 million visits...
The only stats i have is that when the server gets a high load i grab a
snapshot of the Apache processes running and the free mem, so it looks like
this..
This is from yesterday(not sure why that defunt is there)
----------------
top - 15:30:03 up 3 days, 22:10, 3 users, load average: 9.24, 5.73, 5.27
Tasks: 365 total, 1 running, 363 sleeping, 0 stopped, 1 zombie
Cpu(s): 20.6%us, 5.6%sy, 0.0%ni, 71.0%id, 0.2%wa, 0.3%hi, 2.3%si,
0.0%st
Mem: 16435540k total, 4996748k used, 11438792k free, 624820k buffers
Swap: 2096472k total, 0k used, 2096472k free, 1907644k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3352 root 15 0 479m 354m 524 S 7.8 2.2 200:22.25 memcached
21064 apache 15 0 0 0 0 Z 5.8 0.0 0:23.09 httpd <defunct>
21162 apache 15 0 414m 25m 12m S 5.8 0.2 0:22.03 httpd
22107 apache 15 0 412m 24m 11m S 5.8 0.2 0:14.28 httpd
22175 apache 15 0 415m 28m 14m S 5.8 0.2 0:14.48 httpd
22177 apache 15 0 415m 29m 15m S 5.8 0.2 0:14.81 httpd
22294 apache 15 0 414m 25m 12m S 5.8 0.2 0:13.70 httpd
22318 apache 15 0 412m 24m 11m S 5.8 0.2 0:14.16 httpd
23293 apache 15 0 414m 24m 11m S 5.8 0.2 0:06.70 httpd
23366 apache 15 0 415m 28m 14m S 5.8 0.2 0:06.78 httpd
24339 apache 15 0 411m 20m 8400 S 5.8 0.1 0:00.26 httpd
24342 apache 15 0 411m 20m 8220 S 5.8 0.1 0:00.19 httpd
21168 apache 15 0 415m 27m 13m S 3.9 0.2 0:21.94 httpd
21170 apache 15 0 415m 27m 13m S 3.9 0.2 0:21.98 httpd
21182 apache 15 0 415m 29m 15m S 3.9 0.2 0:22.26 httpd
22124 apache 15 0 414m 25m 12m S 3.9 0.2 0:14.94 httpd
22181 apache 15 0 414m 25m 12m S 3.9 0.2 0:14.44 httpd
22193 apache 15 0 415m 27m 13m S 3.9 0.2 0:14.14 httpd
Apache is currently using 6207 Megs of memory...
There are 245 Apache processes running at this time.
--------------------------------
So the load is high, but i have free RAM left and only running 245 Apache
processes, out of a possible 600? I only show the first few lines of top ..
Or do I get this wrong, does each connection have an httpd process that is
run, or are child/subsequent connections, not listed on under a ps ax??
I get the same results on the other web serves too...
I removed access logging, I disable access logging by commenting out the
CustomLog directive in each virtualhost and then in httpd.com I set
CustomLog value to /dev/null, this way I can keep error working too. is this
ok? Here aer some of the httpd.con entries.
Keepalive is off
KeepAliveTimeout 3
MaxKeepAliveRequests 60
HostnameLookups Off
TraceEnable Off
UseCanonicalName Off
Here are loaded modules, I am not sure if I need them all, can I safely turn
any off?
LoadModule actions_module modules/mod_actions.so
LoadModule alias_module modules/mod_alias.so
LoadModule asis_module modules/mod_asis.so
LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule auth_digest_module modules/mod_auth_digest.so
LoadModule authn_alias_module modules/mod_authn_alias.so
LoadModule authn_anon_module modules/mod_authn_anon.so
LoadModule authn_default_module modules/mod_authn_default.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule authz_default_module modules/mod_authz_default.so
LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_owner_module modules/mod_authz_owner.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule autoindex_module modules/mod_autoindex.so
LoadModule cern_meta_module modules/mod_cern_meta.so
LoadModule cgi_module modules/mod_cgi.so
LoadModule deflate_module modules/mod_deflate.so
LoadModule dir_module modules/mod_dir.so
LoadModule env_module modules/mod_env.so
LoadModule expires_module modules/mod_expires.so
LoadModule ext_filter_module modules/mod_ext_filter.so
LoadModule geoip_module /usr/lib64/httpd/modules/mod_geoip.so
LoadModule headers_module modules/mod_headers.so
LoadModule include_module modules/mod_include.so
LoadModule info_module modules/mod_info.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule logio_module modules/mod_logio.so
LoadModule mime_magic_module modules/mod_mime_magic.so
LoadModule mime_module modules/mod_mime.so
LoadModule negotiation_module modules/mod_negotiation.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule status_module modules/mod_status.so
LoadModule userdir_module modules/mod_userdir.so
LoadModule usertrack_module modules/mod_usertrack.so
LoadModule vhost_alias_module modules/mod_vhost_alias.so
Thanks to all in advance.. and have a great day!
I am at the end of my rope.. :(
Thanks
Rob Morin
Systems Administrator
Infinity Labs Inc.
(514) 387-0638 Ext: 207
ilabs-email-sig
RE: [users@httpd] Apache 2.x configuration for high load servers
Posted by Geoff Millikan <gm...@t1shopper.com>.
> I commented out mod_dir and Apache would not start
Yep, you'll need that one. You probably will not need the below list but you kinda have to read up on what each module supports and
see if you're using any of those things.
mod_userdir.so
mod_usertrack.so
mod_status.so
mod_logio.so
mod_cgi.so
mod_cern_meta.so
mod_actions.so
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
RE: [users@httpd] Apache 2.x configuration for high load servers
Posted by Rob Morin <ro...@ilabsinc.com>.
Thanks, that's what they(hosting company) did exactly..
J
Rob Morin
Systems Administrator
Infinity Labs Inc.
(514) 387-0638 Ext: 207
ilabs-email-sig
From: Bostjan Skufca [mailto:bostjan@a2o.si]
Sent: Saturday, May 28, 2011 8:03 PM
To: users@httpd.apache.org
Subject: Re: [users@httpd] Apache 2.x configuration for high load servers
mount -o noatime,nodiratime,remount /path/to/mount/point
b.
On 28 May 2011 20:13, Geoff Millikan <gm...@t1shopper.com> wrote:
> I will mod fstab and reboot one server at a time...
I thought there was a way of dropping atime in real time without a reboot so
you can test performance differences without rebooting
but I could be mistaken. Might want to check out using relatime but I have
no experience with that. I do know that doing noatime
helped our performance.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Apache 2.x configuration for high load servers
Posted by Bostjan Skufca <bo...@a2o.si>.
mount -o noatime,nodiratime,remount /path/to/mount/point
b.
On 28 May 2011 20:13, Geoff Millikan <gm...@t1shopper.com> wrote:
> > I will mod fstab and reboot one server at a time...
>
> I thought there was a way of dropping atime in real time without a reboot
> so you can test performance differences without rebooting
> but I could be mistaken. Might want to check out using relatime but I have
> no experience with that. I do know that doing noatime
> helped our performance.
>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
RE: [users@httpd] Apache 2.x configuration for high load servers
Posted by Rob Morin <ro...@ilabsinc.com>.
When i mentioned this to our hosting company they actually agreed with us,
and went further and said, that we should use something called realtime, but
the kernel version we have with CentOS does not support it, so they used
noatime. They did not have to reboot , they did a mount -o /partition_name
Not sure of the performance increase, but I am sure it helps a bit....
:)
Thanks guys!
Rob Morin
Systems Administrator
Infinity Labs Inc.
(514) 387-0638 Ext: 207
-----Original Message-----
From: Geoff Millikan [mailto:gmillikan@t1shopper.com]
Sent: Saturday, May 28, 2011 2:14 PM
To: users@httpd.apache.org
Subject: RE: [users@httpd] Apache 2.x configuration for high load servers
> I will mod fstab and reboot one server at a time...
I thought there was a way of dropping atime in real time without a reboot so
you can test performance differences without rebooting
but I could be mistaken. Might want to check out using relatime but I have
no experience with that. I do know that doing noatime
helped our performance.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
RE: [users@httpd] Apache 2.x configuration for high load servers
Posted by Geoff Millikan <gm...@t1shopper.com>.
> I will mod fstab and reboot one server at a time...
I thought there was a way of dropping atime in real time without a reboot so you can test performance differences without rebooting
but I could be mistaken. Might want to check out using relatime but I have no experience with that. I do know that doing noatime
helped our performance.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Apache 2.x configuration for high load servers
Posted by sunhux G <su...@gmail.com>.
I'm new to Apache & to my environment too.
We run 4 Apache V2.0.52 & I've seeing high load averages (of 3 to 13)
reported by "top" on the Linux RHES 4.6 for the 1, 5 & 15 minutes avgs
on 3 of our webservers. All the servers' CPU are generally idle except
one webserver which sometimes hit 90-100% CPU utilization.
Our active F5 BIG-2400 LB had been setting 3 of the Apache servers as
"Down" frequently (like 5-30 times/day per websvr on various Apache tcp
ports, ie 80, 86, 443, 444 as shown by the LB's bigd logs).
I login to our LB (which run a customized BSD Unix) & found that whenever
the LB set an Apache server (which LB monitors using http every 5 secs
with 16 secs as timeout) as down, the 1 minute load average on the LB
itself would spike above 1 too (yes, I ran "top" on the LB's command line).
Perhaps I'll run tmstat on the LB as well tomorrow.
When the LB reported an Apache webserver as down, icmp ping from
the LB to the webserver still responds but "telnet websvr_IP port#"
would not respond.
Our network chaps had set the LB to use round-robin algorithm (was
previously & all these while BIG LB's special algorithm, don't know what
it was). It stabilized for a few hours before the symptoms happen again.
Our webservers are all Proliant DL380 G4 dual processor with 2GB RAM.
This frequent "non-response" situation started around first week of May.
All these years, we used the same hardware with only minor changes
to the Apache (security patches) & never had this frequent flapping.
One thing I noted was the LB would flag out a message that it had
FAN 2 rotation error every 20 secs. This fan error surfaced around
early Apr this year; haven't manage to get downtime to replace yet.
Appreciate any inputs on what I can do other than upgrading the
hardware. Let me know any prefork or any settings you would like
me to post here. Honestly, I don't know how to obtain those prefork
& other settings you chaps have been discussing in this thread.
Sun
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
RE: [users@httpd] Apache 2.x configuration for high load servers
Posted by Rob Morin <ro...@ilabsinc.com>.
I understand what your saying... I am not part of the dev team nor do I have
any PHP programming skills beyond test pages and very crude scripts. I do
know that if we were to turn off memcache, our DB server would skyrocket in
load and bring our site to a halt almost and put hugs loads on the server.
>From what I understand, the dev guys store almost everything that can be
stored in the DB.... allot of the queries are unique too, not sure how that
works with mod_cache?? I will also check up on Xdebug and see how that
works from within our code and read up more on mod_cache...
Thanks for all you help guys, its been really appreciated..
And a df -I on one of my web servers looks like this...
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda5 77070336 574507 76495829 1% /
/dev/sda2 524288 43 524245 1% /tmp
/dev/sda1 26104 47 26057 1% /boot
tmpfs 2054442 1 2054441 1% /dev/shm
tmpfs 2054442 1612 2052830 1%
/var/cache/php-eaccelerator/RAM
Rob Morin
Systems Administrator
Infinity Labs Inc.
(514) 387-0638 Ext: 207
-----Original Message-----
From: Geoff Millikan [mailto:gmillikan@t1shopper.com]
Sent: Sunday, May 29, 2011 11:36 PM
To: users@httpd.apache.org
Subject: RE: [users@httpd] Apache 2.x configuration for high load servers
> Nor sure I understand where you say memcache is slow, we uses memcache
> for session variables and for mysql query caching. I did not think
> there was anything as reliable and fast or faster than that? Can you
> elaborate on file-based-cache fro mysql querrie?
Bostjan Skufca replied already however I'll say that mod_disk_cache is a
great tool. But only on those pages which are high
traffic, dynamic and most importantly are not customized based on session or
IP address. You would not need to use any other
caching tools like memcache - because the preassembled, compressed/zipped
page is already assembled, sitting on disk - the page is
just read off the disk and sent to the clients. Queries will not be run,
code will not be executed - basically you're serving a
static page right off the disk.
You can see how this would both speed up the serving of the page and free up
the database (and RAM) to do other things.
Thanks,
Geoff Millikan @ http://www.t1shopper.com/
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
RE: [users@httpd] Apache 2.x configuration for high load servers
Posted by Geoff Millikan <gm...@t1shopper.com>.
> Nor sure I understand where you say memcache is slow, we uses memcache
> for session variables and for mysql query caching. I did not think
> there was anything as reliable and fast or faster than that? Can you
> elaborate on file-based-cache fro mysql querrie?
Bostjan Skufca replied already however I'll say that mod_disk_cache is a great tool. But only on those pages which are high
traffic, dynamic and most importantly are not customized based on session or IP address. You would not need to use any other
caching tools like memcache - because the preassembled, compressed/zipped page is already assembled, sitting on disk - the page is
just read off the disk and sent to the clients. Queries will not be run, code will not be executed - basically you're serving a
static page right off the disk.
You can see how this would both speed up the serving of the page and free up the database (and RAM) to do other things.
Thanks,
Geoff Millikan @ http://www.t1shopper.com/
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
RE: [users@httpd] Apache 2.x configuration for high load servers
Posted by Geoff Millikan <gm...@t1shopper.com>.
> This went well until there were not too many files on the file system...
Our experience here has been this: We're on ext3 and caching (on disk) several hundred thousand files using mod_disk_cache. We got
worried we were going to have too many files but using "df -i" below shows we're at about a million files on the file system so
we're only at 2% usage.
Now we did run into the 32,000 subdirectory limit that ext3 has so we modified our mod_file_cache settings to get around this like
this:
CacheDirLevels 2
CacheDirLength 1
$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
VolGroup00 72712192 972739 71739453 2% /
/dev/sda1 26104 73 26031 1% /boot
tmpfs 1974407 1 1974406 1% /dev/shm
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Apache 2.x configuration for high load servers
Posted by Bostjan Skufca <bo...@a2o.si>.
Inline...
On 27 May 2011 20:52, Rob Morin <ro...@ilabsinc.com> wrote:
> We use a CDN for our images and other static files like CSS and so on…
>
>
>
> Like I mentioned keepalive is off as the Hosting company said we cannot use
> it in conjunction with keepalived on our haproxy setup, I guess I can test
> it on one machine and see if anything goes awry..
>
>
>
> I have not considered compiling it myself yet, I wanted to see how far I
> can go with respect to tuning before I have to rip down apache and compile
> it, we do have a fully managed account with the hosting company but I am
> sure if I ask to compile apache they might disagree with me on that and
> might not support it… most hosting companies now only support packages that
> are in the main repos of that distribution and will not vary from that even
> for compiling..
>
Yes, updates do tend to get more specific here:)
> As for profiling I have asked the Dev guys to see if they can do something
> like that, however New Relic does help them a bit but nothing like xdebug I
> guess..
>
If I am not mistaken, New Relic is metric measured from outside, where
Xdebug gives you detailed profile of your code execution path and delays, it
is actually very informative just to try it out. At one of the companies I
work for we discovered that 80% of page load time the code was just setting
object properties (some braindead design by using methods (and subsequently
__call) to set every variable, don't mention it:)
Nor sure I understand where you say memcache is slow, we uses memcache for
> session variables and for mysql query caching… I did not think there was
> anything as reliable and fast or faster than that?
>
> Can you elaborate on file-based-cache fro mysql querrie? Or point me to
> some docs?
>
Unfortunately I don't have any published measurements, but here it goes: we
had an application that we benchmarked and if we used memcache for caching
mysql queries it gave 100 requests per second tops. Just switching to file
based caching gave us around 150-170 requests per seconds. This went well
until there were not too many files on the filesystem. This also implies two
predispositions:
- you can have independent per-server cache
- you do not cache too many small objects and cache files have to stay in
VFS cache (maybe this could be remedied with some reiserfs which excels at
small files but it is (was?) not suitable because of big kernel lock)
Why is memcache slower than file cache? Because using memcache causes
additional delay by communication between two userland processes (your app
server and memcached: form request, send it, memcache parses it, searches
itself, forms response, sends it, app receives it and parses it) where VFS
brings cache directly from memory (if all fits in there). VFS cache is even
faster than ramdisk if all that needs to be cached fits into your ram.
Take care,
b.
We use munin, so we have most of what you are requesting… I will see about
> making them open to the world for a couple days for you to take a peek at…
>
>
>
> Thanks for your reply, its very much appriciated…
>
>
>
> Have a great weekend!
>
>
>
> Rob Morin
>
> Systems Administrator
>
> Infinity Labs Inc.
>
> (514) 387-0638 Ext: 207
>
> [image: ilabs-email-sig]
>
>
>
>
>
> *From:* Bostjan Skufca [mailto:bostjan@a2o.si]
> *Sent:* Friday, May 27, 2011 12:23 PM
>
> *To:* users@httpd.apache.org
> *Subject:* Re: [users@httpd] Apache 2.x configuration for high load
> servers
>
>
>
> Hi Rob,
>
> couple of thoughts below, please inform me if I make wrong deductions
> somewhere, tnx.
>
> 1. Your static content is served from elsewhere, you are only talking about
> application servers
> 2. Your average response time is 200ms, and given 4cores per server and
> excluding delays you are currently processing 20 requests per second per
> server tops.
> 3. If your load is hitting above number of cores your requests are not
> served as fast as possible
> 4. Keepalive timeout, what is your setting? Given the presumption that you
> don't serve static content yourself it means that you only serve master html
> - you don't need it at all (with given circumstances of course, where each
> open connection in keepalive state consumes one apache process)
> 4a. On the second thought maybe you can use keepalive between load balancer
> and APP servers, to avoid additional delay of connection setup
> 5. Did you consider compiling apache yourself? Build it statically with all
> modules compiled in (and as DSO only those modules you need occasionally)
> and with -O3? The latter calls for careful testing. Same goes for PHP
> 6. Did you try profiling your application? Use xdebug to make a request
> trace and start optimizing there, usually it is far more efficient that
> system-level optimization.
> 7. Memcache is slow compared to file-based cache, given that you have
> enough ram to hold all your files in cache. We've performed the tests and
> unless you don't need distributed cache filesystem cache is better/faster
> solution.
> 8. Maybe latencytop is the next optimization tool, or ftrace if you are
> using linux.
> 9. IO graphs (IOPS, bytes per seconds, disk time per second, time per IOP),
> do you have them?
> 10. Can you provide also load, cpu utilization, network bytes and packets,
> and memory usage graphs?
>
> b.
>
> On 27 May 2011 14:24, Rob Morin <ro...@ilabsinc.com> wrote:
>
> Thanks for all your help... first off I increased MaxRequestsPerChild from
> 1500 to 10000 on one webserver, just as a test to make sure all is good....
>
> As for KeepAlive, we were told we cannot use it due to we are also using
> keepalived on our haproxy load balancer, is this true?? As we have a bunch
> of SSL certs
>
> I already had ServerToken set to prod :)
>
> I am not sure about UseCanonicalName On, as from what I read it might break
> our rewrite rules we use them alott
>
> So I commneted out mod_dir and Apache would not start I got this error, so
> I
> put it back..
>
> Starting httpd: Syntax error on line 22 of /etc/httpd/conf.d/php.conf:
> Invalid command 'DirectoryIndex', perhaps misspelled or defined by a module
> not included in the server configuration
>
> I will disable atime later on this morning, I will mod fstab and reboot one
> server at a time... the load balancer will autoamtiocally remoe that server
> form the cluster when its down so ti will not affect any clients.. :)
>
> I will read up more on mod_file_cache
>
> Thanks again...
>
> Rob Morin
> Systems Administrator
> Infinity Labs Inc.
> (514) 387-0638 Ext: 207
>
>
>
>
> -----Original Message-----
> From: Geoff Millikan [mailto:gmillikan@t1shopper.com]
> Sent: Friday, May 27, 2011 3:35 AM
> To: users@httpd.apache.org
> Subject: RE: [users@httpd] Apache 2.x configuration for high load servers
>
> #Increase this number. The zombies you're seeing
> #is every time an Apache child process dies.
> #The child will die @1500. We run at 20000 and are having no problems.
> MaxRequestsPerChild 1500
>
> #We don't run as hot as you (we average 3 hits/sec and max at like 15
> #in a 24 hour period) but here's our settings:
> StartServers 70
> MinSpareServers 70
> ServerLimit 364
> MaxClients 364
> MaxRequestsPerChild 20000
>
> #Shrink the size of the response header:
> ServerTokens Prod
>
> #Defiantly want keepalive on, it will help page load time for customers.
> #Everything else looks good.
> Keepalive is on
>
> #We have this turned on but we use server side includes SSI and we cache.
> UseCanonicalName On
>
> You have way too many modules enabled. You need user_dir module? Every
> module loaded takes up RAM. You should be able to get the
> "RES" RAM use per process down to under 20 MB unless you're doing something
> funky.
>
> If you're finding that 80% of the time you're running 100 processes, then
> start that many and keep that many going. We found that
> the process that starts up new children takes too long and it was causing
> pages to hang while Apache started up new children. So
> just start as many as you need, even if a lot of them are just idle.
>
> Couple of other tips:
>
> 1. Turn off the "access time" atime on your file system. This will speed
> disk access a lot.
>
> 2. Using mod_file_cache on a big site is hard to get everyone on board for.
> It's complicated and it's easy to cache the wrong
> stuff. However, with a bit of pain, there is a lot of gain to be had there
> in terms of reduced work load on the server and thus
> faster page load times. Suggest you bite the bullet and do it. BTW, the
> CentOS rpm of Apache doesn't work with mod_file_Cahce.
> Lots of bugs. So have to compile from source on this. We're on CentOS 5
> too.
>
> Lots of people will suggest the Worker MPM due to it's lower RAM usage
> however we haven't done it as the PHP web site speaks so
> strongly against it.
>
> Best,
>
> Geoff Millikan @ http://www.t1shopper.com/
>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
>
RE: [users@httpd] Apache 2.x configuration for high load servers
Posted by Rob Morin <ro...@ilabsinc.com>.
We use a CDN for our images and other static files like CSS and so on.
Like I mentioned keepalive is off as the Hosting company said we cannot use
it in conjunction with keepalived on our haproxy setup, I guess I can test
it on one machine and see if anything goes awry..
I have not considered compiling it myself yet, I wanted to see how far I can
go with respect to tuning before I have to rip down apache and compile it,
we do have a fully managed account with the hosting company but I am sure if
I ask to compile apache they might disagree with me on that and might not
support it. most hosting companies now only support packages that are in
the main repos of that distribution and will not vary from that even for
compiling..
As for profiling I have asked the Dev guys to see if they can do something
like that, however New Relic does help them a bit but nothing like xdebug I
guess..
Nor sure I understand where you say memcache is slow, we uses memcache for
session variables and for mysql query caching. I did not think there was
anything as reliable and fast or faster than that?
Can you elaborate on file-based-cache fro mysql querrie? Or point me to some
docs?
We use munin, so we have most of what you are requesting. I will see about
making them open to the world for a couple days for you to take a peek at.
Thanks for your reply, its very much appriciated.
Have a great weekend!
Rob Morin
Systems Administrator
Infinity Labs Inc.
(514) 387-0638 Ext: 207
ilabs-email-sig
From: Bostjan Skufca [mailto:bostjan@a2o.si]
Sent: Friday, May 27, 2011 12:23 PM
To: users@httpd.apache.org
Subject: Re: [users@httpd] Apache 2.x configuration for high load servers
Hi Rob,
couple of thoughts below, please inform me if I make wrong deductions
somewhere, tnx.
1. Your static content is served from elsewhere, you are only talking about
application servers
2. Your average response time is 200ms, and given 4cores per server and
excluding delays you are currently processing 20 requests per second per
server tops.
3. If your load is hitting above number of cores your requests are not
served as fast as possible
4. Keepalive timeout, what is your setting? Given the presumption that you
don't serve static content yourself it means that you only serve master html
- you don't need it at all (with given circumstances of course, where each
open connection in keepalive state consumes one apache process)
4a. On the second thought maybe you can use keepalive between load balancer
and APP servers, to avoid additional delay of connection setup
5. Did you consider compiling apache yourself? Build it statically with all
modules compiled in (and as DSO only those modules you need occasionally)
and with -O3? The latter calls for careful testing. Same goes for PHP
6. Did you try profiling your application? Use xdebug to make a request
trace and start optimizing there, usually it is far more efficient that
system-level optimization.
7. Memcache is slow compared to file-based cache, given that you have enough
ram to hold all your files in cache. We've performed the tests and unless
you don't need distributed cache filesystem cache is better/faster solution.
8. Maybe latencytop is the next optimization tool, or ftrace if you are
using linux.
9. IO graphs (IOPS, bytes per seconds, disk time per second, time per IOP),
do you have them?
10. Can you provide also load, cpu utilization, network bytes and packets,
and memory usage graphs?
b.
On 27 May 2011 14:24, Rob Morin <ro...@ilabsinc.com> wrote:
Thanks for all your help... first off I increased MaxRequestsPerChild from
1500 to 10000 on one webserver, just as a test to make sure all is good....
As for KeepAlive, we were told we cannot use it due to we are also using
keepalived on our haproxy load balancer, is this true?? As we have a bunch
of SSL certs
I already had ServerToken set to prod :)
I am not sure about UseCanonicalName On, as from what I read it might break
our rewrite rules we use them alott
So I commneted out mod_dir and Apache would not start I got this error, so I
put it back..
Starting httpd: Syntax error on line 22 of /etc/httpd/conf.d/php.conf:
Invalid command 'DirectoryIndex', perhaps misspelled or defined by a module
not included in the server configuration
I will disable atime later on this morning, I will mod fstab and reboot one
server at a time... the load balancer will autoamtiocally remoe that server
form the cluster when its down so ti will not affect any clients.. :)
I will read up more on mod_file_cache
Thanks again...
Rob Morin
Systems Administrator
Infinity Labs Inc.
(514) 387-0638 Ext: 207
-----Original Message-----
From: Geoff Millikan [mailto:gmillikan@t1shopper.com]
Sent: Friday, May 27, 2011 3:35 AM
To: users@httpd.apache.org
Subject: RE: [users@httpd] Apache 2.x configuration for high load servers
#Increase this number. The zombies you're seeing
#is every time an Apache child process dies.
#The child will die @1500. We run at 20000 and are having no problems.
MaxRequestsPerChild 1500
#We don't run as hot as you (we average 3 hits/sec and max at like 15
#in a 24 hour period) but here's our settings:
StartServers 70
MinSpareServers 70
ServerLimit 364
MaxClients 364
MaxRequestsPerChild 20000
#Shrink the size of the response header:
ServerTokens Prod
#Defiantly want keepalive on, it will help page load time for customers.
#Everything else looks good.
Keepalive is on
#We have this turned on but we use server side includes SSI and we cache.
UseCanonicalName On
You have way too many modules enabled. You need user_dir module? Every
module loaded takes up RAM. You should be able to get the
"RES" RAM use per process down to under 20 MB unless you're doing something
funky.
If you're finding that 80% of the time you're running 100 processes, then
start that many and keep that many going. We found that
the process that starts up new children takes too long and it was causing
pages to hang while Apache started up new children. So
just start as many as you need, even if a lot of them are just idle.
Couple of other tips:
1. Turn off the "access time" atime on your file system. This will speed
disk access a lot.
2. Using mod_file_cache on a big site is hard to get everyone on board for.
It's complicated and it's easy to cache the wrong
stuff. However, with a bit of pain, there is a lot of gain to be had there
in terms of reduced work load on the server and thus
faster page load times. Suggest you bite the bullet and do it. BTW, the
CentOS rpm of Apache doesn't work with mod_file_Cahce.
Lots of bugs. So have to compile from source on this. We're on CentOS 5
too.
Lots of people will suggest the Worker MPM due to it's lower RAM usage
however we haven't done it as the PHP web site speaks so
strongly against it.
Best,
Geoff Millikan @ http://www.t1shopper.com/
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Apache 2.x configuration for high load servers
Posted by Bostjan Skufca <bo...@a2o.si>.
Hi Rob,
couple of thoughts below, please inform me if I make wrong deductions
somewhere, tnx.
1. Your static content is served from elsewhere, you are only talking about
application servers
2. Your average response time is 200ms, and given 4cores per server and
excluding delays you are currently processing 20 requests per second per
server tops.
3. If your load is hitting above number of cores your requests are not
served as fast as possible
4. Keepalive timeout, what is your setting? Given the presumption that you
don't serve static content yourself it means that you only serve master html
- you don't need it at all (with given circumstances of course, where each
open connection in keepalive state consumes one apache process)
4a. On the second thought maybe you can use keepalive between load balancer
and APP servers, to avoid additional delay of connection setup
5. Did you consider compiling apache yourself? Build it statically with all
modules compiled in (and as DSO only those modules you need occasionally)
and with -O3? The latter calls for careful testing. Same goes for PHP
6. Did you try profiling your application? Use xdebug to make a request
trace and start optimizing there, usually it is far more efficient that
system-level optimization.
7. Memcache is slow compared to file-based cache, given that you have enough
ram to hold all your files in cache. We've performed the tests and unless
you don't need distributed cache filesystem cache is better/faster solution.
8. Maybe latencytop is the next optimization tool, or ftrace if you are
using linux.
9. IO graphs (IOPS, bytes per seconds, disk time per second, time per IOP),
do you have them?
10. Can you provide also load, cpu utilization, network bytes and packets,
and memory usage graphs?
b.
On 27 May 2011 14:24, Rob Morin <ro...@ilabsinc.com> wrote:
> Thanks for all your help... first off I increased MaxRequestsPerChild from
> 1500 to 10000 on one webserver, just as a test to make sure all is good....
>
> As for KeepAlive, we were told we cannot use it due to we are also using
> keepalived on our haproxy load balancer, is this true?? As we have a bunch
> of SSL certs
>
> I already had ServerToken set to prod :)
>
> I am not sure about UseCanonicalName On, as from what I read it might break
> our rewrite rules we use them alott
>
> So I commneted out mod_dir and Apache would not start I got this error, so
> I
> put it back..
>
> Starting httpd: Syntax error on line 22 of /etc/httpd/conf.d/php.conf:
> Invalid command 'DirectoryIndex', perhaps misspelled or defined by a module
> not included in the server configuration
>
> I will disable atime later on this morning, I will mod fstab and reboot one
> server at a time... the load balancer will autoamtiocally remoe that server
> form the cluster when its down so ti will not affect any clients.. :)
>
> I will read up more on mod_file_cache
>
> Thanks again...
>
> Rob Morin
> Systems Administrator
> Infinity Labs Inc.
> (514) 387-0638 Ext: 207
>
>
>
>
> -----Original Message-----
> From: Geoff Millikan [mailto:gmillikan@t1shopper.com]
> Sent: Friday, May 27, 2011 3:35 AM
> To: users@httpd.apache.org
> Subject: RE: [users@httpd] Apache 2.x configuration for high load servers
>
> #Increase this number. The zombies you're seeing
> #is every time an Apache child process dies.
> #The child will die @1500. We run at 20000 and are having no problems.
> MaxRequestsPerChild 1500
>
> #We don't run as hot as you (we average 3 hits/sec and max at like 15
> #in a 24 hour period) but here's our settings:
> StartServers 70
> MinSpareServers 70
> ServerLimit 364
> MaxClients 364
> MaxRequestsPerChild 20000
>
> #Shrink the size of the response header:
> ServerTokens Prod
>
> #Defiantly want keepalive on, it will help page load time for customers.
> #Everything else looks good.
> Keepalive is on
>
> #We have this turned on but we use server side includes SSI and we cache.
> UseCanonicalName On
>
> You have way too many modules enabled. You need user_dir module? Every
> module loaded takes up RAM. You should be able to get the
> "RES" RAM use per process down to under 20 MB unless you're doing something
> funky.
>
> If you're finding that 80% of the time you're running 100 processes, then
> start that many and keep that many going. We found that
> the process that starts up new children takes too long and it was causing
> pages to hang while Apache started up new children. So
> just start as many as you need, even if a lot of them are just idle.
>
> Couple of other tips:
>
> 1. Turn off the "access time" atime on your file system. This will speed
> disk access a lot.
>
> 2. Using mod_file_cache on a big site is hard to get everyone on board for.
> It's complicated and it's easy to cache the wrong
> stuff. However, with a bit of pain, there is a lot of gain to be had there
> in terms of reduced work load on the server and thus
> faster page load times. Suggest you bite the bullet and do it. BTW, the
> CentOS rpm of Apache doesn't work with mod_file_Cahce.
> Lots of bugs. So have to compile from source on this. We're on CentOS 5
> too.
>
> Lots of people will suggest the Worker MPM due to it's lower RAM usage
> however we haven't done it as the PHP web site speaks so
> strongly against it.
>
> Best,
>
> Geoff Millikan @ http://www.t1shopper.com/
>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
RE: [users@httpd] Apache 2.x configuration for high load servers
Posted by Rob Morin <ro...@ilabsinc.com>.
Thanks for all your help... first off I increased MaxRequestsPerChild from
1500 to 10000 on one webserver, just as a test to make sure all is good....
As for KeepAlive, we were told we cannot use it due to we are also using
keepalived on our haproxy load balancer, is this true?? As we have a bunch
of SSL certs
I already had ServerToken set to prod :)
I am not sure about UseCanonicalName On, as from what I read it might break
our rewrite rules we use them alott
So I commneted out mod_dir and Apache would not start I got this error, so I
put it back..
Starting httpd: Syntax error on line 22 of /etc/httpd/conf.d/php.conf:
Invalid command 'DirectoryIndex', perhaps misspelled or defined by a module
not included in the server configuration
I will disable atime later on this morning, I will mod fstab and reboot one
server at a time... the load balancer will autoamtiocally remoe that server
form the cluster when its down so ti will not affect any clients.. :)
I will read up more on mod_file_cache
Thanks again...
Rob Morin
Systems Administrator
Infinity Labs Inc.
(514) 387-0638 Ext: 207
-----Original Message-----
From: Geoff Millikan [mailto:gmillikan@t1shopper.com]
Sent: Friday, May 27, 2011 3:35 AM
To: users@httpd.apache.org
Subject: RE: [users@httpd] Apache 2.x configuration for high load servers
#Increase this number. The zombies you're seeing
#is every time an Apache child process dies.
#The child will die @1500. We run at 20000 and are having no problems.
MaxRequestsPerChild 1500
#We don't run as hot as you (we average 3 hits/sec and max at like 15
#in a 24 hour period) but here's our settings:
StartServers 70
MinSpareServers 70
ServerLimit 364
MaxClients 364
MaxRequestsPerChild 20000
#Shrink the size of the response header:
ServerTokens Prod
#Defiantly want keepalive on, it will help page load time for customers.
#Everything else looks good.
Keepalive is on
#We have this turned on but we use server side includes SSI and we cache.
UseCanonicalName On
You have way too many modules enabled. You need user_dir module? Every
module loaded takes up RAM. You should be able to get the
"RES" RAM use per process down to under 20 MB unless you're doing something
funky.
If you're finding that 80% of the time you're running 100 processes, then
start that many and keep that many going. We found that
the process that starts up new children takes too long and it was causing
pages to hang while Apache started up new children. So
just start as many as you need, even if a lot of them are just idle.
Couple of other tips:
1. Turn off the "access time" atime on your file system. This will speed
disk access a lot.
2. Using mod_file_cache on a big site is hard to get everyone on board for.
It's complicated and it's easy to cache the wrong
stuff. However, with a bit of pain, there is a lot of gain to be had there
in terms of reduced work load on the server and thus
faster page load times. Suggest you bite the bullet and do it. BTW, the
CentOS rpm of Apache doesn't work with mod_file_Cahce.
Lots of bugs. So have to compile from source on this. We're on CentOS 5
too.
Lots of people will suggest the Worker MPM due to it's lower RAM usage
however we haven't done it as the PHP web site speaks so
strongly against it.
Best,
Geoff Millikan @ http://www.t1shopper.com/
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
RE: [users@httpd] Apache 2.x configuration for high load servers
Posted by Geoff Millikan <gm...@t1shopper.com>.
#Increase this number. The zombies you're seeing
#is every time an Apache child process dies.
#The child will die @1500. We run at 20000 and are having no problems.
MaxRequestsPerChild 1500
#We don't run as hot as you (we average 3 hits/sec and max at like 15
#in a 24 hour period) but here's our settings:
StartServers 70
MinSpareServers 70
ServerLimit 364
MaxClients 364
MaxRequestsPerChild 20000
#Shrink the size of the response header:
ServerTokens Prod
#Defiantly want keepalive on, it will help page load time for customers.
#Everything else looks good.
Keepalive is on
#We have this turned on but we use server side includes SSI and we cache.
UseCanonicalName On
You have way too many modules enabled. You need user_dir module? Every module loaded takes up RAM. You should be able to get the
"RES" RAM use per process down to under 20 MB unless you're doing something funky.
If you're finding that 80% of the time you're running 100 processes, then start that many and keep that many going. We found that
the process that starts up new children takes too long and it was causing pages to hang while Apache started up new children. So
just start as many as you need, even if a lot of them are just idle.
Couple of other tips:
1. Turn off the "access time" atime on your file system. This will speed disk access a lot.
2. Using mod_file_cache on a big site is hard to get everyone on board for. It's complicated and it's easy to cache the wrong
stuff. However, with a bit of pain, there is a lot of gain to be had there in terms of reduced work load on the server and thus
faster page load times. Suggest you bite the bullet and do it. BTW, the CentOS rpm of Apache doesn't work with mod_file_Cahce.
Lots of bugs. So have to compile from source on this. We're on CentOS 5 too.
Lots of people will suggest the Worker MPM due to it's lower RAM usage however we haven't done it as the PHP web site speaks so
strongly against it.
Best,
Geoff Millikan @ http://www.t1shopper.com/
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
RE: [users@httpd] Apache 2.x configuration for high load servers
Posted by Geoff Millikan <gm...@t1shopper.com>.
> We also use New Relic and new PHP performance tuning tool, its very cool...
Oh, and remember to remove/comment out any PHP modules you aren't using. Every PHP module you load (like the GD image module which
is loaded by default that people don't often use) is loaded up into the Apache server too.
As we say here, "RAM is like cowbell, you can never have enough."
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
RE: [users@httpd] Apache 2.x configuration for high load servers
Posted by Rob Morin <ro...@ilabsinc.com>.
Ya sorry about the attachments... I put URLs to my personal website rather
than the work related one to be safe :)
http://www.stupidguytalk.org/WEB3.ireel.com-apache_processes-day.png
http://www.stupidguytalk.org/WEB3.ireel.com-apache_accesses-day.png
Sorry about that....
Rob Morin
Systems Administrator
Infinity Labs Inc.
(514) 387-0638 Ext: 207
-----Original Message-----
From: Rob Morin [mailto:rob@ilabsinc.com]
Sent: Wednesday, May 25, 2011 3:00 PM
To: users@httpd.apache.org
Subject: RE: [users@httpd] Apache 2.x configuration for high load servers
Thanks for the prompt reply Mark, I will try to add a few things and clear
up some stuff I might have omitted in my first post...
I have server-status enabled as well as Munin for Apache..
I have attached a munin graphs to this email, not sure if its proper or
not... sorry if not... one is accesses and one is processes per day..
To be safe its about 120 accesses per second , each server is basically the
same give or take 5%, I will enable access logging for one domain and see
what happens with respect to the time to serve a request...
Here is a paste of server status for the same server as the attached pngs of
munin.. our busy time will start in the next couple hours....
Current Time: Wednesday, 25-May-2011 13:48:42 EDT Restart Time: Sunday,
22-May-2011 04:02:05 EDT Parent Server Generation: 1 Server uptime: 3 days 9
hours 46 minutes 37 seconds Total accesses: 24212224 - Total Traffic: 7.1 GB
CPU Usage: u1544.11 s494.57 cu0 cs0 - .692% CPU load
82.2 requests/sec - 25.5 kB/second - 316 B/request
46 requests currently being processed, 384 idle workers
_C____._..____._..__C___..__C___.___.____._.._.__._._._..____.__
__C_._..____._.__.____________.___._______________.___._.__.R.__
._C____CC___C_.__CCR.._._..WW_____.._.C_..C..__.._.__.._______._
__._..__._.__.____.C__C._____..__.__C.____C___._______.C______._
.C_._._.__.________C._____.._..__._..__.C_C__.._.___.__.._____C_
_____._._C__.________.._____.__R_.____.._C_.C.__.CCC.______.__._
._C_____....__.___..__.____.__.__.____._____...____._._._.__.__C
._C______..______._._._________._.___..____....CC_.__C_C_.C__._.
R__...C.__._.___..._.___C_______....__._____._____._____..._.._C
__W.__.__.________..___.
So then my current config I have seems good enough then, from what you have
spoken about so far?
StartServers 150
MinSpareServers 125
MaxSpareServers 250
ServerLimit 600
MaxClients 600
MaxRequestsPerChild 1500
As the load at busy times is about 9, and this is for quad core, so there
might be room for improvement somewhere I guess.. This is why I turned off
logging as we have about 60 domains all writing logfiles via rotatelogs I
fogured removing that extra file open and writing to would help a bit... I
also made a small RAMdisk to store the eAccelerator cache files rather than
use disk... we also use memcache to store sessions and for mysql querries..
We also use Edgecast as our CDN for static content, like images and CSS
files..
We also use New Relic and new PHP performance tuning tool, its very cool...
that shows that a good 95% of our requests take under 200 ms while the rest
can vary between 600 ms and way up to 10 seconds, the 10 second ones are
where we connect to processors to valaidate information on clients..
I will read over that in the URL your pasted and see what I can do to still
increase performance...
As for the logs just to clarify how I did it, all the virtual host config
files have the CustomLog entries commented out, with this in minf that means
Apache will send access log detaisl to whatever the defautl access log file
is , its normally something like /var/log/hhpd/access_log, so in the
httpd.con file under CustomLog combined , I just have %h, then in the
location for customlog rather than /var/log/httpd/access_log I have
/dev/null so this way it will og minimum info and then not write it to disk
and send it out to the bit bucket.... we do nto need logs as we use Google
Analytics in our code. Is this ok, as if I just remove or comment out the
access_log loacation or have no combined entry apache will nto restart....
if I remove mod_log then I will get no errors logs, and those are more
important than access_logs for us...
Hope this sheds more light on my problem...
Thanks again for all your help...
Rob Morin
Systems Administrator
Infinity Labs Inc.
(514) 387-0638 Ext: 207
-----Original Message-----
From: Mark Montague [mailto:mark@catseye.org]
Sent: Wednesday, May 25, 2011 12:06 PM
To: users@httpd.apache.org
Cc: Rob Morin
Subject: Re: [users@httpd] Apache 2.x configuration for high load servers
On May 25, 2011 10:27 , Rob Morin <ro...@ilabsinc.com> wrote:
> We have recently had a 30% increase in traffic and will be expecting
> more. I would like to know the correct way to calculate the proper
> settings for prefork.c my settings, here is what I have now...
> I am going to add more RAM today to make a total of 24 gigs on each
> server, and i am going to add an additional web server in the next
> week or so...
>
> Each day we get about 5 million visits...
>
The settings you use for the prefork MPM are not determined by how many
visits you get per day, but by how many concurrent requests the server in
question needs to handle during your busiest periods. Number of concurrent
requests is in turn determined by how quickly requests come in together with
how long it takes to serve (process, respond to,
handle) each request.
For example, let's say that 90% of the requests you serve require 1 second
to serve, but 10% of the requests take 5 seconds to serve. Let's also assume
that during your busiest period you are getting 100 requests per second.
In each second, then, 100 children will be needed to server these requests
-- 90 of these children will finish (and become available to serve another
request) after 1 second and the remainder will become available again after
5 seconds. This means that after 5 seconds of your busiest anticipated load,
140 children will be tied up serving requests, and hence you'd want to set
ServerLimit comfortably above this value (to allow margin for error and
future growth), assuming that you have the necessary hardware resources
(RAM, processor cores) to effectively do so. In addition, you may want to
set MinSpareServers high enough to handle a spike in load equivalent to one
second's worth of requests, and set MaxSpareServers to free up resources if
more than the number of children required to server two seconds worth of
requests are sitting idle.
Access logs with a custom log format containing %D (time required to serve
the request in microseconds) are your best tool for figuring out the request
patterns that your web server needs to be able to handle.
Also, mod_status is a useful tool for seeing what each child process is
doing at single point in time (which children are serving requests, which
children are idle, and so on).
> Apache is currently using 6207 Megs of memory...
>
> There are 245 Apache processes running at this time.
>
> --------------------------------
>
> So the load is high, but i have free RAM left and only running 245
> Apache processes, out of a possible 600? I only show the first few
> lines of top ..
>
This is anticipated. Apache HTTP Server's prefork MPM will try to keep the
number spare processes (children that are idle and waiting for new
requests) between MinSpareServers and MaxSpareServers at all times.
The rule of thumb I recommend is that you want your system load to be less
than twice the number of processor cores. Other people's advice may differ
from this, so I recommend that you ask around.
If your system load is more than twice the number of processor cores, I
recommend:
- Determine if there is any storage (disk, etc.) I/O bottleneck; if so,
eliminate it.
- Determine if there is any network I/O bottleneck (network link, load
balancer, firewall, etc.); if so, eliminate it.
- Reduce the time required to serve each requests:
-- Reduce time required by any web applications run by the web server
(active content)
-- Reduce time required by the web server itself (web server configuration,
operating system configuration)
The above recommendations may result in you adding more I/O capacity, more
RAM, more processor cores, and/or more machines to your web server cluster.
However, I believe it is generally better to only add more resources once
you have determined that all existing resources are being used as optimally
as possible: improve any web applictions or other active content on your web
site so that they require less resources; change your infrastructure
architecture (what each web server does and how it does it including caching
and proxying); and improve the configuration or your web server (and other
servers, including any database servers) and your operating system.
> Or do I get this wrong, does each connection have an httpd process
> that is run, or are child/subsequent connections, not listed on under
> a ps ax??
>
If you are using the prefork MPM, a number of child processes get created to
handle web server requests (connections). The number of these will vary
dynamically based on how you configure the prefork MPM and also on how many
requests come in how quickly. All of the processes will show up under ps.
> I removed access logging, I disable access logging by commenting out
> the CustomLog directive in each virtualhost and then in httpd.com I
> set CustomLog value to /dev/null, this way I can keep error working
> too. is this ok? Here aer some of the httpd.con entries.
>
Are you sure that you want to not log web server accesses? This removes an
incredibly important diagnostic tool. Also note that if you use CustomLog
with a file of /dev/null, then you are having Apache HTTP Server actually
create each log entry and send it to the kernel, which will then discard it
instead of writing it to disk; if you really want to turn off access
logging, then it would be better to completely remove any CustomLog or
TransferLog directives from your web server configuration.
If you want to improve performance, see the recommendations at
http://httpd.apache.org/docs/2.2/misc/perf-tuning.html
> Here are loaded modules, I am not sure if I need them all, can I
> safely turn any off?
You can and should turn off any modules you are not using; this will
save some memory and may make your web server a little more secure. You
will need to look at what functionality your web site currently relies
upon, together with your web server configuration, in order to determine
which modules you can avoid loading.
--
Mark Montague
mark@catseye.org
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
RE: [users@httpd] Apache 2.x configuration for high load servers
Posted by Rob Morin <ro...@ilabsinc.com>.
Thanks for the prompt reply Mark, I will try to add a few things and clear
up some stuff I might have omitted in my first post...
I have server-status enabled as well as Munin for Apache..
I have attached a munin graphs to this email, not sure if its proper or
not... sorry if not... one is accesses and one is processes per day..
To be safe its about 120 accesses per second , each server is basically the
same give or take 5%, I will enable access logging for one domain and see
what happens with respect to the time to serve a request...
Here is a paste of server status for the same server as the attached pngs of
munin.. our busy time will start in the next couple hours....
Current Time: Wednesday, 25-May-2011 13:48:42 EDT
Restart Time: Sunday, 22-May-2011 04:02:05 EDT
Parent Server Generation: 1
Server uptime: 3 days 9 hours 46 minutes 37 seconds
Total accesses: 24212224 - Total Traffic: 7.1 GB
CPU Usage: u1544.11 s494.57 cu0 cs0 - .692% CPU load
82.2 requests/sec - 25.5 kB/second - 316 B/request
46 requests currently being processed, 384 idle workers
_C____._..____._..__C___..__C___.___.____._.._.__._._._..____.__
__C_._..____._.__.____________.___._______________.___._.__.R.__
._C____CC___C_.__CCR.._._..WW_____.._.C_..C..__.._.__.._______._
__._..__._.__.____.C__C._____..__.__C.____C___._______.C______._
.C_._._.__.________C._____.._..__._..__.C_C__.._.___.__.._____C_
_____._._C__.________.._____.__R_.____.._C_.C.__.CCC.______.__._
._C_____....__.___..__.____.__.__.____._____...____._._._.__.__C
._C______..______._._._________._.___..____....CC_.__C_C_.C__._.
R__...C.__._.___..._.___C_______....__._____._____._____..._.._C
__W.__.__.________..___.
So then my current config I have seems good enough then, from what you have
spoken about so far?
StartServers 150
MinSpareServers 125
MaxSpareServers 250
ServerLimit 600
MaxClients 600
MaxRequestsPerChild 1500
As the load at busy times is about 9, and this is for quad core, so there
might be room for improvement somewhere I guess.. This is why I turned off
logging as we have about 60 domains all writing logfiles via rotatelogs I
fogured removing that extra file open and writing to would help a bit... I
also made a small RAMdisk to store the eAccelerator cache files rather than
use disk... we also use memcache to store sessions and for mysql querries..
We also use Edgecast as our CDN for static content, like images and CSS
files..
We also use New Relic and new PHP performance tuning tool, its very cool...
that shows that a good 95% of our requests take under 200 ms while the rest
can vary between 600 ms and way up to 10 seconds, the 10 second ones are
where we connect to processors to valaidate information on clients..
I will read over that in the URL your pasted and see what I can do to still
increase performance...
As for the logs just to clarify how I did it, all the virtual host config
files have the CustomLog entries commented out, with this in minf that means
Apache will send access log detaisl to whatever the defautl access log file
is , its normally something like /var/log/hhpd/access_log, so in the
httpd.con file under CustomLog combined , I just have %h, then in the
location for customlog rather than /var/log/httpd/access_log I have
/dev/null so this way it will og minimum info and then not write it to disk
and send it out to the bit bucket.... we do nto need logs as we use Google
Analytics in our code. Is this ok, as if I just remove or comment out the
access_log loacation or have no combined entry apache will nto restart....
if I remove mod_log then I will get no errors logs, and those are more
important than access_logs for us...
Hope this sheds more light on my problem...
Thanks again for all your help...
Rob Morin
Systems Administrator
Infinity Labs Inc.
(514) 387-0638 Ext: 207
-----Original Message-----
From: Mark Montague [mailto:mark@catseye.org]
Sent: Wednesday, May 25, 2011 12:06 PM
To: users@httpd.apache.org
Cc: Rob Morin
Subject: Re: [users@httpd] Apache 2.x configuration for high load servers
On May 25, 2011 10:27 , Rob Morin <ro...@ilabsinc.com> wrote:
> We have recently had a 30% increase in traffic and will be expecting
> more. I would like to know the correct way to calculate the proper
> settings for prefork.c my settings, here is what I have now...
> I am going to add more RAM today to make a total of 24 gigs on each
> server, and i am going to add an additional web server in the next
> week or so...
>
> Each day we get about 5 million visits...
>
The settings you use for the prefork MPM are not determined by how many
visits you get per day, but by how many concurrent requests the server
in question needs to handle during your busiest periods. Number of
concurrent requests is in turn determined by how quickly requests come
in together with how long it takes to serve (process, respond to,
handle) each request.
For example, let's say that 90% of the requests you serve require 1
second to serve, but 10% of the requests take 5 seconds to serve. Let's
also assume that during your busiest period you are getting 100 requests
per second.
In each second, then, 100 children will be needed to server these
requests -- 90 of these children will finish (and become available to
serve another request) after 1 second and the remainder will become
available again after 5 seconds. This means that after 5 seconds of your
busiest anticipated load, 140 children will be tied up serving requests,
and hence you'd want to set ServerLimit comfortably above this value (to
allow margin for error and future growth), assuming that you have the
necessary hardware resources (RAM, processor cores) to effectively do
so. In addition, you may want to set MinSpareServers high enough to
handle a spike in load equivalent to one second's worth of requests, and
set MaxSpareServers to free up resources if more than the number of
children required to server two seconds worth of requests are sitting idle.
Access logs with a custom log format containing %D (time required to
serve the request in microseconds) are your best tool for figuring out
the request patterns that your web server needs to be able to handle.
Also, mod_status is a useful tool for seeing what each child process is
doing at single point in time (which children are serving requests,
which children are idle, and so on).
> Apache is currently using 6207 Megs of memory...
>
> There are 245 Apache processes running at this time.
>
> --------------------------------
>
> So the load is high, but i have free RAM left and only running 245
> Apache processes, out of a possible 600? I only show the first few
> lines of top ..
>
This is anticipated. Apache HTTP Server's prefork MPM will try to keep
the number spare processes (children that are idle and waiting for new
requests) between MinSpareServers and MaxSpareServers at all times.
The rule of thumb I recommend is that you want your system load to be
less than twice the number of processor cores. Other people's advice may
differ from this, so I recommend that you ask around.
If your system load is more than twice the number of processor cores, I
recommend:
- Determine if there is any storage (disk, etc.) I/O bottleneck; if so,
eliminate it.
- Determine if there is any network I/O bottleneck (network link, load
balancer, firewall, etc.); if so, eliminate it.
- Reduce the time required to serve each requests:
-- Reduce time required by any web applications run by the web server
(active content)
-- Reduce time required by the web server itself (web server
configuration, operating system configuration)
The above recommendations may result in you adding more I/O capacity,
more RAM, more processor cores, and/or more machines to your web server
cluster. However, I believe it is generally better to only add more
resources once you have determined that all existing resources are being
used as optimally as possible: improve any web applictions or other
active content on your web site so that they require less resources;
change your infrastructure architecture (what each web server does and
how it does it including caching and proxying); and improve the
configuration or your web server (and other servers, including any
database servers) and your operating system.
> Or do I get this wrong, does each connection have an httpd process
> that is run, or are child/subsequent connections, not listed on under
> a ps ax??
>
If you are using the prefork MPM, a number of child processes get
created to handle web server requests (connections). The number of these
will vary dynamically based on how you configure the prefork MPM and
also on how many requests come in how quickly. All of the processes will
show up under ps.
> I removed access logging, I disable access logging by commenting out
> the CustomLog directive in each virtualhost and then in httpd.com I
> set CustomLog value to /dev/null, this way I can keep error working
> too. is this ok? Here aer some of the httpd.con entries.
>
Are you sure that you want to not log web server accesses? This removes
an incredibly important diagnostic tool. Also note that if you use
CustomLog with a file of /dev/null, then you are having Apache HTTP
Server actually create each log entry and send it to the kernel, which
will then discard it instead of writing it to disk; if you really want
to turn off access logging, then it would be better to completely remove
any CustomLog or TransferLog directives from your web server configuration.
If you want to improve performance, see the recommendations at
http://httpd.apache.org/docs/2.2/misc/perf-tuning.html
> Here are loaded modules, I am not sure if I need them all, can I
> safely turn any off?
You can and should turn off any modules you are not using; this will
save some memory and may make your web server a little more secure. You
will need to look at what functionality your web site currently relies
upon, together with your web server configuration, in order to determine
which modules you can avoid loading.
--
Mark Montague
mark@catseye.org
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Apache 2.x configuration for high load servers
Posted by Mark Montague <ma...@catseye.org>.
On May 25, 2011 10:27 , Rob Morin <ro...@ilabsinc.com> wrote:
> We have recently had a 30% increase in traffic and will be expecting
> more. I would like to know the correct way to calculate the proper
> settings for prefork.c my settings, here is what I have now...
> I am going to add more RAM today to make a total of 24 gigs on each
> server, and i am going to add an additional web server in the next
> week or so...
>
> Each day we get about 5 million visits...
>
The settings you use for the prefork MPM are not determined by how many
visits you get per day, but by how many concurrent requests the server
in question needs to handle during your busiest periods. Number of
concurrent requests is in turn determined by how quickly requests come
in together with how long it takes to serve (process, respond to,
handle) each request.
For example, let's say that 90% of the requests you serve require 1
second to serve, but 10% of the requests take 5 seconds to serve. Let's
also assume that during your busiest period you are getting 100 requests
per second.
In each second, then, 100 children will be needed to server these
requests -- 90 of these children will finish (and become available to
serve another request) after 1 second and the remainder will become
available again after 5 seconds. This means that after 5 seconds of your
busiest anticipated load, 140 children will be tied up serving requests,
and hence you'd want to set ServerLimit comfortably above this value (to
allow margin for error and future growth), assuming that you have the
necessary hardware resources (RAM, processor cores) to effectively do
so. In addition, you may want to set MinSpareServers high enough to
handle a spike in load equivalent to one second's worth of requests, and
set MaxSpareServers to free up resources if more than the number of
children required to server two seconds worth of requests are sitting idle.
Access logs with a custom log format containing %D (time required to
serve the request in microseconds) are your best tool for figuring out
the request patterns that your web server needs to be able to handle.
Also, mod_status is a useful tool for seeing what each child process is
doing at single point in time (which children are serving requests,
which children are idle, and so on).
> Apache is currently using 6207 Megs of memory...
>
> There are 245 Apache processes running at this time.
>
> --------------------------------
>
> So the load is high, but i have free RAM left and only running 245
> Apache processes, out of a possible 600? I only show the first few
> lines of top ….
>
This is anticipated. Apache HTTP Server's prefork MPM will try to keep
the number spare processes (children that are idle and waiting for new
requests) between MinSpareServers and MaxSpareServers at all times.
The rule of thumb I recommend is that you want your system load to be
less than twice the number of processor cores. Other people's advice may
differ from this, so I recommend that you ask around.
If your system load is more than twice the number of processor cores, I
recommend:
- Determine if there is any storage (disk, etc.) I/O bottleneck; if so,
eliminate it.
- Determine if there is any network I/O bottleneck (network link, load
balancer, firewall, etc.); if so, eliminate it.
- Reduce the time required to serve each requests:
-- Reduce time required by any web applications run by the web server
(active content)
-- Reduce time required by the web server itself (web server
configuration, operating system configuration)
The above recommendations may result in you adding more I/O capacity,
more RAM, more processor cores, and/or more machines to your web server
cluster. However, I believe it is generally better to only add more
resources once you have determined that all existing resources are being
used as optimally as possible: improve any web applictions or other
active content on your web site so that they require less resources;
change your infrastructure architecture (what each web server does and
how it does it including caching and proxying); and improve the
configuration or your web server (and other servers, including any
database servers) and your operating system.
> Or do I get this wrong, does each connection have an httpd process
> that is run, or are child/subsequent connections, not listed on under
> a ps ax??
>
If you are using the prefork MPM, a number of child processes get
created to handle web server requests (connections). The number of these
will vary dynamically based on how you configure the prefork MPM and
also on how many requests come in how quickly. All of the processes will
show up under ps.
> I removed access logging, I disable access logging by commenting out
> the CustomLog directive in each virtualhost and then in httpd.com I
> set CustomLog value to /dev/null, this way I can keep error working
> too… is this ok? Here aer some of the httpd.con entries…
>
Are you sure that you want to not log web server accesses? This removes
an incredibly important diagnostic tool. Also note that if you use
CustomLog with a file of /dev/null, then you are having Apache HTTP
Server actually create each log entry and send it to the kernel, which
will then discard it instead of writing it to disk; if you really want
to turn off access logging, then it would be better to completely remove
any CustomLog or TransferLog directives from your web server configuration.
If you want to improve performance, see the recommendations at
http://httpd.apache.org/docs/2.2/misc/perf-tuning.html
> Here are loaded modules, I am not sure if I need them all, can I
> safely turn any off?
You can and should turn off any modules you are not using; this will
save some memory and may make your web server a little more secure. You
will need to look at what functionality your web site currently relies
upon, together with your web server configuration, in order to determine
which modules you can avoid loading.
--
Mark Montague
mark@catseye.org
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org