You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Rob Morin <ro...@ilabsinc.com> on 2011/06/07 22:59:58 UTC
[users@httpd] Apache 2.2 connections full
Hello all was not sure how to word my subject line.
I posted a few weeks back on how could I increase performance of apache. I
received quite a few replies and they all helped a bit.
We even added an additional server to our server cluster to total 7 web
servers now, all load balanced by HAProxy
We currently receive about 14 million impressions a day to about 60
different domains/websites..
Our hosting provider uses HAProxy, rather than an appliance, I assume to
reduce costs. However I feel like these HAProxy machines are not reliable at
all..Well not under the supervision of our hosting company , anyways.J
On to my questions..
During peek hours, like a few mins ago, all of our web servers became high
in load, like close to 100! We still had plenty of RAM left and for sure
there was no swapping..
Please see my config below as quick snapshot of the httpd.conf file on our
Centos Servers.
The weird thing is when I would go see the server-status I would see all the
slots filed with either W or C and nothing else.. Then after a what appears
a random amount of time it would all clear up , and then come back again.
Almost like HAProxy would send a bunch of requests to a server, then it
would get full and then(haproxy) say, oh that server is filled I will not
send traffic there, for now, then it would come back and see hey this server
is empty, lets send a bunch of traffic there.
We also use memcache to cache sessions and mysql queries, along with
eAcclerator. I assume that those are functioning properly
Does this make sense, am I on drugs? J
Thanks, and have a great rest of the day!
ServerTokens Prod
ServerRoot "/etc/httpd"
PidFile run/httpd.pid
Timeout 2
KeepAlive Off
TraceEnable Off
MaxKeepAliveRequests 60
KeepAliveTimeout 3
<IfModule prefork.c>
StartServers 125
MinSpareServers 100
MaxSpareServers 225
ServerLimit 400
MaxClients 400
MaxRequestsPerChild 10000
</IfModule>
<IfModule worker.c>
StartServers 2
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 100
</IfModule>
Listen 80
LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule auth_digest_module modules/mod_auth_digest.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule authn_alias_module modules/mod_authn_alias.so
LoadModule authn_anon_module modules/mod_authn_anon.so
LoadModule authn_default_module modules/mod_authn_default.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule authz_owner_module modules/mod_authz_owner.so
LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
LoadModule authz_default_module modules/mod_authz_default.so
LoadModule include_module modules/mod_include.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule env_module modules/mod_env.so
LoadModule ext_filter_module modules/mod_ext_filter.so
LoadModule mime_magic_module modules/mod_mime_magic.so
LoadModule expires_module modules/mod_expires.so
LoadModule deflate_module modules/mod_deflate.so
LoadModule headers_module modules/mod_headers.so
LoadModule usertrack_module modules/mod_usertrack.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule mime_module modules/mod_mime.so
LoadModule status_module modules/mod_status.so
LoadModule autoindex_module modules/mod_autoindex.so
LoadModule vhost_alias_module modules/mod_vhost_alias.so
LoadModule negotiation_module modules/mod_negotiation.so
LoadModule dir_module modules/mod_dir.so
LoadModule alias_module modules/mod_alias.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule cern_meta_module modules/mod_cern_meta.so
LoadModule asis_module modules/mod_asis.so
LoadModule geoip_module /usr/lib64/httpd/modules/mod_geoip.so
Include conf.d/*.conf
Include conf.d/*.vhost
User apache
Group apache
ServerAdmin root@localhost
UseCanonicalName Off
DocumentRoot "/home/www"
<Directory />
Options FollowSymLinks
AllowOverride None
</Directory>
<Directory "/home/www">
Options -Indexes FollowSymLinks
AllowOverride All
Order allow,deny
Allow from all
</Directory>
<IfModule mod_userdir.c>
UserDir disable
</IfModule>
DirectoryIndex index.php index.html
AccessFileName .htaccess
<Files ~ "^\.ht">
Order allow,deny
Deny from all
</Files>
TypesConfig /etc/mime.types
DefaultType text/plain
<IfModule mod_mime_magic.c>
MIMEMagicFile conf/magic
</IfModule>
HostnameLookups Off
ErrorLog logs/error_log
LogLevel notice
LogFormat "%h" combined
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
CustomLog "/dev/null" combined
ServerSignature Off
ScriptAlias /cgi-bin/ "/home/www/cgi-bin/"
<Directory "/home/www/cgi-bin">
AllowOverride None
Options None
Order allow,deny
Allow from all
</Directory>
IndexOptions FancyIndexing VersionSort NameWidth=* HTMLTable
AddIconByEncoding (CMP,/icons/compressed.gif) x-compress x-gzip
AddIconByType (TXT,/icons/text.gif) text/*
AddIconByType (IMG,/icons/image2.gif) image/*
AddIconByType (SND,/icons/sound2.gif) audio/*
AddIconByType (VID,/icons/movie.gif) video/*
AddIcon /icons/binary.gif .bin .exe
AddIcon /icons/binhex.gif .hqx
AddIcon /icons/tar.gif .tar
AddIcon /icons/world2.gif .wrl .wrl.gz .vrml .vrm .iv
AddIcon /icons/compressed.gif .Z .z .tgz .gz .zip
AddIcon /icons/a.gif .ps .ai .eps
AddIcon /icons/layout.gif .html .shtml .htm .pdf
AddIcon /icons/text.gif .txt
AddIcon /icons/c.gif .c
AddIcon /icons/p.gif .pl .py
AddIcon /icons/f.gif .for
AddIcon /icons/dvi.gif .dvi
AddIcon /icons/uuencoded.gif .uu
AddIcon /icons/script.gif .conf .sh .shar .csh .ksh .tcl
AddIcon /icons/tex.gif .tex
AddIcon /icons/bomb.gif core
AddIcon /icons/back.gif ..
AddIcon /icons/hand.right.gif README
AddIcon /icons/folder.gif ^^DIRECTORY^^
AddIcon /icons/blank.gif ^^BLANKICON^^
DefaultIcon /icons/unknown.gif
ReadmeName README.html
HeaderName HEADER.html
AddLanguage ca .ca
AddLanguage cs .cz .cs
AddLanguage da .dk
AddLanguage de .de
AddLanguage el .el
AddLanguage en .en
AddLanguage eo .eo
AddLanguage es .es
AddLanguage et .et
AddLanguage fr .fr
AddLanguage he .he
AddLanguage hr .hr
AddLanguage it .it
AddLanguage ja .ja
AddLanguage ko .ko
AddLanguage ltz .ltz
AddLanguage nl .nl
AddLanguage nn .nn
AddLanguage no .no
AddLanguage pl .po
AddLanguage pt .pt
AddLanguage pt-BR .pt-br
AddLanguage ru .ru
AddLanguage sv .sv
AddLanguage zh-CN .zh-cn
AddLanguage zh-TW .zh-tw
LanguagePriority en fr
ForceLanguagePriority Prefer Fallback
AddDefaultCharset UTF-8
AddDefaultCharset iso-8859-1
AddType application/x-compress .Z
AddType application/x-gzip .gz .tgz
AddHandler type-map var
AddType text/html .shtml
AddOutputFilter INCLUDES .shtml
Alias /error/ "/home/www/error/"
<IfModule mod_negotiation.c>
</IfModule>
BrowserMatch "Mozilla/2" nokeepalive
BrowserMatch "MSIE 4\.0b2;" nokeepalive downgrade-1.0 force-response-1.0
BrowserMatch "RealPlayer 4\.0" force-response-1.0
BrowserMatch "Java/1\.0" force-response-1.0
BrowserMatch "JDK/1\.0" force-response-1.0
BrowserMatch "Microsoft Data Access Internet Publishing Provider"
redirect-carefully
BrowserMatch "MS FrontPage" redirect-carefully
BrowserMatch "^WebDrive" redirect-carefully
BrowserMatch "^WebDAVFS/1.[0123]" redirect-carefully
BrowserMatch "^gnome-vfs/1.0" redirect-carefully
BrowserMatch "^XML Spy" redirect-carefully
BrowserMatch "^Dreamweaver-WebDAV-SCM1" redirect-carefully
ExtendedStatus On
<Location /server-status>
SetHandler server-status
Order deny,allow
Deny from all
Allow from localhost 127.0.0.1 209.172.xxx.xxx 10.2.64.2 10.2.64.3
</Location>
GeoIPEnable On
GeoIPDBFile "/usr/local/share/GeoIP/GeoIPCity.dat"
GeoIPScanProxyHeaders On
AddDefaultCharset iso-8859-1
Rob Morin
Systems Administrator
Infinity Labs Inc.
(514) 387-0638 Ext: 207
ilabs-email-sig
Re: [users@httpd] Apache 2.2 connections full
Posted by Jeroen Geilman <je...@adaptr.nl>.
On 06/07/2011 10:59 PM, Rob Morin wrote:
>
> Hello all was not sure how to word my subject line...
>
> I posted a few weeks back on how could I increase performance of
> apache... I received quite a few replies and they all helped a bit...
>
> We even added an additional server to our server cluster to total 7
> web servers now, all load balanced by HAProxy
>
> We currently receive about 14 million impressions a day to about 60
> different domains/websites....
>
> Our hosting provider uses HAProxy, rather than an appliance, I assume
> to reduce costs. However I feel like these HAProxy machines are not
> reliable at all....Well not under the supervision of our hosting
> company , anyways...J
>
> On to my questions....
>
> During peek hours, like a few mins ago, all of our web servers became
> high in load, like close to 100! We still had plenty of RAM left and
> for sure there was no swapping....
>
So what was the cause of the load ?
Examine dstat output while the load spikes and see if there is a high
percentage spent in IOWait.
Apache does ultimately serve content from disk; is your disk subsystem
up to the task ?
> Please see my config below as quick snapshot of the httpd.conf file
> on our Centos Servers.
>
> The weird thing is when I would go see the server-status I would see
> all the slots filed with either W or C and nothing else.... Then after
> a what appears a random amount of time it would all clear up , and
> then come back again. Almost like HAProxy would send a bunch of
> requests to a server, then it would get full and then(haproxy) say, oh
> that server is filled I will not send traffic there, for now, then it
> would come back and see hey this server is empty, lets send a bunch
> of traffic there...
>
So it may just as well be the frontend that causes these spikes; you
want to establish this with certainty.
Do all apache backend boxes spike at the same time ?
You really need to do a more thorough investigation.
Consider graphing the performance of all boxes, and comparing values
during different times of the day.
An rrdtool grapher like cacti or mrtg will help with this, at marginal
extra load.
If you consider that too heavy, munin has useful data to correlate CPU
with disk I/O and apache stats.
> We also use memcache to cache sessions and mysql queries, along with
> eAcclerator... I assume that those are functioning properly
>
That's a pretty large assumption. Don't.
> Does this make sense, am I on drugs? J
>
You tell us.
> <IfModule prefork.c>
> <IfModule worker.c>
This tells us nothing. WHICH ONE are you actually using ?
If you're using prefork (which is the stupid choice for high load
machines), starting up a hundred new threads at peak times will take
MINUTES.
It looks as if you haven't tuned any of the really vital server
variables for this supposed high load you're trying to handle.
> <Directory />
>
> Options FollowSymLinks
>
> AllowOverride None
>
> </Directory>
>
You really want to deny access here.
> <Directory "/home/www">
>
> Options -Indexes FollowSymLinks
>
This does nothing.
>
> AllowOverride All
>
> Order allow,deny
>
> Allow from all
>
> </Directory>
>
> <IfModule mod_userdir.c>
>
> UserDir disable
>
> </IfModule>
>
> DirectoryIndex index.php index.html
>
> AccessFileName .htaccess
>
> <Files ~ "^\.ht">
>
> Order allow,deny
>
> Deny from all
>
> </Files>
>
> TypesConfig /etc/mime.types
>
> DefaultType text/plain
>
> <IfModule mod_mime_magic.c>
>
> MIMEMagicFile conf/magic
>
> </IfModule>
>
> HostnameLookups Off
>
> ErrorLog logs/error_log
>
> LogLevel notice
>
> LogFormat "%h" combined
>
> LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b" common
>
> LogFormat "%{Referer}i -> %U" referer
>
> LogFormat "%{User-agent}i" agent
>
> CustomLog "/dev/null" combined
>
That's asenine.
> GeoIPEnable On
>
> GeoIPDBFile "/usr/local/share/GeoIP/GeoIPCity.dat"
>
> GeoIPScanProxyHeaders On
>
Is this in any way a heavy/expensive database ?
What is being logged by this ?
What is it used for ?
Depending on usage, it may be as bad as performing DNS lookups.
--
J.