[users@httpd] Performance problems

[Thomas Moyer <tm...@cse.psu.edu>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I am having some problems with performance when using mod_python to  
serve some dynamic content.  The request handler gathers information  
being generated by background processes on the system, and then  
creates a simple XML document that is sent to the client. Here is the  
code I am using with some explanation.

     requestedFilePath = "/" + req.filename.split("/")[-1] 
[:-4].replace("_", "/").replace("-",".")
     sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
     sock.connect(ATTService)
     Attest = message.recvMsg(sock)[0]
     sock.close()

     fmt = "%ds" % len(requestedFilePath)
     msg = struct.pack(fmt, requestedFilePath)
     sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
     sock.connect(MHTService)
     message.sendMsg(sock, msg, fmt)
     MHTList = message.recvMsg(sock)[0]
     sock.close()

     req.content_type = 'text/xml'
     req.write("<ACA>%s%s</ACA>" % (Attest,MHTList))
     return apache.OK

There are 3 different pieces here.  The first block connects to a  
daemon running on the system that is monitoring the system. I connect  
to a Unix domain socket and receive the most recent information it  
has.  I then do the same thing with another daemon that is gathering  
information about files that are available on the system. Finally I  
combine these two blocks of information and send it to the client.

My problem is, when I start using benchmarking tools like JMeter, and  
I put the system under any sort of load (100 clients for example), the  
first recvMsg call (Attest = message.recvMsg(sock)[0]) takes about 12  
seconds to complete per client.  I watch top during the run and see  
that the apache processes are completely occupied (usually showing 120+ 
% CPU usage).  The system running apache is an 8-core machine with  
16GB of RAM so I'm pretty sure there isn't a resource problem for  
either memory or processor.

I'm not really sure where to look next to understand why a simple  
socket recv() call is taking 12 seconds to complete.  The amount of  
data being recv()'d is ~80K of text if that makes a difference.

Below is the config file for Apache:
ServerRoot "/etc/apache2"
LockFile /var/lock/apache2/accept.lock
PidFile ${APACHE_PID_FILE}
Timeout 300
KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimeout 15
<IfModule mpm_prefork_module>
     StartServers          5
     MinSpareServers       5
     MaxSpareServers      10
     MaxClients          150
     MaxRequestsPerChild   0
</IfModule>
<IfModule mpm_worker_module>
     StartServers          2
     MaxClients          150
     MinSpareThreads      25
     MaxSpareThreads      75
     ThreadsPerChild      25
     MaxRequestsPerChild   0
</IfModule>
User ${APACHE_RUN_USER}
Group ${APACHE_RUN_GROUP}
AccessFileName .htaccess
<Files ~ "^\.ht">
     Order allow,deny
     Deny from all
</Files>
DefaultType text/plain
HostnameLookups Off
ErrorLog /var/log/apache2/error.log
LogLevel warn
Include /etc/apache2/mods-enabled/*.load
Include /etc/apache2/mods-enabled/*.conf
Include /etc/apache2/httpd.conf
Include /etc/apache2/ports.conf
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i 
\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
ServerTokens Full
ServerSignature On
PythonOption mod_python.mutex_locks 32
PythonOptimize On
Include /etc/apache2/conf.d/
Include /etc/apache2/sites-enabled/

Enabled Modules:
lias.conf
alias.load
auth_basic.load
authn_file.load
authz_default.load
authz_groupfile.load
authz_host.load
authz_user.load
autoindex.load
cgid.load
dir.load
env.load
mime.load
mod_python.load
negotiation.load
setenvif.load
status.load

And finally the site configuration itself:
NameVirtualHost *
<VirtualHost *>
	ServerAdmin webmaster@localhost
	DocumentRoot /var/www/
	<Directory />
		Options FollowSymLinks
		AllowOverride None
	</Directory>
	<Directory /var/www/>
		Options Indexes FollowSymLinks MultiViews
		AllowOverride None
		Order allow,deny
		allow from all
		AddHandler mod_python .aca
		PythonHandler serveACA | .aca
		PythonPath "sys.path + ['/var/shamon/']"
		PythonDebug On
	</Directory>
	ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
	<Directory "/usr/lib/cgi-bin">
		AllowOverride None
		Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
		Order allow,deny
		Allow from all
	</Directory>
	ErrorLog /var/log/apache2/error.log
	# Possible values include: debug, info, notice, warn, error, crit,
	# alert, emerg.
	LogLevel warn
	CustomLog /var/log/apache2/access.log combined
	ServerSignature On
     Alias /doc/ "/usr/share/doc/"
     <Directory "/usr/share/doc/">
         Options Indexes MultiViews FollowSymLinks
         AllowOverride None
         Order deny,allow
         Deny from all
         Allow from 127.0.0.0/255.0.0.0 ::1/128
     </Directory>
</VirtualHost>

Not sure what other information is needed.

~tom
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (Darwin)

iEYEARECAAYFAkjJDc0ACgkQa7Yypxfw9TdvngCgio05p2p9rHXhI3qB2EC8LgPV
uJEAn0w60bAQJ6QiMsZDrd4Z0c+uTzjr
=4TUr
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org