things to look for in runaway server?

[Brian Behlendorf <br...@organic.com>]

www.links.net is running 0.8.11, is a BSDI 2.0.1 box, and gets about 350K 
hits/day with peaks of 350/minute.  It's for the most part stable, 
with 30-50 children in the pool at peak, but every now and then the 
server goes wild and creates as many children as possible, wedging the 
machine, slowing down requests (though they still get out).  Since I'm 
not always on the machine I'm not there to see when it gets wedged what's 
going on, so we're trying to develop a script that runs every two minutes 
and takes a snapshot of the system.  Given that this should include a 
load average, a number of concurrent httpd's, and the output of a vmstat 
and iostat, is there anything else it should look for?

The logfiles indicate that reverse DNS stopped working (so the log files 
just had IP numbers in it) and every 50 hits had a listing like

206.14.10.76 - - [13/Sep/1995:13:26:23 -0700] "NULL" 200 920

Thoughts?

	Brian

--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--
brian@organic.com  brian@hyperreal.com  http://www.[hyperreal,organic].com/