Servlets/JSP crash, static content & manager app OK

[Roman Fail <rf...@posportal.com>]

This is one for those who understand the internal separation of powers in Tomcat.
 
I have several webapps running in production with Standalone Tomcat 4.1.24, RedHat Linux 8.0 (2.4.18 kernel), Sun JDK 1.4.1_02 HotSpot client.  The min/max heap is 1GB, physical RAM is 2GB, dual P3 processors.  
 
The problem: periodically Tomcat stops serving up servlets and JSP from my webapps.  However, the manager webapp and static pages continue to be served normally.  If I click 'stop' for any of my webapps from the manager app, the page load just hangs forever.  If you refresh the manager, it shows the webapp as stopped.  
 
You would think that I could just stop Tomcat with the $CATALINA_HOME/bin/shutdown.sh script, but that doesn't seem to do anything at all.  Generally I have to 'kill' the pid to get Tomcat to stop, sometimes even a 'kill -9'.  Sessions are not serialized when I call shutdown.sh, but curiously if I try and 'stop' each webapp before calling shutdown.sh, the serialization does occur.
 
My best guess is that I've got a servlet/JSP out there that is doing something very bad.  However, I've examined the JVM with a profiler and found nothing amiss.  I've also started using the AccessLogValve to see the client requests just before the crash and nothing really sticks out.  Sometimes the server will run for 8 hours without a crash, and sometimes it will crash within 60 seconds of a restart.
 
Naturally, I can find nothing useful in the $CATALINA_HOME/logs directory, or the /var/log directory either.  
 
Our company is getting pretty desperate.  It seems that Tomcat is not gracefully handling an error condition.  If we can't solve the problem soon, the boss wants to purchase WebLogic.  While I am loathe to do so, It's getting to a point where we don't have a choice, we've got to save our business.  Of course there are no guarantees that WebLogic will solve the problem either.
 
Any comments or suggestions are welcome!  Thanks a bunch.
 
Roman Fail
POS Portal, Inc.