tomcat segmentation fault

[bc...@mail.liberty.k12.mo.us]

I'm running a Tomcat/Apache web server and a PostgresQL database 
server.  In the past two months or so, the number of webapps I'm running 
has drastically increased, and it has led to many unexpected problems.  
Please bear with me as a summarize before asking my question; hopefully 
this will prove useful to somebody someday:

machine 1:  dual-933MHz, 512 MB RAM, Linux 2.4 (RedHat 7.1), Apache 
1.3.19, Tomcat 4.0.3, Java 1.4
machine 2:  731MHz, 256 MB Ram, Linux 2.2 (RedHat 6.2), PostgresQL 7.1.3

The first problem I ran into was random crashes of PostgresQL.  That 
apparently was caused by too many open files.  I quadrupled the values 
in /proc/sys/fs/file-max and /proc/sys/fs/inode-max and have not had any 
problems with that machine since.

The second problem turned up in my Tomcat logs:  
java.net.SocketException:  Too many open files.  Once I started seeing 
that, I had to reboot Tomcat.  I upped /proc/sys/fs/file-max, but there 
is no /proc/sys/fs/inode-max, and that didn't seem to help.  "ulimit -n 
90000" seems to have fixed that, but...

Tomcat kept crashing, but now it was throwing 
java.lang.OutOfMemoryError: requested x bytes.  I added -Xms64M -Xmx256M 
to increase the Java heap size, and added some garbage collection 
routines.  That didn't seem to fix it.  Then I noticed this in one of 
the logs:
java.lang.OutOfMemoryError:  unable to create new native thread
	at java.lang.Thread.start
	at org.apache.catalina.connector.warp.WarpConnection.start (159)
	at org.apache.catalina.connector.warp.WarpConnector.run (601)
	at java.lang.Thread.run (536)
Which prefigured an OutOfMemoryError and complete crash of Tomcat by 
minutes.

Doing some research, I found that there was a built-in thread limit in 
glibc.  I got the source RPM off the RedHat 7.1 CD, made the necessary 
changes in linuxthreads, and recompiled.  I copied the new 
linuxthreads.so to /lib/i686, and everything seemed to be working fine.  
No crashes, but it was a weekend and nobody uses it on weekends.  So 
this morning I did a 'make install' on glibc (huge mistake; I feel 
strongly that I had some reason for doing that but I can't remember what 
it was), and wham, Tomcat crashes and won't start up again.  I 
reinstalled the glibc RPMs on the RedHat CD to try to get the system 
back to normal, but Tomcat still won't start up.  Which is where my 
question comes in.

Why won't it start up again?  :)  Using Java 1.4, Tomcat fails to start 
up with a segmentation fault.  I switched to Java 1.3.1, and it fails to 
start up with a "cannot find libhpi.so".  Java 1.2.2 works, but I'm 
using some 1.4 features, so that's only a temporary solution.  I can 
still compile and run programs using 1.4; only Tomcat seems to not be 
working.  What have I done and how can I fix it?

Also, what more do I need to do to get these servers to be able to 
handle the load?  Apache is serving about 400 requests per hour on 
average, as much as 1700 during a busy hour, which doesn't seem to me 
like it should be too much for this server to handle.  I usually have 
between 80 and 120 concurrent database connections during peak hours, 
including sessions which are no longer in use and waiting to expire.

Any help would be very much appreciated.

Thanks,
Ben Carterette
bcarterette@liberty.k12.mo.us


--
To unsubscribe:   <ma...@jakarta.apache.org>
For additional commands: <ma...@jakarta.apache.org>
Troubles with the list: <ma...@jakarta.apache.org>