Running Tomcat 5.5.9 in AIX 5.2 with Java 5 64-bit - Tomcat freezes at a high level of workload

[Matthew Lee <mk...@qad.com>]

Hello All,
 
Setup
---------
We are researching into a site running a Webapplication using the 
following setup:

IBM AIX 5.2
Tomcat 5.5.9
Java 5 64-bits (IBM Patch level is set to pap64dev-20061003a and 
downloaded from the IBM URL of : 
http://www-128.ibm.com/developerworks/java/jdk/aix/service.html)
There is 16Gb of memory on the server with > 6 CPU on the server which is
running both Tomcat and the Oracle/Progress DBMS components.
Tomcat startup script has allocated 2 Gb of memory as min and max.
 
Symptoms
--------------
When Tomcat is under heavy load running a webappliccation 
(at around 180 users although it varies), Tomcat 8080 will "freeze", 
whereby no more new uses can access the Webapplication, nor can they 
access the Tomcat Manager page.

During this "freezes/hanging", we can perform a "telnet" to 
<servername>:8080 from a Unix prompt on the server, so it seems that the 
Tomcat 8080 port is able to listen, although not able to process any 
request.

The worst thing about this is when the hanging occurs, the normal 
"shutdown.sh" script cannot shut down the hanging Tomcat process. 
We also find that the "kill" command is ineffective using any signal, 
including "-9". Therefore, when the hanging occurs, the only to restart 
Tomcat and the associated webapp is to reboot the whole server.

While the Tomcat server hangs, all other process seems to run correctly -
for example, we have an Oracle database and Progress DBMS components such 
as Webspeed and Appserver and they all continue to run with no problem.
The CPU time is NOT fully utilized when the Tomcat freezes up.
We have not been able to find any error message associated with the 
catalina.out log when the freeze occurs.
 
Research Done So Far
-------------------------------
We did a search thru the email archive (http://marc.theaimsgroup.com/)
using the keywords of "tomcat aix hangs", the most relevant entry I 
found was the following:
http://marc.theaimsgroup.com/?t=116097948400003&r=1&w=2
But the above uses much older component as follow, and did not seem like a 
good match:
to our setup:
> Tomcat 3.3.1a
> AIX 5.3
> IBM JDK 1.3.1
> Apache/1.3.28
The last reply on the above posting was also not conclusive in what the 
root cause was.
Other things found in Bugzilla which we initially thought were relevant 
were:
http://issues.apache.org/bugzilla/show_bug.cgi?id=40000
http://issues.apache.org/bugzilla/show_bug.cgi?id=34693
http://issues.apache.org/bugzilla/show_bug.cgi?id=32040
Again, upon closer inspection we were unable to find ways to apply the 
info
in the above entries in our situation. 

The one entry where we have extracted some application was the following:
http://issues.apache.org/bugzilla/show_bug.cgi?id=31142
> I tried a telnet task with ANT 1.6.2:
> - AIX 5.1 it's OK 
> - AIX 5.2 it's not OK the task freeze. I must make a CTRL-C to stop it
> 
> 
> You may want to try adding -Djava.net.preferIPv4Stack=true to the Ant 
command 
> line to see if that helps. I have seen some very long delays on AIX 5.2 
with the 
> IBM 1.4 VM when it needs to resolve remote addresses.
So we have now implemented the above in our Tomcat startup script and see 
whether 
the Tomcat server will hold up.
Other things we are trying is to switch to using Java 5 32-bit to run our 
Tomcat 5.5.9
instead of Tomcat 64-bit and see how things goes - the system has been 
stable for 24 hours
so far using this setting.
 
Questions
----------------
Has anyone encountered similarly issues in the past, especially with 
Tomcat 5 on AIX 5.2 + Java 5?

Any suggestion from you good folks on what other setting to configure in 
order to increase our 
chance to get some error messages logged during a system freezes/hanging?
 
Conclusion
----------------
Thanks for reviewing the above and any suggestion you might have on this.

Best Regards,
Matthew

RE: Running Tomcat 5.5.9 in AIX 5.2 with Java 5 64-bit - Tomcat freezes at a high level of workload

[Steffen Heil <li...@steffen-heil.de>]

Hi

First, 5.5.9 is definitly outdated. Update.
[This might help.]

Second, if tomcat is not reaction to kill -9, then it's you VM or even the
OS that has a problem.
Javacode can never to anything against kill -9, which is only handled by the
OS.
[This is only about symptoms.]

Regards,
  Steffen
 

> -----Original Message-----
> From: Matthew Lee [mailto:mkl@qad.com] 
> Sent: Wednesday, November 29, 2006 8:24 AM
> To: users@tomcat.apache.org
> Subject: Running Tomcat 5.5.9 in AIX 5.2 with Java 5 64-bit - 
> Tomcat freezes at a high level of workload 
> 
> Hello All,
>  
> Setup
> ---------
> We are researching into a site running a Webapplication using 
> the following setup:
> 
> IBM AIX 5.2
> Tomcat 5.5.9
> Java 5 64-bits (IBM Patch level is set to pap64dev-20061003a 
> and downloaded from the IBM URL of : 
> http://www-128.ibm.com/developerworks/java/jdk/aix/service.html)
> There is 16Gb of memory on the server with > 6 CPU on the 
> server which is running both Tomcat and the Oracle/Progress 
> DBMS components.
> Tomcat startup script has allocated 2 Gb of memory as min and max.
>  
> Symptoms
> --------------
> When Tomcat is under heavy load running a webappliccation (at 
> around 180 users although it varies), Tomcat 8080 will 
> "freeze", whereby no more new uses can access the 
> Webapplication, nor can they access the Tomcat Manager page.
> 
> During this "freezes/hanging", we can perform a "telnet" to 
> <servername>:8080 from a Unix prompt on the server, so it 
> seems that the Tomcat 8080 port is able to listen, although 
> not able to process any request.
> 
> The worst thing about this is when the hanging occurs, the 
> normal "shutdown.sh" script cannot shut down the hanging 
> Tomcat process. 
> We also find that the "kill" command is ineffective using any 
> signal, including "-9". Therefore, when the hanging occurs, 
> the only to restart Tomcat and the associated webapp is to 
> reboot the whole server.
> 
> While the Tomcat server hangs, all other process seems to run 
> correctly - for example, we have an Oracle database and 
> Progress DBMS components such as Webspeed and Appserver and 
> they all continue to run with no problem.
> The CPU time is NOT fully utilized when the Tomcat freezes up.
> We have not been able to find any error message associated 
> with the catalina.out log when the freeze occurs.
>  
> Research Done So Far
> -------------------------------
> We did a search thru the email archive 
> (http://marc.theaimsgroup.com/) using the keywords of "tomcat 
> aix hangs", the most relevant entry I found was the following:
> http://marc.theaimsgroup.com/?t=116097948400003&r=1&w=2
> But the above uses much older component as follow, and did 
> not seem like a good match:
> to our setup:
> > Tomcat 3.3.1a
> > AIX 5.3
> > IBM JDK 1.3.1
> > Apache/1.3.28
> The last reply on the above posting was also not conclusive 
> in what the root cause was.
> Other things found in Bugzilla which we initially thought 
> were relevant
> were:
> http://issues.apache.org/bugzilla/show_bug.cgi?id=40000
> http://issues.apache.org/bugzilla/show_bug.cgi?id=34693
> http://issues.apache.org/bugzilla/show_bug.cgi?id=32040
> Again, upon closer inspection we were unable to find ways to 
> apply the info in the above entries in our situation. 
> 
> The one entry where we have extracted some application was 
> the following:
> http://issues.apache.org/bugzilla/show_bug.cgi?id=31142
> > I tried a telnet task with ANT 1.6.2:
> > - AIX 5.1 it's OK
> > - AIX 5.2 it's not OK the task freeze. I must make a CTRL-C 
> to stop it
> > 
> > 
> > You may want to try adding -Djava.net.preferIPv4Stack=true 
> to the Ant
> command 
> > line to see if that helps. I have seen some very long delays on AIX 
> > 5.2
> with the 
> > IBM 1.4 VM when it needs to resolve remote addresses.
> So we have now implemented the above in our Tomcat startup 
> script and see whether the Tomcat server will hold up.
> Other things we are trying is to switch to using Java 5 
> 32-bit to run our Tomcat 5.5.9 instead of Tomcat 64-bit and 
> see how things goes - the system has been stable for 24 hours 
> so far using this setting.
>  
> Questions
> ----------------
> Has anyone encountered similarly issues in the past, 
> especially with Tomcat 5 on AIX 5.2 + Java 5?
> 
> Any suggestion from you good folks on what other setting to 
> configure in order to increase our chance to get some error 
> messages logged during a system freezes/hanging?
>  
> Conclusion
> ----------------
> Thanks for reviewing the above and any suggestion you might 
> have on this.
> 
> Best Regards,
> Matthew