Stranded Threads in Standalone Tomcat 5.5.25 Server

["Bontempi, Chris" <Ch...@McKesson.com>]

I know this problem sounds familiar, but I can't find anything in the
archives that quite fits what's going on.  We're running a 5.5.25 Tomcat
server on HP/UX 11.11.  We have several installations like this, and
they are all functioning properly, without this particular problem,
except one.  Our installation process is comprehensive, so it's unlikely
there's a minor piece that is different, but we believe anything is
possible; however, we're completely stuck on what to look at that we
haven't already looked at, and we believe we have run through everything
we can think of.
 
The problem is a steadily increasing number of threads in the tomcat
server, all executing the same Axis 2.0 service -- a know service we
wrote, the exact same code that works properly on many other servers on
a total of three different platforms.  That service is exactly the same
on every system we have installed it on, the Java versions (HP/UX JVM
1.4.2_13) and operating system versions, the Tomcat versions and the
Axis 2 versions are all the same.  But only on one server do threads
linger, until the limit is reached and the server stops accepting
connections.  The ports they are accessing appear in varying states,
including ESTABLISHED, TIME_WAIT and FIN_WAIT_2, although we have
managed to reduce the last state by implementing a 60-second timeout for
the FIN_WAIT_2 state.  If we bump maxThreads in server.xml, it only
delays the limit being reached.  Changing connectionTimeout has no
effect, as most threads live far beyond the default setting anyway.
 
Again, we have checked HP/UX, Tomcat, Java, Axis2 and our application
software, and these things are all the same on every system.  We have
also tried replacing Tomcat, Axis2 and our software with known fresh
installs, but the behavior hasn't changed.  We can't find anything that
fits on this list archive, nor with google.  Can anyone think of what
other parts of the system I can look at, or what else I can try in order
to diagnose the issue here?  We have tried looking at Tomcat with lsof,
but I can't find anything helpful there.  I have read about thread
dumps, but I'm not sure how to do that or what to make of them, but I'm
going to read up on that next.  Again, this does not sound like a known
problem, because we're not using Apache or AJP.
 
Please send any constructive responses to Chris.Bontempi@mckesson.com
<ma...@mckesson.com> 
 
Thanks and regards,
 
Chris
 
Chris Bontempi
Resource Management Solutions
McKesson Provider Technologies
Hamden Home Office (203) 288-3491
Hadley Office (413) 587-6529
 
Confidentiality Notice: This e-mail message, including any attachments,
is for the sole use of the intended recipient(s) and may contain
confidential and privileged information. Any unauthorized review, use,
disclosure, or distribution is prohibited. If you are not the intended
recipient, please contact the sender by reply e-mail and destroy all
copies of the original message.
 

Re: Stranded Threads in Standalone Tomcat 5.5.25 Server

[Christopher Schultz <ch...@christopherschultz.net>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Chris,

On 6/19/2009 5:40 PM, Bontempi, Chris wrote:
> Please send any constructive responses to
> Chris.Bontempi@mckesson.com <ma...@mckesson.com>

If you post to the list, expect a reply to the list. If you want special
attention, pay someone :(

> The problem is a steadily increasing number of threads in the tomcat 
> server, all executing the same Axis 2.0 service -- a know service we 
> wrote, the exact same code that works properly on many other servers
> on a total of three different platforms.

Can you post a thread dump? Assuming all threads are hung in the same
place, you can post a representative sample.

> Again, we have checked HP/UX, Tomcat, Java, Axis2 and our
> application software, and these things are all the same on every
> system.  We have also tried replacing Tomcat, Axis2 and our software
> with known fresh installs, but the behavior hasn't changed.

I would check the HPUX install, too, since you could be running up
against some strange configuration, there, too. Do you have the option
of re-installing using your "comprehensive" install procedure?

> We can't find anything that fits on this list archive, nor with
> google.  Can anyone think of what other parts of the system I can
> look at, or what else I can try in order to diagnose the issue here?

Bad hardware has been known to cause strange issues. Is it possible that
you have a bad hardware component? The regularity of these errors, and
the apparent consistency of the lock-ups seems to argue /against/ bad
hardware, but you didn't provide a thread dump so we don't really know
if each thread suffers the same fate.

> I have read about thread dumps, but I'm not sure how to do that or
> what to make of them, but I'm going to read up on that next.

http://wiki.apache.org/tomcat/HowTo#head-a3243c1d640297a9f2aa9446cbed434efbd4d1d9

Hope that helps,
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkpFZ+8ACgkQ9CaO5/Lv0PCBAgCffySASe2TSHvcajWDAMsFxJIy
rmUAoL8B7Om2YES8P9AXAdeXmnr1BOmx
=GOH2
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org