Tomcat arbitrarily freezes

[Oliver Schoenwald <ol...@fernuni-hagen.de>]

Hello,

we are using Tomcat 5.0.27 to provide the eLearning environment of our 
University.

Our system configuration is as follows:

SUN Fire 240 with 2 cpus at 1.28 GHz (Sparc-3)
8 GB RAM
Solaris 9 (with actual patches)

All requests are handled via Apache 2.0.52 with mod_ssl and using mod_jk2.

The system works well most of the time, but seemingly randomly the 
catalina process starts to use up
more and more cpu performance without actually doing anything useful! 
Memory usage stays normal/stable,
so we analyzed the process and found out the following:

- The cpu performance is used up primarily by the thread "VM Thread".
- This Thread is running nothing else but "synchronization primitives", 
which where called very fast an in high numbers without reading or 
writing anything (as if it is in some kind of endless loop of doing 
nothing).
- The effect (and an symptome which can be monitored repeatedly) is that 
the number of threads that the catalina process is using rises with 
every new request, because no thread gets enough cpu power to come to an 
end. "VM Thread" is taking away anything after a short period of time.

To come to this information, we did the following:

We monitored the machine with prstat -L to see when the cpu usage of the 
catalina process rose to a unusual high value (5% at average, rising up 
to about 60% when VM Thread got into its frenzy). So we could see that 
one single lwp was going into the high cpu usage.
As soon as we could see the rise in cpu usage we used pstack to find out 
which thread is responsible for the lwp-id that was shown by prstat -L.
Then, we used kill -3 to create a thread-dump of the catalina-process 
and searched for the thread-number that was stated by pstack.
That thread-dump showed us that the thread called "VM Thread" seems to 
be the problematic thread.

As far as we found out that problems occured several times with Tomcat 5 
for other users in the world. However, so far no one offered an 
explanation what is going wrong there and what could be done to avoid 
that problem.

We now have prepared a monitoring service that kills and restarts the 
catalina engine as soon as the problem occurs, but that solution is not 
a good one.

Any suggestions of how to solve the problem are very, very welcome. If 
you need more information or have any idea how we should analyze the 
situation more properly, I would be happy to hear of it.


Thank you in advance,


Oliver Schoenwald
FernUniversitaet Hagen
Germany



---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org

Re: Tomcat arbitrarily freezes

[Andreas Vombach <an...@psi.ch>]

... there is always a chance to have hardware problems (bad memory chip) 
so the error will occur when memory consumption increases.

---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org

Re: Tomcat arbitrarily freezes

[Remy Maucherat <re...@gmail.com>]

On Tue, 11 Jan 2005 13:51:13 +0100, Oliver Schoenwald
<ol...@fernuni-hagen.de> wrote:
> Hi!
> 
> five days after my first question and no answer in sight. To bad.
> So far, we have switched from JDK 1.4.2 to JDK 1.5 and from Tomcat
> 5.0.27 to 5.5.4 and the problem persists.
> However, while under JDK 1.4.2 it was always the thread with id #2 under
> JDK 1.5
> it is still the thread "VM Thread", but now under id #4.
> 
> Using the better debugging features of JDK 1.5 (jstack, jmap) we could
> pin down the problem in more detail:
> whenever the problem starts, the output of jstack and jmap shows that
> these debugging-tools have
> more and more problems to give information about certain memory areas.
> When the problem reaches its
> climax, a bug number of memory areas are marked by jstack with an "Error
> occurred during stack walking:"-Message
> or by jmap with "UnmappedAddressException".
> 
> You see, at the moment we are only guessing what problem we have here.
> 
> Maybe someone has an idea how to analyze the problem more properly or
> even know how to solve it (without changing some of the components like
> using another servlet container engine or another JVM).

If you don't get any answers, it is that people don't have any ideas
and / or nothing useful to say.

For example, this is the first time I hear about a problem with this
internal VM thread. While other "Tomcat arbitrarily freezes" issues
are well known on some OSes, this one is brand new to me. You seem to
assert that this is a common problem encountered by many people, and
you also seem certain that your problems come for that (which means
bad luck, as this is an internal VM thread). Can you give links giving
more details about this ?

-- 
xxxxxxxxxxxxxxxxxxxxxxxxx
Rémy Maucherat
Developer & Consultant
JBoss Group (Europe) SàRL
xxxxxxxxxxxxxxxxxxxxxxxxx

---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org

Re: Tomcat arbitrarily freezes

[Oliver Schoenwald <ol...@fernuni-hagen.de>]

Hi!

five days after my first question and no answer in sight. To bad.
So far, we have switched from JDK 1.4.2 to JDK 1.5 and from Tomcat 
5.0.27 to 5.5.4 and the problem persists.
However, while under JDK 1.4.2 it was always the thread with id #2 under 
JDK 1.5
it is still the thread "VM Thread", but now under id #4.

Using the better debugging features of JDK 1.5 (jstack, jmap) we could 
pin down the problem in more detail:
whenever the problem starts, the output of jstack and jmap shows that 
these debugging-tools have
more and more problems to give information about certain memory areas. 
When the problem reaches its
climax, a bug number of memory areas are marked by jstack with an "Error 
occurred during stack walking:"-Message
or by jmap with "UnmappedAddressException".

You see, at the moment we are only guessing what problem we have here.

Maybe someone has an idea how to analyze the problem more properly or 
even know how to solve it (without changing some of the components like 
using another servlet container engine or another JVM).

Thank you for any help,

Oliver Schoenwald
University of Hagen
Germany

Oliver Schoenwald wrote:

> Our system configuration is as follows:
>
> SUN Fire 240 with 2 cpus at 1.28 GHz (Sparc-3)
> 8 GB RAM
> Solaris 9 (with actual patches)
>
> All requests are handled via Apache 2.0.52 with mod_ssl and using 
> mod_jk2.
>
> The system works well most of the time, but seemingly randomly the 
> catalina process starts to use up
> more and more cpu performance without actually doing anything useful! 
> Memory usage stays normal/stable,
> so we analyzed the process and found out the following:
>
> - The cpu performance is used up primarily by the thread "VM Thread".
> - This Thread is running nothing else but "synchronization 
> primitives", which where called very fast an in high numbers without 
> reading or writing anything (as if it is in some kind of endless loop 
> of doing nothing).
> - The effect (and an symptome which can be monitored repeatedly) is 
> that the number of threads that the catalina process is using rises 
> with every new request, because no thread gets enough cpu power to 
> come to an end. "VM Thread" is taking away anything after a short 
> period of time.
>
> We monitored the machine with prstat -L to see when the cpu usage of 
> the catalina process rose to a unusual high value (5% at average, 
> rising up to about 60% when VM Thread got into its frenzy). So we 
> could see that one single lwp was going into the high cpu usage.
> As soon as we could see the rise in cpu usage we used pstack to find 
> out which thread is responsible for the lwp-id that was shown by 
> prstat -L.
> Then, we used kill -3 to create a thread-dump of the catalina-process 
> and searched for the thread-number that was stated by pstack.
> That thread-dump showed us that the thread called "VM Thread" seems to 
> be the problematic thread.
>


---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org