Tomcat hangs after a few days!

[Volker <To...@Trendsetter.de>]

Hi,

I already posted following question on 26th Sep but the replies did not
refer to my problem but another one.

Maybe someone has a helpful hint for me!?

Thx
Volker
-------------------------------------

I have a problem running Jakarta Tomcat 4.1 port under FreeBSD 4.4 with
Blackdown Java 1.3.1 with mod_jk 1.2:

After startup.sh /logs/catalina.out shows up following lines:

[INFO] Registry - -Loading registry information
[INFO] Registry - -Creating new Registry instance
[INFO] Registry - -Creating MBeanServer
[INFO] Http11Protocol - -Initializing Coyote HTTP/1.1 on port 8080
Starting service Tomcat-Standalone
Apache Tomcat/4.1.24
HTTP Tunnel connected to the DigiChat server localhost
HttpTunnelingServlet connected to the groopz server
[INFO] Http11Protocol - -Starting Coyote HTTP/1.1 on port 8080
[INFO] ChannelSocket - -JK2: ajp13 listening on 0.0.0.0/0.0.0.0:8009
[INFO] JkMain - -Jk running ID=0 time=7/21
config=/usr/local/jakarta-tomcat4.1/conf/jk2.properties
[INFO] ChannelSocket - -server has been restarted or reset this
connection
[INFO] ChannelSocket - -server has been restarted or reset this
connection
[INFO] ChannelSocket - -server has been restarted or reset this
connection
[INFO] ChannelSocket - -server has been restarted or reset this
connection

The first days everythings runs correctly. The example servlets can be
executed via port 80 by help of mod_jk.

After a few days suddenly the servlets cannot be executed anymore.
Having a look on the Tomcat process with "top" prints out state
"sbwait".
That state does not change anymore AND I cannot shutdown the tomcat
process by help of shutdown.sh - so I have to kill the process with kill

-9.

"catalina.out" meanwhile has grown up to nearly 1MB because a lot of
lines (content: [INFO] ChannelSocket - -server has been restarted or
reset this connection ) have been inserted.

Killing and restarting Tomcat manually and restarting apache afterwards
for activating mod_jk again results in making all work again.

Does anyone have an idea of what is happening here?


Thanks and regards

Volker







---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org

Re: Tomcat hangs after a few days!

[David Rees <dr...@greenhydrant.com>]

On Wed, October 8, 2003 1at 2:07 am, Volker sent the following
> David Rees schrieb:
>> On Tue, October 7, 2003 1at 1:59 am, Volker sent the following
>>> Following problem can be recognized even if the tomcat process is
>>> not in sbwait mode:
>>>
>>> 1) Sending a -QUIT signal to the tomcat process does not stop the
>>> process.
>>
>> A -QUIT will not shut down Tomcat, but cause the JVM to dump a
>> stack trace to stdout of all threads. This will be useful when the
>> process hangs up. If the process is not responding to the -QUIT
>> command, the JVM has hung up or you have a lot of threads stuck in
>> a tight loop!
>
> I run the jvm in green mode (recommended by John from Volano for
> Blackdown Java under FreeBSD!). So the problem should not derive from
> too many threads!

You can suffer from scheduler starvation if you get too many threads
running in a tight loop!  (I know, it's happenned to me with a green JVM!)
 Symptoms will be 100% CPU, and the JVM will not respond except to the
kill signal.

> Sending the QUIT-signal does not show any result because stdout is
> the terminal and that one does not print out anything when I send a
> -QUIT. It is important to emphasize that the process does not show
> any reaction related to a -QUIT even when tomcat is running
> correctly. My original problem is that tomcat hangs up after a while
> (in that case nothing can be done with signals etc.).
>
> But what I wanted to explain in my last posting: Even when tomcat is
> running correctly (handling the servlet requests) you cannot stop it
> with shutdown.sh or -QUIT. Shutdown.sh only makes the CPU% increase
> over 90% and nothing does work anymore. But the tomcat process still
> is in memory. Only a kill -9 helps.

It's been a while since I've used Blackdown's JVM, but if we can't get a
stack trace from it when Tomcat is hung up (looks like it's easy for you
to reproduce by simply trying to shutdown), it will be difficult to figure
out what's going on.

> Like mentioned above: -QUIT does NOT produce any stack dump on the
> stdout!

On all Sun JVMs (IBM, too?), sending the JVM the QUIT signal will result
in a stack trace like the one described here:

http://developer.java.sun.com/developer/technicalArticles/Programming/Stacktrace/

It is the easiest way for us to remote debug an issue like this.

BTW, does this hang occur with a stock tomcat with only the default
webapps running (IE, no changes to the shipped configuration?)

-Dave

---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org

Re: Tomcat hangs after a few days!

[Volker <to...@trendsetter.de>]

David Rees schrieb:

> On Tue, October 7, 2003 1at 1:59 am, Volker sent the following
> > Following problem can be recognized even if the tomcat process is not in
> > sbwait mode:
> >
> > 1) Sending a -QUIT signal to the tomcat process does not stop the process.
>
> A -QUIT will not shut down Tomcat, but cause the JVM to dump a stack trace
> to stdout of all threads.  This will be useful when the process hangs up.
> If the process is not responding to the -QUIT command, the JVM has hung up or
> you have a lot of threads stuck in a tight loop!

Hi Dave,

I run the jvm in green mode (recommended by John from Volano for Blackdown Java
under FreeBSD!).
So the problem should not derive from too many threads!

Sending the QUIT-signal does not show any result because stdout is the terminal
and that one does not print out anything when I send a -QUIT.
It is important to emphasize that the process does not show any reaction related
to a -QUIT even when tomcat is running correctly.
My original problem is that tomcat hangs up after a while (in that case nothing
can be done with signals etc.).

But what I wanted to explain in my last posting: Even when tomcat is running
correctly (handling the servlet requests) you cannot stop it with shutdown.sh or
-QUIT. Shutdown.sh only makes the CPU% increase over 90% and nothing does work
anymore. But the tomcat process still is in memory. Only a kill -9 helps.

>
> > 2) Using shutdown.sh and -QUIT show up the same result: tomcat process
> > still stays in memory but suddenly the CPU value increases about over 90%.
> >
> > Only a kill -9 stops the process!
> >
> > With tomcat 3.x I did not have such a problem and I could shutdown tomcat
> > without any problems.
>
> Without knowing if and where Tomcat is hung up by reviewing the stack
> dump, we won't be able to get much further.

Like mentioned above: -QUIT does NOT produce any stack dump on the stdout!

Thanks

Volker

>
>
> -Dave
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: tomcat-user-help@jakarta.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org

Re: Tomcat hangs after a few days!

[David Rees <dr...@greenhydrant.com>]

On Tue, October 7, 2003 1at 1:59 am, Volker sent the following
> Following problem can be recognized even if the tomcat process is not in
> sbwait mode:
>
> 1) Sending a -QUIT signal to the tomcat process does not stop the process.

A -QUIT will not shut down Tomcat, but cause the JVM to dump a stack trace
to stdout of all threads.  This will be useful when the process hangs up. 
If the process is not responding to the -QUIT command, the JVM has hung up
or you have a lot of threads stuck in a tight loop!

> 2) Using shutdown.sh and -QUIT show up the same result: tomcat process
> still
> stays in memory but suddenly the CPU value increases about over 90%.
>
> Only a kill -9 stops the process!
>
> With tomcat 3.x I did not have such a problem and I could shutdown tomcat
> without any problems.

Without knowing if and where Tomcat is hung up by reviewing the stack
dump, we won't be able to get much further.

-Dave

---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org

Re: Tomcat hangs after a few days!

[Volker <To...@Trendsetter.de>]

David Rees schrieb:

> Can you send the Tomcat process a -QUIT signal so we can see what the
> state of the JVM is?
>
> It sounds like a JVM bug, and Java on FreeBSD is less than heavily used...
>  Try a different JVM if you can.

Hi David,

I use the most stable jvm I know for FreeBSD: Blackdown Java under Linux
Emulation Mode.

Following problem can be recognized even if the tomcat process is not in
sbwait mode:

1) Sending a -QUIT signal to the tomcat process does not stop the process.
2) Using shutdown.sh and -QUIT show up the same result: tomcat process still
stays in memory but suddenly the CPU value increases about over 90%.

Only a kill -9 stops the process!

With tomcat 3.x I did not have such a problem and I could shutdown tomcat
without any problems.

Any idea?


Thanks and regards

Volker


---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org

Re: Tomcat hangs after a few days!

[David Rees <dr...@greenhydrant.com>]

Can you send the Tomcat process a -QUIT signal so we can see what the
state of the JVM is?

It sounds like a JVM bug, and Java on FreeBSD is less than heavily used...
 Try a different JVM if you can.

-Dave

On Mon, October 6, 2003 at 6:37 am, Volker sent the following
> Hi,
>
> I already posted following question on 26th Sep but the replies did not
> refer to my problem but another one.
>
> Maybe someone has a helpful hint for me!?
>
> Thx
> Volker
> -------------------------------------
>
> I have a problem running Jakarta Tomcat 4.1 port under FreeBSD 4.4 with
> Blackdown Java 1.3.1 with mod_jk 1.2:


---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org