You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by "Robinson, Eric" <er...@psmnv.com> on 2013/03/22 21:36:31 UTC
Diagnosing Silently Frozen Tomcats
When tomcat freezes silently, what is the best way to look inside the frozen tomcat or java instance and see the cause? We have 1500+ tomcat instances, and most of them run great most of the time. But a few times per week, an instance will become unresponsive. There is usually nothing suspicious in the tomcat or gc logs. There are no OOMEs and we don't see any full GC's that take more than 0.25 seconds. The process does not die. It stays running and the port is still listening, and you can even telnet to the port, but tomcat does not answer any requests until you restart it. Is there a way to peer into the frozen instance and see what's up? We use tomcat 6.0.16 and java 6.0.21 on RHEL 5 and RHEL 6.
--
Eric Robinson
Disclaimer - March 22, 2013
This email and any files transmitted with it are confidential and intended solely for users@tomcat.apache.org. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physicians' Managed Care or Physician Select Management. Warning: Although Physicians' Managed Care or Physician Select Management has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments.
This disclaimer was added by Policy Patrol: http://www.policypatrol.com/
Re: Diagnosing Silently Frozen Tomcats
Posted by Michael Gesundheit <mi...@rocketmail.com>.
Just do "sudo kill -3 <PID>"
It will most likely work.
If sudo is not enable - just google it. You will find instructions how
to enable it.
-Michael
--- On Fri, 3/22/13, Mark Eggers <it...@yahoo.com> wrote:
> From: Mark Eggers <it...@yahoo.com>
> Subject: Re: Diagnosing Silently Frozen Tomcats
> To: users@tomcat.apache.org
> Date: Friday, March 22, 2013, 3:36 PM
> On 3/22/2013 3:22 PM, Robinson, Eric
> wrote:
> > Dan said:
> >> Take a full thread dump of the PID. Linux
> kill -3, if Windows
> >> CTRL+BREAK if you have a console window open.
> >>
> >
> > Mark said:
> >> 2. Thread dumps
> >>
> >
> > So I tried kill -3 but it does nothing. The process
> ignores me like
> > it owes me child support. I notice that it is in sleep
> state S1 if
> > that matters. Below is the full output from 'ps ax'
> >
> >
> > 18499 ? Sl
> 1:45 /usr/java/jdk1.6.0_21/bin/java
> -Xms16M
> > -Xmx192M -Djvm=tomcat6_003 -Djava.awt.headless=true
> > -Djava.net.preferIPv4Stack=true
> -Duser.timezone=US/Pacific
> > -Xloggc:/alley/site003/tomcat6/logs/gc.log
> -XX:+PrintGCDateStamps
> > -XX:+PrintGCDetails
> >
> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
> >
> -Djava.util.logging.config.file=/alley/site003/tomcat6/conf/logging.properties
> > -Djava.endorsed.dirs=/alley/site003/tomcat6/endorsed
> -classpath
> > :/alley/site003/tomcat6/bin/bootstrap.jar
> > -Dcatalina.base=/alley/site003/tomcat6
> > -Dcatalina.home=/alley/site003/tomcat6
> > -Djava.io.tmpdir=/alley/site003/tomcat6/temp
> > org.apache.catalina.startup.Bootstrap start
> >
> >
> > Any thoughts on why kill won't kill it?
> >
> > --Eric
>
> 1. Make sure you have permissions
>
> If you're not the owner of the process or root, you'll get a
> 'permission
> denied'
>
> 2. look in the log files
>
> $CATALINA_BASE/logs/catalina.out is the standard place for
> the thread
> dump. Who knows where it is on rpm-managed installations . .
> . I always
> install mine from scratch.
>
> . . . . just my two cents.
> /mde/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Diagnosing Silently Frozen Tomcats
Posted by Mark Eggers <it...@yahoo.com>.
On 3/22/2013 3:22 PM, Robinson, Eric wrote:
> Dan said:
>> Take a full thread dump of the PID. Linux kill -3, if Windows
>> CTRL+BREAK if you have a console window open.
>>
>
> Mark said:
>> 2. Thread dumps
>>
>
> So I tried kill -3 but it does nothing. The process ignores me like
> it owes me child support. I notice that it is in sleep state S1 if
> that matters. Below is the full output from 'ps ax'
>
>
> 18499 ? Sl 1:45 /usr/java/jdk1.6.0_21/bin/java -Xms16M
> -Xmx192M -Djvm=tomcat6_003 -Djava.awt.headless=true
> -Djava.net.preferIPv4Stack=true -Duser.timezone=US/Pacific
> -Xloggc:/alley/site003/tomcat6/logs/gc.log -XX:+PrintGCDateStamps
> -XX:+PrintGCDetails
> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
> -Djava.util.logging.config.file=/alley/site003/tomcat6/conf/logging.properties
> -Djava.endorsed.dirs=/alley/site003/tomcat6/endorsed -classpath
> :/alley/site003/tomcat6/bin/bootstrap.jar
> -Dcatalina.base=/alley/site003/tomcat6
> -Dcatalina.home=/alley/site003/tomcat6
> -Djava.io.tmpdir=/alley/site003/tomcat6/temp
> org.apache.catalina.startup.Bootstrap start
>
>
> Any thoughts on why kill won't kill it?
>
> --Eric
1. Make sure you have permissions
If you're not the owner of the process or root, you'll get a 'permission
denied'
2. look in the log files
$CATALINA_BASE/logs/catalina.out is the standard place for the thread
dump. Who knows where it is on rpm-managed installations . . . I always
install mine from scratch.
. . . . just my two cents.
/mde/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Diagnosing Silently Frozen Tomcats
Posted by "Robinson, Eric" <er...@psmnv.com>.
>
> ... and a link to TFM, if Eric didn't see [the other] Mark's response:
> http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_d
> ump_of_my_running_webapp_.3F
>
> - -chris
I did see that and I even read it before posting. It said kill -3 sends a SIGQUIT. I guess I jumped to the conclusion that QUIT meant quit. :-)
--Eric
Disclaimer - March 22, 2013
This email and any files transmitted with it are confidential and intended solely for Tomcat Users List. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physicians' Managed Care or Physician Select Management. Warning: Although Physicians' Managed Care or Physician Select Management has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments.
This disclaimer was added by Policy Patrol: http://www.policypatrol.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Diagnosing Silently Frozen Tomcats
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Mark,
On 3/22/13 6:26 PM, Mark Thomas wrote:
> On 22/03/2013 22:22, Robinson, Eric wrote:
>> Dan said:
>>> Take a full thread dump of the PID. Linux kill -3, if Windows
>>> CTRL+BREAK if you have a console window open.
>>>
>>
>> Mark said:
>>> 2. Thread dumps
>>>
>>
>> So I tried kill -3 but it does nothing. The process ignores me
>> like it owes me child support. I notice that it is in sleep state
>> S1 if that matters. Below is the full output from 'ps ax'
>>
>>
>> 18499 ? Sl 1:45 /usr/java/jdk1.6.0_21/bin/java -Xms16M
>> -Xmx192M -Djvm=tomcat6_003 -Djava.awt.headless=true
>> -Djava.net.preferIPv4Stack=true -Duser.timezone=US/Pacific
>> -Xloggc:/alley/site003/tomcat6/logs/gc.log -XX:+PrintGCDateStamps
>> -XX:+PrintGCDetails
>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>> -Djava.util.logging.config.file=/alley/site003/tomcat6/conf/logging.properties
>> -Djava.endorsed.dirs=/alley/site003/tomcat6/endorsed -classpath
>> :/alley/site003/tomcat6/bin/bootstrap.jar
>> -Dcatalina.base=/alley/site003/tomcat6
>> -Dcatalina.home=/alley/site003/tomcat6
>> -Djava.io.tmpdir=/alley/site003/tomcat6/temp
>> org.apache.catalina.startup.Bootstrap start
>>
>>
>> Any thoughts on why kill won't kill it?
>
> RTFM
>
> kill -3 doesn't kill the process it generates a thread dump to
> stdout.
... and a link to TFM, if Eric didn't see [the other] Mark's response:
http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_my_running_webapp_.3F
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJRTN3nAAoJEBzwKT+lPKRYlxwQAL9sJIW+RaxrI8J3rGwfArk/
Nyh0l7eiZvKBpKrpORrtjoNyc21kCerWYU5Bptt4S+X2gYaQh4tUlcyzrfd1jb9L
74jXxqr4zHkXhOZZWu9iZKefZvXwnohpb2HxAnD660W6SAhN0JEqrV5oyHSsW2tX
9rC9Mc0hOO7aFCFnal7G6GLTPLMbRSWcfWxV3zuidtv7WLmsIMju687F/CvuGR6t
oR0KF9SD3pMGHexJ7q7/U5VdfcVOBkyrteBJqrnHr6ydfhUzGIhOJZdx00b6yWmK
x1ssuzlSwIfr1Hx5YjeS0AKyUdDnXwqPEFoQIM1TJjb9bMshDRqkNu42w3GwiVsB
rcjpQayoQIWyJ1aPOb5B2JHHHQGU+DaPVKJHtZRc+C4BXDKgivW+BWJmDSlKi5tI
ph6YFKu0ESvJ5+EgIDz23gruZhh4fMTQkF9cRwa21fMiyycya31rYZg35+EpkKc0
C8e7v6mMHKmt7MxK3Jme4OxJcdwjL1e1yCLOIDlxHRq/r0flAaNkyesZn0atloDH
/366WTSAheHZrYngjyJ3mCO/pTttfqIYNJuFOUjv4YeaGLRV+61ec7np9KNImiCp
1PfeBlLSd7SZXWiKACDVCetqt5RWQr1OQcshCAKM8SKaB9SmuOUH2tW4FFU/6R4K
QP9LE5zCtD+5P+1KiUNX
=Rdgb
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Diagnosing Silently Frozen Tomcats
Posted by "Robinson, Eric" <er...@psmnv.com>.
> RTFM
>
> kill -3 doesn't kill the process it generates a thread dump to stdout.
>
> Mark
Ha! It's probably been 30 years since the last time anyone told me to RTFM. Fair enough, I suppose I deserved it, although in my defense I did check the man page first. :-)
Thanks!
--Eric
Disclaimer - March 22, 2013
This email and any files transmitted with it are confidential and intended solely for Tomcat Users List. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physicians' Managed Care or Physician Select Management. Warning: Although Physicians' Managed Care or Physician Select Management has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments.
This disclaimer was added by Policy Patrol: http://www.policypatrol.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Diagnosing Silently Frozen Tomcats
Posted by Mark Thomas <ma...@apache.org>.
On 22/03/2013 22:22, Robinson, Eric wrote:
> Dan said:
>> Take a full thread dump of the PID. Linux kill -3, if
>> Windows CTRL+BREAK if you have a console window open.
>>
>
> Mark said:
>> 2. Thread dumps
>>
>
> So I tried kill -3 but it does nothing. The process ignores me like it owes me child support. I notice that it is in sleep state S1 if that matters. Below is the full output from 'ps ax'
>
>
> 18499 ? Sl 1:45 /usr/java/jdk1.6.0_21/bin/java -Xms16M -Xmx192M -Djvm=tomcat6_003 -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Duser.timezone=US/Pacific -Xloggc:/alley/site003/tomcat6/logs/gc.log -XX:+PrintGCDateStamps -XX:+PrintGCDetails -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/alley/site003/tomcat6/conf/logging.properties -Djava.endorsed.dirs=/alley/site003/tomcat6/endorsed -classpath :/alley/site003/tomcat6/bin/bootstrap.jar -Dcatalina.base=/alley/site003/tomcat6 -Dcatalina.home=/alley/site003/tomcat6 -Djava.io.tmpdir=/alley/site003/tomcat6/temp org.apache.catalina.startup.Bootstrap start
>
>
> Any thoughts on why kill won't kill it?
RTFM
kill -3 doesn't kill the process it generates a thread dump to stdout.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Diagnosing Silently Frozen Tomcats
Posted by "Robinson, Eric" <er...@psmnv.com>.
Dan said:
> Take a full thread dump of the PID. Linux kill -3, if
> Windows CTRL+BREAK if you have a console window open.
>
Mark said:
> 2. Thread dumps
>
So I tried kill -3 but it does nothing. The process ignores me like it owes me child support. I notice that it is in sleep state S1 if that matters. Below is the full output from 'ps ax'
18499 ? Sl 1:45 /usr/java/jdk1.6.0_21/bin/java -Xms16M -Xmx192M -Djvm=tomcat6_003 -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Duser.timezone=US/Pacific -Xloggc:/alley/site003/tomcat6/logs/gc.log -XX:+PrintGCDateStamps -XX:+PrintGCDetails -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/alley/site003/tomcat6/conf/logging.properties -Djava.endorsed.dirs=/alley/site003/tomcat6/endorsed -classpath :/alley/site003/tomcat6/bin/bootstrap.jar -Dcatalina.base=/alley/site003/tomcat6 -Dcatalina.home=/alley/site003/tomcat6 -Djava.io.tmpdir=/alley/site003/tomcat6/temp org.apache.catalina.startup.Bootstrap start
Any thoughts on why kill won't kill it?
--Eric
Disclaimer - March 22, 2013
This email and any files transmitted with it are confidential and intended solely for Tomcat Users List. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physicians' Managed Care or Physician Select Management. Warning: Although Physicians' Managed Care or Physician Select Management has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments.
This disclaimer was added by Policy Patrol: http://www.policypatrol.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Diagnosing Silently Frozen Tomcats
Posted by "Carrillo, Dan" <dc...@idanalytics.com>.
Take a full thread dump of the PID. Linux kill -3, if Windows CTRL+BREAK if you have a console window open.
This will allow you to see the state of all threads and see whether it's your own application class (likely), or Tomcat itself.
-----Original Message-----
From: Robinson, Eric [mailto:eric.robinson@psmnv.com]
Sent: Friday, March 22, 2013 1:37 PM
To: users@tomcat.apache.org
Subject: Diagnosing Silently Frozen Tomcats
When tomcat freezes silently, what is the best way to look inside the frozen tomcat or java instance and see the cause? We have 1500+ tomcat instances, and most of them run great most of the time. But a few times per week, an instance will become unresponsive. There is usually nothing suspicious in the tomcat or gc logs. There are no OOMEs and we don't see any full GC's that take more than 0.25 seconds. The process does not die. It stays running and the port is still listening, and you can even telnet to the port, but tomcat does not answer any requests until you restart it. Is there a way to peer into the frozen instance and see what's up? We use tomcat 6.0.16 and java 6.0.21 on RHEL 5 and RHEL 6.
--
Eric Robinson
Disclaimer - March 22, 2013
This email and any files transmitted with it are confidential and intended solely for users@tomcat.apache.org. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physicians' Managed Care or Physician Select Management. Warning: Although Physicians' Managed Care or Physician Select Management has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments.
This disclaimer was added by Policy Patrol: http://www.policypatrol.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Diagnosing Silently Frozen Tomcats
Posted by Mark Eggers <it...@yahoo.com>.
On 3/22/2013 1:36 PM, Robinson, Eric wrote:
> When tomcat freezes silently, what is the best way to look inside the
> frozen tomcat or java instance and see the cause? We have 1500+
> tomcat instances, and most of them run great most of the time. But a
> few times per week, an instance will become unresponsive. There is
> usually nothing suspicious in the tomcat or gc logs. There are no
> OOMEs and we don't see any full GC's that take more than 0.25
> seconds. The process does not die. It stays running and the port is
> still listening, and you can even telnet to the port, but tomcat does
> not answer any requests until you restart it. Is there a way to peer
> into the frozen instance and see what's up? We use tomcat 6.0.16 and
> java 6.0.21 on RHEL 5 and RHEL 6.
>
1. Upgrade - many important security fixes
2. Thread dumps
http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_my_running_webapp_.3F
. . . just my two cents.
/mde/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org