You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Daniel Gibby <dg...@edirectpublishing.com> on 2004/02/17 18:20:47 UTC
thread dump analysis
My tomcat 4.1.29 instance running J2RE 1.4.1 IBM build
cxia321411-20030930 on RedHat 9 kernel 2.4.18-14
keeps gaining processor usage until finally can't answer requests
successfully.
The machine has a relatively light load.
I did a kill -3 on the process that showed up on top and got a stack
trace... the problem is I have no idea how to analyze the thread dump to
see what is consuming CPU.
I'm sure something must be spinning its wheels, but I don't know how to
tell... I can just see that when I run top my tomcat process has 99.9 %
of the CPU and the load average is 8.00 8.00 8.00
I've fixed problems in the past on a separate java application (not
tomcat) where I can tell what the problem is in the thread dump because
a thread waiting to be notified is also the one that has a lock on it to
notify the thing that is waiting to notify it... (that didn't make
sense, I know... but anyway it is basically a circle where it won't ever
get woken up.)
However, in this tomcat case, I can't see anything like that where
something is waiting in circles... even though I wouldn't rule that out.
My experience on reading thread dumps is limited... Anyway, can someone
who has better experience tell me what is consuming the CPU? Restarting
tomcat brings the load back down, and it slowly goes up again... like
over a few days to a weeks time it is back up to 8.00 Load Average.
I won't include the whole file. I trimmed the file down to 1350 lines by
getting rid of a lot of 2HPMEMMAPLINE lines and the section titled:
0SECTION CL subcomponent dump routine
but I think that is still too long to post here.
I'm hoping that someone can tell me what to include and what to exclude
and I'll reply with the appropriate parts of the dump.
Thanks,
Daniel
NULL
------------------------------------------------------------------------
0SECTION TITLE subcomponent dump routine
NULL ===============================
1TISIGINFO signal 3 received
1TIDATETIME Date: 2004/02/17 at 08:53:22
1TIFILENAME Javacore filename: /tmp/javacore.20040217.085322.23429.txt
NULL
------------------------------------------------------------------------
0SECTION XHPI subcomponent dump routine
NULL ==============================
1HPTIME Tue Feb 17 08:53:22 2004
1HPSIGRECV SIGQUIT received in ?? at (nil) in ??.
1HPFULLVERSION J2RE 1.4.1 IBM build cxia321411-20030930
NULL
1HPOPENV Operating Environment
NULL ---------------------
2HPHOSTNAME Host : somehost.com.(none)
2HPOSLEVEL OS Level : 2.4.18-14.#1 Wed Sep 4 13:35:50 EDT 2002
2HPLIBCVER glibc Version : 2.2.93
2HPCPUS Processors -
3HPARCH Architecture : (not implemented)
3HPNUMCPUS How Many : (not implemented)
3HPCPUSENABLED Enabled : 1
NULL
1HPMEMINFO Memory Info
NULL -----------
2HPMEMLINE total: used: free: shared: buffers: cached:
2HPMEMLINE Mem: 1055625216 1015181312 40443904 0 83464192
614227968
2HPMEMLINE Swap: 1052827648 929792 1051897856
2HPMEMLINE MemTotal: 1030884 kB
2HPMEMLINE MemFree: 39496 kB
2HPMEMLINE MemShared: 0 kB
2HPMEMLINE Buffers: 81508 kB
2HPMEMLINE Cached: 599596 kB
2HPMEMLINE SwapCached: 236 kB
2HPMEMLINE Active: 552968 kB
2HPMEMLINE Inact_dirty: 344020 kB
2HPMEMLINE Inact_clean: 50304 kB
2HPMEMLINE Inact_target: 189456 kB
2HPMEMLINE HighTotal: 130880 kB
2HPMEMLINE HighFree: 1024 kB
2HPMEMLINE LowTotal: 900004 kB
2HPMEMLINE LowFree: 38472 kB
2HPMEMLINE SwapTotal: 1028152 kB
2HPMEMLINE SwapFree: 1027244 kB
2HPMEMLINE Committed_AS: 1067972 kB
NULL
1HPUSERLIMITS User Limits (in bytes except for NOFILE and NPROC) -
NULL -----------
2HPUSERLIMIT RLIMIT_FSIZE : infinity
2HPUSERLIMIT RLIMIT_DATA : infinity
2HPUSERLIMIT RLIMIT_STACK : 2093056
2HPUSERLIMIT RLIMIT_CORE : 0
2HPUSERLIMIT RLIMIT_NOFILE : 1024
2HPUSERLIMIT RLIMIT_NPROC : 7168
NULL
1HPSIGHANDLERS JVM Signal Handlers
NULL -------------------
2HPSIGHANDLER HUP : unknown handler
2HPSIGHANDLER INT : unknown handler
2HPSIGHANDLER QUIT : unknown handler
2HPSIGHANDLER ILL : unknown handler
2HPSIGHANDLER TRAP : unknown handler
2HPSIGHANDLER ABRT : unknown handler
2HPSIGHANDLER FPE : unknown handler
2HPSIGHANDLER KILL : default handler
---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
Re: thread dump analysis
Posted by Daniel Gibby <dg...@edirectpublishing.com>.
I'm going to try and disable the jcrontab servlets and see if the
problem persists. After that, I'll try and schedule some upgrades.
Thanks for your help David!
Daniel
David Rees wrote:
> David Rees wrote, On 2/17/2004 12:43 PM:
>
>> On Tue, February 17, 2004 1at 2:04 pm, Daniel Gibby wrote:
>>
>>> Well, I'd rather not show the world what my java processes are doing in
>>> case there is something proprietary in there.
>>>
>>> I'll send it to you personally.
>>
>>
>> OK, but it's tough for people to help troubleshoot your issue unless you
>> do so. ;-)
>
>
> I had a look at Daniel's thread dump, it did not appear to be an issue
> with Tomcat, but rather an issue with either some jcrontab threads or
> a JVM/OS issue as he is running the IBM 1.4.1 JDK on a stock RH 8 system.
>
> -Dave
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
Re: thread dump analysis
Posted by David Rees <dr...@greenhydrant.com>.
David Rees wrote, On 2/17/2004 12:43 PM:
> On Tue, February 17, 2004 1at 2:04 pm, Daniel Gibby wrote:
>
>>Well, I'd rather not show the world what my java processes are doing in
>>case there is something proprietary in there.
>>
>>I'll send it to you personally.
>
> OK, but it's tough for people to help troubleshoot your issue unless you
> do so. ;-)
I had a look at Daniel's thread dump, it did not appear to be an issue
with Tomcat, but rather an issue with either some jcrontab threads or a
JVM/OS issue as he is running the IBM 1.4.1 JDK on a stock RH 8 system.
-Dave
---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
Re: thread dump analysis
Posted by David Rees <dr...@greenhydrant.com>.
On Tue, February 17, 2004 1at 2:04 pm, Daniel Gibby wrote:
> Well, I'd rather not show the world what my java processes are doing in
> case there is something proprietary in there.
>
> I'll send it to you personally.
OK, but it's tough for people to help troubleshoot your issue unless you
do so. ;-)
-Dave
---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
Re: thread dump analysis
Posted by Daniel Gibby <dg...@edirectpublishing.com>.
Well, I'd rather not show the world what my java processes are doing in
case there is something proprietary in there.
I'll send it to you personally.
Daniel
David Rees wrote:
>On Tue, February 17, 2004 at 9:20 am, Daniel Gibby wrote:
>
>
>>I did a kill -3 on the process that showed up on top and got a stack
>>trace... the problem is I have no idea how to analyze the thread dump to
>>see what is consuming CPU.
>>I'm sure something must be spinning its wheels, but I don't know how to
>>tell... I can just see that when I run top my tomcat process has 99.9 %
>>of the CPU and the load average is 8.00 8.00 8.00
>>
>>
>>
><snip>
>
>
>>I'm hoping that someone can tell me what to include and what to exclude
>>and I'll reply with the appropriate parts of the dump.
>>
>>
>
>Best to just post the entire dump somewhere on the web where everyone can
>look at it. Sounds like something somewhere is getting threads stuck in a
>loop, should be pretty easy to spot if you get the dump during a non-busy
>time.
>
>Not sure how big of an attachment the list will accept, but you might be
>able to compress it and attach it to the list.
>
>If it's too big, just send it to me privately and I will have a look.
>
>-Dave
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
>For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
Re: thread dump analysis
Posted by David Rees <dr...@greenhydrant.com>.
On Tue, February 17, 2004 at 9:20 am, Daniel Gibby wrote:
>
> I did a kill -3 on the process that showed up on top and got a stack
> trace... the problem is I have no idea how to analyze the thread dump to
> see what is consuming CPU.
> I'm sure something must be spinning its wheels, but I don't know how to
> tell... I can just see that when I run top my tomcat process has 99.9 %
> of the CPU and the load average is 8.00 8.00 8.00
>
<snip>
>
> I'm hoping that someone can tell me what to include and what to exclude
> and I'll reply with the appropriate parts of the dump.
Best to just post the entire dump somewhere on the web where everyone can
look at it. Sounds like something somewhere is getting threads stuck in a
loop, should be pretty easy to spot if you get the dump during a non-busy
time.
Not sure how big of an attachment the list will accept, but you might be
able to compress it and attach it to the list.
If it's too big, just send it to me privately and I will have a look.
-Dave
---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
Re: thread dump analysis
Posted by Daniel Gibby <dg...@edirectpublishing.com>.
I definitely do have LD_KERNEL_ASSUME in my environment
If you search for LD_KERNEL and my name on google, you will see that I
was part of a discussion a while back that was glad to find that out.
Daniel Gibby
Oscar Carrillo wrote:
>Hi,
>
>Saw your message on the boards.
>
>Did you make sure you have this environment variable set?
>
>On systems that I need it, I put it in my tomcat startup file.
>
>LD_KERNEL_ASSUME="2.2.5"
>
>Check out my howto page for my brief notes on threading and it's potential
>problems with JVMs and threads.
>
>http://www.linuxjava.net/howto/webapp/
>
>Oscar
>
>On Tue, 17 Feb 2004, Daniel Gibby wrote:
>
>
>
>>My tomcat 4.1.29 instance running J2RE 1.4.1 IBM build
>>cxia321411-20030930 on RedHat 9 kernel 2.4.18-14
>>keeps gaining processor usage until finally can't answer requests
>>successfully.
>>
>>The machine has a relatively light load.
>>
>>I did a kill -3 on the process that showed up on top and got a stack
>>trace... the problem is I have no idea how to analyze the thread dump to
>>see what is consuming CPU.
>>I'm sure something must be spinning its wheels, but I don't know how to
>>tell... I can just see that when I run top my tomcat process has 99.9 %
>>of the CPU and the load average is 8.00 8.00 8.00
>>
>>I've fixed problems in the past on a separate java application (not
>>tomcat) where I can tell what the problem is in the thread dump because
>>a thread waiting to be notified is also the one that has a lock on it to
>>notify the thing that is waiting to notify it... (that didn't make
>>sense, I know... but anyway it is basically a circle where it won't ever
>>get woken up.)
>>However, in this tomcat case, I can't see anything like that where
>>something is waiting in circles... even though I wouldn't rule that out.
>>My experience on reading thread dumps is limited... Anyway, can someone
>>who has better experience tell me what is consuming the CPU? Restarting
>>tomcat brings the load back down, and it slowly goes up again... like
>>over a few days to a weeks time it is back up to 8.00 Load Average.
>>
>>I won't include the whole file. I trimmed the file down to 1350 lines by
>>getting rid of a lot of 2HPMEMMAPLINE lines and the section titled:
>>0SECTION CL subcomponent dump routine
>>but I think that is still too long to post here.
>>
>>I'm hoping that someone can tell me what to include and what to exclude
>>and I'll reply with the appropriate parts of the dump.
>>
>>Thanks,
>>Daniel
>>
>>NULL
>>------------------------------------------------------------------------
>>0SECTION TITLE subcomponent dump routine
>>NULL ===============================
>>1TISIGINFO signal 3 received
>>1TIDATETIME Date: 2004/02/17 at 08:53:22
>>1TIFILENAME Javacore filename: /tmp/javacore.20040217.085322.23429.txt
>>NULL
>>------------------------------------------------------------------------
>>0SECTION XHPI subcomponent dump routine
>>NULL ==============================
>>1HPTIME Tue Feb 17 08:53:22 2004
>>1HPSIGRECV SIGQUIT received in ?? at (nil) in ??.
>>1HPFULLVERSION J2RE 1.4.1 IBM build cxia321411-20030930
>>NULL
>>1HPOPENV Operating Environment
>>NULL ---------------------
>>2HPHOSTNAME Host : somehost.com.(none)
>>2HPOSLEVEL OS Level : 2.4.18-14.#1 Wed Sep 4 13:35:50 EDT 2002
>>2HPLIBCVER glibc Version : 2.2.93
>>2HPCPUS Processors -
>>3HPARCH Architecture : (not implemented)
>>3HPNUMCPUS How Many : (not implemented)
>>3HPCPUSENABLED Enabled : 1
>>NULL
>>1HPMEMINFO Memory Info
>>NULL -----------
>>2HPMEMLINE total: used: free: shared: buffers: cached:
>>2HPMEMLINE Mem: 1055625216 1015181312 40443904 0 83464192
>>614227968
>>2HPMEMLINE Swap: 1052827648 929792 1051897856
>>2HPMEMLINE MemTotal: 1030884 kB
>>2HPMEMLINE MemFree: 39496 kB
>>2HPMEMLINE MemShared: 0 kB
>>2HPMEMLINE Buffers: 81508 kB
>>2HPMEMLINE Cached: 599596 kB
>>2HPMEMLINE SwapCached: 236 kB
>>2HPMEMLINE Active: 552968 kB
>>2HPMEMLINE Inact_dirty: 344020 kB
>>2HPMEMLINE Inact_clean: 50304 kB
>>2HPMEMLINE Inact_target: 189456 kB
>>2HPMEMLINE HighTotal: 130880 kB
>>2HPMEMLINE HighFree: 1024 kB
>>2HPMEMLINE LowTotal: 900004 kB
>>2HPMEMLINE LowFree: 38472 kB
>>2HPMEMLINE SwapTotal: 1028152 kB
>>2HPMEMLINE SwapFree: 1027244 kB
>>2HPMEMLINE Committed_AS: 1067972 kB
>>NULL
>>1HPUSERLIMITS User Limits (in bytes except for NOFILE and NPROC) -
>>NULL -----------
>>2HPUSERLIMIT RLIMIT_FSIZE : infinity
>>2HPUSERLIMIT RLIMIT_DATA : infinity
>>2HPUSERLIMIT RLIMIT_STACK : 2093056
>>2HPUSERLIMIT RLIMIT_CORE : 0
>>2HPUSERLIMIT RLIMIT_NOFILE : 1024
>>2HPUSERLIMIT RLIMIT_NPROC : 7168
>>NULL
>>1HPSIGHANDLERS JVM Signal Handlers
>>NULL -------------------
>>2HPSIGHANDLER HUP : unknown handler
>>2HPSIGHANDLER INT : unknown handler
>>2HPSIGHANDLER QUIT : unknown handler
>>2HPSIGHANDLER ILL : unknown handler
>>2HPSIGHANDLER TRAP : unknown handler
>>2HPSIGHANDLER ABRT : unknown handler
>>2HPSIGHANDLER FPE : unknown handler
>>2HPSIGHANDLER KILL : default handler
>>
>>
>>
>>
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
>>For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
>>
>>
>>
>
>
>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
>For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
Re: thread dump analysis
Posted by Oscar Carrillo <to...@daydream.stanford.edu>.
Hi,
Saw your message on the boards.
Did you make sure you have this environment variable set?
On systems that I need it, I put it in my tomcat startup file.
LD_KERNEL_ASSUME="2.2.5"
Check out my howto page for my brief notes on threading and it's potential
problems with JVMs and threads.
http://www.linuxjava.net/howto/webapp/
Oscar
On Tue, 17 Feb 2004, Daniel Gibby wrote:
> My tomcat 4.1.29 instance running J2RE 1.4.1 IBM build
> cxia321411-20030930 on RedHat 9 kernel 2.4.18-14
> keeps gaining processor usage until finally can't answer requests
> successfully.
>
> The machine has a relatively light load.
>
> I did a kill -3 on the process that showed up on top and got a stack
> trace... the problem is I have no idea how to analyze the thread dump to
> see what is consuming CPU.
> I'm sure something must be spinning its wheels, but I don't know how to
> tell... I can just see that when I run top my tomcat process has 99.9 %
> of the CPU and the load average is 8.00 8.00 8.00
>
> I've fixed problems in the past on a separate java application (not
> tomcat) where I can tell what the problem is in the thread dump because
> a thread waiting to be notified is also the one that has a lock on it to
> notify the thing that is waiting to notify it... (that didn't make
> sense, I know... but anyway it is basically a circle where it won't ever
> get woken up.)
> However, in this tomcat case, I can't see anything like that where
> something is waiting in circles... even though I wouldn't rule that out.
> My experience on reading thread dumps is limited... Anyway, can someone
> who has better experience tell me what is consuming the CPU? Restarting
> tomcat brings the load back down, and it slowly goes up again... like
> over a few days to a weeks time it is back up to 8.00 Load Average.
>
> I won't include the whole file. I trimmed the file down to 1350 lines by
> getting rid of a lot of 2HPMEMMAPLINE lines and the section titled:
> 0SECTION CL subcomponent dump routine
> but I think that is still too long to post here.
>
> I'm hoping that someone can tell me what to include and what to exclude
> and I'll reply with the appropriate parts of the dump.
>
> Thanks,
> Daniel
>
> NULL
> ------------------------------------------------------------------------
> 0SECTION TITLE subcomponent dump routine
> NULL ===============================
> 1TISIGINFO signal 3 received
> 1TIDATETIME Date: 2004/02/17 at 08:53:22
> 1TIFILENAME Javacore filename: /tmp/javacore.20040217.085322.23429.txt
> NULL
> ------------------------------------------------------------------------
> 0SECTION XHPI subcomponent dump routine
> NULL ==============================
> 1HPTIME Tue Feb 17 08:53:22 2004
> 1HPSIGRECV SIGQUIT received in ?? at (nil) in ??.
> 1HPFULLVERSION J2RE 1.4.1 IBM build cxia321411-20030930
> NULL
> 1HPOPENV Operating Environment
> NULL ---------------------
> 2HPHOSTNAME Host : somehost.com.(none)
> 2HPOSLEVEL OS Level : 2.4.18-14.#1 Wed Sep 4 13:35:50 EDT 2002
> 2HPLIBCVER glibc Version : 2.2.93
> 2HPCPUS Processors -
> 3HPARCH Architecture : (not implemented)
> 3HPNUMCPUS How Many : (not implemented)
> 3HPCPUSENABLED Enabled : 1
> NULL
> 1HPMEMINFO Memory Info
> NULL -----------
> 2HPMEMLINE total: used: free: shared: buffers: cached:
> 2HPMEMLINE Mem: 1055625216 1015181312 40443904 0 83464192
> 614227968
> 2HPMEMLINE Swap: 1052827648 929792 1051897856
> 2HPMEMLINE MemTotal: 1030884 kB
> 2HPMEMLINE MemFree: 39496 kB
> 2HPMEMLINE MemShared: 0 kB
> 2HPMEMLINE Buffers: 81508 kB
> 2HPMEMLINE Cached: 599596 kB
> 2HPMEMLINE SwapCached: 236 kB
> 2HPMEMLINE Active: 552968 kB
> 2HPMEMLINE Inact_dirty: 344020 kB
> 2HPMEMLINE Inact_clean: 50304 kB
> 2HPMEMLINE Inact_target: 189456 kB
> 2HPMEMLINE HighTotal: 130880 kB
> 2HPMEMLINE HighFree: 1024 kB
> 2HPMEMLINE LowTotal: 900004 kB
> 2HPMEMLINE LowFree: 38472 kB
> 2HPMEMLINE SwapTotal: 1028152 kB
> 2HPMEMLINE SwapFree: 1027244 kB
> 2HPMEMLINE Committed_AS: 1067972 kB
> NULL
> 1HPUSERLIMITS User Limits (in bytes except for NOFILE and NPROC) -
> NULL -----------
> 2HPUSERLIMIT RLIMIT_FSIZE : infinity
> 2HPUSERLIMIT RLIMIT_DATA : infinity
> 2HPUSERLIMIT RLIMIT_STACK : 2093056
> 2HPUSERLIMIT RLIMIT_CORE : 0
> 2HPUSERLIMIT RLIMIT_NOFILE : 1024
> 2HPUSERLIMIT RLIMIT_NPROC : 7168
> NULL
> 1HPSIGHANDLERS JVM Signal Handlers
> NULL -------------------
> 2HPSIGHANDLER HUP : unknown handler
> 2HPSIGHANDLER INT : unknown handler
> 2HPSIGHANDLER QUIT : unknown handler
> 2HPSIGHANDLER ILL : unknown handler
> 2HPSIGHANDLER TRAP : unknown handler
> 2HPSIGHANDLER ABRT : unknown handler
> 2HPSIGHANDLER FPE : unknown handler
> 2HPSIGHANDLER KILL : default handler
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org