You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Mark Stevens <ma...@googlemail.com> on 2007/07/24 12:32:21 UTC
Tomcat consuming entire CPU.
Hi all,
Sorry if this has been covered already in other posts,
I'm fishing for some tips to debug an intermittent issue with Tomcat load.
My JRE:java version "1.3.1_14"
The server is running Oracle, Tomcat, Apache 2.2 and a Content
Management System, Operating system is Solaris 8 running on sun4u Sun
Fire V240 with 4GB of RAM.
Every so often the java process for Tomcat consumes an entire CPU, and
requires Tomcat to be killed, and then restarted, this is happened
quiet load with only a couple of HTTP requests coming through every
second.
I'm wondering if this down to code, or just not having enough free
RAM, Xms and Xmx are both set to 1024M, I've heard it's not good to
set Xms too high.
Anyhow, can anyone suggest a good, and easy to use tool to profile my
webapps memory and CPU usage, I've had a quick look at J-Profiler,
but was wondering if there were any other favorites amongst the group
users.
Thanks in advance.
Mark.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat consuming entire CPU.
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Mark Stevens [mailto:mark.stevens99@googlemail.com]
> Subject: Re: Tomcat consuming entire CPU.
>
> Every time I capture the thread using much higher percentage CPU than
> all other threads, it's pointing to '
>
> "VM Thread" prio=5 tid=0xb12f8 nid=0x4 runnable
This is a high-priority background thread that is mostly an internal JVM
operations dispatcher. It's not a Java thread, which is why there's no
stack trace. I think (not positive) that it's also the thread that
prints the thread dump, so it's always going to appear as runnable in
that output.
Although you stated the heap was not full, I wonder if the PermGen is at
its maximum? If you run JConsole on your live Tomcat, go to the Memory
tab, and look at the set of non-heap columns on the bottom right. If
any of these are approaching 100%, that could be the problem.
> when trussing the main Tomcat process it's performing
> hundreds/thousands of
>
> stat64("<jar file including path>", 0xDBE7EDA8) = 0 against the Jar
> files in the directory for each jar contained in the webapp I'm
> troubleshooting.
Sounds like you still have some development-mode configuration settings
in place. Check for <WatchedResource> settings in your <Context>
elements.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat consuming entire CPU.
Posted by Mark Stevens <ma...@googlemail.com>.
Hi again,
I think I've located the Thread that is consuming the CPU, although
I'm none the wiser.
Every time I capture the thread using much higher percentage CPU than
all other threads, it's pointing to '
"VM Thread" prio=5 tid=0xb12f8 nid=0x4 runnable
With no stack trace associated to it, is this common?
when trussing the main Tomcat process it's performing hundreds/thousands of
stat64("<jar file including path>", 0xDBE7EDA8) = 0 against the Jar
files in the directory for each jar contained in the webapp I'm
troubleshooting.
I'm generating the load by sending loads of requests for a single
page, it seems like a lot of stat64's for the same thing that I would
assume should be cached in Tomcat unless specified otherwise?
Anyone seen this type of behaviour?
Many thanks.
Mark.
On 25/07/07, Mark Stevens <ma...@googlemail.com> wrote:
> Thanks very much all for the advice,
>
> I've managed to get my thread dumps nicely formatted side by side in
> HTML using TDAN (Thread Dump Analyzer)
>
> I'll work out how to then get thread ID's using 'ps' and then should
> be in business.
>
> I'm also going to speak with DBA about limiting Oracles(Database)
> memory, and get Tomcat moved from was into RAM, I think this will help
> general performance, but isn't the root cause.
>
>
>
>
> Thanks again,
>
> Mark.
>
>
>
>
> On 25/07/07, Juha Laiho <Ju...@iki.fi> wrote:
> > Mark Stevens wrote:
> > > I don't think we are using sendfile, to be honest, I've had no
> > > involvement in development of the code, I'm just requested to look
> > > after the server.
> > >
> > > I'm going to try and work out how to read thread dumps, hopefully this
> > > will help, I'm finding my X11 connection too slow for J-Profiler to be
> > > usable.
> >
> > Ok, some more details on what to look for (pretty much already
> > said by Charles in an earlier message).
> >
> > When you encounter the CPU hogging situation, run (several times, if
> > needed) such form of "ps" command that shows data on the individual
> > threads. This'll give you the thread ID of the CPU-hungry thread (most
> > possibly there is just one thread in this state), when you look at the
> > CPU time used by thread - one is incrementing, others are not.
> >
> > Then run "kill -QUIT <tomcat_process_id>" to get the thread dump in
> > catalina.out logfile.
> >
> > When you have the thread dump, you'll at least get a name of the thread
> > and the Java method name being executed in the thread which is eating
> > your CPU. This gives, if nothing else, a possibility to see whether
> > the thread is looping in Java library code, application code, or Tomcat
> > server code. A caveat on the data formats; at least on Linux platforms
> > the thread ids from "ps" are in decimal, whereas in Java thread dump
> > they are in hexadecimal, so you'll need to do a dec->hex conversion to
> > the thread id from ps before searching in the thread dump.
> >
> > > Something else I noticed in my Live envrionment is that Oracle is
> > > hogging all the memory and tomcat is being forced into swap.
> >
> > Oracle as in database - or application server? If database, then you'll
> > need to get your DBA people to re-tune Oracle to leave some memory for
> > others as well. If you're the DBA as well, see Oracle documentation on
> > tuning SGA and db buffer cache sizes.
> > --
> > ..Juha
> >
> > ---------------------------------------------------------------------
> > To start a new topic, e-mail: users@tomcat.apache.org
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> >
> >
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat consuming entire CPU.
Posted by Mark Stevens <ma...@googlemail.com>.
Thanks very much all for the advice,
I've managed to get my thread dumps nicely formatted side by side in
HTML using TDAN (Thread Dump Analyzer)
I'll work out how to then get thread ID's using 'ps' and then should
be in business.
I'm also going to speak with DBA about limiting Oracles(Database)
memory, and get Tomcat moved from was into RAM, I think this will help
general performance, but isn't the root cause.
Thanks again,
Mark.
On 25/07/07, Juha Laiho <Ju...@iki.fi> wrote:
> Mark Stevens wrote:
> > I don't think we are using sendfile, to be honest, I've had no
> > involvement in development of the code, I'm just requested to look
> > after the server.
> >
> > I'm going to try and work out how to read thread dumps, hopefully this
> > will help, I'm finding my X11 connection too slow for J-Profiler to be
> > usable.
>
> Ok, some more details on what to look for (pretty much already
> said by Charles in an earlier message).
>
> When you encounter the CPU hogging situation, run (several times, if
> needed) such form of "ps" command that shows data on the individual
> threads. This'll give you the thread ID of the CPU-hungry thread (most
> possibly there is just one thread in this state), when you look at the
> CPU time used by thread - one is incrementing, others are not.
>
> Then run "kill -QUIT <tomcat_process_id>" to get the thread dump in
> catalina.out logfile.
>
> When you have the thread dump, you'll at least get a name of the thread
> and the Java method name being executed in the thread which is eating
> your CPU. This gives, if nothing else, a possibility to see whether
> the thread is looping in Java library code, application code, or Tomcat
> server code. A caveat on the data formats; at least on Linux platforms
> the thread ids from "ps" are in decimal, whereas in Java thread dump
> they are in hexadecimal, so you'll need to do a dec->hex conversion to
> the thread id from ps before searching in the thread dump.
>
> > Something else I noticed in my Live envrionment is that Oracle is
> > hogging all the memory and tomcat is being forced into swap.
>
> Oracle as in database - or application server? If database, then you'll
> need to get your DBA people to re-tune Oracle to leave some memory for
> others as well. If you're the DBA as well, see Oracle documentation on
> tuning SGA and db buffer cache sizes.
> --
> ..Juha
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat consuming entire CPU.
Posted by Juha Laiho <Ju...@iki.fi>.
Mark Stevens wrote:
> I don't think we are using sendfile, to be honest, I've had no
> involvement in development of the code, I'm just requested to look
> after the server.
>
> I'm going to try and work out how to read thread dumps, hopefully this
> will help, I'm finding my X11 connection too slow for J-Profiler to be
> usable.
Ok, some more details on what to look for (pretty much already
said by Charles in an earlier message).
When you encounter the CPU hogging situation, run (several times, if
needed) such form of "ps" command that shows data on the individual
threads. This'll give you the thread ID of the CPU-hungry thread (most
possibly there is just one thread in this state), when you look at the
CPU time used by thread - one is incrementing, others are not.
Then run "kill -QUIT <tomcat_process_id>" to get the thread dump in
catalina.out logfile.
When you have the thread dump, you'll at least get a name of the thread
and the Java method name being executed in the thread which is eating
your CPU. This gives, if nothing else, a possibility to see whether
the thread is looping in Java library code, application code, or Tomcat
server code. A caveat on the data formats; at least on Linux platforms
the thread ids from "ps" are in decimal, whereas in Java thread dump
they are in hexadecimal, so you'll need to do a dec->hex conversion to
the thread id from ps before searching in the thread dump.
> Something else I noticed in my Live envrionment is that Oracle is
> hogging all the memory and tomcat is being forced into swap.
Oracle as in database - or application server? If database, then you'll
need to get your DBA people to re-tune Oracle to leave some memory for
others as well. If you're the DBA as well, see Oracle documentation on
tuning SGA and db buffer cache sizes.
--
..Juha
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat consuming entire CPU.
Posted by Mark Stevens <ma...@googlemail.com>.
Hi, and thanks for responses,
I don't think we are using sendfile, to be honest, I've had no
involvement in development of the code, I'm just requested to look
after the server.
I'm going to try and work out how to read thread dumps, hopefully this
will help, I'm finding my X11 connection too slow for J-Profiler to be
usable.
I apologise for not being to Savvy with my environment versions I get
a bit confused with all the different Java environments like
JDK/JRE/SDK/J2SE/J2EE etc....
The version of java executed by Tomcat that consumes CPU is ....
/usr/java/bin/java -version
java version "1.3.1_14"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.1_14-b03)
Java HotSpot(TM) Client VM (build 1.3.1_14-b03, mixed mode)
Tomcat is version 4.1
I'll see if moving version's to 1.3.1_20 changes things.
I can reproduce similar behavior to live environment on dev consuming
an entire CPU by running Apache Benchmark to simulate 20 concurrent
sessions requesting the homepage 10000 times.....
./ab -c20 -n 10000 http://<url><tomcat port>/
Something else I noticed in my Live envrionment is that Oracle is
hogging all the memory and tomcat is being forced into swap.
I'll crack on with investigations and post to group if I find anything
worth sharing.
Many thanks again,
Mark.
On 24/07/07, Caldarale, Charles R <Ch...@unisys.com> wrote:
> > From: Mark Stevens [mailto:mark.stevens99@googlemail.com]
> > Subject: Tomcat consuming entire CPU.
> >
> > My JRE:java version "1.3.1_14"
>
> You might want to try a JRE that is a bit more up to date than that.
> The 1.3.1 leg is up to _20 and even that is unsupported on everything
> other than Solaris 8 (falls into the "Vintage Support Period"). Do you
> have the required Solaris patches installed? Also, shouldn't that be a
> JDK, not a JRE?
>
> > The server is running Oracle, Tomcat, Apache 2.2 and a Content
> > Management System, Operating system is Solaris 8 running on sun4u Sun
> > Fire V240 with 4GB of RAM.
>
> What Tomcat version? Current ones require a 1.4 or 1.5 JRE.
>
> > Every so often the java process for Tomcat consumes an entire CPU, and
> > requires Tomcat to be killed, and then restarted, this is happened
> > quiet load with only a couple of HTTP requests coming through every
> > second.
>
> What does a thread dump show?
>
> > I've heard it's not good to set Xms too high.
>
> You can't set it larger than whatever the maximum contiguous virtual
> space for a process is, but that's the only consideration. Regardless,
> that's unlikely to be your problem.
>
> - Chuck
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> MATERIAL and is thus for use only by the intended recipient. If you
> received this in error, please contact the sender and delete the e-mail
> and its attachments from all computers.
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat consuming entire CPU.
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Mark Stevens [mailto:mark.stevens99@googlemail.com]
> Subject: Tomcat consuming entire CPU.
>
> My JRE:java version "1.3.1_14"
You might want to try a JRE that is a bit more up to date than that.
The 1.3.1 leg is up to _20 and even that is unsupported on everything
other than Solaris 8 (falls into the "Vintage Support Period"). Do you
have the required Solaris patches installed? Also, shouldn't that be a
JDK, not a JRE?
> The server is running Oracle, Tomcat, Apache 2.2 and a Content
> Management System, Operating system is Solaris 8 running on sun4u Sun
> Fire V240 with 4GB of RAM.
What Tomcat version? Current ones require a 1.4 or 1.5 JRE.
> Every so often the java process for Tomcat consumes an entire CPU, and
> requires Tomcat to be killed, and then restarted, this is happened
> quiet load with only a couple of HTTP requests coming through every
> second.
What does a thread dump show?
> I've heard it's not good to set Xms too high.
You can't set it larger than whatever the maximum contiguous virtual
space for a process is, but that's the only consideration. Regardless,
that's unlikely to be your problem.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat consuming entire CPU.
Posted by Chris Lear <ch...@laculine.com>.
* Mark Stevens wrote (24/07/07 11:32):
> Hi all,
>
> Sorry if this has been covered already in other posts,
>
> I'm fishing for some tips to debug an intermittent issue with Tomcat load.
>
> My JRE:java version "1.3.1_14"
>
> The server is running Oracle, Tomcat, Apache 2.2 and a Content
> Management System, Operating system is Solaris 8 running on sun4u Sun
> Fire V240 with 4GB of RAM.
>
>
> Every so often the java process for Tomcat consumes an entire CPU, and
> requires Tomcat to be killed, and then restarted, this is happened
> quiet load with only a couple of HTTP requests coming through every
> second.
>
> I'm wondering if this down to code, or just not having enough free
> RAM, Xms and Xmx are both set to 1024M, I've heard it's not good to
> set Xms too high.
>
>
> Anyhow, can anyone suggest a good, and easy to use tool to profile my
> webapps memory and CPU usage, I've had a quick look at J-Profiler,
> but was wondering if there were any other favorites amongst the group
> users.
I experienced these exact symptoms and reported it here:
http://issues.apache.org/bugzilla/show_bug.cgi?id=42925.
Are you using sendfile (default when using Apache Portable Runtime)?
Removing that was the workaround for me, and it's still working fine.
However, my setup is different, so it's possible that this is a red
herring, but I might as well mention it.
You could also have a look at this thread with some different problems
that might be more Solaris-related:
http://mail-archives.apache.org/mod_mbox/tomcat-users/200609.mbox/%3c45118136.4030709@jouy.inra.fr%3e
For debugging, I found that getting thread dumps (kill -3) worked fairly
well, though it took me quite a while to work out how to use them. I
could use ps or top to find that one thread was the problem, and get the
thread id, then cross-reference that to the thread dump to find which
thread it was. It was always sendfile, which uses native code, so I just
removed it from the config. Problem not solved, but there you are.
Chris
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org