You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Matthew Laird <la...@sfu.ca> on 2008/10/16 22:30:21 UTC
Tomcat not using multiple cores
We're pulling our hair out with a Tomcat issue.
We have an in-house application running on Tomcat 5.5 with Sun JDK 1.6.
The machine is an x86 dual-CPU, quad core (8 cores total) with 16GB of
RAM. We're running OpenSuSE 10.2, 32-bit. Java memory size set to 2GB,
multi-threaded GC enabled.
What occurs is when a user clicks a certain kind of analysis on the
website, data is retrieved from a database and then a lot of formatting
is done before returning it to the user. This typically causes 100% CPU
usage for this thread for a few minutes (bioinformatics application,
that part isn't going to change).
Unfortunately what then occurs is all other threads suddenly become
unusably slow. The entire web application grinds to a halt until this
thread that's running hot completes.
Looking at top, it appears that these threads aren't spreading among all
the cores. I see one core go to 100% usage, and the others stay at 100%
idle. So we're running multi-thread, but because everything is staying
on the same core, we're still getting thread contention that's bringing
the entire application to its knees.
The only time I began to see the other cores actually start being used
is when I enabled multi-threaded GC. But that doesn't give much
improvement since the threads responding the web requests are still all
on the same core.
I'm not sure how to convince the Tomcat/Java container to spread its
threads among the cores.
Thanks.
--
Matthew Laird
Lead Software Developer, Bioinformatics
Brinkman Laboratory, MBB Dept.
Simon Fraser University
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat not using multiple cores
Posted by Brantley Hobbs <sb...@uga.edu>.
This will be a helpful document:
http://java.sun.com/docs/hotspot/gc5.0/ergo5.html
Brantley
Matthew Laird wrote:
> We're pulling our hair out with a Tomcat issue.
>
> We have an in-house application running on Tomcat 5.5 with Sun JDK 1.6.
> The machine is an x86 dual-CPU, quad core (8 cores total) with 16GB of
> RAM. We're running OpenSuSE 10.2, 32-bit. Java memory size set to 2GB,
> multi-threaded GC enabled.
>
> What occurs is when a user clicks a certain kind of analysis on the
> website, data is retrieved from a database and then a lot of formatting
> is done before returning it to the user. This typically causes 100% CPU
> usage for this thread for a few minutes (bioinformatics application,
> that part isn't going to change).
>
> Unfortunately what then occurs is all other threads suddenly become
> unusably slow. The entire web application grinds to a halt until this
> thread that's running hot completes.
>
> Looking at top, it appears that these threads aren't spreading among all
> the cores. I see one core go to 100% usage, and the others stay at 100%
> idle. So we're running multi-thread, but because everything is staying
> on the same core, we're still getting thread contention that's bringing
> the entire application to its knees.
>
> The only time I began to see the other cores actually start being used
> is when I enabled multi-threaded GC. But that doesn't give much
> improvement since the threads responding the web requests are still all
> on the same core.
>
> I'm not sure how to convince the Tomcat/Java container to spread its
> threads among the cores.
>
> Thanks.
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat not using multiple cores
Posted by Leon Rosenberg <ro...@googlemail.com>.
On Fri, Oct 17, 2008 at 3:05 AM, Matthew Laird <la...@sfu.ca> wrote:
>
> Caldarale, Charles R wrote:
>
>> "The only time I began to see the other cores actually start being used is
>> when I enabled multi-threaded GC. But that doesn't give much improvement
>> since the threads responding the web requests are still all on the same
>> core."
>>
>> The most likely cause is internal synchronization in the webapp or the
>> database it references.
>
> I've heard similar from someone else but I'm not sure how that's possible.
> The app is not at all threaded, is 100% read-only from the database (aside
> from creation of temp tables which have no interaction between client
> requests) and the MySQL server is on another machine humming along without
> any bottlenecks that I can see.
And how many connections to the database do you use and which kind of
connections?
Leon
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat not using multiple cores
Posted by János Löbb <ja...@yale.edu>.
On Oct 16, 2008, at 9:05 PM, Matthew Laird wrote:
>
> Caldarale, Charles R wrote:
>
>> "The only time I began to see the other cores actually start being
>> used is when I enabled multi-threaded GC. But that doesn't give
>> much improvement since the threads responding the web requests are
>> still all on the same core."
>> The most likely cause is internal synchronization in the webapp or
>> the database it references.
>
> I've heard similar from someone else but I'm not sure how that's
> possible. The app is not at all threaded, is 100% read-only from
> the database (aside from creation of temp tables which have no
> interaction between client requests) and the MySQL server is on
> another machine humming along without any bottlenecks that I can see.
>
> But of course I'm not a Java or JVM expert and have no idea what
> kind of interlinks can exist between different client connections.
> But to my knowledge they're all pretty straight forward handlers,
> "get data from database, format, return to user"
>
> I'm leaning more towards GC issues. I setup Tomcat on a 64-bit
> machine and tried a few configurations:
>
> -Xms4096M -Xmx4096M -server -XX:+DisableExplicitGC -XX:
> +UseConcMarkSweepGC
>
> -Xms4096M -Xmx4096M -server
>
> -Xms2048M -Xmx2048M -server
>
> As I went through each of those the app became more and more
> sluggish and a single core finally in the last configuration did
> this 100% CPU usage again.
>
> Now, two issues I see. First, it's not being aggressive enough at
> spreading the load among different cores. Second, GC does seem to
> be an issue.
>
> Unless I'm missing something, which I might be.
>
> Thanks.
>
Make sure IP6 is not enabled if You are an IP4 network. We had
problems like You describe because of it.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat not using multiple cores
Posted by Brantley Hobbs <sb...@uga.edu>.
Matt,
Chuck is right. You want to get real information. Try the verbose GC
output flag on the JVM and see what that says. If you /don't/ see a
bunch of major collections in the output while that task is running,
then GC is not your problem.
B.
Caldarale, Charles R wrote:
>> From: Matthew Laird [mailto:lairdm@sfu.ca]
>> Subject: Re: Tomcat not using multiple cores
>>
>> But of course I'm not a Java or JVM expert and have no idea
>> what kind of interlinks can exist between different client
>> connections.
>>
>
> Again - get a thread dump when at 100% on one core and 0% on the others so you will have real information rather than speculation.
>
> - Chuck
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat not using multiple cores
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Matthew Laird [mailto:lairdm@sfu.ca]
> Subject: Re: Tomcat not using multiple cores
>
> But of course I'm not a Java or JVM expert and have no idea
> what kind of interlinks can exist between different client
> connections.
Again - get a thread dump when at 100% on one core and 0% on the others so you will have real information rather than speculation.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat not using multiple cores
Posted by Matthew Laird <la...@sfu.ca>.
Caldarale, Charles R wrote:
> "The only time I began to see the other cores actually start being used is when I enabled multi-threaded GC. But that doesn't give much improvement since the threads responding the web requests are still all on the same core."
>
> The most likely cause is internal synchronization in the webapp or the database it references.
I've heard similar from someone else but I'm not sure how that's
possible. The app is not at all threaded, is 100% read-only from the
database (aside from creation of temp tables which have no interaction
between client requests) and the MySQL server is on another machine
humming along without any bottlenecks that I can see.
But of course I'm not a Java or JVM expert and have no idea what kind of
interlinks can exist between different client connections. But to my
knowledge they're all pretty straight forward handlers, "get data from
database, format, return to user"
I'm leaning more towards GC issues. I setup Tomcat on a 64-bit machine
and tried a few configurations:
-Xms4096M -Xmx4096M -server -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC
-Xms4096M -Xmx4096M -server
-Xms2048M -Xmx2048M -server
As I went through each of those the app became more and more sluggish
and a single core finally in the last configuration did this 100% CPU
usage again.
Now, two issues I see. First, it's not being aggressive enough at
spreading the load among different cores. Second, GC does seem to be an
issue.
Unless I'm missing something, which I might be.
Thanks.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat not using multiple cores
Posted by Brantley Hobbs <sb...@uga.edu>.
This is the exact thing that makes me think that that's the problem. I
saw the exact same symptoms: No matter how many CPU's I threw at it and
no matter how I threaded my app, I was only consuming one CPU in total.
I know it sounds counter-intuitive, and I could easily be wrong (I'm no
JVM memory expert), but this is almost verbatim the exact same problem I
had, and it was solved with GC tuning.
I'm no gambler, but if I were I'd be willing bet that when he enables GC
verbose output that he'll see more and more time going to major collections.
Brantley
Caldarale, Charles R wrote:
>> From: Brantley Hobbs [mailto:sbhobbs@uga.edu]
>> Subject: Re: Tomcat not using multiple cores
>>
>> I think that this is a garbage collection issue.
>>
>
> Extremely unlikely, since the OP has already stated:
>
> "The only time I began to see the other cores actually start being used is when I enabled multi-threaded GC. But that doesn't give much improvement since the threads responding the web requests are still all on the same core."
>
> The most likely cause is internal synchronization in the webapp or the database it references.
>
> - Chuck
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat not using multiple cores
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Brantley Hobbs [mailto:sbhobbs@uga.edu]
> Subject: Re: Tomcat not using multiple cores
>
> I think that this is a garbage collection issue.
Extremely unlikely, since the OP has already stated:
"The only time I began to see the other cores actually start being used is when I enabled multi-threaded GC. But that doesn't give much improvement since the threads responding the web requests are still all on the same core."
The most likely cause is internal synchronization in the webapp or the database it references.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat not using multiple cores
Posted by Brantley Hobbs <sb...@uga.edu>.
I think that this is a garbage collection issue.
Enable the garbage collection output to see if that's the case (the
"-verbose:gc" flag) . I struggled with this myself just a month ago and
ended up learning more about Java GC than I ever wanted to know. The
gist of it is that when the JVM runs out of heap memory and does a major
collection, it will pause all threads while it's collecting. It doesn't
matter what the priority of the thread is or whether or not you are
using a parallel GC. As you approach heap memory limits, the GC is
working its tail off trying to recover memory wherever it can, pausing
all JVM activity as necessary. I have a feeling that you'll see more
and more GC calls as the app grinds to a halt.
A couple of points:
Disabling GC hints can sometimes dramatically increase performance if
you're not actually running out of memory. Don't assume that just
because *your* code isn't requesting a collection that a third-party
library isn't.
As someone pointed out elsewhere, you're going to want to switch to a 64
bit OS and JRE at some point, if you want to take full advantage of all
that RAM.
In case you're curious, these are the JRE flags that we ended up running
with:
-XX:+UseParallelGC
-XX:+AggressiveOpts
-XX:+DisableExplicitGC
-XX:+UseAdaptiveSizePolicy
-XX:MaxPermSize=512m
(Obviously the MaxPermSize flag is dependent on how your applications
allocate memory.)
A profiler will be helpful too. I've had some success using the one
built-in to Netbeans.
Good Luck!
Brantley
Matthew Laird wrote:
> We're pulling our hair out with a Tomcat issue.
>
> We have an in-house application running on Tomcat 5.5 with Sun JDK 1.6.
> The machine is an x86 dual-CPU, quad core (8 cores total) with 16GB of
> RAM. We're running OpenSuSE 10.2, 32-bit. Java memory size set to 2GB,
> multi-threaded GC enabled.
>
> What occurs is when a user clicks a certain kind of analysis on the
> website, data is retrieved from a database and then a lot of formatting
> is done before returning it to the user. This typically causes 100% CPU
> usage for this thread for a few minutes (bioinformatics application,
> that part isn't going to change).
>
> Unfortunately what then occurs is all other threads suddenly become
> unusably slow. The entire web application grinds to a halt until this
> thread that's running hot completes.
>
> Looking at top, it appears that these threads aren't spreading among all
> the cores. I see one core go to 100% usage, and the others stay at 100%
> idle. So we're running multi-thread, but because everything is staying
> on the same core, we're still getting thread contention that's bringing
> the entire application to its knees.
>
> The only time I began to see the other cores actually start being used
> is when I enabled multi-threaded GC. But that doesn't give much
> improvement since the threads responding the web requests are still all
> on the same core.
>
> I'm not sure how to convince the Tomcat/Java container to spread its
> threads among the cores.
>
> Thanks.
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat not using multiple cores
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Matthew Laird [mailto:lairdm@sfu.ca]
> Subject: Re: Tomcat not using multiple cores
>
> From Tomcat, as far as I understand you can only do 2GB per Tomcat
> instance. Please correct me if I'm wrong.
Not true. The limitation is due to the OS, not the JVM and certainly not Tomcat. A 32-bit OS gives each process only 2 or 3 GB of virtual space to work with, and this space must hold the Java heap, code, non-Java data space, and myriad resource buffers of varying types.
A 64-bit OS provides a much larger virtual space for each process, so you can configure huge heap spaces for Java programs, including Tomcat.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat not using multiple cores
Posted by Matthew Laird <la...@sfu.ca>.
From the OS, no.
From Tomcat, as far as I understand you can only do 2GB per Tomcat
instance. Please correct me if I'm wrong.
Jim Cox wrote:
> On Thu, Oct 16, 2008 at 10:30 PM, Matthew Laird <la...@sfu.ca> wrote:
> [...lines snipped...]
>> We have an in-house application running on Tomcat 5.5 with Sun JDK 1.6.
>> The machine is an x86 dual-CPU, quad core (8 cores total) with 16GB of
>> RAM. We're running OpenSuSE 10.2, 32-bit. Java memory size set to 2GB,
>> multi-threaded GC enabled.
> [...rest of post snipped...]
>
> Apologizing in advance for straying a off-topic, but have you had any issues
> seeing the full 16GB with a 32-bit Linux install?
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat not using multiple cores
Posted by Jim Cox <sh...@gmail.com>.
On Thu, Oct 16, 2008 at 10:30 PM, Matthew Laird <la...@sfu.ca> wrote:
[...lines snipped...]
> We have an in-house application running on Tomcat 5.5 with Sun JDK 1.6.
> The machine is an x86 dual-CPU, quad core (8 cores total) with 16GB of
> RAM. We're running OpenSuSE 10.2, 32-bit. Java memory size set to 2GB,
> multi-threaded GC enabled.
[...rest of post snipped...]
Apologizing in advance for straying a off-topic, but have you had any issues
seeing the full 16GB with a 32-bit Linux install?
Re: Tomcat not using multiple cores
Posted by Leon Rosenberg <ro...@googlemail.com>.
Hello Matthew,
Without wanting to advertise own product, I think moskito can help you
a lot here. It was specially designed to detect
cpu eaters among call trees. Feel free to contact me of list if you
want to give it a try ;-)
http://moskito.anotheria.net/moskitodemo/mui/mskShowAllProducers
regards
Leon
On Thu, Oct 16, 2008 at 10:30 PM, Matthew Laird <la...@sfu.ca> wrote:
> We're pulling our hair out with a Tomcat issue.
>
> We have an in-house application running on Tomcat 5.5 with Sun JDK 1.6.
> The machine is an x86 dual-CPU, quad core (8 cores total) with 16GB of
> RAM. We're running OpenSuSE 10.2, 32-bit. Java memory size set to 2GB,
> multi-threaded GC enabled.
>
> What occurs is when a user clicks a certain kind of analysis on the
> website, data is retrieved from a database and then a lot of formatting
> is done before returning it to the user. This typically causes 100% CPU
> usage for this thread for a few minutes (bioinformatics application,
> that part isn't going to change).
>
> Unfortunately what then occurs is all other threads suddenly become
> unusably slow. The entire web application grinds to a halt until this
> thread that's running hot completes.
>
> Looking at top, it appears that these threads aren't spreading among all
> the cores. I see one core go to 100% usage, and the others stay at 100%
> idle. So we're running multi-thread, but because everything is staying
> on the same core, we're still getting thread contention that's bringing
> the entire application to its knees.
>
> The only time I began to see the other cores actually start being used
> is when I enabled multi-threaded GC. But that doesn't give much
> improvement since the threads responding the web requests are still all
> on the same core.
>
> I'm not sure how to convince the Tomcat/Java container to spread its
> threads among the cores.
>
> Thanks.
>
> --
> Matthew Laird
> Lead Software Developer, Bioinformatics
> Brinkman Laboratory, MBB Dept.
> Simon Fraser University
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat not using multiple cores
Posted by Dan Armbrust <da...@gmail.com>.
My guesses:
Your application has bad sync locking between some threads, which is
causing one thread to block the others.
Your application has bad database access code, which allows one thread
to block others while waiting on the DB.
Your application is disk bound, and the single long process is hashing
the disk drives so much that is has dragged down the rest of the OS,
making it very slow to respond to the IO request of the other, small
tasks.
You haven't configured the GC well - however, I would expect this to
cause sporadic problems, rather than continuous ones.
There are numerous JVM tools out there to help you with all of these
issues - from the simple garbage collection monitor that comes with
java (JConsole http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html)
to full blown tools like JProfiler (which last I checked, had a free
trial)
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat not using multiple cores
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Matthew Laird [mailto:lairdm@sfu.ca]
> Subject: Tomcat not using multiple cores
>
> Unfortunately what then occurs is all other threads suddenly become
> unusably slow. The entire web application grinds to a halt until this
> thread that's running hot completes.
We've run Tomcat on 32 cores and can keep them all busy - the problem is almost certainly in your webapp. Take a thread dump (with locking information enabled) and see what everyone is waiting for. The jstack utility is good for this.
> I'm not sure how to convince the Tomcat/Java container to spread its
> threads among the cores.
Convince your application; it's not Tomcat or the JVM.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org