You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by li...@dankeeley.co.uk on 2007/11/08 16:41:27 UTC
Tomcat - threads / throughput limits?
Hi,
We have an architecture where we have many tomcat app servers load
balanced by apache at the front.
In resolving our current bottleneck i used JProfiler to see what the
tomcat applications were doing and when under high load there are a lot of
threads which are blocked on this:
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run()
This suggests that they are spending all their time waiting for tomcat to
run them, either way they're not runnable anyway.
Our app pushes through huge quantites of very quick requests - typically
each request takes a total of 50ms. We have a limit of 300 connections on
our AJP connector.
The point of blocked threads occurs way before we hit that limit. At the
same time neither the machine is busy, nor are there any obvious problems
with memory usage for the instance.
And indeed; Adding more app servers keeps the load to each one down and
does resolve the problem - i.e. we can increase throughput.
So; My question is, is there a reccomended limit to how many threads
tomcat can handle before it spends more time managing threads than it does
doing real work?
Thanks!
Dan
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat - threads / throughput limits?
Posted by David kerber <dc...@verizon.net>.
It doesn't answer your question in general, but I increased mine from
300 to 900 threads on a dual-processor, dual-core machine, and saw
significantly increased throughput with short, simple requests. I
didn't try any in-between settings, though, so I don't know if there
might be a sweet spot somewhere between those settings, and I don't know
if going higher would help or not, because the setting it's at right now
handles the full load with no waiting.
D
lists@dankeeley.co.uk wrote:
> Hi,
>
> We have an architecture where we have many tomcat app servers load
> balanced by apache at the front.
>
> In resolving our current bottleneck i used JProfiler to see what the
> tomcat applications were doing and when under high load there are a lot of
> threads which are blocked on this:
>
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run()
>
> This suggests that they are spending all their time waiting for tomcat to
> run them, either way they're not runnable anyway.
>
> Our app pushes through huge quantites of very quick requests - typically
> each request takes a total of 50ms. We have a limit of 300 connections on
> our AJP connector.
>
> The point of blocked threads occurs way before we hit that limit. At the
> same time neither the machine is busy, nor are there any obvious problems
> with memory usage for the instance.
>
> And indeed; Adding more app servers keeps the load to each one down and
> does resolve the problem - i.e. we can increase throughput.
>
> So; My question is, is there a reccomended limit to how many threads
> tomcat can handle before it spends more time managing threads than it does
> doing real work?
>
> Thanks!
> Dan
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat - threads / throughput limits?
Posted by li...@dankeeley.co.uk.
>
> Have you tried running your profiler against a tomcat not doing ANY
> requests?
>
> I agree with Jim on this one - These are threads WAITING on a
> connection...
> and not doing anything.
>
> Have a look in tomcat manager, and then you can see how many 'free'
> connections
> you have.
>
>
We only see the blocked threads when the instance is under heavy load.
When the instances is "clean" and newly started most threads are idel.
when it's under medium load you see a few are processing and the vast
majority of the rest are all in "net i/o"
Then under very heavy load you start to see the blocking...
Well thats what we see anyway!
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat - threads / throughput limits?
Posted by Andrew Miehs <an...@2sheds.de>.
On 08/11/2007, at 4:51 PM, Jim Cox wrote:
> On Nov 8, 2007 10:41 AM, <li...@dankeeley.co.uk> wrote:
>> In resolving our current bottleneck i used JProfiler to see what the
>> tomcat applications were doing and when under high load there are a
>> lot of
>> threads which are blocked on this:
>>
>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run()
>>
>> This suggests that they are spending all their time waiting for
>> tomcat to
>> run them, either way they're not runnable anyway.
>>
>
> "http80-Processor131" daemon prio=5 tid=0x00baa7c0 nid=0x10f9 in
> Object.wait() [0xa5e81000..0xa5e819c8]
> at java.lang.Object.wait(Native Method)
> - waiting on <0xc9d99d20> (a
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
>
> FWIW, I always interpreted those as HTTP servicing threads waiting for
> something to do (i.e. not a bad thing at all).
Have you tried running your profiler against a tomcat not doing ANY
requests?
I agree with Jim on this one - These are threads WAITING on a
connection...
and not doing anything.
Have a look in tomcat manager, and then you can see how many 'free'
connections
you have.
Cheers
Andrew
Re: Tomcat - threads / throughput limits?
Posted by Jim Cox <sh...@gmail.com>.
On Nov 8, 2007 10:41 AM, <li...@dankeeley.co.uk> wrote:
> Hi,
>
> We have an architecture where we have many tomcat app servers load
> balanced by apache at the front.
>
> In resolving our current bottleneck i used JProfiler to see what the
> tomcat applications were doing and when under high load there are a lot of
> threads which are blocked on this:
>
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run()
>
> This suggests that they are spending all their time waiting for tomcat to
> run them, either way they're not runnable anyway.
>
> Our app pushes through huge quantites of very quick requests - typically
> each request takes a total of 50ms. We have a limit of 300 connections on
> our AJP connector.
>
> The point of blocked threads occurs way before we hit that limit. At the
> same time neither the machine is busy, nor are there any obvious problems
> with memory usage for the instance.
>
> And indeed; Adding more app servers keeps the load to each one down and
> does resolve the problem - i.e. we can increase throughput.
>
> So; My question is, is there a reccomended limit to how many threads
> tomcat can handle before it spends more time managing threads than it does
> doing real work?
>
> Thanks!
> Dan
Not quite answering your question, but re your "blocked" threads, do
you mean something like the following in a stack dump?
"http80-Processor131" daemon prio=5 tid=0x00baa7c0 nid=0x10f9 in
Object.wait() [0xa5e81000..0xa5e819c8]
at java.lang.Object.wait(Native Method)
- waiting on <0xc9d99d20> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:429)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:660)
- locked <0xc9d99d20> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:534)
FWIW, I always interpreted those as HTTP servicing threads waiting for
something to do (i.e. not a bad thing at all).
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org