You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by keeplearning <p_...@yahoo.com> on 2009/09/09 06:12:30 UTC
very high CPU and load...
I am running a load test with 2 web (apache) and 2 appservers (tomcat). I am
noticing that after 250 users, I start seeing very high CPU usage and very
high load. Below is the o/p from top command. I also saw this message in
catalina.out: "SEVERE: All threads (250) are currently busy, waiting.
Increase maxThreads (250) or check the servlet status"
top - 00:31:21 up 138 days, 7:44, 0 users, load average: 20.68, 14.47,
6.99
Tasks: 88 total, 2 running, 86 sleeping, 0 stopped, 0 zombie
Cpu(s): 95.1%us, 2.5%sy, 0.0%ni, 0.9%id, 0.0%wa, 0.1%hi, 1.4%si,
0.0%st
Mem: 16439088k total, 3260004k used, 13179084k free, 187148k buffers
Swap: 2031608k total, 0k used, 2031608k free, 1528756k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4771 tomcat 16 0 3119m 1.3g 25m S 396 8.2 41:56.61 java
1 root 18 0 10308 660 552 S 0 0.0 0:01.54 init
2 root RT 0 0 0 0 S 0 0.0 0:00.07 migration/0
3 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0
4 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/0
5 root RT 0 0 0 0 S 0 0.0 0:00.05 migration/1
6 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/1
7 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/1
--
View this message in context: http://www.nabble.com/very-high-CPU-and-load...-tp25358279p25358279.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: very high CPU and load...
Posted by Peter Crowther <pe...@melandra.com>.
2009/9/9 keeplearning <p_...@yahoo.com>
> With what you said, shouldn't 2 tomcat servers handle 500 connections (250
> *2)?
>
Your two httpds may not load-balance perfectly - so it's quite possible that
both will direct traffic to the same Tomcat server. So, that one Tomcat
server may have to handle all the traffic.
- Peter
Re: very high CPU and load...
Posted by Rainer Jung <ra...@kippdata.de>.
On 09.09.2009 19:44, keeplearning wrote:
>
> I have apache 2.2.3. Not sure how to find the tomcat version. We use
> mod_proxy_ajp connector. During the test, I see high CPU usage and load on
> both appservers. So, I don't think it's the issue of everything going to one
> appserver.
>
> With what i read around, increasing maxClients does not help much. Ofcourse
> I will try that but not that hopeful.
I don't know which is your platform, bot e.g. on Unix/Linux type
platforms you can use ps with additional flags (e.g. -L) to list CPU
usage per thread. That way you can first identify, whether your high cpu
usage comes from one special thread or is distributed over lots of
threads. Then you can do a couple of thread dumps (kill -QUIT), which go
to catalina.out, and check, which of the threads were the ones, that had
the high cpu load (and what they were doing).
Depending on the platform you might have to convert the decimal thread
number from ps to a hex number to compare with the tid or nid in the
thread dumps.
If you are using the tcnative connector, then make sure to use the
latest version. It's the only peace in Tomcat I remember that produced
high cpu load as a result of a bug. It's much more likely though, that
the reason lies somewhere within your application.
Regards,
Rainer
> Tim Funk-2 wrote:
>>
>> At this point, no one on the list will have a clue since we don't know
>> 1) apache version
>> 2) tomcat version
>> 3) type of connector used, mod_proxy_ajp, mod_jk, or mod_proxy_http
>>
>> But if tomcat is saying All threads (250) are currently busy, waiting. -
>> that means it is only configured to handle 250 workers.
>>
>> -Tim
>>
>> keeplearning wrote:
>>> Where do I see # of apache workers and tomcat workers in the config file?
>>> With what you said, shouldn't 2 tomcat servers handle 500 connections
>>> (250
>>> *2)?
>>>
>>> Thanks a lot
>>> Prakash
>>>
>>>
>>>
>>> funkman wrote:
>>>> you have a config issue. I bet you have 250 apache workers (per server)
>>>> and 250 tomcat workers(per server).
>>>>
>>>> But there are 500 apache workers (250 * 2). So in the worst case - you
>>>> need tomcat to handle 500 connections.
>>>>
>>>> -Tim
>>>>
>>>> keeplearning wrote:
>>>>> I am running a load test with 2 web (apache) and 2 appservers (tomcat).
>>>>> I
>>>>> am
>>>>> noticing that after 250 users, I start seeing very high CPU usage and
>>>>> very
>>>>> high load. Below is the o/p from top command. I also saw this message
>>>>> in
>>>>> catalina.out: "SEVERE: All threads (250) are currently busy, waiting.
>>>>> Increase maxThreads (250) or check the servlet status"
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: very high CPU and load...
Posted by keeplearning <p_...@yahoo.com>.
I have apache 2.2.3. Not sure how to find the tomcat version. We use
mod_proxy_ajp connector. During the test, I see high CPU usage and load on
both appservers. So, I don't think it's the issue of everything going to one
appserver.
With what i read around, increasing maxClients does not help much. Ofcourse
I will try that but not that hopeful.
Tim Funk-2 wrote:
>
> At this point, no one on the list will have a clue since we don't know
> 1) apache version
> 2) tomcat version
> 3) type of connector used, mod_proxy_ajp, mod_jk, or mod_proxy_http
>
> But if tomcat is saying All threads (250) are currently busy, waiting. -
> that means it is only configured to handle 250 workers.
>
> -Tim
>
> keeplearning wrote:
>> Where do I see # of apache workers and tomcat workers in the config file?
>> With what you said, shouldn't 2 tomcat servers handle 500 connections
>> (250
>> *2)?
>>
>> Thanks a lot
>> Prakash
>>
>>
>>
>> funkman wrote:
>>> you have a config issue. I bet you have 250 apache workers (per server)
>>> and 250 tomcat workers(per server).
>>>
>>> But there are 500 apache workers (250 * 2). So in the worst case - you
>>> need tomcat to handle 500 connections.
>>>
>>> -Tim
>>>
>>> keeplearning wrote:
>>>> I am running a load test with 2 web (apache) and 2 appservers (tomcat).
>>>> I
>>>> am
>>>> noticing that after 250 users, I start seeing very high CPU usage and
>>>> very
>>>> high load. Below is the o/p from top command. I also saw this message
>>>> in
>>>> catalina.out: "SEVERE: All threads (250) are currently busy, waiting.
>>>> Increase maxThreads (250) or check the servlet status"
>>>>
>>>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
--
View this message in context: http://www.nabble.com/very-high-CPU-and-load...-tp25358279p25369994.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: very high CPU and load...
Posted by Tim Funk <fu...@apache.org>.
At this point, no one on the list will have a clue since we don't know
1) apache version
2) tomcat version
3) type of connector used, mod_proxy_ajp, mod_jk, or mod_proxy_http
But if tomcat is saying All threads (250) are currently busy, waiting. -
that means it is only configured to handle 250 workers.
-Tim
keeplearning wrote:
> Where do I see # of apache workers and tomcat workers in the config file?
> With what you said, shouldn't 2 tomcat servers handle 500 connections (250
> *2)?
>
> Thanks a lot
> Prakash
>
>
>
> funkman wrote:
>> you have a config issue. I bet you have 250 apache workers (per server)
>> and 250 tomcat workers(per server).
>>
>> But there are 500 apache workers (250 * 2). So in the worst case - you
>> need tomcat to handle 500 connections.
>>
>> -Tim
>>
>> keeplearning wrote:
>>> I am running a load test with 2 web (apache) and 2 appservers (tomcat). I
>>> am
>>> noticing that after 250 users, I start seeing very high CPU usage and
>>> very
>>> high load. Below is the o/p from top command. I also saw this message in
>>> catalina.out: "SEVERE: All threads (250) are currently busy, waiting.
>>> Increase maxThreads (250) or check the servlet status"
>>>
>>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: very high CPU and load...
Posted by keeplearning <p_...@yahoo.com>.
Where do I see # of apache workers and tomcat workers in the config file?
With what you said, shouldn't 2 tomcat servers handle 500 connections (250
*2)?
Thanks a lot
Prakash
funkman wrote:
>
> you have a config issue. I bet you have 250 apache workers (per server)
> and 250 tomcat workers(per server).
>
> But there are 500 apache workers (250 * 2). So in the worst case - you
> need tomcat to handle 500 connections.
>
> -Tim
>
> keeplearning wrote:
>> I am running a load test with 2 web (apache) and 2 appservers (tomcat). I
>> am
>> noticing that after 250 users, I start seeing very high CPU usage and
>> very
>> high load. Below is the o/p from top command. I also saw this message in
>> catalina.out: "SEVERE: All threads (250) are currently busy, waiting.
>> Increase maxThreads (250) or check the servlet status"
>>
>>
>>
>> top - 00:31:21 up 138 days, 7:44, 0 users, load average: 20.68, 14.47,
>> 6.99
>> Tasks: 88 total, 2 running, 86 sleeping, 0 stopped, 0 zombie
>> Cpu(s): 95.1%us, 2.5%sy, 0.0%ni, 0.9%id, 0.0%wa, 0.1%hi, 1.4%si,
>> 0.0%st
>> Mem: 16439088k total, 3260004k used, 13179084k free, 187148k buffers
>> Swap: 2031608k total, 0k used, 2031608k free, 1528756k cached
>>
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>> 4771 tomcat 16 0 3119m 1.3g 25m S 396 8.2 41:56.61 java
>> 1 root 18 0 10308 660 552 S 0 0.0 0:01.54 init
>> 2 root RT 0 0 0 0 S 0 0.0 0:00.07 migration/0
>> 3 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0
>> 4 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/0
>> 5 root RT 0 0 0 0 S 0 0.0 0:00.05 migration/1
>> 6 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/1
>> 7 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/1
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
--
View this message in context: http://www.nabble.com/very-high-CPU-and-load...-tp25358279p25366405.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: very high CPU and load...
Posted by Tim Funk <fu...@apache.org>.
you have a config issue. I bet you have 250 apache workers (per server)
and 250 tomcat workers(per server).
But there are 500 apache workers (250 * 2). So in the worst case - you
need tomcat to handle 500 connections.
-Tim
keeplearning wrote:
> I am running a load test with 2 web (apache) and 2 appservers (tomcat). I am
> noticing that after 250 users, I start seeing very high CPU usage and very
> high load. Below is the o/p from top command. I also saw this message in
> catalina.out: "SEVERE: All threads (250) are currently busy, waiting.
> Increase maxThreads (250) or check the servlet status"
>
>
>
> top - 00:31:21 up 138 days, 7:44, 0 users, load average: 20.68, 14.47,
> 6.99
> Tasks: 88 total, 2 running, 86 sleeping, 0 stopped, 0 zombie
> Cpu(s): 95.1%us, 2.5%sy, 0.0%ni, 0.9%id, 0.0%wa, 0.1%hi, 1.4%si,
> 0.0%st
> Mem: 16439088k total, 3260004k used, 13179084k free, 187148k buffers
> Swap: 2031608k total, 0k used, 2031608k free, 1528756k cached
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 4771 tomcat 16 0 3119m 1.3g 25m S 396 8.2 41:56.61 java
> 1 root 18 0 10308 660 552 S 0 0.0 0:01.54 init
> 2 root RT 0 0 0 0 S 0 0.0 0:00.07 migration/0
> 3 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0
> 4 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/0
> 5 root RT 0 0 0 0 S 0 0.0 0:00.05 migration/1
> 6 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/1
> 7 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/1
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: very high CPU and load...
Posted by André Warnier <aw...@ice-sa.com>.
keeplearning wrote:
> I am running a load test with 2 web (apache) and 2 appservers (tomcat). I am
> noticing that after 250 users, I start seeing very high CPU usage and very
> high load. Below is the o/p from top command. I also saw this message in
> catalina.out: "SEVERE: All threads (250) are currently busy, waiting.
> Increase maxThreads (250) or check the servlet status"
>
Well, yes.
The last message above is pretty explicit.
What is it that is mysterious ?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org