You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by John Minchuk <mi...@gmail.com> on 2011/12/22 00:56:40 UTC
Tomcat AJP Thread Spike and System Hang
Quick overview of our setup. Http requests flow from our load balancers, to
squid proxys, to Apaches, to our Tomcat servers. We migrated to this setup
from an Oracle App Server.
Apache: 2.2.3
Tomcat: 7.0.11.0
JVM: 1.6.0_22-b04
Linux: 2.6.18-194.17.1.el5
Our production environment has max threads set at 200, the number of
threads usually hovers around 150. About twice a day, at seemingly
unrelated times we get a sudden spike in the number of ajp threads open.
Eventually this hits our max of 200. At this point Tomcat still seems
responsive, but the number of our httpd processes spikes until Apache locks
ups. At this point we have monitoring software that kills and restarts
Apache. We then manually restart Tomcat.
Here is a graph of the AJP Threads running. You can see a sudden jump to
200 threads. The other dips are most likely reloads triggered by our
configuration management software (puppet).
http://sporkit.com/thread_spike/spike.jpg
Also interesting to note, these threads (all 200) appear to be in the keep
alive state.
http://sporkit.com/thread_spike/threads.jpg
Our access logs don't indicate a high number of visits, or any one
particular page that might cause this issue (that I can see).
At this point we are stumped. Do we spend our time tracking down memory
leaks? Is there something we could do to at least mitigate the problem
over the holidays? Any input greatly appreciated.
Re: Tomcat AJP Thread Spike and System Hang
Posted by Igor Cicimov <ic...@gmail.com>.
What module are you using on apache side mod_proxy or mod_jk? Also you
might post the Connector settings from tomcat. Sorry typing from my mobile
just think that info might be useful to some one who can help you more.
On Dec 22, 2011 10:57 AM, "John Minchuk" <mi...@gmail.com> wrote:
> Quick overview of our setup. Http requests flow from our load balancers, to
> squid proxys, to Apaches, to our Tomcat servers. We migrated to this setup
> from an Oracle App Server.
>
> Apache: 2.2.3
> Tomcat: 7.0.11.0
> JVM: 1.6.0_22-b04
> Linux: 2.6.18-194.17.1.el5
>
> Our production environment has max threads set at 200, the number of
> threads usually hovers around 150. About twice a day, at seemingly
> unrelated times we get a sudden spike in the number of ajp threads open.
> Eventually this hits our max of 200. At this point Tomcat still seems
> responsive, but the number of our httpd processes spikes until Apache locks
> ups. At this point we have monitoring software that kills and restarts
> Apache. We then manually restart Tomcat.
>
> Here is a graph of the AJP Threads running. You can see a sudden jump to
> 200 threads. The other dips are most likely reloads triggered by our
> configuration management software (puppet).
>
> http://sporkit.com/thread_spike/spike.jpg
>
> Also interesting to note, these threads (all 200) appear to be in the keep
> alive state.
>
> http://sporkit.com/thread_spike/threads.jpg
>
> Our access logs don't indicate a high number of visits, or any one
> particular page that might cause this issue (that I can see).
>
> At this point we are stumped. Do we spend our time tracking down memory
> leaks? Is there something we could do to at least mitigate the problem
> over the holidays? Any input greatly appreciated.
>
Re: Tomcat AJP Thread Spike and System Hang
Posted by ma...@apache.org.
John Minchuk <mi...@gmail.com> wrote:
>Quick overview of our setup. Http request
flow from our load
>balancers, to
>squid proxys, to Apaches, to our Tomcat servers. We migrated to this
>setup
>from an Oracle App Server.
>
>Apache: 2.2.3
>Tomcat: 7.0.11.0
>JVM: 1.6.0_22-b04
>Linux: 2.6.18-194.17.1.el5
>
>Our production environment has max threads set at 200, the number of
>threads usually hovers around 150. About twice a day, at seemingly
>unrelated times we get a sudden spike in the number of ajp threads
>open.
>Eventually this hits our max of 200. At this point Tomcat still seems
>responsive, but the number of our httpd processes spikes until Apache
>locks
>ups. At this point we have monitoring software that kills and restarts
>Apache. We then manually restart Tomcat.
>
>Here is a graph of the AJP Threads running. You can see a sudden jump
>to
>200 threads. The other dips are most likely reloads triggered by our
>configuration management software (puppet).
>
>http://sporkit.com/thread_spike/spike.jpg
>
>Also interesting to note, these threads (all 200) appear to be in the
>keep
>alive state.
>
>http://sporkit.com/thread_spike/threads.jpg
>
>Our access logs don't indicate a high number of visits, or any one
>particular page that might cause this issue (that I can see).
>
>At this point we are stumped. Do we spend our time tracking down
>memory
>leaks? Is there something we could do to at least mitigate the problem
>over the holidays? Any input greatly appreciated.
Without any configuration information it is impossible to be sure what the problem is.
My best guess is that httpd is configured to use persistent connections to Tomcat and Tomcat is using the BIO AJP connector. When connections == maxThreads everthing locks up. A thread dump on Tomcat will confirm this is what is happening.
If this is the case, possible solutions are (in my order of preference)
- disable connection reuse in https.conf (ignore the warnings about performance issues in the docs: the risks are overstated)
- switch to APR/native AJP in Tomcat
- switch to NIO AJP (requires Tomcat upgrade)
You also need to consider some upgrades. The versions you are running are rather old.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org