You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by "Smith, Mitchell" <mi...@cwc.com> on 2014/10/22 02:06:56 UTC
Excessive threads crashing tomcat
Hi
I have an issue where my applicaion servers have recently started failing
with OOM - cannot create native thread errors. However the dump files all
show he majority of threads (over 90%) like the following:
"pool-21088-thread-1" - Thread t@36407
>
> java.lang.Thread.State: WAITING
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <2bc1d4f0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> Locked ownable synchronizers:
> - None
>
>
>
My first assumption is that these are for a connection pool, however I am
seeing close to 1000 (one thousand) and these never appear to timeout, or
close.
I could of course increase the number of threads the OS allows, but this of
course has memory implications etc and is not really solving the root cause.
Can anyone identify what these are? I see some posts online indicating
these are the connection pool waiting for new requests. But this doesnt
seem to match my configuation for acceptable connections
Any help would be greatly appreciated
Thanks
*Mitchell *
Cable & Wireless Communications Plc is a company registered in England & Wales with number:
07130199 and offices located at 3rd Floor, 26 Red Lion Square, London WC1R 4HQ
Re: Excessive threads crashing tomcat
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Mitchell,
On 10/21/14 8:08 PM, Smith, Mitchell wrote:
> Tomcat7 Java1.7_025
Thanks for adding the version numbers.
How are you configuring Tomcat's thread pool? Either through one or
more <Connector>s or with one or more <Executor>s. Can you post the
(sanitized) configurations for any of those that are in
CATALINA_BASE/conf/server.xml?
> On 21 October 2014 20:06, Smith, Mitchell <mi...@cwc.com>
> wrote:
>
>> Hi
>>
>> I have an issue where my applicaion servers have recently
>> started failing with OOM - cannot create native thread errors.
That's the important part: "cannot create native thread". So, you
haven't run out of memory, you have run out of threads, file
descriptors, or stack space.
The solution is to either reduce the number of threads in your JVM or
to modify your process's limits to allow more resources to be used.
The decision of which action to take depends upon your requirements...
>> However the dump files all show he majority of threads (over
>> 90%) like the following:
>>
>> "pool-21088-thread-1" - Thread t@36407
>>>
>>> java.lang.Thread.State: WAITING at sun.misc.Unsafe.park(Native
>>> Method) - parking to wait for <2bc1d4f0> (a
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>
>>>
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>>>
>>>
at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
>>>
>>>
at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>>>
>>>
at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>>>
>>>
at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>>>
>>>
at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>
>>>
at java.lang.Thread.run(Thread.java:724)
>>> Locked ownable synchronizers: - None
>>>
>>>
>>>
>> My first assumption is that these are for a connection pool,
>> however I am seeing close to 1000 (one thousand) and these never
>> appear to timeout, or close.
Yes, these are Tomcat's request-processor threads. 1000 seems like a
lot. The default is 250 per connector so you must have modified your
configuration.
They never "time-out" or close because they are waiting for work to be
done (e.g. for more requests to be handled).
If you have many many idle worker threads, perhaps you should reduce
the size of your thread pool.
>> I could of course increase the number of threads the OS allows,
>> but this of course has memory implications etc and is not really
>> solving the root cause.
The real question is whether or not you need all those threads at all.
Are you finding that you have peak-load that your server can't handle?
If not, try scaling-back the number of threads you actually use.
Do you have any instrumentation in production to see how much traffic
your server is actually handling? Do you know how many requests/min or
req/sec you are handling? How about max concurrent requests during a
certain period of time?
>> Can anyone identify what these are? I see some posts online
>> indicating these are the connection pool waiting for new
>> requests. But this doesnt seem to match my configuation for
>> acceptable connections
Post your configuration, and we'll take a look.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
iQIcBAEBCAAGBQJUR7ZFAAoJEBzwKT+lPKRYDJwP/R1wWgYUf94H3xHkl/F67LNh
VQkceESfamZ9ADNo1jr9oSIyKmHfcy1St7ih8+izdfJSX8tDwbzN/QnFBIoTDrDM
j8JkODHR0VKRa1Wyt34kPpwThx8ABRLIfh4jA/8+XDKv/57UPcOAGRpc6YzuaXNW
h0IS82qoLsgjLrSFzH/cUGToFKVN1H2RhnMrWCRWNkcqosKIGdddf3VwNxE2gW9F
D7vGmxfVUMff88y5craIENmrw7v1q8mQxhQCS0qeUFEzgQA8AVHlE54rk5sGbqXB
ItKaMlpkQEllaz5h7FGpcSmA5jrHhy1SiC0gOAad21kmqgXOaiJdMtd91mDBqylE
zxd5XGeh8TLSCeKFjRFox326TsovkHxtSonneWOTkTIb1qqpFGeFbdNDjpSMNnlC
Q3atlo4hkEgh+QxCH+ntuCzG3ziXWyQuWuvqF4VyHxa9oX+8SkWRaOoqmcADyZbm
bHSIcUHBpQ5UihK3j2ViChGF+DKvTMbFXUIRKjGbjS7ey6krkRXM9tJKGDlYWMVn
JKDcvmE6aSFpXTehxFRAvPmaNNegneimEPM/QECtw1cAYQmqjUWwugHdM2Gb6rzH
OZltocdoUYLRFBIIUwV9yVdJXhlmeFl2s/RvM31kr+ujoKA+zC5DqtUv5b08t9kM
DAkZI5/Xtdo+fZ7xjFpV
=m7XY
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Excessive threads crashing tomcat
Posted by "Smith, Mitchell" <mi...@cwc.com>.
Tomcat7
Java1.7_025
On 21 October 2014 20:06, Smith, Mitchell <mi...@cwc.com> wrote:
> Hi
>
> I have an issue where my applicaion servers have recently started failing
> with OOM - cannot create native thread errors. However the dump files all
> show he majority of threads (over 90%) like the following:
>
> "pool-21088-thread-1" - Thread t@36407
>>
>> java.lang.Thread.State: WAITING
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for <2bc1d4f0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:724)
>> Locked ownable synchronizers:
>> - None
>>
>>
>>
> My first assumption is that these are for a connection pool, however I am
> seeing close to 1000 (one thousand) and these never appear to timeout, or
> close.
>
>
> I could of course increase the number of threads the OS allows, but this
> of course has memory implications etc and is not really solving the root
> cause.
>
> Can anyone identify what these are? I see some posts online indicating
> these are the connection pool waiting for new requests. But this doesnt
> seem to match my configuation for acceptable connections
>
> Any help would be greatly appreciated
>
> Thanks
>
> *Mitchell *
>
>
Cable & Wireless Communications Plc is a company registered in England & Wales with number:
07130199 and offices located at 3rd Floor, 26 Red Lion Square, London WC1R 4HQ