You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Greg Huber <gr...@gmail.com> on 2015/05/03 12:25:53 UTC
High cpu on Tomcat 8
Hello,
After an upgrade to Tomcat 8.0.21 and (Oracle jdk1.8.0_40) I seem to be
having an erratic high cpu issue, often when the server gets busy. The
application was OK tomcat 7 and has not been modified since the upgrade.
I use mod_jk / apache
#
# workers.properties
#
# Define 1 real worker using ajp13
worker.list=worker1
# Set properties for worker1 (ajp13)
worker.worker1.type=ajp13
worker.worker1.host=localhost
worker.worker1.port=8009
worker.worker1.lbfactor=50
worker.worker1.socket_keepalive=1
Here are my startup options:
Tomcat 7
JAVA_OPTS="-Xms128M -Xmx512m -XX:MaxPermSize=256m"
Tomcat 8 (java 8 does not support MaxPermSize)
JAVA_OPTS="-Xms128M -Xmx512m"
If I trace the thread it seems to be related to ajp-apr-8009-Poller
"ajp-apr-8009-Poller" #26 daemon prio=5 os_prio=0 tid=0x00007ffe300bd000
nid=0xc82 runnable [0x00007ffdd1fd1000]
java.lang.Thread.State: RUNNABLE
at sun.misc.Unsafe.unpark(Native Method)
at java.util.concurrent.locks.LockSupport.unpark(LockSupport.java:141)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.unparkSuccessor(AbstractQueuedSynchronizer.java:662)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1264)
at
java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:457)
at
java.util.concurrent.LinkedBlockingQueue.signalNotEmpty(LinkedBlockingQueue.java:176)
at
java.util.concurrent.LinkedBlockingQueue.offer(LinkedBlockingQueue.java:430)
at org.apache.tomcat.util.threads.TaskQueue.offer(TaskQueue.java:74)
at org.apache.tomcat.util.threads.TaskQueue.offer(TaskQueue.java:31)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1361)
at
org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:161)
at
org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:141)
at
org.apache.tomcat.util.net.AprEndpoint.processSocket(AprEndpoint.java:896)
at org.apache.tomcat.util.net.AprEndpoint$Poller.null (Redefined)
at java.lang.Thread.run(Redefined)
....
"ajp-apr-8009-Poller" #26 daemon prio=5 os_prio=0 tid=0x00007ffe300bd000
nid=0xc82 runnable [0x00007ffdd1fd1000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e4a05160> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
at
java.util.concurrent.LinkedBlockingQueue.signalNotEmpty(LinkedBlockingQueue.java:172)
at
java.util.concurrent.LinkedBlockingQueue.offer(LinkedBlockingQueue.java:430)
at org.apache.tomcat.util.threads.TaskQueue.offer(TaskQueue.java:74)
at org.apache.tomcat.util.threads.TaskQueue.offer(TaskQueue.java:31)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1361)
at
org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:161)
at
org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:141)
at
org.apache.tomcat.util.net.AprEndpoint.processSocket(AprEndpoint.java:896)
at org.apache.tomcat.util.net.AprEndpoint$Poller.null (Redefined)
at java.lang.Thread.run(Redefined)
Killing the thread stops the cpu, but then Tomcat does not work. Any ideas
what would be causing this?
Cheers Greg
Re: High cpu on Tomcat 8
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Chuck,
On 5/4/15 10:23 AM, Caldarale, Charles R wrote:
>> From: Christopher Schultz [mailto:chris@christopherschultz.net]
>> Subject: Re: High cpu on Tomcat 8
>
>> Car analogy: it's the distributor cap of all the bytes flying
>> around the container.
>
> You're dating yourself :-)
>
> Haven't seen a distributor on a car in many years.
The analogy is still reasonably accurate :)
Wait 'till I explain which part of Tomcat is the Leiden Jar.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
Comment: GPGTools - http://gpgtools.org
iQIcBAEBCAAGBQJVR4WkAAoJEBzwKT+lPKRYZDwP/2e99U6sxIRWsI+HdXp09yMz
uNrvoBeH31KpbZcn5SfIbXcxppBSU0rKrDWyqbDLezVVso4c3b+4FnS0VRVtMzwU
0+jID3BBpi/iIy//x3JkUjewTIuFisj4AI+LBD3Z6D3W9kMITmgWKeUSJ2ZcEVpY
GiIIf78hfhz8uMLGO82LGbnWCjyRzuDMpMxUl5iIBtJd/GWUhaJ0p8t1dfT11LhG
44enEDqiZYoYzw2Cv8qY7T/a0EMLcBktjOBkzwK0y1CICiBIgefqsCc+oEYfRV5f
A6Xa6BTIf3MG9s77tndO3vQNNKjIv+tsgQemk9rigH6eraPP8nZ4wvP1eILTgfS3
hr8Zi9xQBhI2QWsj65t/qtU1d461VicUdywwAK/enQXxd9G57cjM7gI/Eei8zv3Z
Hbk4fUWpBurMSNmFWMrc7KYFg58CiL5IMOERoDuOYA2+wjjglrZODGNVLpdBillQ
FI17Hl/XRQJ9Xpk8oq5TykEQ6SPOdPrYJc3jqqo1j0Gzw8NjAYVaBLm+r+jRYew1
5Uiv9UmI9QhNUiDGJu8EbmkTTykKj0Preh/F97lEIg9B8clOepNnOnEHE6hmQ5rU
gKwF3h1CZ6Y5lBcenAcTzDpib3wSZ4fSZNbC9T+1f/iPMCAS1k/wc4gk3442xB2M
ValgdrHXYwAS+iUnPjts
=mVFU
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: High cpu on Tomcat 8
Posted by André Warnier <aw...@ice-sa.com>.
Caldarale, Charles R wrote:
>> From: Christopher Schultz [mailto:chris@christopherschultz.net]
>> Subject: Re: High cpu on Tomcat 8
>
>> Car analogy: it's the distributor cap of all the bytes flying around
>> the container.
>
> You're dating yourself :-)
>
> Haven't seen a distributor on a car in many years.
>
Hey, my car has one.
Which probably dates me too, and my car..
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: High cpu on Tomcat 8
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Christopher Schultz [mailto:chris@christopherschultz.net]
> Subject: Re: High cpu on Tomcat 8
> Car analogy: it's the distributor cap of all the bytes flying around
> the container.
You're dating yourself :-)
Haven't seen a distributor on a car in many years.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: High cpu on Tomcat 8
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Greg,
On 5/3/15 2:30 PM, Greg Huber wrote:
> Thanks for the reply, I will up the memory on the heap space and
> have another go with the profiler if happens again. When I was
> looking at the thread dumps there were no other active threads
> other than the ajp-apr-8009-Poller so maybe it is only a memory
> issue.
No, the Poller thread can be quite active, but you will only see it in
a few configurations because it mostly blocks on select(), and then
notifies other threads that their I/O work is done.
Car analogy: it's the distributor cap of all the bytes flying around
the container.
- -chris
> On 3 May 2015 at 17:35, Felix Schumacher
> <fe...@internetallee.de> wrote:
>
>>
>>
>> Am 3. Mai 2015 12:25:53 MESZ, schrieb Greg Huber
>> <gr...@gmail.com>:
>>> Hello,
>>>
>>> After an upgrade to Tomcat 8.0.21 and (Oracle jdk1.8.0_40) I
>>> seem to be having an erratic high cpu issue, often when the
>>> server gets busy. The application was OK tomcat 7 and has not
>>> been modified since the upgrade.
>>>
>>> I use mod_jk / apache
>>>
>>> # # workers.properties #
>>>
>>> # Define 1 real worker using ajp13 worker.list=worker1 # Set
>>> properties for worker1 (ajp13) worker.worker1.type=ajp13
>>> worker.worker1.host=localhost worker.worker1.port=8009
>>> worker.worker1.lbfactor=50 worker.worker1.socket_keepalive=1
>>>
>>> Here are my startup options:
>>>
>>> Tomcat 7 JAVA_OPTS="-Xms128M -Xmx512m -XX:MaxPermSize=256m"
>>>
>>> Tomcat 8 (java 8 does not support MaxPermSize)
>>>
>>> JAVA_OPTS="-Xms128M -Xmx512m"
>>
>> I believe java 8 combines the permgen into the heap space, so it
>> is possible, that you run out of space now that you use java 8.
>>
>> Use jstat, jvisualvm or jconsole to look at your gc cycles. They
>> can consume a lot of cpu.
>>
>>>
>>> If I trace the thread it seems to be related to
>>> ajp-apr-8009-Poller
>>>
>>> "ajp-apr-8009-Poller" #26 daemon prio=5 os_prio=0
>>> tid=0x00007ffe300bd000 nid=0xc82 runnable [0x00007ffdd1fd1000]
>>> java.lang.Thread.State: RUNNABLE at
>>> sun.misc.Unsafe.unpark(Native Method)
>>
>> This thread does nothing.
>>
>>
>>> at
>>> java.util.concurrent.locks.LockSupport.unpark(LockSupport.java:141)
>>>
>>>
at
>>
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.unparkSuccesso
r(AbstractQueuedSynchronizer.java:662)
>>>
>>>
at
>>
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.release(Abstra
ctQueuedSynchronizer.java:1264)
>>>
>>>
at
>>> java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:4
57)
>>>
>>>
at
>>
>>> java.util.concurrent.LinkedBlockingQueue.signalNotEmpty(LinkedBlocki
ngQueue.java:176)
>>>
>>>
at
>>
>>> java.util.concurrent.LinkedBlockingQueue.offer(LinkedBlockingQueue.j
ava:430)
>>>
>>>
at org.apache.tomcat.util.threads.TaskQueue.offer(TaskQueue.java:74)
>>> at
>>> org.apache.tomcat.util.threads.TaskQueue.offer(TaskQueue.java:31)
>>>
>>>
at
>>
>>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.j
ava:1361)
>>>
>>>
at
>>
>>> org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(ThreadPool
Executor.java:161)
>>>
>>>
at
>>
>>> org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(ThreadPool
Executor.java:141)
>>>
>>>
at
>>> org.apache.tomcat.util.net.AprEndpoint.processSocket(AprEndpoint.jav
a:896)
>>>
>>>
at org.apache.tomcat.util.net.AprEndpoint$Poller.null (Redefined)
>>> at java.lang.Thread.run(Redefined)
>>>
>>> ....
>>>
>>> "ajp-apr-8009-Poller" #26 daemon prio=5 os_prio=0
>>> tid=0x00007ffe300bd000 nid=0xc82 runnable [0x00007ffdd1fd1000]
>>> java.lang.Thread.State: WAITING (parking) at
>>> sun.misc.Unsafe.park(Native Method)
>>
>> This thread does nothing, either.
>>
>>> - parking to wait for <0x00000000e4a05160> (a
>>> java.util.concurrent.locks.ReentrantLock$NonfairSync) at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>
>>>
at
>>
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckIn
terrupt(AbstractQueuedSynchronizer.java:836)
>>>
>>>
at
>>
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(
AbstractQueuedSynchronizer.java:870)
>>>
>>>
at
>>
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Abstra
ctQueuedSynchronizer.java:1199)
>>>
>>>
at
>>
>>> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantL
ock.java:209)
>>>
>>>
at
>>> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285
)
>>>
>>>
at
>>
>>> java.util.concurrent.LinkedBlockingQueue.signalNotEmpty(LinkedBlocki
ngQueue.java:172)
>>>
>>>
at
>>
>>> java.util.concurrent.LinkedBlockingQueue.offer(LinkedBlockingQueue.j
ava:430)
>>>
>>>
at org.apache.tomcat.util.threads.TaskQueue.offer(TaskQueue.java:74)
>>> at
>>> org.apache.tomcat.util.threads.TaskQueue.offer(TaskQueue.java:31)
>>>
>>>
at
>>
>>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.j
ava:1361)
>>>
>>>
at
>>
>>> org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(ThreadPool
Executor.java:161)
>>>
>>>
at
>>
>>> org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(ThreadPool
Executor.java:141)
>>>
>>>
at
>>> org.apache.tomcat.util.net.AprEndpoint.processSocket(AprEndpoint.jav
a:896)
>>>
>>>
at org.apache.tomcat.util.net.AprEndpoint$Poller.null (Redefined)
>>> at java.lang.Thread.run(Redefined)
>>>
>>>
>>> Killing the thread stops the cpu, but then Tomcat does not
>>> work. Any ideas what would be causing this?
>>
>> You might look for other threads, that do something. I would be
>> especially suspicious about array resizing operations.
>>
>> Regards Felix
>>
>>>
>>>
>>> Cheers Greg
>>
>>
>> ---------------------------------------------------------------------
>>
>>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
Comment: GPGTools - http://gpgtools.org
iQIcBAEBCAAGBQJVR37uAAoJEBzwKT+lPKRYvS8P/14lHaikJZc2laMKDIZI4JIb
beYklkmTjG2JSJiYmpT624LKHUPH86APRaBbeU/89BWPfmu/fy3q8Ho+RHsaaIij
JppPlPl/A3kKPxkG8ptUws85WAXQpMD9VodhHOiJLJO/Tk7+kCiLTM0cQ+1d8iDE
KDNHS8YWzo0LPTnrvGn0evHWAKptfxmMxPMpDXwckcvIsKJMa3zI7jfcPQ5wQuv2
X6sTjnRp5Gxqaypx5ZY/JiLmS+SIzUdyAP4oVSa3pJ1hU4XJbXDfU9hj4+tjxMOm
N+eGB8M7AtLSLIommBmtJJjTeZHtjbQiOLsHUD9CIIN/OqjU2CHgmynKcWK1+/TD
j3TMQ8kJjGsXD/fqq4diKs8e1G3btTcIkUZWMlMw/cG10sUkono6vsmqJkUZ2XhE
yCmMii+vPSfamX+biitD3noH6EqP39utHVWyeE2+cuBRcfW8H83OEM3TNKzZfdcI
XbwKiPpEEoWJjMjyMyX0dLKLYq+Z7obldJlknQommTaZePhkPf3+Ykuyfbd+ckLi
6CVcH47ED+EaFswHc5Ik8l0/4SCSJkWBTiYKpUSSu9FAIFM1VRywQY6CMv9IhTJm
12+ow0MEtEQXy6yy6kw77jpNm8+xUf4Y+Fpv7O7j1BuJp84VR23znuANnzYjj0uq
0d0+Q08YRHrkYMHfC4ya
=fZkm
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: High cpu on Tomcat 8
Posted by Greg Huber <gr...@gmail.com>.
Thanks for the reply, I will up the memory on the heap space and have
another go with the profiler if happens again. When I was looking at the
thread dumps there were no other active threads other than the
ajp-apr-8009-Poller so maybe it is only a memory issue.
Cheers Greg.
On 3 May 2015 at 17:35, Felix Schumacher <fe...@internetallee.de>
wrote:
>
>
> Am 3. Mai 2015 12:25:53 MESZ, schrieb Greg Huber <gr...@gmail.com>:
> >Hello,
> >
> >After an upgrade to Tomcat 8.0.21 and (Oracle jdk1.8.0_40) I seem to be
> >having an erratic high cpu issue, often when the server gets busy.
> >The
> >application was OK tomcat 7 and has not been modified since the
> >upgrade.
> >
> >I use mod_jk / apache
> >
> >#
> ># workers.properties
> >#
> >
> ># Define 1 real worker using ajp13
> >worker.list=worker1
> ># Set properties for worker1 (ajp13)
> >worker.worker1.type=ajp13
> >worker.worker1.host=localhost
> >worker.worker1.port=8009
> >worker.worker1.lbfactor=50
> >worker.worker1.socket_keepalive=1
> >
> >Here are my startup options:
> >
> >Tomcat 7
> >JAVA_OPTS="-Xms128M -Xmx512m -XX:MaxPermSize=256m"
> >
> >Tomcat 8 (java 8 does not support MaxPermSize)
> >
> >JAVA_OPTS="-Xms128M -Xmx512m"
>
> I believe java 8 combines the permgen into the heap space, so it is
> possible, that you run out of space now that you use java 8.
>
> Use jstat, jvisualvm or jconsole to look at your gc cycles. They can
> consume a lot of cpu.
>
> >
> >If I trace the thread it seems to be related to ajp-apr-8009-Poller
> >
> >"ajp-apr-8009-Poller" #26 daemon prio=5 os_prio=0
> >tid=0x00007ffe300bd000
> >nid=0xc82 runnable [0x00007ffdd1fd1000]
> > java.lang.Thread.State: RUNNABLE
> > at sun.misc.Unsafe.unpark(Native Method)
>
> This thread does nothing.
>
>
> > at java.util.concurrent.locks.LockSupport.unpark(LockSupport.java:141)
> > at
>
> >java.util.concurrent.locks.AbstractQueuedSynchronizer.unparkSuccessor(AbstractQueuedSynchronizer.java:662)
> > at
>
> >java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1264)
> > at
> >java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:457)
> > at
>
> >java.util.concurrent.LinkedBlockingQueue.signalNotEmpty(LinkedBlockingQueue.java:176)
> > at
>
> >java.util.concurrent.LinkedBlockingQueue.offer(LinkedBlockingQueue.java:430)
> > at org.apache.tomcat.util.threads.TaskQueue.offer(TaskQueue.java:74)
> > at org.apache.tomcat.util.threads.TaskQueue.offer(TaskQueue.java:31)
> > at
>
> >java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1361)
> > at
>
> >org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:161)
> > at
>
> >org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:141)
> > at
> >org.apache.tomcat.util.net.AprEndpoint.processSocket(AprEndpoint.java:896)
> > at org.apache.tomcat.util.net.AprEndpoint$Poller.null (Redefined)
> > at java.lang.Thread.run(Redefined)
> >
> >....
> >
> >"ajp-apr-8009-Poller" #26 daemon prio=5 os_prio=0
> >tid=0x00007ffe300bd000
> >nid=0xc82 runnable [0x00007ffdd1fd1000]
> > java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
>
> This thread does nothing, either.
>
> > - parking to wait for <0x00000000e4a05160> (a
> >java.util.concurrent.locks.ReentrantLock$NonfairSync)
> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> > at
>
> >java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> > at
>
> >java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
> > at
>
> >java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
> > at
>
> >java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
> >at
> >java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
> > at
>
> >java.util.concurrent.LinkedBlockingQueue.signalNotEmpty(LinkedBlockingQueue.java:172)
> > at
>
> >java.util.concurrent.LinkedBlockingQueue.offer(LinkedBlockingQueue.java:430)
> > at org.apache.tomcat.util.threads.TaskQueue.offer(TaskQueue.java:74)
> > at org.apache.tomcat.util.threads.TaskQueue.offer(TaskQueue.java:31)
> > at
>
> >java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1361)
> > at
>
> >org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:161)
> > at
>
> >org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:141)
> > at
> >org.apache.tomcat.util.net.AprEndpoint.processSocket(AprEndpoint.java:896)
> > at org.apache.tomcat.util.net.AprEndpoint$Poller.null (Redefined)
> > at java.lang.Thread.run(Redefined)
> >
> >
> >Killing the thread stops the cpu, but then Tomcat does not work. Any
> >ideas
> >what would be causing this?
>
> You might look for other threads, that do something. I would be especially
> suspicious about array resizing operations.
>
> Regards
> Felix
>
> >
> >
> >Cheers Greg
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
RE: High cpu on Tomcat 8
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Felix Schumacher [mailto:felix.schumacher@internetallee.de]
> Subject: Re: High cpu on Tomcat 8
> I believe java 8 combines the permgen into the heap space
Not the whole story. A small part of what was in PermGen is now allocated in the Java heap, but the majority is moved to areas of native memory (Metaspace) for class data. More info here:
http://mail.openjdk.java.net/pipermail/hotspot-dev/2012-September/006679.html
https://blogs.oracle.com/poonam/entry/about_g1_garbage_collector_permanent
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: High cpu on Tomcat 8
Posted by Felix Schumacher <fe...@internetallee.de>.
Am 3. Mai 2015 12:25:53 MESZ, schrieb Greg Huber <gr...@gmail.com>:
>Hello,
>
>After an upgrade to Tomcat 8.0.21 and (Oracle jdk1.8.0_40) I seem to be
>having an erratic high cpu issue, often when the server gets busy.
>The
>application was OK tomcat 7 and has not been modified since the
>upgrade.
>
>I use mod_jk / apache
>
>#
># workers.properties
>#
>
># Define 1 real worker using ajp13
>worker.list=worker1
># Set properties for worker1 (ajp13)
>worker.worker1.type=ajp13
>worker.worker1.host=localhost
>worker.worker1.port=8009
>worker.worker1.lbfactor=50
>worker.worker1.socket_keepalive=1
>
>Here are my startup options:
>
>Tomcat 7
>JAVA_OPTS="-Xms128M -Xmx512m -XX:MaxPermSize=256m"
>
>Tomcat 8 (java 8 does not support MaxPermSize)
>
>JAVA_OPTS="-Xms128M -Xmx512m"
I believe java 8 combines the permgen into the heap space, so it is possible, that you run out of space now that you use java 8.
Use jstat, jvisualvm or jconsole to look at your gc cycles. They can consume a lot of cpu.
>
>If I trace the thread it seems to be related to ajp-apr-8009-Poller
>
>"ajp-apr-8009-Poller" #26 daemon prio=5 os_prio=0
>tid=0x00007ffe300bd000
>nid=0xc82 runnable [0x00007ffdd1fd1000]
> java.lang.Thread.State: RUNNABLE
> at sun.misc.Unsafe.unpark(Native Method)
This thread does nothing.
> at java.util.concurrent.locks.LockSupport.unpark(LockSupport.java:141)
> at
>java.util.concurrent.locks.AbstractQueuedSynchronizer.unparkSuccessor(AbstractQueuedSynchronizer.java:662)
> at
>java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1264)
> at
>java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:457)
> at
>java.util.concurrent.LinkedBlockingQueue.signalNotEmpty(LinkedBlockingQueue.java:176)
> at
>java.util.concurrent.LinkedBlockingQueue.offer(LinkedBlockingQueue.java:430)
> at org.apache.tomcat.util.threads.TaskQueue.offer(TaskQueue.java:74)
> at org.apache.tomcat.util.threads.TaskQueue.offer(TaskQueue.java:31)
> at
>java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1361)
> at
>org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:161)
> at
>org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:141)
> at
>org.apache.tomcat.util.net.AprEndpoint.processSocket(AprEndpoint.java:896)
> at org.apache.tomcat.util.net.AprEndpoint$Poller.null (Redefined)
> at java.lang.Thread.run(Redefined)
>
>....
>
>"ajp-apr-8009-Poller" #26 daemon prio=5 os_prio=0
>tid=0x00007ffe300bd000
>nid=0xc82 runnable [0x00007ffdd1fd1000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
This thread does nothing, either.
> - parking to wait for <0x00000000e4a05160> (a
>java.util.concurrent.locks.ReentrantLock$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
>java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
>java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
> at
>java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
> at
>java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
>at
>java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
> at
>java.util.concurrent.LinkedBlockingQueue.signalNotEmpty(LinkedBlockingQueue.java:172)
> at
>java.util.concurrent.LinkedBlockingQueue.offer(LinkedBlockingQueue.java:430)
> at org.apache.tomcat.util.threads.TaskQueue.offer(TaskQueue.java:74)
> at org.apache.tomcat.util.threads.TaskQueue.offer(TaskQueue.java:31)
> at
>java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1361)
> at
>org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:161)
> at
>org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:141)
> at
>org.apache.tomcat.util.net.AprEndpoint.processSocket(AprEndpoint.java:896)
> at org.apache.tomcat.util.net.AprEndpoint$Poller.null (Redefined)
> at java.lang.Thread.run(Redefined)
>
>
>Killing the thread stops the cpu, but then Tomcat does not work. Any
>ideas
>what would be causing this?
You might look for other threads, that do something. I would be especially suspicious about array resizing operations.
Regards
Felix
>
>
>Cheers Greg
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: High cpu on Tomcat 8
Posted by Greg Huber <gr...@gmail.com>.
> > When the CPU usage goes high, does the server actually slow down?
> I do not think so, it makes the server slowdown
>> Sounds like you're contradicting yourself; you do not think it slows
down, or it does slow down?
It would slow down as the server is at 80%, but if it was the webapp in a
deadlock it would not respond, so it is some other thread/task. From the
profiler the only one active thread was the was the ajp-apr-8009-Poller.
I am hoping the memory is the resolution, maybe I will simulate some high
load (or wait till Friday/Saturday) and see if its resolved.
Cheers Greg
On 5 May 2015 at 12:48, Caldarale, Charles R <Ch...@unisys.com>
wrote:
> > From: Greg Huber [mailto:gregh3269@gmail.com]
> > Subject: Re: High cpu on Tomcat 8
>
> > > Have you set a pollerThreadCount?
>
> > I have had a look and I cannot find where this is set. Is there any
> > documentation on this?
>
> The pollerThreadCount applies only to the HTTP version of the <Connector>,
> not the AJP one. Red herring.
>
> > > When the CPU usage goes high, does the server actually slow down?
>
> > I do not think so, it makes the server slowdown
>
> Sounds like you're contradicting yourself; you do not think it slows down,
> or it does slow down?
>
> - Chuck
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> MATERIAL and is thus for use only by the intended recipient. If you
> received this in error, please contact the sender and delete the e-mail and
> its attachments from all computers.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
RE: High cpu on Tomcat 8
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Greg Huber [mailto:gregh3269@gmail.com]
> Subject: Re: High cpu on Tomcat 8
> > Have you set a pollerThreadCount?
> I have had a look and I cannot find where this is set. Is there any
> documentation on this?
The pollerThreadCount applies only to the HTTP version of the <Connector>, not the AJP one. Red herring.
> > When the CPU usage goes high, does the server actually slow down?
> I do not think so, it makes the server slowdown
Sounds like you're contradicting yourself; you do not think it slows down, or it does slow down?
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: High cpu on Tomcat 8
Posted by Greg Huber <gr...@gmail.com>.
>Have you set a pollerThreadCount? If so, what is it? If not, you might
>want to consider setting it to "2", but probably not any higher, and
>see if it improves things.
I have had a look and I cannot find where this is set. Is there any
documentation on this?
>When the CPU usage goes high, does the server actually slow down?
I do not think so, it makes the server slowdown (and all the fans come on)
Cheers Greg
On 4 May 2015 at 15:13, Christopher Schultz <ch...@christopherschultz.net>
wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Greg,
>
> On 5/4/15 7:13 AM, Greg Huber wrote:
> > Thanks, I am going to up the memory. The profiler I used only
> > highlighted the ajp-apr-8009-Poller as active. Terminating the
> > thread stopped the high cpu.
>
> ... and probably killed your ability to process requests, unless you
> configured more than one Poller thread.
>
> Have you set a pollerThreadCount? If so, what is it? If not, you might
> want to consider setting it to "2", but probably not any higher, and
> see if it improves things.
>
> The Poller thread is responsible for handling all blocking-style I/O
> both into and out of your servlets. When your site gets busy, this
> thread will be doing a lot of work.
>
> When the CPU usage goes high, does the server actually slow down?
>
> - -chris
>
> > On 4 May 2015 at 10:18, Mark Thomas <ma...@apache.org> wrote:
> >
> >> On 03/05/2015 11:25, Greg Huber wrote:
> >>> Hello,
> >>>
> >>> After an upgrade to Tomcat 8.0.21 and (Oracle jdk1.8.0_40) I
> >>> seem to be having an erratic high cpu issue, often when the
> >>> server gets busy. The application was OK tomcat 7 and has not
> >>> been modified since the upgrade.
> >>
> >> Use ps to get the thread ID of the thread that is using the CPU.
> >> Take a thread dump and find what that thread is doing (you'll
> >> need to convert the thread ID from decimal to hex). It is the
> >> stack trace of that thread that will be interesting.
> >>
> >> Mark
> >>
> >>
> >> ---------------------------------------------------------------------
> >>
> >>
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> For additional commands, e-mail: users-help@tomcat.apache.org
> >>
> >>
> >
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2
> Comment: GPGTools - http://gpgtools.org
>
> iQIcBAEBCAAGBQJVR35yAAoJEBzwKT+lPKRYamAQAIYQMdBZLRueevXz71rqJxpA
> Ij1lEpK4FlXrY1hukAKEX0k/yyiLc2UkXeI0DZtstKNiDDyEo+KmsykvjlTjUmAt
> mvyhicQ3zhvlNaLIFYBwUIHNqzx+dBmgF/w75pkxKrDOj7MMx7gIFxPGXlTj2+XH
> 1tt8uWgvHhElKnROjG+jU2bG3/wqZyXfSnT+SsfNhQQE6r0W3MRqJh/0X808GgWO
> bSJdfk2Dz03/OksrEzK9cVV5/f4zB2Ggce/Uw+4qtZ0jj0jhRd9JXdaJlRFpPfbM
> EdjDeOVmsJz6oqP+IvSEvtJjQY9RR6iJB8SkyWph64stxRQeeOBFzUsBIDWLTK+d
> kB4/9HgGpnld8LaDEr3hrY2uXmtjEVwgkVzs1TKVpFipaACePuHG/3aa81/j0mMC
> wP1iLSzt3SrjI2Z0dXlOszcB5DlQIiInqFG3PpTD8Wfr63hjX7m43zEdepamTX7d
> eIjyu+TGX1Z+8yZabQzt+IPqGlk2uozafFiJOyxvwAbfBFqmF+rTKxOnYLMS67U7
> nFx50rXx/Xq1TCCsWbX4L1s0Y7Gh1G3DAtVTCLFKI+O3oW5aSUTed0trwUcE+oEP
> VXYkRvSqDTcxJp+fXszz/yJGJxo3Yy46wfgX4WgGf9FZBdJ8XNchzOTPZp/qlqNa
> WrehBe11KsKgy21Hc+Lz
> =Hooe
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: High cpu on Tomcat 8
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Greg,
On 5/4/15 7:13 AM, Greg Huber wrote:
> Thanks, I am going to up the memory. The profiler I used only
> highlighted the ajp-apr-8009-Poller as active. Terminating the
> thread stopped the high cpu.
... and probably killed your ability to process requests, unless you
configured more than one Poller thread.
Have you set a pollerThreadCount? If so, what is it? If not, you might
want to consider setting it to "2", but probably not any higher, and
see if it improves things.
The Poller thread is responsible for handling all blocking-style I/O
both into and out of your servlets. When your site gets busy, this
thread will be doing a lot of work.
When the CPU usage goes high, does the server actually slow down?
- -chris
> On 4 May 2015 at 10:18, Mark Thomas <ma...@apache.org> wrote:
>
>> On 03/05/2015 11:25, Greg Huber wrote:
>>> Hello,
>>>
>>> After an upgrade to Tomcat 8.0.21 and (Oracle jdk1.8.0_40) I
>>> seem to be having an erratic high cpu issue, often when the
>>> server gets busy. The application was OK tomcat 7 and has not
>>> been modified since the upgrade.
>>
>> Use ps to get the thread ID of the thread that is using the CPU.
>> Take a thread dump and find what that thread is doing (you'll
>> need to convert the thread ID from decimal to hex). It is the
>> stack trace of that thread that will be interesting.
>>
>> Mark
>>
>>
>> ---------------------------------------------------------------------
>>
>>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
Comment: GPGTools - http://gpgtools.org
iQIcBAEBCAAGBQJVR35yAAoJEBzwKT+lPKRYamAQAIYQMdBZLRueevXz71rqJxpA
Ij1lEpK4FlXrY1hukAKEX0k/yyiLc2UkXeI0DZtstKNiDDyEo+KmsykvjlTjUmAt
mvyhicQ3zhvlNaLIFYBwUIHNqzx+dBmgF/w75pkxKrDOj7MMx7gIFxPGXlTj2+XH
1tt8uWgvHhElKnROjG+jU2bG3/wqZyXfSnT+SsfNhQQE6r0W3MRqJh/0X808GgWO
bSJdfk2Dz03/OksrEzK9cVV5/f4zB2Ggce/Uw+4qtZ0jj0jhRd9JXdaJlRFpPfbM
EdjDeOVmsJz6oqP+IvSEvtJjQY9RR6iJB8SkyWph64stxRQeeOBFzUsBIDWLTK+d
kB4/9HgGpnld8LaDEr3hrY2uXmtjEVwgkVzs1TKVpFipaACePuHG/3aa81/j0mMC
wP1iLSzt3SrjI2Z0dXlOszcB5DlQIiInqFG3PpTD8Wfr63hjX7m43zEdepamTX7d
eIjyu+TGX1Z+8yZabQzt+IPqGlk2uozafFiJOyxvwAbfBFqmF+rTKxOnYLMS67U7
nFx50rXx/Xq1TCCsWbX4L1s0Y7Gh1G3DAtVTCLFKI+O3oW5aSUTed0trwUcE+oEP
VXYkRvSqDTcxJp+fXszz/yJGJxo3Yy46wfgX4WgGf9FZBdJ8XNchzOTPZp/qlqNa
WrehBe11KsKgy21Hc+Lz
=Hooe
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: High cpu on Tomcat 8
Posted by Greg Huber <gr...@gmail.com>.
Thanks, I am going to up the memory. The profiler I used only highlighted
the ajp-apr-8009-Poller as active. Terminating the thread stopped the high
cpu.
Cheers Greg
On 4 May 2015 at 10:18, Mark Thomas <ma...@apache.org> wrote:
> On 03/05/2015 11:25, Greg Huber wrote:
> > Hello,
> >
> > After an upgrade to Tomcat 8.0.21 and (Oracle jdk1.8.0_40) I seem to be
> > having an erratic high cpu issue, often when the server gets busy. The
> > application was OK tomcat 7 and has not been modified since the upgrade.
>
> Use ps to get the thread ID of the thread that is using the CPU.
> Take a thread dump and find what that thread is doing (you'll need to
> convert the thread ID from decimal to hex). It is the stack trace of
> that thread that will be interesting.
>
> Mark
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: High cpu on Tomcat 8
Posted by Mark Thomas <ma...@apache.org>.
On 03/05/2015 11:25, Greg Huber wrote:
> Hello,
>
> After an upgrade to Tomcat 8.0.21 and (Oracle jdk1.8.0_40) I seem to be
> having an erratic high cpu issue, often when the server gets busy. The
> application was OK tomcat 7 and has not been modified since the upgrade.
Use ps to get the thread ID of the thread that is using the CPU.
Take a thread dump and find what that thread is doing (you'll need to
convert the thread ID from decimal to hex). It is the stack trace of
that thread that will be interesting.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org