You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Rich Mayfield <ri...@veeva.com> on 2015/06/30 20:06:46 UTC

Tomcat 8 AprEndpoint Poller thread consuming high CPU

Moved from Tomcat 7.0.55 to Tomcat 8.0.23 and now see the AJP Poller
thread churning through a lot of CPU in a pretty much quiet system.

This seems like a bug.

* Restarting does not resolve the issue
* I've seen some mention of bumping up the memory. Doubling the max
heap (-Xmx from 506m to 1012m) does not help. This is a relatively
small service that is essentially idle - or at least it would be with
the exception of this Poller thread.

The connector configuration is unchanged.

<Connector port="28080" protocol="AJP/1.3"
               secure="true"
               scheme="https"
               connectionTimeout="20000"
               packetSize="20000"
               maxThreads="1024"
               URIEncoding="UTF-8"
               redirectPort="443"
               useSendfile="false"
               compression="force"
               compressableMimeTypes="text/html,text/xml,text/plain,text/javascript,text/css,application/json,application/xml"
/>

The thread consuming all the CPU

"ajp-apr-28080-Poller" #62 daemon prio=5 os_prio=0
tid=0x00002ba40293e000 nid=0x407d waiting on condition
[0x00002ba3fc200000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000eaf94570> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
        at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
        at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
        at java.util.concurrent.LinkedBlockingQueue.signalNotEmpty(LinkedBlockingQueue.java:172)
        at java.util.concurrent.LinkedBlockingQueue.offer(LinkedBlockingQueue.java:430)
        at org.apache.tomcat.util.threads.TaskQueue.offer(TaskQueue.java:74)
        at org.apache.tomcat.util.threads.TaskQueue.offer(TaskQueue.java:31)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1361)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:161)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:141)
        at org.apache.tomcat.util.net.AprEndpoint.processSocket(AprEndpoint.java:896)
        at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1883)
        at java.lang.Thread.run(Thread.java:745)


# rpm -qa | grep apr
apr-1.5.1-1
apr-util-1.5.3-1

# rpm -qa | grep http
httpd-2.2.27-1

Server version: Apache Tomcat/8.0.23
Server built:   May 19 2015 14:58:38 UTC
Server number:  8.0.23.0
OS Name:        Linux
OS Version:     2.6.18-371.el5
Architecture:   amd64
JVM Version:    1.8.0_40-b25
JVM Vendor:     Oracle Corporation

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: Tomcat 8 AprEndpoint Poller thread consuming high CPU

Posted by Konstantin Kolinko <kn...@gmail.com>.
2015-06-30 21:06 GMT+03:00 Rich Mayfield <ri...@veeva.com>:
> Moved from Tomcat 7.0.55 to Tomcat 8.0.23 and now see the AJP Poller
> thread churning through a lot of CPU in a pretty much quiet system.
>
> This seems like a bug.
>
> * Restarting does not resolve the issue

Is it easy to observe the issue? Does it happen soon after a restart,
or you are waiting for a long time?

> * I've seen some mention of bumping up the memory. Doubling the max
> heap (-Xmx from 506m to 1012m) does not help. This is a relatively
> small service that is essentially idle - or at least it would be with
> the exception of this Poller thread.
>
> The connector configuration is unchanged.
>
> <Connector port="28080" protocol="AJP/1.3"

1. Note that the above value of "protocol" means auto-selecting
between APR and BIO implementations.
>From thread name (below) it selected APR implementation provided by
Tomcat-Native library.

You may try choosing another implementation of the protocol.

Version of Tomcat-Native = ?  (It is logged when Tomcat starts)

>                secure="true"
>                scheme="https"
>                connectionTimeout="20000"
>                packetSize="20000"
>                maxThreads="1024"
>                URIEncoding="UTF-8"
>                redirectPort="443"
>                useSendfile="false"
>                compression="force"
>                compressableMimeTypes="text/html,text/xml,text/plain,text/javascript,text/css,application/json,application/xml"
> />
>
> The thread consuming all the CPU

2. How do you know that *this* thread consumes the cpu?
>From the stack trace it seems rather idle.
Does its state (stack trace) change over time?

Otherwise, this looks like a bug.

> "ajp-apr-28080-Poller" #62 daemon prio=5 os_prio=0
> tid=0x00002ba40293e000 nid=0x407d waiting on condition
> [0x00002ba3fc200000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000eaf94570> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
>         at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
>         at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
>         at java.util.concurrent.LinkedBlockingQueue.signalNotEmpty(LinkedBlockingQueue.java:172)
>         at java.util.concurrent.LinkedBlockingQueue.offer(LinkedBlockingQueue.java:430)
>         at org.apache.tomcat.util.threads.TaskQueue.offer(TaskQueue.java:74)
>         at org.apache.tomcat.util.threads.TaskQueue.offer(TaskQueue.java:31)
>         at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1361)
>         at org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:161)
>         at org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:141)
>         at org.apache.tomcat.util.net.AprEndpoint.processSocket(AprEndpoint.java:896)
>         at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1883)
>         at java.lang.Thread.run(Thread.java:745)
>
>
> # rpm -qa | grep apr
> apr-1.5.1-1
> apr-util-1.5.3-1
>
> # rpm -qa | grep http
> httpd-2.2.27-1
>
> Server version: Apache Tomcat/8.0.23
> Server built:   May 19 2015 14:58:38 UTC
> Server number:  8.0.23.0
> OS Name:        Linux
> OS Version:     2.6.18-371.el5
> Architecture:   amd64
> JVM Version:    1.8.0_40-b25
> JVM Vendor:     Oracle Corporation
>

3. Tomcat 8.0.24 has been released several days ago.

Its connector ("Coyote") section of changelog does not mention any
similar errors, although there were several changes related to
APR/native.


Best regards,
Konstantin Kolinko

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: Tomcat 8 AprEndpoint Poller thread consuming high CPU

Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Rich,

On 6/30/15 2:06 PM, Rich Mayfield wrote:
> Moved from Tomcat 7.0.55 to Tomcat 8.0.23 and now see the AJP
> Poller thread churning through a lot of CPU in a pretty much quiet
> system.
> 
> This seems like a bug.
> 
> * Restarting does not resolve the issue * I've seen some mention of
> bumping up the memory. Doubling the max heap (-Xmx from 506m to
> 1012m) does not help. This is a relatively small service that is
> essentially idle - or at least it would be with the exception of
> this Poller thread.
> 
> The connector configuration is unchanged.
> 
> <Connector port="28080" protocol="AJP/1.3" secure="true" 
> scheme="https" connectionTimeout="20000" packetSize="20000" 
> maxThreads="1024" URIEncoding="UTF-8" redirectPort="443" 
> useSendfile="false" compression="force" 
> compressableMimeTypes="text/html,text/xml,text/plain,text/javascript,t
ext/css,application/json,application/xml"
>
> 
/>
> 
> The thread consuming all the CPU
> 
> "ajp-apr-28080-Poller" #62 daemon prio=5 os_prio=0 
> tid=0x00002ba40293e000 nid=0x407d waiting on condition 
> [0x00002ba3fc200000] java.lang.Thread.State: WAITING (parking) at
> sun.misc.Unsafe.park(Native Method) - parking to wait for
> <0x00000000eaf94570> (a 
> java.util.concurrent.locks.ReentrantLock$NonfairSync) at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) 
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInte
rrupt(AbstractQueuedSynchronizer.java:836)
>
> 
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Abst
ractQueuedSynchronizer.java:870)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Abstract
QueuedSynchronizer.java:1199)
>
> 
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.
java:209)
> at
> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
>
> 
at
java.util.concurrent.LinkedBlockingQueue.signalNotEmpty(LinkedBlockingQu
eue.java:172)
> at
> java.util.concurrent.LinkedBlockingQueue.offer(LinkedBlockingQueue.jav
a:430)
>
> 
at org.apache.tomcat.util.threads.TaskQueue.offer(TaskQueue.java:74)
> at
> org.apache.tomcat.util.threads.TaskQueue.offer(TaskQueue.java:31) 
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.jav
a:1361)
>
> 
at
org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(ThreadPoolExec
utor.java:161)
> at
> org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(ThreadPoolEx
ecutor.java:141)
>
> 
at
org.apache.tomcat.util.net.AprEndpoint.processSocket(AprEndpoint.java:89
6)
> at
> org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:188
3)
>
> 
at java.lang.Thread.run(Thread.java:745)
> 
> 
> # rpm -qa | grep apr apr-1.5.1-1 apr-util-1.5.3-1
> 
> # rpm -qa | grep http httpd-2.2.27-1
> 
> Server version: Apache Tomcat/8.0.23 Server built:   May 19 2015
> 14:58:38 UTC Server number:  8.0.23.0 OS Name:        Linux OS
> Version:     2.6.18-371.el5 Architecture:   amd64 JVM Version:
> 1.8.0_40-b25 JVM Vendor:     Oracle Corporation


Do you have more than one "Poller" thread for the APR connector?

That thread is currently blocked on a monitor. Are you saying that it
looks like it wakes up all the time and then goes back to sleep?

The poller thread is responsible for routing I/O requests between the
sockets and the request processors. Basically, it figures out which
thread's data is available and then hands it off. So, the poller
thread is often busy, but it doesn't do a great deal of actual work
itself.

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org

iQIcBAEBCAAGBQJVnYIKAAoJEBzwKT+lPKRYg0kP/1dlwElcKwLCVwHzZ6S8gG/n
hIX3Xj0Z0K4+odIVSpjHYGfCObNOtZFmUohXxCqRhrtvOf6VIkmy3FYmR+Uth4Et
iMdAI+w1g4XhkkdPo8vThJngHrK4wzXlJSWmoMX9Tq7fGKBu0nWkT6c9PgNFWK3E
glTUIOoQqKt6qJnMZU8Z3wCLjQFcPoLB7/VfAugK/BfA6YpZ6UuUx3hNT0qQ2LoV
T0QX0qESJjvmhkUgKUUS4VyRd7J8NfXsbk5w/NFX1AgCfOpEOduhsd0K2SxZI5/Z
sydBduMXdubSN59ckcRFFeOjxCwQ1r07nYl3KH2H2SwiK02V8dCXyObW/ZIDPbyN
HAnfE9K9wTzZUeJpyDYew8BRcPKgZS6WXYgcovHAZZtMCyTJTDMjmMQai0rlh+M5
++MhLqUmsvUL4JhdhMzfb/CFbKBDBRPM0oVSWIfAh/eALIj7Og1iM0wl7oPrOHHL
n+ww9+fix7Ik1hRD1DBjlk2eB6NQUQ2p6UZIa9fGgYp9rTHpkDaayFOzEc0iheoS
twwlP3a+GJ2TfCsjR1KPfsNIuCpXRiYVDpxm1PJY5/9UssiqiNRS/8yVVZX9E7fU
TAlgNMBZu81eswGWB6bhlmifFtOhMTmqwm/AViqUkMB4IcVoLDtWwKyXU3eSzdmu
MmsSGJ2O1r2aQhO1Er9Y
=s0s6
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org