You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Robert Anderson <ra...@gmail.com> on 2015/07/16 16:37:53 UTC
Many Threads in TIMED_WAITING (parking) - [org.apache.tomcat.util.threads.TaskQueue]
Hi,
Sometimes, in moments of high traffic for our patterns (170 req/sec), we
have a lot of threads like that:
"ajp-apr-8009-exec-115 ^ 16/07/2015 - 11:13:37 - End of Execute" daemon
prio=10 tid=0x00002aaab5c36800 nid=0x12f9 waiting on condition
[0x000000004ddcb000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000609a53198> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Thus, Tomcat hangs (all available threads stay in that state) and we have
to restart the service.
Server version: Apache Tomcat/7.0.63
Server built: Jun 30 2015 08:08:33 UTC
Server number: 7.0.63.0
OS Name: Linux
OS Version: 2.6.18-194.17.1.el5
Architecture: amd64
JVM Version: 1.7.0_80-b15
JVM Vendor: Oracle Corporation
We are using tomcat-native (APR connector) and Apache (2.2.3) as load
balancer with mod_jk.
server.xml
<?xml version='1.0' encoding='utf-8'?>
<Server port="8005" shutdown="SHUTDOWN">
<Listener className="org.apache.catalina.core.AprLifecycleListener"
SSLEngine="off" />
<Listener className="org.apache.catalina.core.JasperListener" />
<Listener
className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
<Listener
className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
<Listener
className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />
<GlobalNamingResources>
<Resource name="UserDatabase" auth="Container"
type="org.apache.catalina.UserDatabase"
description="User database that can be updated and saved"
factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
pathname="conf/tomcat-users.xml" />
</GlobalNamingResources>
<Service name="Catalina">
<Connector port="8080" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8443" />
<Connector port="8009" enableLookups="false" connectionTimeout="60000"
protocol="AJP/1.3" redirectPort="8443" />
<Engine name="Catalina" defaultHost="localhost" jvmRoute="tomsrv04">
<Realm className="org.apache.catalina.realm.LockOutRealm">
<Realm className="org.apache.catalina.realm.UserDatabaseRealm"
resourceName="UserDatabase" digest="SHA"/>
</Realm>
<Host name="localhost" appBase="webapps"
unpackWARs="true" autoDeploy="true">
<Valve className="org.apache.catalina.valves.StuckThreadDetectionValve"
threshold="60" interruptThreadThreshold="120" />
</Host>
</Engine>
</Service>
</Server>
Any suggestions?
Thanks in advance.
Robert
Re: Many Threads in TIMED_WAITING (parking) - [org.apache.tomcat.util.threads.TaskQueue]
Posted by Robert Anderson <ra...@gmail.com>.
Interesting. What's the CPU usage like at the time? If it's high, you
might want to look at gc logging to see if you've got a lot of GC happening.
CPU is low.
You'll probably also want to configure the access logs to log the amount of
time the request took to process.
I'll analyze this option.
Thanks.
2015-07-16 13:44 GMT-03:00 Daniel Mikusa <dm...@pivotal.io>:
> On Thu, Jul 16, 2015 at 12:04 PM, Robert Anderson <ra...@gmail.com>
> wrote:
>
> > Thanks, Dan.
> >
> > This is not a problem. The thread is in the pool waiting for work. It's
> > what you'd normally see on a server that's not handling any requests.
> >
> > It's awkward. Everything is going normal and suddenly, many threads start
> > waiting on the same monitor. After restart Tomcat, everything works
> without
> > even make any changes in Apache (service or config files).
> >
>
> The scenario you're describing can happen and it doesn't necessarily mean
> there's a problem with Tomcat. If you're seeing lots of threads waiting
> for work, it just means that you've got lots of free workers. The server
> could still be really busy if you got a couple threads running and working
> like mad.
>
> A full thread dump would be helpful, to see what else is happening at the
> time. You can also look at multiple thread dumps 10 - 15 seconds apart to
> see what the threads are doing over time (i.e. is thread #X at the same
> place across all the threads?)
>
>
> > - When the problem occurs, check the access logs for both and see if the
> > request is making it from HTTPD to Tomcat.
> >
> > OK. I'll check
> >
>
> You'll probably also want to configure the access logs to log the amount of
> time the request took to process. That'll give you some understanding of
> how much time was spent at each hop. In Tomcat that's done by adding "%D"
> to the access log pattern.
>
>
>
> https://tomcat.apache.org/tomcat-7.0-doc/config/valve.html#Access_Log_Valve/Attributes
>
> HTTPD is the same, but it logs in micro seconds instead of milliseconds.
>
> http://httpd.apache.org/docs/2.2/mod/mod_log_config.html#formats
>
>
> >
> > - Also when the problem occurs, try hitting Tomcat directly while the
> > problem is occurring. See if you can access your app when you bypass
> > HTTPD.
> >
> > I've already tried this. Tomcat becomes very very very slow when it
> > happens.
> >
>
> Interesting. What's the CPU usage like at the time? If it's high, you
> might want to look at gc logging to see if you've got a lot of GC
> happening.
>
> Dan
>
>
>
> >
> > - Post your HTTPD config
> >
> > worker.properties:
> >
> > worker.list=balancer
> >
> > worker.tomsrv01.type=ajp13
> > worker.tomsrv01.host=x.y.z.1
> > worker.tomsrv01.port=8009
> > worker.tomsrv01.lbfactor=1
> > worker.tomsrv01.connect_timeout=10000
> > worker.tomsrv01.prepost_timeout=10000
> > worker.tomsrv01.socket_timeout=1800
> > worker.tomsrv01.connection_pool_timeout=60
> >
> > worker.tomsrv02.type=ajp13
> > worker.tomsrv02.host=x.y.z.2
> > worker.tomsrv02.port=8009
> > worker.tomsrv02.lbfactor=1
> > worker.tomsrv02.connect_timeout=10000
> > worker.tomsrv02.prepost_timeout=10000
> > worker.tomsrv02.socket_timeout=1800
> > worker.tomsrv02.connection_pool_timeout=60
> >
> > worker.tomsrv04.type=ajp13
> > worker.tomsrv04.host=x.y.z.3
> > worker.tomsrv04.port=8009
> > worker.tomsrv04.lbfactor=1
> > worker.tomsrv04.connect_timeout=10000
> > worker.tomsrv04.prepost_timeout=10000
> > worker.tomsrv04.socket_timeout=1800
> > worker.tomsrv04.connection_pool_timeout=60
> >
> > worker.balancer.type=lb
> > worker.balancer.balance_workers=tomsrv01,tomsrv02,tomsrv04
> > worker.balancer.sticky_session=1
> >
> > - Make sure timeouts are configured appropriately -
> >
> > Ok
> >
> >
> > Do you have a firewall in between HTTPD & Tomcat? Is there anything that
> > could be breaking the connection?
> >
> > No.
> >
> >
> > Best regards.
> >
> >
> > 2015-07-16 12:44 GMT-03:00 Daniel Mikusa <dm...@pivotal.io>:
> >
> > > On Thu, Jul 16, 2015 at 10:37 AM, Robert Anderson <ra...@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > Sometimes, in moments of high traffic for our patterns (170 req/sec),
> > we
> > > > have a lot of threads like that:
> > > >
> > > > "ajp-apr-8009-exec-115 ^ 16/07/2015 - 11:13:37 - End of Execute"
> daemon
> > > > prio=10 tid=0x00002aaab5c36800 nid=0x12f9 waiting on condition
> > > > [0x000000004ddcb000]
> > > > java.lang.Thread.State: TIMED_WAITING (parking)
> > > > at sun.misc.Unsafe.park(Native Method)
> > > > - parking to wait for <0x0000000609a53198> (a
> > > >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > > > at
> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> > > > at
> > > >
> > >
> >
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> > > > at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86)
> > > > at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> > > > at java.lang.Thread.run(Thread.java:745)
> > > >
> > >
> > > This is not a problem. The thread is in the pool waiting for work.
> It's
> > > what you'd normally see on a server that's not handling any requests.
> > >
> > >
> > > >
> > > > Thus, Tomcat hangs (all available threads stay in that state) and we
> > have
> > > > to restart the service.
> > > >
> > > > Server version: Apache Tomcat/7.0.63
> > > > Server built: Jun 30 2015 08:08:33 UTC
> > > > Server number: 7.0.63.0
> > > > OS Name: Linux
> > > > OS Version: 2.6.18-194.17.1.el5
> > > > Architecture: amd64
> > > > JVM Version: 1.7.0_80-b15
> > > > JVM Vendor: Oracle Corporation
> > > >
> > > > We are using tomcat-native (APR connector) and Apache (2.2.3) as load
> > > > balancer with mod_jk.
> > > >
> > > >
> > > > server.xml
> > > >
> > > > <?xml version='1.0' encoding='utf-8'?>
> > > > <Server port="8005" shutdown="SHUTDOWN">
> > > > <Listener className="org.apache.catalina.core.AprLifecycleListener"
> > > > SSLEngine="off" />
> > > > <Listener className="org.apache.catalina.core.JasperListener" />
> > > > <Listener
> > > > className="org.apache.catalina.core.JreMemoryLeakPreventionListener"
> />
> > > > <Listener
> > > >
> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
> > > />
> > > > <Listener
> > > >
> className="org.apache.catalina.core.ThreadLocalLeakPreventionListener"
> > />
> > > > <GlobalNamingResources>
> > > > <Resource name="UserDatabase" auth="Container"
> > > > type="org.apache.catalina.UserDatabase"
> > > > description="User database that can be updated and
> saved"
> > > >
> > > factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
> > > > pathname="conf/tomcat-users.xml" />
> > > > </GlobalNamingResources>
> > > >
> > > > <Service name="Catalina">
> > > >
> > > > <Connector port="8080" protocol="HTTP/1.1"
> > > > connectionTimeout="20000"
> > > > redirectPort="8443" />
> > > > <Connector port="8009" enableLookups="false"
> > > connectionTimeout="60000"
> > > > protocol="AJP/1.3" redirectPort="8443" />
> > > >
> > > > <Engine name="Catalina" defaultHost="localhost"
> > jvmRoute="tomsrv04">
> > > >
> > > > <Realm className="org.apache.catalina.realm.LockOutRealm">
> > > > <Realm
> className="org.apache.catalina.realm.UserDatabaseRealm"
> > > > resourceName="UserDatabase" digest="SHA"/>
> > > > </Realm>
> > > >
> > > > <Host name="localhost" appBase="webapps"
> > > > unpackWARs="true" autoDeploy="true">
> > > >
> > > > <Valve
> className="org.apache.catalina.valves.StuckThreadDetectionValve"
> > > > threshold="60" interruptThreadThreshold="120" />
> > > > </Host>
> > > > </Engine>
> > > > </Service>
> > > > </Server>
> > > >
> > > >
> > > > Any suggestions?
> > > >
> > >
> > > This could be an issue with HTTPD talking to Tomcat.
> > > - When the problem occurs, check the access logs for both and see if
> > the
> > > request is making it from HTTPD to Tomcat.
> > > - Also when the problem occurs, try hitting Tomcat directly while
> the
> > > problem is occurring. See if you can access your app when you bypass
> > > HTTPD.
> > > - Post your HTTPD config
> > > - Make sure timeouts are configured appropriately -
> > > http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html
> > >
> > > Do you have a firewall in between HTTPD & Tomcat? Is there anything
> that
> > > could be breaking the connection?
> > >
> > > Dan
> > >
> >
>
Re: Many Threads in TIMED_WAITING (parking) - [org.apache.tomcat.util.threads.TaskQueue]
Posted by Daniel Mikusa <dm...@pivotal.io>.
On Thu, Jul 16, 2015 at 12:04 PM, Robert Anderson <ra...@gmail.com>
wrote:
> Thanks, Dan.
>
> This is not a problem. The thread is in the pool waiting for work. It's
> what you'd normally see on a server that's not handling any requests.
>
> It's awkward. Everything is going normal and suddenly, many threads start
> waiting on the same monitor. After restart Tomcat, everything works without
> even make any changes in Apache (service or config files).
>
The scenario you're describing can happen and it doesn't necessarily mean
there's a problem with Tomcat. If you're seeing lots of threads waiting
for work, it just means that you've got lots of free workers. The server
could still be really busy if you got a couple threads running and working
like mad.
A full thread dump would be helpful, to see what else is happening at the
time. You can also look at multiple thread dumps 10 - 15 seconds apart to
see what the threads are doing over time (i.e. is thread #X at the same
place across all the threads?)
> - When the problem occurs, check the access logs for both and see if the
> request is making it from HTTPD to Tomcat.
>
> OK. I'll check
>
You'll probably also want to configure the access logs to log the amount of
time the request took to process. That'll give you some understanding of
how much time was spent at each hop. In Tomcat that's done by adding "%D"
to the access log pattern.
https://tomcat.apache.org/tomcat-7.0-doc/config/valve.html#Access_Log_Valve/Attributes
HTTPD is the same, but it logs in micro seconds instead of milliseconds.
http://httpd.apache.org/docs/2.2/mod/mod_log_config.html#formats
>
> - Also when the problem occurs, try hitting Tomcat directly while the
> problem is occurring. See if you can access your app when you bypass
> HTTPD.
>
> I've already tried this. Tomcat becomes very very very slow when it
> happens.
>
Interesting. What's the CPU usage like at the time? If it's high, you
might want to look at gc logging to see if you've got a lot of GC happening.
Dan
>
> - Post your HTTPD config
>
> worker.properties:
>
> worker.list=balancer
>
> worker.tomsrv01.type=ajp13
> worker.tomsrv01.host=x.y.z.1
> worker.tomsrv01.port=8009
> worker.tomsrv01.lbfactor=1
> worker.tomsrv01.connect_timeout=10000
> worker.tomsrv01.prepost_timeout=10000
> worker.tomsrv01.socket_timeout=1800
> worker.tomsrv01.connection_pool_timeout=60
>
> worker.tomsrv02.type=ajp13
> worker.tomsrv02.host=x.y.z.2
> worker.tomsrv02.port=8009
> worker.tomsrv02.lbfactor=1
> worker.tomsrv02.connect_timeout=10000
> worker.tomsrv02.prepost_timeout=10000
> worker.tomsrv02.socket_timeout=1800
> worker.tomsrv02.connection_pool_timeout=60
>
> worker.tomsrv04.type=ajp13
> worker.tomsrv04.host=x.y.z.3
> worker.tomsrv04.port=8009
> worker.tomsrv04.lbfactor=1
> worker.tomsrv04.connect_timeout=10000
> worker.tomsrv04.prepost_timeout=10000
> worker.tomsrv04.socket_timeout=1800
> worker.tomsrv04.connection_pool_timeout=60
>
> worker.balancer.type=lb
> worker.balancer.balance_workers=tomsrv01,tomsrv02,tomsrv04
> worker.balancer.sticky_session=1
>
> - Make sure timeouts are configured appropriately -
>
> Ok
>
>
> Do you have a firewall in between HTTPD & Tomcat? Is there anything that
> could be breaking the connection?
>
> No.
>
>
> Best regards.
>
>
> 2015-07-16 12:44 GMT-03:00 Daniel Mikusa <dm...@pivotal.io>:
>
> > On Thu, Jul 16, 2015 at 10:37 AM, Robert Anderson <ra...@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > Sometimes, in moments of high traffic for our patterns (170 req/sec),
> we
> > > have a lot of threads like that:
> > >
> > > "ajp-apr-8009-exec-115 ^ 16/07/2015 - 11:13:37 - End of Execute" daemon
> > > prio=10 tid=0x00002aaab5c36800 nid=0x12f9 waiting on condition
> > > [0x000000004ddcb000]
> > > java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for <0x0000000609a53198> (a
> > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > > at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> > > at
> > >
> > >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> > > at
> > >
> >
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> > > at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86)
> > > at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32)
> > > at
> > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> > > at
> > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> > > at
> > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > at
> > >
> > >
> >
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> > > at java.lang.Thread.run(Thread.java:745)
> > >
> >
> > This is not a problem. The thread is in the pool waiting for work. It's
> > what you'd normally see on a server that's not handling any requests.
> >
> >
> > >
> > > Thus, Tomcat hangs (all available threads stay in that state) and we
> have
> > > to restart the service.
> > >
> > > Server version: Apache Tomcat/7.0.63
> > > Server built: Jun 30 2015 08:08:33 UTC
> > > Server number: 7.0.63.0
> > > OS Name: Linux
> > > OS Version: 2.6.18-194.17.1.el5
> > > Architecture: amd64
> > > JVM Version: 1.7.0_80-b15
> > > JVM Vendor: Oracle Corporation
> > >
> > > We are using tomcat-native (APR connector) and Apache (2.2.3) as load
> > > balancer with mod_jk.
> > >
> > >
> > > server.xml
> > >
> > > <?xml version='1.0' encoding='utf-8'?>
> > > <Server port="8005" shutdown="SHUTDOWN">
> > > <Listener className="org.apache.catalina.core.AprLifecycleListener"
> > > SSLEngine="off" />
> > > <Listener className="org.apache.catalina.core.JasperListener" />
> > > <Listener
> > > className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
> > > <Listener
> > > className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
> > />
> > > <Listener
> > > className="org.apache.catalina.core.ThreadLocalLeakPreventionListener"
> />
> > > <GlobalNamingResources>
> > > <Resource name="UserDatabase" auth="Container"
> > > type="org.apache.catalina.UserDatabase"
> > > description="User database that can be updated and saved"
> > >
> > factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
> > > pathname="conf/tomcat-users.xml" />
> > > </GlobalNamingResources>
> > >
> > > <Service name="Catalina">
> > >
> > > <Connector port="8080" protocol="HTTP/1.1"
> > > connectionTimeout="20000"
> > > redirectPort="8443" />
> > > <Connector port="8009" enableLookups="false"
> > connectionTimeout="60000"
> > > protocol="AJP/1.3" redirectPort="8443" />
> > >
> > > <Engine name="Catalina" defaultHost="localhost"
> jvmRoute="tomsrv04">
> > >
> > > <Realm className="org.apache.catalina.realm.LockOutRealm">
> > > <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
> > > resourceName="UserDatabase" digest="SHA"/>
> > > </Realm>
> > >
> > > <Host name="localhost" appBase="webapps"
> > > unpackWARs="true" autoDeploy="true">
> > >
> > > <Valve className="org.apache.catalina.valves.StuckThreadDetectionValve"
> > > threshold="60" interruptThreadThreshold="120" />
> > > </Host>
> > > </Engine>
> > > </Service>
> > > </Server>
> > >
> > >
> > > Any suggestions?
> > >
> >
> > This could be an issue with HTTPD talking to Tomcat.
> > - When the problem occurs, check the access logs for both and see if
> the
> > request is making it from HTTPD to Tomcat.
> > - Also when the problem occurs, try hitting Tomcat directly while the
> > problem is occurring. See if you can access your app when you bypass
> > HTTPD.
> > - Post your HTTPD config
> > - Make sure timeouts are configured appropriately -
> > http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html
> >
> > Do you have a firewall in between HTTPD & Tomcat? Is there anything that
> > could be breaking the connection?
> >
> > Dan
> >
>
Re: Many Threads in TIMED_WAITING (parking) - [org.apache.tomcat.util.threads.TaskQueue]
Posted by Robert Anderson <ra...@gmail.com>.
Thanks, Dan.
This is not a problem. The thread is in the pool waiting for work. It's
what you'd normally see on a server that's not handling any requests.
It's awkward. Everything is going normal and suddenly, many threads start
waiting on the same monitor. After restart Tomcat, everything works without
even make any changes in Apache (service or config files).
- When the problem occurs, check the access logs for both and see if the
request is making it from HTTPD to Tomcat.
OK. I'll check
- Also when the problem occurs, try hitting Tomcat directly while the
problem is occurring. See if you can access your app when you bypass HTTPD.
I've already tried this. Tomcat becomes very very very slow when it happens.
- Post your HTTPD config
worker.properties:
worker.list=balancer
worker.tomsrv01.type=ajp13
worker.tomsrv01.host=x.y.z.1
worker.tomsrv01.port=8009
worker.tomsrv01.lbfactor=1
worker.tomsrv01.connect_timeout=10000
worker.tomsrv01.prepost_timeout=10000
worker.tomsrv01.socket_timeout=1800
worker.tomsrv01.connection_pool_timeout=60
worker.tomsrv02.type=ajp13
worker.tomsrv02.host=x.y.z.2
worker.tomsrv02.port=8009
worker.tomsrv02.lbfactor=1
worker.tomsrv02.connect_timeout=10000
worker.tomsrv02.prepost_timeout=10000
worker.tomsrv02.socket_timeout=1800
worker.tomsrv02.connection_pool_timeout=60
worker.tomsrv04.type=ajp13
worker.tomsrv04.host=x.y.z.3
worker.tomsrv04.port=8009
worker.tomsrv04.lbfactor=1
worker.tomsrv04.connect_timeout=10000
worker.tomsrv04.prepost_timeout=10000
worker.tomsrv04.socket_timeout=1800
worker.tomsrv04.connection_pool_timeout=60
worker.balancer.type=lb
worker.balancer.balance_workers=tomsrv01,tomsrv02,tomsrv04
worker.balancer.sticky_session=1
- Make sure timeouts are configured appropriately -
Ok
Do you have a firewall in between HTTPD & Tomcat? Is there anything that
could be breaking the connection?
No.
Best regards.
2015-07-16 12:44 GMT-03:00 Daniel Mikusa <dm...@pivotal.io>:
> On Thu, Jul 16, 2015 at 10:37 AM, Robert Anderson <ra...@gmail.com>
> wrote:
>
> > Hi,
> >
> > Sometimes, in moments of high traffic for our patterns (170 req/sec), we
> > have a lot of threads like that:
> >
> > "ajp-apr-8009-exec-115 ^ 16/07/2015 - 11:13:37 - End of Execute" daemon
> > prio=10 tid=0x00002aaab5c36800 nid=0x12f9 waiting on condition
> > [0x000000004ddcb000]
> > java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for <0x0000000609a53198> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> > at
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> > at
> >
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> > at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86)
> > at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at
> >
> >
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> > at java.lang.Thread.run(Thread.java:745)
> >
>
> This is not a problem. The thread is in the pool waiting for work. It's
> what you'd normally see on a server that's not handling any requests.
>
>
> >
> > Thus, Tomcat hangs (all available threads stay in that state) and we have
> > to restart the service.
> >
> > Server version: Apache Tomcat/7.0.63
> > Server built: Jun 30 2015 08:08:33 UTC
> > Server number: 7.0.63.0
> > OS Name: Linux
> > OS Version: 2.6.18-194.17.1.el5
> > Architecture: amd64
> > JVM Version: 1.7.0_80-b15
> > JVM Vendor: Oracle Corporation
> >
> > We are using tomcat-native (APR connector) and Apache (2.2.3) as load
> > balancer with mod_jk.
> >
> >
> > server.xml
> >
> > <?xml version='1.0' encoding='utf-8'?>
> > <Server port="8005" shutdown="SHUTDOWN">
> > <Listener className="org.apache.catalina.core.AprLifecycleListener"
> > SSLEngine="off" />
> > <Listener className="org.apache.catalina.core.JasperListener" />
> > <Listener
> > className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
> > <Listener
> > className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
> />
> > <Listener
> > className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />
> > <GlobalNamingResources>
> > <Resource name="UserDatabase" auth="Container"
> > type="org.apache.catalina.UserDatabase"
> > description="User database that can be updated and saved"
> >
> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
> > pathname="conf/tomcat-users.xml" />
> > </GlobalNamingResources>
> >
> > <Service name="Catalina">
> >
> > <Connector port="8080" protocol="HTTP/1.1"
> > connectionTimeout="20000"
> > redirectPort="8443" />
> > <Connector port="8009" enableLookups="false"
> connectionTimeout="60000"
> > protocol="AJP/1.3" redirectPort="8443" />
> >
> > <Engine name="Catalina" defaultHost="localhost" jvmRoute="tomsrv04">
> >
> > <Realm className="org.apache.catalina.realm.LockOutRealm">
> > <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
> > resourceName="UserDatabase" digest="SHA"/>
> > </Realm>
> >
> > <Host name="localhost" appBase="webapps"
> > unpackWARs="true" autoDeploy="true">
> >
> > <Valve className="org.apache.catalina.valves.StuckThreadDetectionValve"
> > threshold="60" interruptThreadThreshold="120" />
> > </Host>
> > </Engine>
> > </Service>
> > </Server>
> >
> >
> > Any suggestions?
> >
>
> This could be an issue with HTTPD talking to Tomcat.
> - When the problem occurs, check the access logs for both and see if the
> request is making it from HTTPD to Tomcat.
> - Also when the problem occurs, try hitting Tomcat directly while the
> problem is occurring. See if you can access your app when you bypass
> HTTPD.
> - Post your HTTPD config
> - Make sure timeouts are configured appropriately -
> http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html
>
> Do you have a firewall in between HTTPD & Tomcat? Is there anything that
> could be breaking the connection?
>
> Dan
>
Re: Many Threads in TIMED_WAITING (parking) - [org.apache.tomcat.util.threads.TaskQueue]
Posted by Daniel Mikusa <dm...@pivotal.io>.
On Thu, Jul 16, 2015 at 10:37 AM, Robert Anderson <ra...@gmail.com>
wrote:
> Hi,
>
> Sometimes, in moments of high traffic for our patterns (170 req/sec), we
> have a lot of threads like that:
>
> "ajp-apr-8009-exec-115 ^ 16/07/2015 - 11:13:37 - End of Execute" daemon
> prio=10 tid=0x00002aaab5c36800 nid=0x12f9 waiting on condition
> [0x000000004ddcb000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x0000000609a53198> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86)
> at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32)
> at
>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at
>
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> at java.lang.Thread.run(Thread.java:745)
>
This is not a problem. The thread is in the pool waiting for work. It's
what you'd normally see on a server that's not handling any requests.
>
> Thus, Tomcat hangs (all available threads stay in that state) and we have
> to restart the service.
>
> Server version: Apache Tomcat/7.0.63
> Server built: Jun 30 2015 08:08:33 UTC
> Server number: 7.0.63.0
> OS Name: Linux
> OS Version: 2.6.18-194.17.1.el5
> Architecture: amd64
> JVM Version: 1.7.0_80-b15
> JVM Vendor: Oracle Corporation
>
> We are using tomcat-native (APR connector) and Apache (2.2.3) as load
> balancer with mod_jk.
>
>
> server.xml
>
> <?xml version='1.0' encoding='utf-8'?>
> <Server port="8005" shutdown="SHUTDOWN">
> <Listener className="org.apache.catalina.core.AprLifecycleListener"
> SSLEngine="off" />
> <Listener className="org.apache.catalina.core.JasperListener" />
> <Listener
> className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
> <Listener
> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
> <Listener
> className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />
> <GlobalNamingResources>
> <Resource name="UserDatabase" auth="Container"
> type="org.apache.catalina.UserDatabase"
> description="User database that can be updated and saved"
> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
> pathname="conf/tomcat-users.xml" />
> </GlobalNamingResources>
>
> <Service name="Catalina">
>
> <Connector port="8080" protocol="HTTP/1.1"
> connectionTimeout="20000"
> redirectPort="8443" />
> <Connector port="8009" enableLookups="false" connectionTimeout="60000"
> protocol="AJP/1.3" redirectPort="8443" />
>
> <Engine name="Catalina" defaultHost="localhost" jvmRoute="tomsrv04">
>
> <Realm className="org.apache.catalina.realm.LockOutRealm">
> <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
> resourceName="UserDatabase" digest="SHA"/>
> </Realm>
>
> <Host name="localhost" appBase="webapps"
> unpackWARs="true" autoDeploy="true">
>
> <Valve className="org.apache.catalina.valves.StuckThreadDetectionValve"
> threshold="60" interruptThreadThreshold="120" />
> </Host>
> </Engine>
> </Service>
> </Server>
>
>
> Any suggestions?
>
This could be an issue with HTTPD talking to Tomcat.
- When the problem occurs, check the access logs for both and see if the
request is making it from HTTPD to Tomcat.
- Also when the problem occurs, try hitting Tomcat directly while the
problem is occurring. See if you can access your app when you bypass HTTPD.
- Post your HTTPD config
- Make sure timeouts are configured appropriately -
http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html
Do you have a firewall in between HTTPD & Tomcat? Is there anything that
could be breaking the connection?
Dan
Re: Many Threads in TIMED_WAITING (parking) - [org.apache.tomcat.util.threads.TaskQueue]
Posted by Robert Anderson <ra...@gmail.com>.
Hi Mark,
How can I disable it?
2015-07-16 13:56 GMT-03:00 Mark Thomas <ma...@apache.org>:
> On 16 July 2015 16:37:53 CEST, Robert Anderson <ra...@gmail.com> wrote:
> >Hi,
> >
> >Sometimes, in moments of high traffic for our patterns (170 req/sec),
> >we
> >have a lot of threads like that:
> >
> >"ajp-apr-8009-exec-115 ^ 16/07/2015 - 11:13:37 - End of Execute" daemon
> >prio=10 tid=0x00002aaab5c36800 nid=0x12f9 waiting on condition
> >[0x000000004ddcb000]
> > java.lang.Thread.State: TIMED_WAITING (parking)
> >at sun.misc.Unsafe.park(Native Method)
> >- parking to wait for <0x0000000609a53198> (a
> >java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >at
> >java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> >at
>
> >java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> >at
>
> >java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> >at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86)
> >at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32)
> >at
>
> >java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> >at
>
> >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> >at
>
> >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >at
>
> >org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> >at java.lang.Thread.run(Thread.java:745)
> >
> >Thus, Tomcat hangs (all available threads stay in that state) and we
> >have
> >to restart the service.
> >
> >Server version: Apache Tomcat/7.0.63
> >Server built: Jun 30 2015 08:08:33 UTC
> >Server number: 7.0.63.0
> >OS Name: Linux
> >OS Version: 2.6.18-194.17.1.el5
> >Architecture: amd64
> >JVM Version: 1.7.0_80-b15
> >JVM Vendor: Oracle Corporation
> >
> >We are using tomcat-native (APR connector) and Apache (2.2.3) as load
> >balancer with mod_jk.
> >
> >
> >server.xml
> >
> ><?xml version='1.0' encoding='utf-8'?>
> ><Server port="8005" shutdown="SHUTDOWN">
> > <Listener className="org.apache.catalina.core.AprLifecycleListener"
> >SSLEngine="off" />
> > <Listener className="org.apache.catalina.core.JasperListener" />
> > <Listener
> >className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
> > <Listener
> >className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
> >/>
> > <Listener
> >className="org.apache.catalina.core.ThreadLocalLeakPreventionListener"
> >/>
> > <GlobalNamingResources>
> > <Resource name="UserDatabase" auth="Container"
> > type="org.apache.catalina.UserDatabase"
> > description="User database that can be updated and saved"
> > factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
> > pathname="conf/tomcat-users.xml" />
> > </GlobalNamingResources>
> >
> > <Service name="Catalina">
> >
> > <Connector port="8080" protocol="HTTP/1.1"
> > connectionTimeout="20000"
> > redirectPort="8443" />
> > <Connector port="8009" enableLookups="false" connectionTimeout="60000"
> >protocol="AJP/1.3" redirectPort="8443" />
> >
> > <Engine name="Catalina" defaultHost="localhost" jvmRoute="tomsrv04">
> >
> > <Realm className="org.apache.catalina.realm.LockOutRealm">
> > <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
> > resourceName="UserDatabase" digest="SHA"/>
> > </Realm>
> >
> > <Host name="localhost" appBase="webapps"
> > unpackWARs="true" autoDeploy="true">
> >
> ><Valve className="org.apache.catalina.valves.StuckThreadDetectionValve"
> >threshold="60" interruptThreadThreshold="120" />
> > </Host>
> > </Engine>
> > </Service>
> ></Server>
> >
> >
> >Any suggestions?
> >
> >
> >Thanks in advance.
> >
> >
> >Robert
>
> Short since I'm on my phone.
>
> Connection exhaustion?
>
> Disable connection reuse or use NIO.
>
> Mark
Re: Many Threads in TIMED_WAITING (parking) - [org.apache.tomcat.util.threads.TaskQueue]
Posted by Mark Thomas <ma...@apache.org>.
On 16 July 2015 16:37:53 CEST, Robert Anderson <ra...@gmail.com> wrote:
>Hi,
>
>Sometimes, in moments of high traffic for our patterns (170 req/sec),
>we
>have a lot of threads like that:
>
>"ajp-apr-8009-exec-115 ^ 16/07/2015 - 11:13:37 - End of Execute" daemon
>prio=10 tid=0x00002aaab5c36800 nid=0x12f9 waiting on condition
>[0x000000004ddcb000]
> java.lang.Thread.State: TIMED_WAITING (parking)
>at sun.misc.Unsafe.park(Native Method)
>- parking to wait for <0x0000000609a53198> (a
>java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>at
>java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>at
>java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>at
>java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86)
>at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32)
>at
>java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>at
>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>at
>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>at
>org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
>at java.lang.Thread.run(Thread.java:745)
>
>Thus, Tomcat hangs (all available threads stay in that state) and we
>have
>to restart the service.
>
>Server version: Apache Tomcat/7.0.63
>Server built: Jun 30 2015 08:08:33 UTC
>Server number: 7.0.63.0
>OS Name: Linux
>OS Version: 2.6.18-194.17.1.el5
>Architecture: amd64
>JVM Version: 1.7.0_80-b15
>JVM Vendor: Oracle Corporation
>
>We are using tomcat-native (APR connector) and Apache (2.2.3) as load
>balancer with mod_jk.
>
>
>server.xml
>
><?xml version='1.0' encoding='utf-8'?>
><Server port="8005" shutdown="SHUTDOWN">
> <Listener className="org.apache.catalina.core.AprLifecycleListener"
>SSLEngine="off" />
> <Listener className="org.apache.catalina.core.JasperListener" />
> <Listener
>className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
> <Listener
>className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
>/>
> <Listener
>className="org.apache.catalina.core.ThreadLocalLeakPreventionListener"
>/>
> <GlobalNamingResources>
> <Resource name="UserDatabase" auth="Container"
> type="org.apache.catalina.UserDatabase"
> description="User database that can be updated and saved"
> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
> pathname="conf/tomcat-users.xml" />
> </GlobalNamingResources>
>
> <Service name="Catalina">
>
> <Connector port="8080" protocol="HTTP/1.1"
> connectionTimeout="20000"
> redirectPort="8443" />
> <Connector port="8009" enableLookups="false" connectionTimeout="60000"
>protocol="AJP/1.3" redirectPort="8443" />
>
> <Engine name="Catalina" defaultHost="localhost" jvmRoute="tomsrv04">
>
> <Realm className="org.apache.catalina.realm.LockOutRealm">
> <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
> resourceName="UserDatabase" digest="SHA"/>
> </Realm>
>
> <Host name="localhost" appBase="webapps"
> unpackWARs="true" autoDeploy="true">
>
><Valve className="org.apache.catalina.valves.StuckThreadDetectionValve"
>threshold="60" interruptThreadThreshold="120" />
> </Host>
> </Engine>
> </Service>
></Server>
>
>
>Any suggestions?
>
>
>Thanks in advance.
>
>
>Robert
Short since I'm on my phone.
Connection exhaustion?
Disable connection reuse or use NIO.
Mark