Tomcat endpoints are becoming extremely unresponsive

[Craig Lynch <rc...@gmail.com>]

We run embedded tomcat on version 8, and for some reason are consistently
seeing extreme slowness across all Tomcat endpoints at very consistent
intervals of three hours. Once a site gets into the slow state, it is never
able to recover, and stays unresponsive (requests take tens of minutes to
hours) until the service is manually restarted.



There are no resource issues that I've been able to detect (heap seems
fine, no apparent memory leaks, cpu is fine, network/db connections aren't
exhausted, etc). Tomcat does seem to receive the requests, but for some
reason does not seem to be processing them.



There is an exception that occurs right around the time the service goes
into a bad state, which is the reason I believe this to be a Tomcat issue.
The stack trace is as follows:



Exception in thread "mc-26" java.lang.IllegalMonitorStateException



    at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(Unknown
Source)



    at
java.util.concurrent.locks.AbstractQueuedSynchronizer.release(Unknown
Source)



    at java.util.concurrent.locks.ReentrantLock.unlock(Unknown Source)



    at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)



    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)



    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)



    at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)



    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)



    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)



    at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)



    at java.lang.Thread.run(Unknown Source)



Having done some additional testing, I can elaborate on the state of things
when this exception occurs:

* The ReentrantLock that is throwing the exception is meant to throw the
IllegalMonitorStateException when a thread other than the thread that
created the lock tries to release it. Interestingly, when tryRelease() is
called, the owning thread is actually null, which means that the lock isn't
currently taken or owned by anyone.



* The lock's state is 0, which is consistent with the lock's owning thread
being null.



* The tryRelease funtion takes an int argument "releases", which is 1
(you'd find that in the stacktrace anyway, since it's passed in as a
constant further up, but mentioning it might save some time).



I realize that the LinkedBlockingQueue and ReentrantLock are java core
concurrency libraries, but it seems like Tomcat is getting into a bad state
once the error occurs, and is unable to exit the bad state. The executing
thread that gets this exception dies, and almost all other threads end up
staying almost all the time in an unsafe parked state. The
IllegalMonitorStateException also generally occurs in several threads after
it's shown up for the first time.



I'm not sure how to describe how to reproduce this issue, other than saying
that everyone at my company with our service installed experiences it very
reliably every three hours. We've been thus far unable to determine what
causes it, however. My personal theory is that somehow the TaskQueue is
getting into a state where it can only rarely give tasks to the executor
threads, but I don't know what causes things to enter that state. From what
we can tell, it does seem to be related to our Jersey endpoints.



If you need any additional information, just let me know and I'll be happy
to provide anything that might be useful. I've been getting most of my
information from thread/heap dumps as well as modifying local versions of
Tomcat to provide additional logging.



Most of our services are running Tomcat 8.0.32, but seem to still exhibit
the problem on versions at least as early as 8.0.15 and as late as 8.0.34.



I submitted this as a bug (59737), which Mark Thomas immediately closed
saying that there’s nothing to indicate that this is a Tomcat bug. I’m
happy to provide any additional information that might be useful, most of
I’ve looked at so far seems to point to Tomcat, though I would be ecstatic
if someone could point me to where I’m messing up J

Re: Tomcat endpoints are becoming extremely unresponsive

[Mark Thomas <ma...@apache.org>]

On 20/06/2016 20:58, Craig Lynch wrote:
> We run embedded tomcat on version 8, and for some reason are consistently
> seeing extreme slowness across all Tomcat endpoints at very consistent
> intervals of three hours. Once a site gets into the slow state, it is never
> able to recover, and stays unresponsive (requests take tens of minutes to
> hours) until the service is manually restarted.
> 
> There are no resource issues that I've been able to detect (heap seems
> fine, no apparent memory leaks, cpu is fine, network/db connections aren't
> exhausted, etc). Tomcat does seem to receive the requests, but for some
> reason does not seem to be processing them.
> 
> There is an exception that occurs right around the time the service goes
> into a bad state, which is the reason I believe this to be a Tomcat issue.
> The stack trace is as follows:
> 
> Exception in thread "mc-26" java.lang.IllegalMonitorStateException
>     at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(Unknown
> Source)
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.release(Unknown
> Source)
>     at java.util.concurrent.locks.ReentrantLock.unlock(Unknown Source)
>     at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
>     at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)
>     at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
>     at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
>     at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>     at
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
>     at java.lang.Thread.run(Unknown Source)

And when you Googled "IllegalMonitorStateException LinkedBlockingQueue"
what did you find?

Mark


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org