You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Harshad Kamat <hn...@gmail.com> on 2010/03/02 03:27:03 UTC
Tomcat Acceptor Thread goes into wait() to never come back - Tomcat
6.0
Hi,
I believe I've found a race condition in Tomcat that causes the http port to
be non-responsive. It exists in 6.0 and also in 5.5 (although the code has
been refactored).
I could not find any reference to it in the Bug database or the mailing list
archives.
Consider a tomcat instance with maxThreads set to 2, i.e. you have 2 tomcat
threads to service incoming requests.
The sequence of events is as follows:
1. Thread 1 and Thread 2 are both servicing a request each.
2. A third request comes in.
3. In class JIOEndpoint.java, the acceptor thread calls methods
processSocket() which then calls getWorkerThread() which then calls
createWorkerThread().
4. createWorkerThread() returns null since both threads are busy processing
the two requests.
5. Here is the race condition in method getWorkerThread() in the code shown
below
protected Worker getWorkerThread(){
...
Worker workerThread = createWorkerThread();
while (workerThread == null) {
try {
synchronized (workers) {
workers.wait();
}
}
...
}
The acceptor thread executes the "while(workerThread == null)" statement and
is then switched out by the CPU.
The two threads executing the two requests complete and go into
Worker.await() waiting for the next job after executing method
recycleWorkerThread().
The acceptor thread is switched back into CPU and executes the synchronized
block and goes into the wait().
At this point, there aren't any Worker threads out there processing requests
and therefore there isn't any thread to wake up the acceptor thread.
The application is non-responsive after this.
Thoughts?
A simple solution would be to check if curThreadsBusy > 0 in the
synchronized block before going into wait() in method getWorkerThread().
Thanks,
Harshad
Stack Traces below:
"bda19102143" id=1578 in WAITING on
lock=org.apache.tomcat.util.net.JIoEndpoint$Worker@13aa4ee3^M
at java.lang.Object.wait(Native Method)^M
at java.lang.Object.wait(Object.java:485)^M
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)^M
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)^M
at java.lang.Thread.run(Thread.java:619)^M
"http-8091-Acceptor-0" id=43 in WAITING on
lock=org.apache.tomcat.util.net.JIoEndpoint$WorkerStack@13bd7b6a^M
at java.lang.Object.wait(Native Method)^M
at java.lang.Object.wait(Object.java:485)^M
at
org.apache.tomcat.util.net.JIoEndpoint.getWorkerThread(JIoEndpoint.java:700)^M
at
org.apache.tomcat.util.net.JIoEndpoint.processSocket(JIoEndpoint.java:731)^M
at
org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:313)^M
at java.lang.Thread.run(Thread.java:619)^M
RE: Tomcat Acceptor Thread goes into wait() to never come back -
Tomcat 6.0
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Harshad Kamat [mailto:hnkamat@gmail.com]
> Subject: Tomcat Acceptor Thread goes into wait() to never come back -
> Tomcat 6.0
>
> I believe I've found a race condition in Tomcat that causes the http
> port to be non-responsive.
I believe you are correct. To close the timing window, the lock needs to remain up across the call to createWorkerThread() and the test for null. Something like this would work:
protected Worker getWorkerThread() {
Worker workerThread;
// Allocate a new worker thread
synchronized (workers) {
while ((workerThread = createWorkerThread()) == null) {
try {
workers.wait();
} catch (InterruptedException e) {
// Ignore
}
}
}
return workerThread;
}
While we're at it, the synchronized clause in createWorkerThread() can now be eliminated, and the method simplified (and a lot of useless and confusing parentheses removed):
protected Worker createWorkerThread() {
if (workers.size() > 0) {
curThreadsBusy++;
return workers.pop();
}
if (maxThreads < 0 || curThreads < maxThreads) {
curThreadsBusy++;
if (curThreadsBusy == maxThreads) {
log.info(sm.getString("endpoint.info.maxThreads",
Integer.toString(maxThreads), address,
Integer.toString(port)));
}
return newWorkerThread();
}
return null;
}
The two methods could be combined, but it would probably be uglier. One could also fix the symptom by putting a time limit on the wait() call, but that offends the sensibilities.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat Acceptor Thread goes into wait() to never come back -
Tomcat 6.0
Posted by Hae Loong Chan <ha...@gmail.com>.
Hi,
If the threads are in the race condition, can it be any possibility
the Tomcat will become responsive again - say if the current
processing request thread is dropped due to request timeout and the
browser is closed?
On Tue, Mar 2, 2010 at 10:27 AM, Harshad Kamat <hn...@gmail.com> wrote:
> Hi,
>
> I believe I've found a race condition in Tomcat that causes the http port to
> be non-responsive. It exists in 6.0 and also in 5.5 (although the code has
> been refactored).
> I could not find any reference to it in the Bug database or the mailing list
> archives.
>
> Consider a tomcat instance with maxThreads set to 2, i.e. you have 2 tomcat
> threads to service incoming requests.
> The sequence of events is as follows:
> 1. Thread 1 and Thread 2 are both servicing a request each.
> 2. A third request comes in.
> 3. In class JIOEndpoint.java, the acceptor thread calls methods
> processSocket() which then calls getWorkerThread() which then calls
> createWorkerThread().
> 4. createWorkerThread() returns null since both threads are busy processing
> the two requests.
> 5. Here is the race condition in method getWorkerThread() in the code shown
> below
>
> protected Worker getWorkerThread(){
> ...
> Worker workerThread = createWorkerThread();
> while (workerThread == null) {
> try {
> synchronized (workers) {
> workers.wait();
> }
> }
> ...
> }
>
> The acceptor thread executes the "while(workerThread == null)" statement and
> is then switched out by the CPU.
> The two threads executing the two requests complete and go into
> Worker.await() waiting for the next job after executing method
> recycleWorkerThread().
> The acceptor thread is switched back into CPU and executes the synchronized
> block and goes into the wait().
>
> At this point, there aren't any Worker threads out there processing requests
> and therefore there isn't any thread to wake up the acceptor thread.
> The application is non-responsive after this.
>
> Thoughts?
>
> A simple solution would be to check if curThreadsBusy > 0 in the
> synchronized block before going into wait() in method getWorkerThread().
>
> Thanks,
> Harshad
>
>
> Stack Traces below:
>
> "bda19102143" id=1578 in WAITING on
> lock=org.apache.tomcat.util.net.JIoEndpoint$Worker@13aa4ee3^M
> at java.lang.Object.wait(Native Method)^M
> at java.lang.Object.wait(Object.java:485)^M
> at
> org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)^M
> at
> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)^M
> at java.lang.Thread.run(Thread.java:619)^M
>
> "http-8091-Acceptor-0" id=43 in WAITING on
> lock=org.apache.tomcat.util.net.JIoEndpoint$WorkerStack@13bd7b6a^M
> at java.lang.Object.wait(Native Method)^M
> at java.lang.Object.wait(Object.java:485)^M
> at
> org.apache.tomcat.util.net.JIoEndpoint.getWorkerThread(JIoEndpoint.java:700)^M
> at
> org.apache.tomcat.util.net.JIoEndpoint.processSocket(JIoEndpoint.java:731)^M
> at
> org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:313)^M
> at java.lang.Thread.run(Thread.java:619)^M
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org