Missing timeout in PoolTcpEndpoint ?

[Tal Dayan <ta...@zapta.com>]

Hello,

Our production system (Tomcat 3.2.1, IBM 1.3 build cx130-20001124, Redhat
6.2) suffers from a sever problem
of hanging threads and we are trying to come with a quick fix for it.

When Tomcat runs (Standalone mode) it accumulate threads that hangs forever.
Later, when the number of hanging threads
reaches the max number of threads, it does not response anymore.

A typical stack of an hanging threads looks like

"Thread-62" (TID:0x402e5cc0, sys_thread_t:0x86a9078, state:S, native
ID:0x1e446) prio=5
        at java.net.SocketInputStream.socketRead(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:104)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:192)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:210)
        at
org.apache.tomcat.service.http.HttpRequestAdapter.doRead(HttpRequestAdapter.
java:115)
        at
org.apache.tomcat.core.BufferedServletInputStream.doRead(BufferedServletInpu
tStream.java:106)
        at
org.apache.tomcat.core.BufferedServletInputStream.read(BufferedServletInputS
tream.java:128)
        at
javax.servlet.ServletInputStream.readLine(ServletInputStream.java:138)
        at
org.apache.tomcat.service.http.HttpRequestAdapter.readNextRequest(HttpReques
tAdapter.java:129)
        at
org.apache.tomcat.service.http.HttpConnectionHandler.processConnection(HttpC
onnectionHandler.java:195)
        at
org.apache.tomcat.service.TcpWorkerThread.runIt(PoolTcpEndpoint.java:416)
        at
org.apache.tomcat.util.ThreadPool$ControlRunnable.run(ThreadPool.java:498)
        at java.lang.Thread.run(Thread.java:498)

Also, when we do 'netstat -an' we see permanent HTTP connection of about the
same number of the stuck threads, and a 'tcpdump' shows for hours no
activity from the clients at the other side of the 'connection'.

An examination of the source codes shows that PoolTcpEndpoint has a private
member for timeout but it is not used. A similar member in SimpleTcpEndpoint
is used in a call Socket.setSoTimeout(). Since setSoTimeout() controls the
timeout of a single read() operation, setting it few minutes will guarantee
that the hanging threads will eventually get released.

Any idea address this problem will be greatly appreciated. Was this problem
fixed in a newer version  of Tomcat (which is table enough for production)?
Should we apply the patch ourselves ? Should we file a bug for it ?

This is a major problem for us and our production system hardly survives a
single day, so your help will be greatly appreciated.

Thanks,

Tal

RE: Missing timeout in PoolTcpEndpoint ?

[Tal Dayan <ta...@zapta.com>]

I just submitted a bug for it that describes how to reproduce it:

http://nagoya.apache.org/bugzilla/show_bug.cgi?id=1006

Tal