You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Patty O'Reilly <or...@qualcomm.com> on 2005/04/14 01:13:40 UTC
ThreadPool logFull
Hi,
I'm running tomcat 5.0.30.
After running for a week or so the server begins to slow down and
finally crashes because it is out of threads. I can see the java
processes accumulating day by day.
Apr 8, 2005 3:11:16 PM org.apache.tomcat.util.threads.ThreadPool logFull
SEVERE: All threads (200) are currently busy, waiting. Increase maxThreads (200)
or check the servlet status
I'm pretty new to tomcat. Not sure if the error is in my apache
server's workers.properties, or the tomcat servers server.xml, or....
Can anyone point me in the right direction. This service is lightly
loaded. 200 threads should be plenty.
--patty
---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
Re: ThreadPool logFull
Posted by Patty O'Reilly <or...@qualcomm.com>.
I did the thread dump. Cool.
There are 25 threads that look like this:
"http-8100-Processor13" daemon prio=1 tid=0x083c6090 nid=0x99a in Object.wait() [bcdff000..bcdff8d0]
at java.lang.Object.wait(Native Method)
- waiting on <0x33feebc0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:429)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
- locked <0x33feebc0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:534)
There is 1 thread that look like this:
"http-8100-Processor25" daemon prio=1 tid=0x084badf8 nid=0x9a6 runnable [bb5ff000..bb5ff8d0]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0x33ef71f0> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:368)
at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:549)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:534)
There are 102 of these runnable processes and two waiting:
"TP-Processor39" daemon prio=1 tid=0x0809de40 nid=0x35d0 runnable [b53ff000..b53ff8d0]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:222)
at java.io.BufferedInputStream.read(BufferedInputStream.java:277)
- locked <0x3449e188> (a java.io.BufferedInputStream)
at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:598)
at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:535)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:663)
at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:866)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:534)
The total number of processes is now at 143 for this tomcat instance.
Yesterday evening it was at around 95. Most of the hits are the result of
our monitoring system.
I don't know how to make sense of the thread dumps. The application was
written by one of our developers. It performs a simple export compliance
task. The app simply grabs the user's name and checks to see if it is in
a database, if it isn't, the app displays an agreement, which the users
either accepts or rejects. If the user accepts the agreement, their name
is added to the database. After that, the user is redirected to either a
logout page if they don't accept the agreement, or they are redirected
to another server, if their name appears in the database.
Can you help with the tread dump?
--patty
On Thu, 14 Apr 2005, David Rees wrote:
> Date: Thu, 14 Apr 2005 04:56:17 -0700
> From: David Rees <dr...@gmail.com>
> Reply-To: Tomcat Users List <to...@jakarta.apache.org>,
> David Rees <dr...@gmail.com>
> To: Tomcat Users List <to...@jakarta.apache.org>
> Subject: Re: ThreadPool logFull
>
> On 4/13/05, Patty O'Reilly <or...@qualcomm.com> wrote:
> >
> > I'm running tomcat 5.0.30.
> >
> > After running for a week or so the server begins to slow down and
> > finally crashes because it is out of threads. I can see the java
> > processes accumulating day by day.
>
> Send the process a -QUIT signal if on Linux, do the equivalent on
> Windows and you can get stack traces showing where each thread is
> getting stuck instead of returning to the thread pool for new
> processing.
>
> > I'm pretty new to tomcat. Not sure if the error is in my apache
> > server's workers.properties, or the tomcat servers server.xml, or....
>
> It's probably not a Tomcat issue, but your something in your
> application which is getting hung up.
>
> -Dave
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
Re: ThreadPool logFull
Posted by David Rees <dr...@gmail.com>.
On 4/13/05, Patty O'Reilly <or...@qualcomm.com> wrote:
>
> I'm running tomcat 5.0.30.
>
> After running for a week or so the server begins to slow down and
> finally crashes because it is out of threads. I can see the java
> processes accumulating day by day.
Send the process a -QUIT signal if on Linux, do the equivalent on
Windows and you can get stack traces showing where each thread is
getting stuck instead of returning to the thread pool for new
processing.
> I'm pretty new to tomcat. Not sure if the error is in my apache
> server's workers.properties, or the tomcat servers server.xml, or....
It's probably not a Tomcat issue, but your something in your
application which is getting hung up.
-Dave
---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
Re: ThreadPool logFull -- BufferedInputStream
Posted by Patty O'Reilly <or...@qualcomm.com>.
This problem was resolved (thanks to a suggestion from Mladen Turk)
by adding connectionTimeout="20000" to the AJP connector in the
server.xml file.
--patty
On Mon, 18 Apr 2005, Patty O'Reilly wrote:
> Date: Mon, 18 Apr 2005 13:53:24 -0700 (PDT)
> From: Patty O'Reilly <or...@qualcomm.com>
> Reply-To: Tomcat Users List <to...@jakarta.apache.org>
> To: Tomcat Users List <to...@jakarta.apache.org>
> Subject: ThreadPool logFull -- BufferedInputStream
>
> I haven't heard anything back so I thought I'd rephrase my question.
>
> I'm running tomcat-5.0.30 with apache 2.0.52 and j2sdk1.4.2_04 on
> Red Hat AS 3 update 2. I am seeing runaway thread creation where
> threads continue to accumulate until they are maxed out.
>
> After doing a thread dump, which I have very little experience
> reading, it appears that there may be a socket bug in either Java
> or RedHat, but I'm not sure.
>
> Most threads are locked on BufferedInputStream. This is what a typical
> thread looks like:
>
> "TP-Processor104" daemon prio=1 tid=0x080dc218 nid=0x4c87 runnable [ad1ff000..a$
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:129)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:222)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:277)
> - locked <0x339bd098> (a java.io.BufferedInputStream)
> at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:598)
> at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:535)
> at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:6$
> at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:866)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.$
> at java.lang.Thread.run(Thread.java:534)
>
> The last thing called appears to be SocketInputStream.socketRead0
> (Native Method). The application is a simple jsp using taglibs.
>
> Can anyone shed some light on this. I'm brand new to Tomcat.
>
> Thanks
>
> --patty
>
> On Wed, 13 Apr 2005, Patty O'Reilly wrote:
>
> > Date: Wed, 13 Apr 2005 16:13:40 -0700 (PDT)
> > From: Patty O'Reilly <or...@qualcomm.com>
> > Reply-To: Tomcat Users List <to...@jakarta.apache.org>
> > To: tomcat-user@jakarta.apache.org
> > Subject: ThreadPool logFull
> >
> > Hi,
> >
> > I'm running tomcat 5.0.30.
> >
> > After running for a week or so the server begins to slow down and
> > finally crashes because it is out of threads. I can see the java
> > processes accumulating day by day.
> >
> > Apr 8, 2005 3:11:16 PM org.apache.tomcat.util.threads.ThreadPool logFull
> > SEVERE: All threads (200) are currently busy, waiting. Increase maxThreads (200)
> > or check the servlet status
> >
> > I'm pretty new to tomcat. Not sure if the error is in my apache
> > server's workers.properties, or the tomcat servers server.xml, or....
> >
> > Can anyone point me in the right direction. This service is lightly
> > loaded. 200 threads should be plenty.
> >
> > --patty
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
ThreadPool logFull -- BufferedInputStream
Posted by Patty O'Reilly <or...@qualcomm.com>.
I haven't heard anything back so I thought I'd rephrase my question.
I'm running tomcat-5.0.30 with apache 2.0.52 and j2sdk1.4.2_04 on
Red Hat AS 3 update 2. I am seeing runaway thread creation where
threads continue to accumulate until they are maxed out.
After doing a thread dump, which I have very little experience
reading, it appears that there may be a socket bug in either Java
or RedHat, but I'm not sure.
Most threads are locked on BufferedInputStream. This is what a typical
thread looks like:
"TP-Processor104" daemon prio=1 tid=0x080dc218 nid=0x4c87 runnable [ad1ff000..a$
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:222)
at java.io.BufferedInputStream.read(BufferedInputStream.java:277)
- locked <0x339bd098> (a java.io.BufferedInputStream)
at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:598)
at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:535)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:6$
at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:866)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.$
at java.lang.Thread.run(Thread.java:534)
The last thing called appears to be SocketInputStream.socketRead0
(Native Method). The application is a simple jsp using taglibs.
Can anyone shed some light on this. I'm brand new to Tomcat.
Thanks
--patty
On Wed, 13 Apr 2005, Patty O'Reilly wrote:
> Date: Wed, 13 Apr 2005 16:13:40 -0700 (PDT)
> From: Patty O'Reilly <or...@qualcomm.com>
> Reply-To: Tomcat Users List <to...@jakarta.apache.org>
> To: tomcat-user@jakarta.apache.org
> Subject: ThreadPool logFull
>
> Hi,
>
> I'm running tomcat 5.0.30.
>
> After running for a week or so the server begins to slow down and
> finally crashes because it is out of threads. I can see the java
> processes accumulating day by day.
>
> Apr 8, 2005 3:11:16 PM org.apache.tomcat.util.threads.ThreadPool logFull
> SEVERE: All threads (200) are currently busy, waiting. Increase maxThreads (200)
> or check the servlet status
>
> I'm pretty new to tomcat. Not sure if the error is in my apache
> server's workers.properties, or the tomcat servers server.xml, or....
>
> Can anyone point me in the right direction. This service is lightly
> loaded. 200 threads should be plenty.
>
> --patty
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org