You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by "Gattu, Praveen" <pr...@amazon.com> on 2008/11/26 01:06:08 UTC
Threadpool doesn't contain any threads
Hi - I am noticing a weird issue with our Tomcat installation. Here is my production environment
Tomcat version - 5.5
JDK run time - 1.6
OS : Redhat Enterprise 3.0
Everyday we are noticing the tomcat not responding on a port. On further inspection we noticed that the thread pool that is serving this port has no threads. Right around the time tomcat goes into this state, we noticed that there is a thread death, and get the following stack trace
org.apache.tomcat.util.threads.ThreadPool: Caught exception (java.lang.ThreadDeath) executing org.apache.tomcat.util.net<http://org.apache.tomcat.util.net>.LeaderFollowerWorkerThread@be86ea,
terminating thread { java.lang.ThreadDeath | at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:488) | at org.apache.tomcat.util.net<http://org.apache.tomcat.util.net.LeaderFollowerWorkerThread.ru>.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:73) | at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:746) | at java.lang.Thread.run(Thread.java:619) }
After this the thread pool never gets started again. I see that this exception is essentially thrown in PoolTcpEndpoint's acceptSocket function, when the function encounters IOException accepting connections on the socket.
Did anyone else notice this issue. Another minor caveat is, we built Tomcat using JDK 1.5 but are running it on JDK 1.6. Could this be causing any issue?
Also I used Jconsole and noticed that currentThreadBusy and currentThreadCount are both -1.
If anyone can think of any pointers please let me know.
Thanks for any help,
Praveen
Re: Threadpool doesn't contain any threads
Posted by Kees Jan Koster <kj...@gmail.com>.
Dear Praveen,
And what about my question below?
>>> Did anyone else notice this issue. Another minor caveat is, we built
>>> Tomcat using JDK 1.5 but are running it on JDK 1.6. Could this be
>>> causing any issue?
>>
>> Hum. You built your own Tomcat? Is there a test environment that you
>> can test this application with a pre-built one from Apache.org? Just
>> to make sure.
--
Kees Jan
http://java-monitor.com/forum/
kjkoster@kjkoster.org
06-51838192
The secret of success lies in the stability of the goal. -- Benjamin
Disraeli
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Threadpool doesn't contain any threads
Posted by "Gattu, Praveen" <pr...@amazon.com>.
I don't think it is related to JMX. I took a stack dump and I don't see threads related to that thread pool. To answer Kees questions, it happens randomly and there is no specific request pattern. However whenever this happens I see a "Thread death" exception which I included in my previous email. When I tracked it through I see that this is caused when PoolTcpEndpoint encounters an IOException on socket accept, and it essentially either reinitializes endpoint or if that fails, stops the endpoint(which causes to shutdown the thread pool), initializes the endpoint and starts the endpoint.
My guess is in the "start" phase when it is trying to start the thread pool, its going into some bad state.
There is no other activity going on, also we have a large fleet and this issue props up randomly on some of the boxes. Since I cannot reproduce this behavior predictably I don't know if I can say I can test it out with a prebuilt apache build.
Thanks for the comments. Any other pointers please send it my way.
-----Original Message-----
From: Filip Hanik - Dev Lists [mailto:devlists@hanik.com]
Sent: Wednesday, November 26, 2008 8:28 AM
To: Tomcat Users List
Subject: Re: Threadpool doesn't contain any threads
could just be flaky jmx code, meaning we don't publish completely
accurate numbers
Filip
Kees Jan Koster wrote:
> Dear Praveen,
>
>> Everyday we are noticing the tomcat not responding on a port. On
>> further inspection we noticed that the thread pool that is serving
>> this port has no threads. Right around the time tomcat goes into this
>> state, we noticed that there is a thread death, and get the following
>> stack trace
>
> Does this happen randomly on the day, or at a specific time of day? Is
> it reproducible on a test machine?
>
> What other activity is going on around the time that Tomcat dies? Cron
> jobs? Backups? Network changes?
>
>> Did anyone else notice this issue. Another minor caveat is, we built
>> Tomcat using JDK 1.5 but are running it on JDK 1.6. Could this be
>> causing any issue?
>
> Hum. You built your own Tomcat? Is there a test environment that you
> can test this application with a pre-built one from Apache.org? Just
> to make sure.
>
>> Also I used Jconsole and noticed that currentThreadBusy and
>> currentThreadCount are both -1.
>
> I'm not sure that these figures can be trusted once the thread pool
> has died.
> --
> Kees Jan
>
> http://java-monitor.com/forum/
> kjkoster@kjkoster.org
> 06-51838192
>
> Human beings make life so interesting. Do you know that in a universe
> so full of wonders,
> they have managed to invent boredom. Quite astonishing... -- Terry
> Partchett
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Threadpool doesn't contain any threads
Posted by Filip Hanik - Dev Lists <de...@hanik.com>.
could just be flaky jmx code, meaning we don't publish completely
accurate numbers
Filip
Kees Jan Koster wrote:
> Dear Praveen,
>
>> Everyday we are noticing the tomcat not responding on a port. On
>> further inspection we noticed that the thread pool that is serving
>> this port has no threads. Right around the time tomcat goes into this
>> state, we noticed that there is a thread death, and get the following
>> stack trace
>
> Does this happen randomly on the day, or at a specific time of day? Is
> it reproducible on a test machine?
>
> What other activity is going on around the time that Tomcat dies? Cron
> jobs? Backups? Network changes?
>
>> Did anyone else notice this issue. Another minor caveat is, we built
>> Tomcat using JDK 1.5 but are running it on JDK 1.6. Could this be
>> causing any issue?
>
> Hum. You built your own Tomcat? Is there a test environment that you
> can test this application with a pre-built one from Apache.org? Just
> to make sure.
>
>> Also I used Jconsole and noticed that currentThreadBusy and
>> currentThreadCount are both -1.
>
> I'm not sure that these figures can be trusted once the thread pool
> has died.
> --
> Kees Jan
>
> http://java-monitor.com/forum/
> kjkoster@kjkoster.org
> 06-51838192
>
> Human beings make life so interesting. Do you know that in a universe
> so full of wonders,
> they have managed to invent boredom. Quite astonishing... -- Terry
> Partchett
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Threadpool doesn't contain any threads
Posted by Kees Jan Koster <kj...@gmail.com>.
Dear Praveen,
> Everyday we are noticing the tomcat not responding on a port. On
> further inspection we noticed that the thread pool that is serving
> this port has no threads. Right around the time tomcat goes into
> this state, we noticed that there is a thread death, and get the
> following stack trace
Does this happen randomly on the day, or at a specific time of day? Is
it reproducible on a test machine?
What other activity is going on around the time that Tomcat dies? Cron
jobs? Backups? Network changes?
> Did anyone else notice this issue. Another minor caveat is, we built
> Tomcat using JDK 1.5 but are running it on JDK 1.6. Could this be
> causing any issue?
Hum. You built your own Tomcat? Is there a test environment that you
can test this application with a pre-built one from Apache.org? Just
to make sure.
> Also I used Jconsole and noticed that currentThreadBusy and
> currentThreadCount are both -1.
I'm not sure that these figures can be trusted once the thread pool
has died.
--
Kees Jan
http://java-monitor.com/forum/
kjkoster@kjkoster.org
06-51838192
Human beings make life so interesting. Do you know that in a universe
so full of wonders,
they have managed to invent boredom. Quite astonishing... -- Terry
Partchett
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Threadpool doesn't contain any threads
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Gattu, Praveen [mailto:praveeng@amazon.com]
> Subject: Threadpool doesn't contain any threads
>
> Did anyone else notice this issue. Another minor caveat is,
> we built Tomcat using JDK 1.5 but are running it on JDK 1.6.
Can't answer your primary question, but as for building Tomcat, it must be done on 1.5, but it will run properly on 1.5 or 1.6.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org