You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by CS Wong <li...@gmail.com> on 2009/08/25 08:22:06 UTC
Tomcat cluster fails and generates tons of logs
Periodically, I'm getting problems with my Tomcat 6 cluster (2 nodes). One
of the nodes would just go haywire and generate a ton of logs repeating the
following:
Aug 25, 2009 11:44:10 AM org.apache.catalina.ha.session.DeltaRequest reset
SEVERE: Unable to remove element
java.util.NoSuchElementException
at java.util.LinkedList.remove(LinkedList.java:788)
at java.util.LinkedList.removeFirst(LinkedList.java:134)
at org.apache.catalina.ha.session.DeltaRequest.reset(DeltaRequest.java:201)
at org.apache.catalina.ha.session.DeltaRequest.execute(DeltaRequest.java:195)
at org.apache.catalina.ha.session.DeltaManager.handleSESSION_DELTA(DeltaManager.java:1364)
at org.apache.catalina.ha.session.DeltaManager.messageReceived(DeltaManager.java:1320)
at org.apache.catalina.ha.session.DeltaManager.messageDataReceived(DeltaManager.java:1083)
at org.apache.catalina.ha.session.ClusterSessionListener.messageReceived(ClusterSessionListener.java:87)
at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:916)
at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:897)
at org.apache.catalina.tribes.group.GroupChannel.messageReceived(GroupChannel.java:264)
at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79)
at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.messageReceived(TcpFailureDetector.java:110)
at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79)
at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79)
at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79)
at org.apache.catalina.tribes.group.ChannelCoordinator.messageReceived(ChannelCoordinator.java:241)
at org.apache.catalina.tribes.transport.ReceiverBase.messageDataReceived(ReceiverBase.java:225)
at org.apache.catalina.tribes.transport.nio.NioReplicationTask.drainChannel(NioReplicationTask.java:188)
at org.apache.catalina.tribes.transport.nio.NioReplicationTask.run(NioReplicationTask.java:91)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
That's the only thing that it shows. The other node in the cluster is still
active at this time. There's nothing to do but to restart. The large amount
of logs has caused disk space issues more than a couple of times too.
Does anybody have any idea what's wrong here?
Thanks! Wong
Re: Tomcat cluster fails and generates tons of logs
Posted by Filip Hanik - Dev Lists <de...@hanik.com>.
what version of Tomcat, and we will get it fixed
On 08/25/2009 12:22 AM, CS Wong wrote:
> Periodically, I'm getting problems with my Tomcat 6 cluster (2 nodes). One
> of the nodes would just go haywire and generate a ton of logs repeating the
> following:
>
> Aug 25, 2009 11:44:10 AM org.apache.catalina.ha.session.DeltaRequest reset
> SEVERE: Unable to remove element
> java.util.NoSuchElementException
> at java.util.LinkedList.remove(LinkedList.java:788)
> at java.util.LinkedList.removeFirst(LinkedList.java:134)
> at org.apache.catalina.ha.session.DeltaRequest.reset(DeltaRequest.java:201)
> at org.apache.catalina.ha.session.DeltaRequest.execute(DeltaRequest.java:195)
> at org.apache.catalina.ha.session.DeltaManager.handleSESSION_DELTA(DeltaManager.java:1364)
> at org.apache.catalina.ha.session.DeltaManager.messageReceived(DeltaManager.java:1320)
> at org.apache.catalina.ha.session.DeltaManager.messageDataReceived(DeltaManager.java:1083)
> at org.apache.catalina.ha.session.ClusterSessionListener.messageReceived(ClusterSessionListener.java:87)
> at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:916)
> at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:897)
> at org.apache.catalina.tribes.group.GroupChannel.messageReceived(GroupChannel.java:264)
> at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79)
> at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.messageReceived(TcpFailureDetector.java:110)
> at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79)
> at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79)
> at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79)
> at org.apache.catalina.tribes.group.ChannelCoordinator.messageReceived(ChannelCoordinator.java:241)
> at org.apache.catalina.tribes.transport.ReceiverBase.messageDataReceived(ReceiverBase.java:225)
> at org.apache.catalina.tribes.transport.nio.NioReplicationTask.drainChannel(NioReplicationTask.java:188)
> at org.apache.catalina.tribes.transport.nio.NioReplicationTask.run(NioReplicationTask.java:91)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
>
> That's the only thing that it shows. The other node in the cluster is still
> active at this time. There's nothing to do but to restart. The large amount
> of logs has caused disk space issues more than a couple of times too.
>
> Does anybody have any idea what's wrong here?
>
> Thanks! Wong
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat cluster fails and generates tons of logs
Posted by CS Wong <li...@gmail.com>.
ok, we'll try this out then.
One question about the regression, would it occur if the 2 nodes are in
different Solaris containers (both having different IPs) but on the same
physical host?
Thanks a lot!
Wong
On Wed, Aug 26, 2009 at 10:39 AM, Filip Hanik - Dev Lists <
devlists@hanik.com> wrote:
> hi Wong, yes, that one does implement a higher level of thread safety, and
> most likely would resolve your problem.
> With 6.0.20, there is a regression where tomcat nodes on the same host wont
> discover each other
> https://issues.apache.org/bugzilla/show_bug.cgi?id=47308
>
> Filip
>
>
> On 08/25/2009 07:22 PM, CS Wong wrote:
>
>> A brief look through "svn log
>>
>> http://svn.apache.org/repos/asf/tomcat/trunk/java/org/apache/catalina/ha/session/DeltaRequest.java
>> "
>> turns up this:
>> ------------------------------------------------------------------------
>> r618823 | fhanik | 2008-02-06 07:29:56 +0800 (Wed, 06 Feb 2008) | 3 lines
>>
>> Remove synchronization on the DeltaRequest object, and let the object that
>> manages the delta request (session/manager) to handle the locking
>> properly,
>> using the session lock
>> There is a case with a non sticky load balancer where using synchronized
>> and
>> a lock (essentially two locks) can end up in a dead lock
>> ------------------------------------------------------------------------
>>
>> This is the only one where the commit comments seem to indicate anything
>> related to my issue. Given that 6.0.14 was released on 14 Aug 2007 (
>> http://www.mail-archive.com/announce@apache.org/msg00386.html), it may be
>> applicable.
>>
>> Would just like to know your opinion, is it likely that this is the issue
>> I'm facing? Thanks!
>>
>> Wong
>>
>>
>> On Wed, Aug 26, 2009 at 8:48 AM, CS Wong<li...@gmail.com> wrote:
>>
>>
>>
>>> Thanks, Filip.
>>> I'm running 6.0.14 right now. Would you have any idea whether any changes
>>> in the code since then would have fixed something like this? I can try to
>>> push for an upgrade to 6.0.20 but the app owners would probably want to
>>> know
>>> whether it would be fixed for sure since they have to go through a rather
>>> troublesome round of testing which takes up quite a bit of time. It helps
>>> that they know that the problem won't reoccur once this has been done.
>>>
>>> Thanks,
>>> Wong
>>>
>>>
>>> On Tue, Aug 25, 2009 at 11:35 PM, Filip Hanik - Dev Lists<
>>> devlists@hanik.com> wrote:
>>>
>>>
>>>
>>>> I've taken a look at the code.
>>>> The fix for this is easy, but it doesn't explain why it happens. This is
>>>> a
>>>> concurrency issue, but if you're not running the latest tomcat version,
>>>> then
>>>> it could already have been fixed.
>>>>
>>>> best
>>>> Filip
>>>>
>>>>
>>>> On 08/25/2009 01:55 AM, CS Wong wrote:
>>>>
>>>>
>>>>
>>>>> Hi Michael,
>>>>> The logs are the bit that went haywire. The applications at this point
>>>>> still
>>>>> work but often, there's not enough time to troubleshoot much else. The
>>>>> logs
>>>>> can increase by 5-6GB in a matter of an hour or so and hence, we often
>>>>> just
>>>>> kill the service (normal shutdown.sh doesn't respond any more at this
>>>>> point,
>>>>> we have to kill -9 it) in panic and delete the logs before the entire
>>>>> server
>>>>> goes kaboom. This time, I managed to tail out some of the logs, for
>>>>> which
>>>>> I
>>>>> pasted an extract (same repeating pattern of errors):
>>>>>
>>>>> Aug 25, 2009 11:44:02 AM org.apache.catalina.ha.session.DeltaRequest
>>>>> reset
>>>>> SEVERE: Unable to remove element
>>>>> java.util.NoSuchElementException
>>>>> at java.util.LinkedList.remove(LinkedList.java:788)
>>>>> at java.util.LinkedList.removeFirst(LinkedList.java:134)
>>>>> at
>>>>>
>>>>> org.apache.catalina.ha.session.DeltaRequest.reset(DeltaRequest.java:201)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.catalina.ha.session.DeltaRequest.execute(DeltaRequest.java:195)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.catalina.ha.session.DeltaManager.handleSESSION_DELTA(DeltaManager.java:1364)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.catalina.ha.session.DeltaManager.messageReceived(DeltaManager.java:1320)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.catalina.ha.session.DeltaManager.messageDataReceived(DeltaManager.java:1083)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.catalina.ha.session.ClusterSessionListener.messageReceived(ClusterSessionListener.java:87)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:916)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:897)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.catalina.tribes.group.GroupChannel.messageReceived(GroupChannel.java:264)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.messageReceived(TcpFailureDetector.java:110)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.catalina.tribes.group.ChannelCoordinator.messageReceived(ChannelCoordinator.java:241)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.catalina.tribes.transport.ReceiverBase.messageDataReceived(ReceiverBase.java:225)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.catalina.tribes.transport.nio.NioReplicationTask.drainChannel(NioReplicationTask.java:188)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.catalina.tribes.transport.nio.NioReplicationTask.run(NioReplicationTask.java:91)
>>>>> at
>>>>>
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
>>>>> at
>>>>>
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>>
>>>>> Wong
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Aug 25, 2009 at 3:36 PM, Michael Ludwig<ml...@as-guides.com>
>>>>> wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> CS Wong schrieb:
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Periodically, I'm getting problems with my Tomcat 6 cluster (2
>>>>>>> nodes).
>>>>>>> One of the nodes would just go haywire
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Could you elaborate on what "going haywire" means?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> Below, you write:
>>>>>>
>>>>>> [The NoSuchElementException is] the only thing that it shows. The
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> other node in the cluster is still active at this time. There's
>>>>>>> nothing to do but to restart. The large amount of logs has caused
>>>>>>> disk space issues more than a couple of times too.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> So is that server not active any more? Unresponsive? Hyperactive
>>>>>> writing
>>>>>> to the log file? Looping?
>>>>>>
>>>>>> and generate a ton of logs repeating the following:
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Aug 25, 2009 11:44:10 AM org.apache.catalina.ha.session.DeltaRequest
>>>>>>> reset
>>>>>>> SEVERE: Unable to remove element
>>>>>>> java.util.NoSuchElementException
>>>>>>> at java.util.LinkedList.remove(LinkedList.java:788)
>>>>>>> at java.util.LinkedList.removeFirst(LinkedList.java:134)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.catalina.ha.session.DeltaRequest.reset(DeltaRequest.java:201)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.catalina.ha.session.DeltaRequest.execute(DeltaRequest.java:195)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.catalina.ha.session.DeltaManager.handleSESSION_DELTA(DeltaManager.java:1364)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.catalina.ha.session.DeltaManager.messageReceived(DeltaManager.java:1320)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.catalina.ha.session.DeltaManager.messageDataReceived(DeltaManager.java:1083)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.catalina.ha.session.ClusterSessionListener.messageReceived(ClusterSessionListener.java:87)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> I only found this, which seems to have led you here:
>>>>>>
>>>>>> http://stackoverflow.com/questions/1326336/
>>>>>>
>>>>>> Maybe it is helpful to others who know about Tomcat internals.
>>>>>>
>>>>>> --
>>>>>> Michael Ludwig
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Tomcat cluster fails and generates tons of logs
Posted by Filip Hanik - Dev Lists <de...@hanik.com>.
hi Wong, yes, that one does implement a higher level of thread safety,
and most likely would resolve your problem.
With 6.0.20, there is a regression where tomcat nodes on the same host
wont discover each other
https://issues.apache.org/bugzilla/show_bug.cgi?id=47308
Filip
On 08/25/2009 07:22 PM, CS Wong wrote:
> A brief look through "svn log
> http://svn.apache.org/repos/asf/tomcat/trunk/java/org/apache/catalina/ha/session/DeltaRequest.java"
> turns up this:
> ------------------------------------------------------------------------
> r618823 | fhanik | 2008-02-06 07:29:56 +0800 (Wed, 06 Feb 2008) | 3 lines
>
> Remove synchronization on the DeltaRequest object, and let the object that
> manages the delta request (session/manager) to handle the locking properly,
> using the session lock
> There is a case with a non sticky load balancer where using synchronized and
> a lock (essentially two locks) can end up in a dead lock
> ------------------------------------------------------------------------
>
> This is the only one where the commit comments seem to indicate anything
> related to my issue. Given that 6.0.14 was released on 14 Aug 2007 (
> http://www.mail-archive.com/announce@apache.org/msg00386.html), it may be
> applicable.
>
> Would just like to know your opinion, is it likely that this is the issue
> I'm facing? Thanks!
>
> Wong
>
>
> On Wed, Aug 26, 2009 at 8:48 AM, CS Wong<li...@gmail.com> wrote:
>
>
>> Thanks, Filip.
>> I'm running 6.0.14 right now. Would you have any idea whether any changes
>> in the code since then would have fixed something like this? I can try to
>> push for an upgrade to 6.0.20 but the app owners would probably want to know
>> whether it would be fixed for sure since they have to go through a rather
>> troublesome round of testing which takes up quite a bit of time. It helps
>> that they know that the problem won't reoccur once this has been done.
>>
>> Thanks,
>> Wong
>>
>>
>> On Tue, Aug 25, 2009 at 11:35 PM, Filip Hanik - Dev Lists<
>> devlists@hanik.com> wrote:
>>
>>
>>> I've taken a look at the code.
>>> The fix for this is easy, but it doesn't explain why it happens. This is a
>>> concurrency issue, but if you're not running the latest tomcat version, then
>>> it could already have been fixed.
>>>
>>> best
>>> Filip
>>>
>>>
>>> On 08/25/2009 01:55 AM, CS Wong wrote:
>>>
>>>
>>>> Hi Michael,
>>>> The logs are the bit that went haywire. The applications at this point
>>>> still
>>>> work but often, there's not enough time to troubleshoot much else. The
>>>> logs
>>>> can increase by 5-6GB in a matter of an hour or so and hence, we often
>>>> just
>>>> kill the service (normal shutdown.sh doesn't respond any more at this
>>>> point,
>>>> we have to kill -9 it) in panic and delete the logs before the entire
>>>> server
>>>> goes kaboom. This time, I managed to tail out some of the logs, for which
>>>> I
>>>> pasted an extract (same repeating pattern of errors):
>>>>
>>>> Aug 25, 2009 11:44:02 AM org.apache.catalina.ha.session.DeltaRequest
>>>> reset
>>>> SEVERE: Unable to remove element
>>>> java.util.NoSuchElementException
>>>> at java.util.LinkedList.remove(LinkedList.java:788)
>>>> at java.util.LinkedList.removeFirst(LinkedList.java:134)
>>>> at
>>>> org.apache.catalina.ha.session.DeltaRequest.reset(DeltaRequest.java:201)
>>>> at
>>>>
>>>> org.apache.catalina.ha.session.DeltaRequest.execute(DeltaRequest.java:195)
>>>> at
>>>>
>>>> org.apache.catalina.ha.session.DeltaManager.handleSESSION_DELTA(DeltaManager.java:1364)
>>>> at
>>>>
>>>> org.apache.catalina.ha.session.DeltaManager.messageReceived(DeltaManager.java:1320)
>>>> at
>>>>
>>>> org.apache.catalina.ha.session.DeltaManager.messageDataReceived(DeltaManager.java:1083)
>>>> at
>>>>
>>>> org.apache.catalina.ha.session.ClusterSessionListener.messageReceived(ClusterSessionListener.java:87)
>>>> at
>>>>
>>>> org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:916)
>>>> at
>>>>
>>>> org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:897)
>>>> at
>>>>
>>>> org.apache.catalina.tribes.group.GroupChannel.messageReceived(GroupChannel.java:264)
>>>> at
>>>>
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79)
>>>> at
>>>>
>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.messageReceived(TcpFailureDetector.java:110)
>>>> at
>>>>
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79)
>>>> at
>>>>
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79)
>>>> at
>>>>
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79)
>>>> at
>>>>
>>>> org.apache.catalina.tribes.group.ChannelCoordinator.messageReceived(ChannelCoordinator.java:241)
>>>> at
>>>>
>>>> org.apache.catalina.tribes.transport.ReceiverBase.messageDataReceived(ReceiverBase.java:225)
>>>> at
>>>>
>>>> org.apache.catalina.tribes.transport.nio.NioReplicationTask.drainChannel(NioReplicationTask.java:188)
>>>> at
>>>>
>>>> org.apache.catalina.tribes.transport.nio.NioReplicationTask.run(NioReplicationTask.java:91)
>>>> at
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
>>>> at
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>>
>>>> Wong
>>>>
>>>>
>>>>
>>>> On Tue, Aug 25, 2009 at 3:36 PM, Michael Ludwig<ml...@as-guides.com>
>>>> wrote:
>>>>
>>>>
>>>>
>>>>
>>>>> CS Wong schrieb:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> Periodically, I'm getting problems with my Tomcat 6 cluster (2 nodes).
>>>>>> One of the nodes would just go haywire
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> Could you elaborate on what "going haywire" means?
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> Below, you write:
>>>>>
>>>>> [The NoSuchElementException is] the only thing that it shows. The
>>>>>
>>>>>
>>>>>
>>>>>> other node in the cluster is still active at this time. There's
>>>>>> nothing to do but to restart. The large amount of logs has caused
>>>>>> disk space issues more than a couple of times too.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> So is that server not active any more? Unresponsive? Hyperactive writing
>>>>> to the log file? Looping?
>>>>>
>>>>> and generate a ton of logs repeating the following:
>>>>>
>>>>>
>>>>>
>>>>>> Aug 25, 2009 11:44:10 AM org.apache.catalina.ha.session.DeltaRequest
>>>>>> reset
>>>>>> SEVERE: Unable to remove element
>>>>>> java.util.NoSuchElementException
>>>>>> at java.util.LinkedList.remove(LinkedList.java:788)
>>>>>> at java.util.LinkedList.removeFirst(LinkedList.java:134)
>>>>>> at
>>>>>>
>>>>>> org.apache.catalina.ha.session.DeltaRequest.reset(DeltaRequest.java:201)
>>>>>> at
>>>>>>
>>>>>> org.apache.catalina.ha.session.DeltaRequest.execute(DeltaRequest.java:195)
>>>>>> at
>>>>>>
>>>>>> org.apache.catalina.ha.session.DeltaManager.handleSESSION_DELTA(DeltaManager.java:1364)
>>>>>> at
>>>>>>
>>>>>> org.apache.catalina.ha.session.DeltaManager.messageReceived(DeltaManager.java:1320)
>>>>>> at
>>>>>>
>>>>>> org.apache.catalina.ha.session.DeltaManager.messageDataReceived(DeltaManager.java:1083)
>>>>>> at
>>>>>>
>>>>>> org.apache.catalina.ha.session.ClusterSessionListener.messageReceived(ClusterSessionListener.java:87)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> I only found this, which seems to have led you here:
>>>>>
>>>>> http://stackoverflow.com/questions/1326336/
>>>>>
>>>>> Maybe it is helpful to others who know about Tomcat internals.
>>>>>
>>>>> --
>>>>> Michael Ludwig
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat cluster fails and generates tons of logs
Posted by CS Wong <li...@gmail.com>.
A brief look through "svn log
http://svn.apache.org/repos/asf/tomcat/trunk/java/org/apache/catalina/ha/session/DeltaRequest.java"
turns up this:
------------------------------------------------------------------------
r618823 | fhanik | 2008-02-06 07:29:56 +0800 (Wed, 06 Feb 2008) | 3 lines
Remove synchronization on the DeltaRequest object, and let the object that
manages the delta request (session/manager) to handle the locking properly,
using the session lock
There is a case with a non sticky load balancer where using synchronized and
a lock (essentially two locks) can end up in a dead lock
------------------------------------------------------------------------
This is the only one where the commit comments seem to indicate anything
related to my issue. Given that 6.0.14 was released on 14 Aug 2007 (
http://www.mail-archive.com/announce@apache.org/msg00386.html), it may be
applicable.
Would just like to know your opinion, is it likely that this is the issue
I'm facing? Thanks!
Wong
On Wed, Aug 26, 2009 at 8:48 AM, CS Wong <li...@gmail.com> wrote:
> Thanks, Filip.
> I'm running 6.0.14 right now. Would you have any idea whether any changes
> in the code since then would have fixed something like this? I can try to
> push for an upgrade to 6.0.20 but the app owners would probably want to know
> whether it would be fixed for sure since they have to go through a rather
> troublesome round of testing which takes up quite a bit of time. It helps
> that they know that the problem won't reoccur once this has been done.
>
> Thanks,
> Wong
>
>
> On Tue, Aug 25, 2009 at 11:35 PM, Filip Hanik - Dev Lists <
> devlists@hanik.com> wrote:
>
>> I've taken a look at the code.
>> The fix for this is easy, but it doesn't explain why it happens. This is a
>> concurrency issue, but if you're not running the latest tomcat version, then
>> it could already have been fixed.
>>
>> best
>> Filip
>>
>>
>> On 08/25/2009 01:55 AM, CS Wong wrote:
>>
>>> Hi Michael,
>>> The logs are the bit that went haywire. The applications at this point
>>> still
>>> work but often, there's not enough time to troubleshoot much else. The
>>> logs
>>> can increase by 5-6GB in a matter of an hour or so and hence, we often
>>> just
>>> kill the service (normal shutdown.sh doesn't respond any more at this
>>> point,
>>> we have to kill -9 it) in panic and delete the logs before the entire
>>> server
>>> goes kaboom. This time, I managed to tail out some of the logs, for which
>>> I
>>> pasted an extract (same repeating pattern of errors):
>>>
>>> Aug 25, 2009 11:44:02 AM org.apache.catalina.ha.session.DeltaRequest
>>> reset
>>> SEVERE: Unable to remove element
>>> java.util.NoSuchElementException
>>> at java.util.LinkedList.remove(LinkedList.java:788)
>>> at java.util.LinkedList.removeFirst(LinkedList.java:134)
>>> at
>>> org.apache.catalina.ha.session.DeltaRequest.reset(DeltaRequest.java:201)
>>> at
>>>
>>> org.apache.catalina.ha.session.DeltaRequest.execute(DeltaRequest.java:195)
>>> at
>>>
>>> org.apache.catalina.ha.session.DeltaManager.handleSESSION_DELTA(DeltaManager.java:1364)
>>> at
>>>
>>> org.apache.catalina.ha.session.DeltaManager.messageReceived(DeltaManager.java:1320)
>>> at
>>>
>>> org.apache.catalina.ha.session.DeltaManager.messageDataReceived(DeltaManager.java:1083)
>>> at
>>>
>>> org.apache.catalina.ha.session.ClusterSessionListener.messageReceived(ClusterSessionListener.java:87)
>>> at
>>>
>>> org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:916)
>>> at
>>>
>>> org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:897)
>>> at
>>>
>>> org.apache.catalina.tribes.group.GroupChannel.messageReceived(GroupChannel.java:264)
>>> at
>>>
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79)
>>> at
>>>
>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.messageReceived(TcpFailureDetector.java:110)
>>> at
>>>
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79)
>>> at
>>>
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79)
>>> at
>>>
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79)
>>> at
>>>
>>> org.apache.catalina.tribes.group.ChannelCoordinator.messageReceived(ChannelCoordinator.java:241)
>>> at
>>>
>>> org.apache.catalina.tribes.transport.ReceiverBase.messageDataReceived(ReceiverBase.java:225)
>>> at
>>>
>>> org.apache.catalina.tribes.transport.nio.NioReplicationTask.drainChannel(NioReplicationTask.java:188)
>>> at
>>>
>>> org.apache.catalina.tribes.transport.nio.NioReplicationTask.run(NioReplicationTask.java:91)
>>> at
>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
>>> at
>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>> at java.lang.Thread.run(Thread.java:619)
>>>
>>> Wong
>>>
>>>
>>>
>>> On Tue, Aug 25, 2009 at 3:36 PM, Michael Ludwig<ml...@as-guides.com>
>>> wrote:
>>>
>>>
>>>
>>>> CS Wong schrieb:
>>>>
>>>>
>>>>
>>>>> Periodically, I'm getting problems with my Tomcat 6 cluster (2 nodes).
>>>>> One of the nodes would just go haywire
>>>>>
>>>>>
>>>>>
>>>> Could you elaborate on what "going haywire" means?
>>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>> Below, you write:
>>>>
>>>> [The NoSuchElementException is] the only thing that it shows. The
>>>>
>>>>
>>>>> other node in the cluster is still active at this time. There's
>>>>> nothing to do but to restart. The large amount of logs has caused
>>>>> disk space issues more than a couple of times too.
>>>>>
>>>>>
>>>>>
>>>> So is that server not active any more? Unresponsive? Hyperactive writing
>>>> to the log file? Looping?
>>>>
>>>> and generate a ton of logs repeating the following:
>>>>
>>>>
>>>>> Aug 25, 2009 11:44:10 AM org.apache.catalina.ha.session.DeltaRequest
>>>>> reset
>>>>> SEVERE: Unable to remove element
>>>>> java.util.NoSuchElementException
>>>>> at java.util.LinkedList.remove(LinkedList.java:788)
>>>>> at java.util.LinkedList.removeFirst(LinkedList.java:134)
>>>>> at
>>>>>
>>>>> org.apache.catalina.ha.session.DeltaRequest.reset(DeltaRequest.java:201)
>>>>> at
>>>>>
>>>>> org.apache.catalina.ha.session.DeltaRequest.execute(DeltaRequest.java:195)
>>>>> at
>>>>>
>>>>> org.apache.catalina.ha.session.DeltaManager.handleSESSION_DELTA(DeltaManager.java:1364)
>>>>> at
>>>>>
>>>>> org.apache.catalina.ha.session.DeltaManager.messageReceived(DeltaManager.java:1320)
>>>>> at
>>>>>
>>>>> org.apache.catalina.ha.session.DeltaManager.messageDataReceived(DeltaManager.java:1083)
>>>>> at
>>>>>
>>>>> org.apache.catalina.ha.session.ClusterSessionListener.messageReceived(ClusterSessionListener.java:87)
>>>>>
>>>>>
>>>>>
>>>> I only found this, which seems to have led you here:
>>>>
>>>> http://stackoverflow.com/questions/1326336/
>>>>
>>>> Maybe it is helpful to others who know about Tomcat internals.
>>>>
>>>> --
>>>> Michael Ludwig
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
Re: Tomcat cluster fails and generates tons of logs
Posted by CS Wong <li...@gmail.com>.
Thanks, Filip.
I'm running 6.0.14 right now. Would you have any idea whether any changes in
the code since then would have fixed something like this? I can try to push
for an upgrade to 6.0.20 but the app owners would probably want to know
whether it would be fixed for sure since they have to go through a rather
troublesome round of testing which takes up quite a bit of time. It helps
that they know that the problem won't reoccur once this has been done.
Thanks,
Wong
On Tue, Aug 25, 2009 at 11:35 PM, Filip Hanik - Dev Lists <
devlists@hanik.com> wrote:
> I've taken a look at the code.
> The fix for this is easy, but it doesn't explain why it happens. This is a
> concurrency issue, but if you're not running the latest tomcat version, then
> it could already have been fixed.
>
> best
> Filip
>
>
> On 08/25/2009 01:55 AM, CS Wong wrote:
>
>> Hi Michael,
>> The logs are the bit that went haywire. The applications at this point
>> still
>> work but often, there's not enough time to troubleshoot much else. The
>> logs
>> can increase by 5-6GB in a matter of an hour or so and hence, we often
>> just
>> kill the service (normal shutdown.sh doesn't respond any more at this
>> point,
>> we have to kill -9 it) in panic and delete the logs before the entire
>> server
>> goes kaboom. This time, I managed to tail out some of the logs, for which
>> I
>> pasted an extract (same repeating pattern of errors):
>>
>> Aug 25, 2009 11:44:02 AM org.apache.catalina.ha.session.DeltaRequest reset
>> SEVERE: Unable to remove element
>> java.util.NoSuchElementException
>> at java.util.LinkedList.remove(LinkedList.java:788)
>> at java.util.LinkedList.removeFirst(LinkedList.java:134)
>> at
>> org.apache.catalina.ha.session.DeltaRequest.reset(DeltaRequest.java:201)
>> at
>> org.apache.catalina.ha.session.DeltaRequest.execute(DeltaRequest.java:195)
>> at
>>
>> org.apache.catalina.ha.session.DeltaManager.handleSESSION_DELTA(DeltaManager.java:1364)
>> at
>>
>> org.apache.catalina.ha.session.DeltaManager.messageReceived(DeltaManager.java:1320)
>> at
>>
>> org.apache.catalina.ha.session.DeltaManager.messageDataReceived(DeltaManager.java:1083)
>> at
>>
>> org.apache.catalina.ha.session.ClusterSessionListener.messageReceived(ClusterSessionListener.java:87)
>> at
>>
>> org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:916)
>> at
>>
>> org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:897)
>> at
>>
>> org.apache.catalina.tribes.group.GroupChannel.messageReceived(GroupChannel.java:264)
>> at
>>
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79)
>> at
>>
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.messageReceived(TcpFailureDetector.java:110)
>> at
>>
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79)
>> at
>>
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79)
>> at
>>
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79)
>> at
>>
>> org.apache.catalina.tribes.group.ChannelCoordinator.messageReceived(ChannelCoordinator.java:241)
>> at
>>
>> org.apache.catalina.tribes.transport.ReceiverBase.messageDataReceived(ReceiverBase.java:225)
>> at
>>
>> org.apache.catalina.tribes.transport.nio.NioReplicationTask.drainChannel(NioReplicationTask.java:188)
>> at
>>
>> org.apache.catalina.tribes.transport.nio.NioReplicationTask.run(NioReplicationTask.java:91)
>> at
>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
>> at
>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>> at java.lang.Thread.run(Thread.java:619)
>>
>> Wong
>>
>>
>>
>> On Tue, Aug 25, 2009 at 3:36 PM, Michael Ludwig<ml...@as-guides.com>
>> wrote:
>>
>>
>>
>>> CS Wong schrieb:
>>>
>>>
>>>
>>>> Periodically, I'm getting problems with my Tomcat 6 cluster (2 nodes).
>>>> One of the nodes would just go haywire
>>>>
>>>>
>>>>
>>> Could you elaborate on what "going haywire" means?
>>>
>>>
>>
>>
>>
>>
>>
>>
>>> Below, you write:
>>>
>>> [The NoSuchElementException is] the only thing that it shows. The
>>>
>>>
>>>> other node in the cluster is still active at this time. There's
>>>> nothing to do but to restart. The large amount of logs has caused
>>>> disk space issues more than a couple of times too.
>>>>
>>>>
>>>>
>>> So is that server not active any more? Unresponsive? Hyperactive writing
>>> to the log file? Looping?
>>>
>>> and generate a ton of logs repeating the following:
>>>
>>>
>>>> Aug 25, 2009 11:44:10 AM org.apache.catalina.ha.session.DeltaRequest
>>>> reset
>>>> SEVERE: Unable to remove element
>>>> java.util.NoSuchElementException
>>>> at java.util.LinkedList.remove(LinkedList.java:788)
>>>> at java.util.LinkedList.removeFirst(LinkedList.java:134)
>>>> at
>>>> org.apache.catalina.ha.session.DeltaRequest.reset(DeltaRequest.java:201)
>>>> at
>>>>
>>>> org.apache.catalina.ha.session.DeltaRequest.execute(DeltaRequest.java:195)
>>>> at
>>>>
>>>> org.apache.catalina.ha.session.DeltaManager.handleSESSION_DELTA(DeltaManager.java:1364)
>>>> at
>>>>
>>>> org.apache.catalina.ha.session.DeltaManager.messageReceived(DeltaManager.java:1320)
>>>> at
>>>>
>>>> org.apache.catalina.ha.session.DeltaManager.messageDataReceived(DeltaManager.java:1083)
>>>> at
>>>>
>>>> org.apache.catalina.ha.session.ClusterSessionListener.messageReceived(ClusterSessionListener.java:87)
>>>>
>>>>
>>>>
>>> I only found this, which seems to have led you here:
>>>
>>> http://stackoverflow.com/questions/1326336/
>>>
>>> Maybe it is helpful to others who know about Tomcat internals.
>>>
>>> --
>>> Michael Ludwig
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>>
>>>
>>
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Tomcat cluster fails and generates tons of logs
Posted by Filip Hanik - Dev Lists <de...@hanik.com>.
I've taken a look at the code.
The fix for this is easy, but it doesn't explain why it happens. This is
a concurrency issue, but if you're not running the latest tomcat
version, then it could already have been fixed.
best
Filip
On 08/25/2009 01:55 AM, CS Wong wrote:
> Hi Michael,
> The logs are the bit that went haywire. The applications at this point still
> work but often, there's not enough time to troubleshoot much else. The logs
> can increase by 5-6GB in a matter of an hour or so and hence, we often just
> kill the service (normal shutdown.sh doesn't respond any more at this point,
> we have to kill -9 it) in panic and delete the logs before the entire server
> goes kaboom. This time, I managed to tail out some of the logs, for which I
> pasted an extract (same repeating pattern of errors):
>
> Aug 25, 2009 11:44:02 AM org.apache.catalina.ha.session.DeltaRequest reset
> SEVERE: Unable to remove element
> java.util.NoSuchElementException
> at java.util.LinkedList.remove(LinkedList.java:788)
> at java.util.LinkedList.removeFirst(LinkedList.java:134)
> at org.apache.catalina.ha.session.DeltaRequest.reset(DeltaRequest.java:201)
> at
> org.apache.catalina.ha.session.DeltaRequest.execute(DeltaRequest.java:195)
> at
> org.apache.catalina.ha.session.DeltaManager.handleSESSION_DELTA(DeltaManager.java:1364)
> at
> org.apache.catalina.ha.session.DeltaManager.messageReceived(DeltaManager.java:1320)
> at
> org.apache.catalina.ha.session.DeltaManager.messageDataReceived(DeltaManager.java:1083)
> at
> org.apache.catalina.ha.session.ClusterSessionListener.messageReceived(ClusterSessionListener.java:87)
> at
> org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:916)
> at
> org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:897)
> at
> org.apache.catalina.tribes.group.GroupChannel.messageReceived(GroupChannel.java:264)
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79)
> at
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.messageReceived(TcpFailureDetector.java:110)
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79)
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79)
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79)
> at
> org.apache.catalina.tribes.group.ChannelCoordinator.messageReceived(ChannelCoordinator.java:241)
> at
> org.apache.catalina.tribes.transport.ReceiverBase.messageDataReceived(ReceiverBase.java:225)
> at
> org.apache.catalina.tribes.transport.nio.NioReplicationTask.drainChannel(NioReplicationTask.java:188)
> at
> org.apache.catalina.tribes.transport.nio.NioReplicationTask.run(NioReplicationTask.java:91)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
>
> Wong
>
>
>
> On Tue, Aug 25, 2009 at 3:36 PM, Michael Ludwig<ml...@as-guides.com> wrote:
>
>
>> CS Wong schrieb:
>>
>>
>>> Periodically, I'm getting problems with my Tomcat 6 cluster (2 nodes).
>>> One of the nodes would just go haywire
>>>
>>>
>> Could you elaborate on what "going haywire" means?
>>
>
>
>
>
>
>> Below, you write:
>>
>> [The NoSuchElementException is] the only thing that it shows. The
>>
>>> other node in the cluster is still active at this time. There's
>>> nothing to do but to restart. The large amount of logs has caused
>>> disk space issues more than a couple of times too.
>>>
>>>
>> So is that server not active any more? Unresponsive? Hyperactive writing
>> to the log file? Looping?
>>
>> and generate a ton of logs repeating the following:
>>
>>> Aug 25, 2009 11:44:10 AM org.apache.catalina.ha.session.DeltaRequest reset
>>> SEVERE: Unable to remove element
>>> java.util.NoSuchElementException
>>> at java.util.LinkedList.remove(LinkedList.java:788)
>>> at java.util.LinkedList.removeFirst(LinkedList.java:134)
>>> at
>>> org.apache.catalina.ha.session.DeltaRequest.reset(DeltaRequest.java:201)
>>> at
>>> org.apache.catalina.ha.session.DeltaRequest.execute(DeltaRequest.java:195)
>>> at
>>> org.apache.catalina.ha.session.DeltaManager.handleSESSION_DELTA(DeltaManager.java:1364)
>>> at
>>> org.apache.catalina.ha.session.DeltaManager.messageReceived(DeltaManager.java:1320)
>>> at
>>> org.apache.catalina.ha.session.DeltaManager.messageDataReceived(DeltaManager.java:1083)
>>> at
>>> org.apache.catalina.ha.session.ClusterSessionListener.messageReceived(ClusterSessionListener.java:87)
>>>
>>>
>> I only found this, which seems to have led you here:
>>
>> http://stackoverflow.com/questions/1326336/
>>
>> Maybe it is helpful to others who know about Tomcat internals.
>>
>> --
>> Michael Ludwig
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat cluster fails and generates tons of logs
Posted by CS Wong <li...@gmail.com>.
Hi Michael,
The logs are the bit that went haywire. The applications at this point still
work but often, there's not enough time to troubleshoot much else. The logs
can increase by 5-6GB in a matter of an hour or so and hence, we often just
kill the service (normal shutdown.sh doesn't respond any more at this point,
we have to kill -9 it) in panic and delete the logs before the entire server
goes kaboom. This time, I managed to tail out some of the logs, for which I
pasted an extract (same repeating pattern of errors):
Aug 25, 2009 11:44:02 AM org.apache.catalina.ha.session.DeltaRequest reset
SEVERE: Unable to remove element
java.util.NoSuchElementException
at java.util.LinkedList.remove(LinkedList.java:788)
at java.util.LinkedList.removeFirst(LinkedList.java:134)
at org.apache.catalina.ha.session.DeltaRequest.reset(DeltaRequest.java:201)
at
org.apache.catalina.ha.session.DeltaRequest.execute(DeltaRequest.java:195)
at
org.apache.catalina.ha.session.DeltaManager.handleSESSION_DELTA(DeltaManager.java:1364)
at
org.apache.catalina.ha.session.DeltaManager.messageReceived(DeltaManager.java:1320)
at
org.apache.catalina.ha.session.DeltaManager.messageDataReceived(DeltaManager.java:1083)
at
org.apache.catalina.ha.session.ClusterSessionListener.messageReceived(ClusterSessionListener.java:87)
at
org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:916)
at
org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:897)
at
org.apache.catalina.tribes.group.GroupChannel.messageReceived(GroupChannel.java:264)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79)
at
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.messageReceived(TcpFailureDetector.java:110)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79)
at
org.apache.catalina.tribes.group.ChannelCoordinator.messageReceived(ChannelCoordinator.java:241)
at
org.apache.catalina.tribes.transport.ReceiverBase.messageDataReceived(ReceiverBase.java:225)
at
org.apache.catalina.tribes.transport.nio.NioReplicationTask.drainChannel(NioReplicationTask.java:188)
at
org.apache.catalina.tribes.transport.nio.NioReplicationTask.run(NioReplicationTask.java:91)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Wong
On Tue, Aug 25, 2009 at 3:36 PM, Michael Ludwig <ml...@as-guides.com> wrote:
> CS Wong schrieb:
>
>> Periodically, I'm getting problems with my Tomcat 6 cluster (2 nodes).
>> One of the nodes would just go haywire
>>
>
> Could you elaborate on what "going haywire" means?
>
>
> Below, you write:
>
> [The NoSuchElementException is] the only thing that it shows. The
>> other node in the cluster is still active at this time. There's
>> nothing to do but to restart. The large amount of logs has caused
>> disk space issues more than a couple of times too.
>>
>
> So is that server not active any more? Unresponsive? Hyperactive writing
> to the log file? Looping?
>
> and generate a ton of logs repeating the following:
>>
>> Aug 25, 2009 11:44:10 AM org.apache.catalina.ha.session.DeltaRequest reset
>> SEVERE: Unable to remove element
>> java.util.NoSuchElementException
>> at java.util.LinkedList.remove(LinkedList.java:788)
>> at java.util.LinkedList.removeFirst(LinkedList.java:134)
>> at
>> org.apache.catalina.ha.session.DeltaRequest.reset(DeltaRequest.java:201)
>> at
>> org.apache.catalina.ha.session.DeltaRequest.execute(DeltaRequest.java:195)
>> at
>> org.apache.catalina.ha.session.DeltaManager.handleSESSION_DELTA(DeltaManager.java:1364)
>> at
>> org.apache.catalina.ha.session.DeltaManager.messageReceived(DeltaManager.java:1320)
>> at
>> org.apache.catalina.ha.session.DeltaManager.messageDataReceived(DeltaManager.java:1083)
>> at
>> org.apache.catalina.ha.session.ClusterSessionListener.messageReceived(ClusterSessionListener.java:87)
>>
>
> I only found this, which seems to have led you here:
>
> http://stackoverflow.com/questions/1326336/
>
> Maybe it is helpful to others who know about Tomcat internals.
>
> --
> Michael Ludwig
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Tomcat cluster fails and generates tons of logs
Posted by Michael Ludwig <ml...@as-guides.com>.
CS Wong schrieb:
> Periodically, I'm getting problems with my Tomcat 6 cluster (2 nodes).
> One of the nodes would just go haywire
Could you elaborate on what "going haywire" means?
Below, you write:
> [The NoSuchElementException is] the only thing that it shows. The
> other node in the cluster is still active at this time. There's
> nothing to do but to restart. The large amount of logs has caused
> disk space issues more than a couple of times too.
So is that server not active any more? Unresponsive? Hyperactive writing
to the log file? Looping?
> and generate a ton of logs repeating the following:
>
> Aug 25, 2009 11:44:10 AM org.apache.catalina.ha.session.DeltaRequest reset
> SEVERE: Unable to remove element
> java.util.NoSuchElementException
> at java.util.LinkedList.remove(LinkedList.java:788)
> at java.util.LinkedList.removeFirst(LinkedList.java:134)
> at org.apache.catalina.ha.session.DeltaRequest.reset(DeltaRequest.java:201)
> at org.apache.catalina.ha.session.DeltaRequest.execute(DeltaRequest.java:195)
> at org.apache.catalina.ha.session.DeltaManager.handleSESSION_DELTA(DeltaManager.java:1364)
> at org.apache.catalina.ha.session.DeltaManager.messageReceived(DeltaManager.java:1320)
> at org.apache.catalina.ha.session.DeltaManager.messageDataReceived(DeltaManager.java:1083)
> at org.apache.catalina.ha.session.ClusterSessionListener.messageReceived(ClusterSessionListener.java:87)
I only found this, which seems to have led you here:
http://stackoverflow.com/questions/1326336/
Maybe it is helpful to others who know about Tomcat internals.
--
Michael Ludwig
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org