You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by gp456 <pa...@werum.de> on 2008/02/13 13:55:38 UTC
Tomcat Cluster Error -
Hello,
we have a similar problem as Raul Garcia has posted in
http://www.nabble.com/Tomcat-6---Cluster-error.-to14921385.html#a14921385
this message, but we're receiving the error of a disappearing member after
appr. 12 hours!!
Our load is also similar: 1 message/sec.
We are using a hardware loadbalancer in front of two apache/tomcat pairs.
The apache are connected via ajp
to the local tomcat or (if the local tomcat is down) to the tomcat on the
other node.
Our config:
Node1/2:
Apache 2.0.59
Tomcat 6.0.13
Tomcat conf Node 1:
...
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
channelSendOptions="8">
<Manager className="org.apache.catalina.ha.session.DeltaManager"
expireSessionsOnShutdown="false"
notifyListenersOnReplication="true"/>
<Channel
className="org.apache.catalina.tribes.group.GroupChannel">
<Membership
className="org.apache.catalina.tribes.membership.McastService"
address="239.252.100.12"
port="14022"
frequency="1000"
dropTime="30000"/>
<Receiver
className="org.apache.catalina.tribes.transport.nio.NioReceiver"
address="223.99.114.21"
port="14022"
autoBind="100"
selectorTimeout="5000"
maxThreads="6"/>
<Sender
className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
<Transport
className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
timeout="60000" maxRetryAttempts="10" keepAliveCount="0" />
</Sender>
<Interceptor
className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
<Interceptor
className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
<Interceptor
className="org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor"
interval="1000" />
</Channel>
<Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
filter=""/>
<Valve
className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
<!-- Deployer
className="org.apache.catalina.ha.deploy.FarmWarDeployer"
tempDir="/tmp/war-temp/"
deployDir="/tmp/war-deploy/"
watchDir="/tmp/war-listen/"
watchEnabled="false"/ -->
<ClusterListener
className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
<ClusterListener
className="org.apache.catalina.ha.session.ClusterSessionListener"/>
</Cluster>
...
Tomcat conf Node 2:
...
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
channelSendOptions="8">
<Manager className="org.apache.catalina.ha.session.DeltaManager"
expireSessionsOnShutdown="false"
notifyListenersOnReplication="true"/>
<Channel
className="org.apache.catalina.tribes.group.GroupChannel">
<Membership
className="org.apache.catalina.tribes.membership.McastService"
address="239.252.100.12"
port="14022"
frequency="1000"
dropTime="30000"/>
<Receiver
className="org.apache.catalina.tribes.transport.nio.NioReceiver"
address="223.99.114.20"
port="14022"
autoBind="100"
selectorTimeout="5000"
maxThreads="6"/>
<Sender
className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
<Transport
className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
timeout="60000" maxRetryAttempts="10" keepAliveCount="0" />
</Sender>
<Interceptor
className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
<Interceptor
className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
<Interceptor
className="org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor"
interval="1000" />
</Channel>
<Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
filter=""/>
<Valve
className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
<!-- Deployer
className="org.apache.catalina.ha.deploy.FarmWarDeployer"
tempDir="/tmp/war-temp/"
deployDir="/tmp/war-deploy/"
watchDir="/tmp/war-listen/"
watchEnabled="false"/ -->
<ClusterListener
className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
<ClusterListener
className="org.apache.catalina.ha.session.ClusterSessionListener"/>
</Cluster>
...
Log On Node 1:
...
[CLUSTERLOG] main - INFO - 02/12/2008 10:05:15,875 - McastService - Sleeping
for 2000 milliseconds to establish cluster membership, start level:8
[CLUSTERLOG] pool-2-thread-1 - INFO - 02/12/2008 10:05:15,908 - BufferPool -
Created a buffer pool with max size:104857600 bytes of
type:org.apache.catalina
.tribes.io.BufferPool15Impl
[CLUSTERLOG] main - INFO - 02/12/2008 10:05:17,877 - McastService - Done
sleeping, membership established, start level:8
[CLUSTERLOG] main - INFO - 02/12/2008 10:05:17,884 - JvmRouteBinderValve -
JvmRouteBinderValve started
[CLUSTERLOG] pool-2-thread-3 - WARN - 02/12/2008 10:05:19,328 -
ClusterListener - Context manager doesn't exist:
[CLUSTERLOG] main - INFO - 02/12/2008 10:05:20,015 - DeltaManager - Register
manager to cluster element Host with name localhost
[CLUSTERLOG] main - INFO - 02/12/2008 10:05:20,015 - DeltaManager - Starting
clustering manager at
[CLUSTERLOG] main - WARN - 02/12/2008 10:05:20,026 - DeltaManager - Manager
[], requesting session state from
org.apache.catalina.tribes.membership.MemberIm
pl[tcp://223.99.114.21:14022,223.99.114.21,14022, alive=167423,id={10 -86 53
5 6 -125 71 -7 -109 5 -36 4 48 -46 45 65 }, payload={}, command={},
domain={},
]. This operation will timeout if no session state has been received within
60 seconds.
[CLUSTERLOG] pool-1-thread-1 - INFO - 02/12/2008 10:05:20,045 -
ThroughputInterceptor - ThroughputInterceptor Report[
Tx Msg:1 messages
Sent:0,00 MB (total)
Sent:0,00 MB (application)
Time:0,02 seconds
Tx Speed:0,03 MB/sec (total)
TxSpeed:0,03 MB/sec (application)
Error Msg:0
Rx Msg:2 messages
Rx Speed:0,00 MB/sec (since 1st msg)
Received:0,00 MB]
...
after some time we're getting this entry:
[CLUSTERLOG] pool-2-thread-4 - WARN - 02/12/2008 15:00:52,044 -
ClusterListener - Context manager doesn't exist:
[CLUSTERLOG] pool-2-thread-5 - WARN - 02/12/2008 15:00:52,910 -
ClusterListener - Context manager doesn't exist:
[CLUSTERLOG] pool-2-thread-2 - WARN - 02/12/2008 15:00:54,299 -
ClusterListener - Context manager doesn't exist:
[CLUSTERLOG] pool-2-thread-6 - WARN - 02/12/2008 15:00:54,707 -
ClusterListener - Context manager doesn't exist:
[CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] - INFO -
02/12/2008 15:00:56,011 - DeltaManager - Register manager to cluster
element H
ost with name localhost
[CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] - INFO -
02/12/2008 15:00:56,011 - DeltaManager - Starting clustering manager at
[CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] - WARN -
02/12/2008 15:00:56,012 - DeltaManager - Manager [], requesting session
state f
rom
org.apache.catalina.tribes.membership.MemberImpl[tcp://223.99.114.21:14022,223.99.114.21,14022,
alive=17902419,id={10 -86 53 5 6 -125 71 -7 -109 5 -36 4
48 -46 45 65 }, payload={}, command={}, domain={}, ]. This operation will
timeout if no session state has been received within 60 seconds.
[CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] - INFO -
02/12/2008 15:01:02,245 - DeltaManager - Manager []; session state send at
12.0
2.08 15:00 received in 6.233 ms.
[CLUSTERLOG] pool-2-thread-2 - INFO - 02/12/2008 15:01:13,178 -
ThroughputInterceptor - ThroughputInterceptor Report[
Tx Msg:20003 messages
Sent:143,22 MB (total)
Sent:143,22 MB (application)
Time:54,66 seconds
Tx Speed:2,62 MB/sec (total)
TxSpeed:2,62 MB/sec (application)
Error Msg:0
Rx Msg:20000 messages
Rx Speed:0,01 MB/sec (since 1st msg)
Received:143,04 MB]
[CLUSTERLOG] pool-2-thread-5 - INFO - 02/12/2008 15:23:49,278 -
ThroughputInterceptor - ThroughputInterceptor Report[
Tx Msg:20606 messages
Sent:156,13 MB (total)
Sent:156,13 MB (application)
Time:61,55 seconds
Tx Speed:2,54 MB/sec (total)
TxSpeed:2,54 MB/sec (application)
Error Msg:9
Rx Msg:21000 messages
Rx Speed:0,01 MB/sec (since 1st msg)
Received:150,14 MB]
and now we're receiving and sending message but also Error messages....
and then:
[CLUSTERLOG] NioReceiver - WARN - 02/12/2008 17:34:23,895 - NioReceiver -
Channel key is registered, but has had no interest ops for the last 3000 ms.
(canc
elled:false):sun.nio.ch.SelectionKeyImpl@e161d6 last access:2008-02-12
17:34:17.957
and finally:
[CLUSTERLOG] 4564 - INFO - 02/12/2008 22:04:28,652 - TcpFailureDetector -
Received
memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://
223.99.114.21:14022,223.99.114.21,14022, alive=43315450,id={10 -86 53 5 6
-125 71 -7 -109 5 -36 4 48 -46 45 65 }, payload={}, command={}, domain={},
]] mess
age. Will verify.
[CLUSTERLOG] 4564 - INFO - 02/12/2008 22:04:29,654 - TcpFailureDetector -
Verification complete. Member
disappeared[org.apache.catalina.tribes.membership.Me
mberImpl[tcp://223.99.114.21:14022,223.99.114.21,14022,
alive=43316452,id={10 -86 53 5 6 -125 71 -7 -109 5 -36 4 48 -46 45 65 },
payload={}, command={}, dom
ain={}, ]]
[CLUSTERLOG] 4564 - INFO - 02/12/2008 22:04:29,654 - SimpleTcpCluster -
Received member
disappeared:org.apache.catalina.tribes.membership.MemberImpl[tcp://2
23.99.114.21:14022,223.99.114.21,14022, alive=43316452,id={10 -86 53 5 6
-125 71 -7 -109 5 -36 4 48 -46 45 65 }, payload={}, command={}, domain={}, ]
[CLUSTERLOG] 4564 - ERROR - 02/12/2008 22:04:29,654 - SimpleTcpCluster -
Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out(60000
ms.).; Faulty members:tcp://223.99.114.21:14022;
at
org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:97)
at
org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
at
org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80)
at
org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
at
org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor.sendMessage(ThroughputInterceptor.java:61)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
at
org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:60)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
at
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
at
org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
at
org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
at
org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:835)
at
org.apache.catalina.ha.tcp.SimpleTcpCluster.sendClusterDomain(SimpleTcpCluster.java:814)
at
org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:551)
at
org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:535)
at
org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:517)
at
org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:428)
at
org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:362)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:261)
at
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
at
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:686)
at java.lang.Thread.run(Thread.java:595)
[CLUSTERLOG] 4575 - ERROR - 02/12/2008 22:04:34,920 - SimpleTcpCluster -
Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: No destination given; No faulty
members identified.
at
org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:194)
at
org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
...
the error messages are repeating...
Log On Node 2:
[CLUSTERLOG] main - WARN - 02/12/2008 10:02:35,175 - SimpleTcpCluster -
Dynamic setProperty(className,value) has been disabled, please use explicit
properties
for the element you are trying to identify
[CLUSTERLOG] main - INFO - 02/12/2008 10:02:35,954 - SimpleTcpCluster -
Cluster is about to start
[CLUSTERLOG] main - INFO - 02/12/2008 10:02:35,973 - ReceiverBase - Receiver
Server Socket bound to:/223.99.114.21:14022
[CLUSTERLOG] main - INFO - 02/12/2008 10:02:36,025 - McastService - Setting
cluster mcast soTimeout to 1000
[CLUSTERLOG] main - INFO - 02/12/2008 10:02:36,037 - McastService - Sleeping
for 2000 milliseconds to establish cluster membership, start level:4
[CLUSTERLOG] Thread-3 - INFO - 02/12/2008 10:02:36,913 - SimpleTcpCluster -
Replication member
added:org.apache.catalina.tribes.membership.MemberImpl[tcp://22
3.99.114.20:14022,223.99.114.20,14022, alive=322404622,id={109 -10 62 -21 19
102 69 -42 -92 -123 -59 -15 37 16 50 -5 }, payload={}, command={},
domain={}, ]
[CLUSTERLOG] main - INFO - 02/12/2008 10:02:38,039 - McastService - Done
sleeping, membership established, start level:4
[CLUSTERLOG] main - INFO - 02/12/2008 10:02:38,044 - McastService - Sleeping
for 2000 milliseconds to establish cluster membership, start level:8
[CLUSTERLOG] pool-2-thread-1 - INFO - 02/12/2008 10:02:38,074 - BufferPool -
Created a buffer pool with max size:104857600 bytes of
type:org.apache.catalina.t
ribes.io.BufferPool15Impl
[CLUSTERLOG] pool-2-thread-3 - WARN - 02/12/2008 10:02:38,312 -
ClusterListener - Context manager doesn't exist:
[CLUSTERLOG] pool-2-thread-4 - WARN - 02/12/2008 10:02:38,816 -
ClusterListener - Context manager doesn't exist:
[CLUSTERLOG] pool-2-thread-5 - WARN - 02/12/2008 10:02:39,277 -
ClusterListener - Context manager doesn't exist:
[CLUSTERLOG] pool-2-thread-6 - WARN - 02/12/2008 10:02:39,337 -
ClusterListener - Context manager doesn't exist:
[CLUSTERLOG] main - INFO - 02/12/2008 10:02:40,046 - McastService - Done
sleeping, membership established, start level:8
[CLUSTERLOG] main - INFO - 02/12/2008 10:02:40,053 - JvmRouteBinderValve -
JvmRouteBinderValve started
[CLUSTERLOG] pool-2-thread-1 - WARN - 02/12/2008 10:02:42,816 -
ClusterListener - Context manager doesn't exist:
[CLUSTERLOG] main - INFO - 02/12/2008 10:02:43,093 - DeltaManager - Register
manager to cluster element Host with name localhost
[CLUSTERLOG] main - INFO - 02/12/2008 10:02:43,093 - DeltaManager - Starting
clustering manager at
[CLUSTERLOG] main - WARN - 02/12/2008 10:02:43,100 - DeltaManager - Manager
[], requesting session state from
org.apache.catalina.tribes.membership.MemberImpl
[tcp://223.99.114.20:14022,223.99.114.20,14022, alive=322410633,id={109 -10
62 -21 19 102 69 -42 -92 -123 -59 -15 37 16 50 -5 }, payload={}, command={},
domai
n={}, ]. This operation will timeout if no session state has been received
within 60 seconds.
[CLUSTERLOG] pool-1-thread-1 - INFO - 02/12/2008 10:02:43,112 -
ThroughputInterceptor - ThroughputInterceptor Report[
Tx Msg:1 messages
Sent:0,00 MB (total)
Sent:0,00 MB (application)
Time:0,01 seconds
Tx Speed:0,04 MB/sec (total)
TxSpeed:0,04 MB/sec (application)
Error Msg:0
Rx Msg:6 messages
Rx Speed:0,00 MB/sec (since 1st msg)
Received:0,01 MB]
[CLUSTERLOG] main - INFO - 02/12/2008 10:02:43,719 - DeltaManager - Manager
[]; session state send at 12.02.08 10:02 received in 624 ms.
[CLUSTERLOG] Thread-31 - INFO - 02/12/2008 10:04:55,941 - TcpFailureDetector
- Verification complete. Member
disappeared[org.apache.catalina.tribes.membership
.MemberImpl[tcp://223.99.114.20:14022,223.99.114.20,14022,
alive=322543666,id={109 -10 62 -21 19 102 69 -42 -92 -123 -59 -15 37 16 50
-5 }, payload={}, comman
d={66 65 66 89 45 65 76 69 88 ...(9)}, domain={}, ]]
[CLUSTERLOG] Thread-30 - INFO - 02/12/2008 10:04:55,941 - TcpFailureDetector
- Verification complete. Member
disappeared[org.apache.catalina.tribes.membership
.MemberImpl[tcp://223.99.114.20:14022,223.99.114.20,14022,
alive=322543666,id={109 -10 62 -21 19 102 69 -42 -92 -123 -59 -15 37 16 50
-5 }, payload={}, comman
d={66 65 66 89 45 65 76 69 88 ...(9)}, domain={}, ]]
[CLUSTERLOG] Thread-31 - INFO - 02/12/2008 10:04:55,941 - SimpleTcpCluster -
Received member
disappeared:org.apache.catalina.tribes.membership.MemberImpl[tcp:
//223.99.114.20:14022,223.99.114.20,14022, alive=322543666,id={109 -10 62
-21 19 102 69 -42 -92 -123 -59 -15 37 16 50 -5 }, payload={}, command={66 65
66 89 4
5 65 76 69 88 ...(9)}, domain={}, ]
[CLUSTERLOG] Thread-30 - INFO - 02/12/2008 10:04:55,941 - SimpleTcpCluster -
Received member
disappeared:org.apache.catalina.tribes.membership.MemberImpl[tcp:
//223.99.114.20:14022,223.99.114.20,14022, alive=322543666,id={109 -10 62
-21 19 102 69 -42 -92 -123 -59 -15 37 16 50 -5 }, payload={}, command={66 65
66 89 4
5 65 76 69 88 ...(9)}, domain={}, ]
[CLUSTERLOG] Thread-36 - INFO - 02/12/2008 10:05:19,299 - SimpleTcpCluster -
Replication member
added:org.apache.catalina.tribes.membership.MemberImpl[tcp://2
23.99.114.20:14022,223.99.114.20,14022, alive=2028,id={19 -115 -1 -101 8 91
66 -91 -112 105 -121 109 -82 11 56 50 }, payload={}, command={}, domain={},
]
[CLUSTERLOG] pool-2-thread-4 - INFO - 02/12/2008 10:25:11,742 -
ThroughputInterceptor - ThroughputInterceptor Report[
Tx Msg:892 messages
Sent:5,82 MB (total)
Sent:5,82 MB (application)
Time:1,33 seconds
Tx Speed:4,39 MB/sec (total)
TxSpeed:4,39 MB/sec (application)
Error Msg:0
Rx Msg:1000 messages
Rx Speed:0,00 MB/sec (since 1st msg)
Received:4,47 MB]
...
[CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] - INFO -
02/12/2008 15:01:16,005 - DeltaManager - Manager [] expiring sessions upon
shutdo
wn
[CLUSTERLOG] pool-2-thread-1 - WARN - 02/12/2008 15:01:20,014 -
ClusterListener - Context manager doesn't exist:
[CLUSTERLOG] pool-2-thread-1 - WARN - 02/12/2008 15:01:20,015 -
ClusterListener - Context manager doesn't exist:
[CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] - INFO -
02/12/2008 15:01:20,219 - DeltaManager - Register manager to cluster
element Hos
t with name localhost
[CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] - INFO -
02/12/2008 15:01:20,220 - DeltaManager - Starting clustering manager at
[CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] - WARN -
02/12/2008 15:01:20,220 - DeltaManager - Manager [], requesting session
state fro
m
org.apache.catalina.tribes.membership.MemberImpl[tcp://223.99.114.20:14022,223.99.114.20,14022,
alive=17762449,id={19 -115 -1 -101 8 91 66 -91 -112 105 -121
109 -82 11 56 50 }, payload={}, command={}, domain={}, ]. This operation
will timeout if no session state has been received within 60 seconds.
[CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] - INFO -
02/12/2008 15:01:24,093 - DeltaManager - Manager []; session state send at
12.02.
08 15:01 received in 3.872 ms.
[CLUSTERLOG] pool-1-thread-1 - INFO - 02/12/2008 15:22:41,143 -
ThroughputInterceptor - ThroughputInterceptor Report[
Tx Msg:21000 messages
Sent:150,30 MB (total)
Sent:150,30 MB (application)
Time:70,58 seconds
Tx Speed:2,13 MB/sec (total)
TxSpeed:2,13 MB/sec (application)
Error Msg:0
Rx Msg:20534 messages
Rx Speed:0,01 MB/sec (since 1st msg)
Received:156,92 MB]
after this entry we're NOT receiving any messages (Rx Msg):
[CLUSTERLOG] pool-1-thread-2 - INFO - 02/12/2008 15:40:02,612 -
ThroughputInterceptor - ThroughputInterceptor Report[
Tx Msg:22000 messages
Sent:154,75 MB (total)
Sent:154,75 MB (application)
Time:72,92 seconds
Tx Speed:2,12 MB/sec (total)
TxSpeed:2,12 MB/sec (application)
Error Msg:0
Rx Msg:20534 messages
Rx Speed:0,01 MB/sec (since 1st msg)
Received:156,92 MB]
[CLUSTERLOG] pool-1-thread-2 - INFO - 02/12/2008 15:54:15,184 -
ThroughputInterceptor - ThroughputInterceptor Report[
Tx Msg:23022 messages
Sent:156,30 MB (total)
Sent:156,30 MB (application)
Time:74,46 seconds
Tx Speed:2,10 MB/sec (total)
TxSpeed:2,10 MB/sec (application)
Error Msg:0
Rx Msg:20534 messages
Rx Speed:0,01 MB/sec (since 1st msg)
Received:156,92 MB]
...
Many thanks for your help!!!
--
View this message in context: http://www.nabble.com/Tomcat-Cluster-Error---tp15457470p15457470.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat Cluster Error - Operation has timed out
Posted by Filip Hanik - Dev Lists <de...@hanik.com>.
you already asked this question, and I already answered.
http://marc.info/?l=tomcat-user&m=120290898727190&w=2
Filip
gp456 wrote:
> Hello,
>
> we have a similar problem as Raul Garcia has posted in
> http://www.nabble.com/Tomcat-6---Cluster-error.-to14921385.html#a14921385
> this message, but we're receiving the error of a disappearing member after
> appr. 12 hours!!
> Our load is also similar: 1 message/sec.
>
> We are using a hardware loadbalancer in front of two apache/tomcat pairs.
> The apache are connected via ajp
> to the local tomcat or (if the local tomcat is down) to the tomcat on the
> other node.
> Our config:
> Node1/2:
> Apache 2.0.59
> Tomcat 6.0.13
>
> Tomcat conf Node 1:
>
> ...
> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
> channelSendOptions="8">
>
> <Manager className="org.apache.catalina.ha.session.DeltaManager"
> expireSessionsOnShutdown="false"
> notifyListenersOnReplication="true"/>
>
> <Channel
> className="org.apache.catalina.tribes.group.GroupChannel">
> <Membership
> className="org.apache.catalina.tribes.membership.McastService"
> address="239.252.100.12"
> port="14022"
> frequency="1000"
> dropTime="30000"/>
> <Receiver
> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
> address="223.99.114.21"
> port="14022"
> autoBind="100"
> selectorTimeout="5000"
> maxThreads="6"/>
>
> <Sender
> className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
> <Transport
> className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
> timeout="60000" maxRetryAttempts="10" keepAliveCount="0" />
> </Sender>
> <Interceptor
> className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
> <Interceptor
> className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
> <Interceptor
> className="org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor"
> interval="1000" />
> </Channel>
>
> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
> filter=""/>
> <Valve
> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
>
> <!-- Deployer
> className="org.apache.catalina.ha.deploy.FarmWarDeployer"
> tempDir="/tmp/war-temp/"
> deployDir="/tmp/war-deploy/"
> watchDir="/tmp/war-listen/"
> watchEnabled="false"/ -->
>
> <ClusterListener
> className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
> <ClusterListener
> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
> </Cluster>
> ...
>
> Tomcat conf Node 2:
> ...
> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
> channelSendOptions="8">
>
> <Manager className="org.apache.catalina.ha.session.DeltaManager"
> expireSessionsOnShutdown="false"
> notifyListenersOnReplication="true"/>
>
> <Channel
> className="org.apache.catalina.tribes.group.GroupChannel">
> <Membership
> className="org.apache.catalina.tribes.membership.McastService"
> address="239.252.100.12"
> port="14022"
> frequency="1000"
> dropTime="30000"/>
> <Receiver
> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
> address="223.99.114.20"
> port="14022"
> autoBind="100"
> selectorTimeout="5000"
> maxThreads="6"/>
>
> <Sender
> className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
> <Transport
> className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
> timeout="60000" maxRetryAttempts="10" keepAliveCount="0" />
> </Sender>
> <Interceptor
> className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
> <Interceptor
> className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
> <Interceptor
> className="org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor"
> interval="1000" />
> </Channel>
>
> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
> filter=""/>
> <Valve
> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
>
> <!-- Deployer
> className="org.apache.catalina.ha.deploy.FarmWarDeployer"
> tempDir="/tmp/war-temp/"
> deployDir="/tmp/war-deploy/"
> watchDir="/tmp/war-listen/"
> watchEnabled="false"/ -->
>
> <ClusterListener
> className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
> <ClusterListener
> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
> </Cluster>
> ...
>
> Log On Node 1:
>
> ...
> [CLUSTERLOG] main - INFO - 02/12/2008 10:05:15,875 - McastService - Sleeping
> for 2000 milliseconds to establish cluster membership, start level:8
> [CLUSTERLOG] pool-2-thread-1 - INFO - 02/12/2008 10:05:15,908 - BufferPool -
> Created a buffer pool with max size:104857600 bytes of
> type:org.apache.catalina
> .tribes.io.BufferPool15Impl
> [CLUSTERLOG] main - INFO - 02/12/2008 10:05:17,877 - McastService - Done
> sleeping, membership established, start level:8
> [CLUSTERLOG] main - INFO - 02/12/2008 10:05:17,884 - JvmRouteBinderValve -
> JvmRouteBinderValve started
> [CLUSTERLOG] pool-2-thread-3 - WARN - 02/12/2008 10:05:19,328 -
> ClusterListener - Context manager doesn't exist:
> [CLUSTERLOG] main - INFO - 02/12/2008 10:05:20,015 - DeltaManager - Register
> manager to cluster element Host with name localhost
> [CLUSTERLOG] main - INFO - 02/12/2008 10:05:20,015 - DeltaManager - Starting
> clustering manager at
> [CLUSTERLOG] main - WARN - 02/12/2008 10:05:20,026 - DeltaManager - Manager
> [], requesting session state from
> org.apache.catalina.tribes.membership.MemberIm
> pl[tcp://223.99.114.21:14022,223.99.114.21,14022, alive=167423,id={10 -86 53
> 5 6 -125 71 -7 -109 5 -36 4 48 -46 45 65 }, payload={}, command={},
> domain={},
> ]. This operation will timeout if no session state has been received within
> 60 seconds.
> [CLUSTERLOG] pool-1-thread-1 - INFO - 02/12/2008 10:05:20,045 -
> ThroughputInterceptor - ThroughputInterceptor Report[
> Tx Msg:1 messages
> Sent:0,00 MB (total)
> Sent:0,00 MB (application)
> Time:0,02 seconds
> Tx Speed:0,03 MB/sec (total)
> TxSpeed:0,03 MB/sec (application)
> Error Msg:0
> Rx Msg:2 messages
> Rx Speed:0,00 MB/sec (since 1st msg)
> Received:0,00 MB]
> ...
> after some time we're getting this entry:
>
> [CLUSTERLOG] pool-2-thread-4 - WARN - 02/12/2008 15:00:52,044 -
> ClusterListener - Context manager doesn't exist:
> [CLUSTERLOG] pool-2-thread-5 - WARN - 02/12/2008 15:00:52,910 -
> ClusterListener - Context manager doesn't exist:
> [CLUSTERLOG] pool-2-thread-2 - WARN - 02/12/2008 15:00:54,299 -
> ClusterListener - Context manager doesn't exist:
> [CLUSTERLOG] pool-2-thread-6 - WARN - 02/12/2008 15:00:54,707 -
> ClusterListener - Context manager doesn't exist:
> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] - INFO -
> 02/12/2008 15:00:56,011 - DeltaManager - Register manager to cluster
> element H
> ost with name localhost
> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] - INFO -
> 02/12/2008 15:00:56,011 - DeltaManager - Starting clustering manager at
> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] - WARN -
> 02/12/2008 15:00:56,012 - DeltaManager - Manager [], requesting session
> state f
> rom
> org.apache.catalina.tribes.membership.MemberImpl[tcp://223.99.114.21:14022,223.99.114.21,14022,
> alive=17902419,id={10 -86 53 5 6 -125 71 -7 -109 5 -36 4
> 48 -46 45 65 }, payload={}, command={}, domain={}, ]. This operation will
> timeout if no session state has been received within 60 seconds.
> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] - INFO -
> 02/12/2008 15:01:02,245 - DeltaManager - Manager []; session state send at
> 12.0
> 2.08 15:00 received in 6.233 ms.
> [CLUSTERLOG] pool-2-thread-2 - INFO - 02/12/2008 15:01:13,178 -
> ThroughputInterceptor - ThroughputInterceptor Report[
> Tx Msg:20003 messages
> Sent:143,22 MB (total)
> Sent:143,22 MB (application)
> Time:54,66 seconds
> Tx Speed:2,62 MB/sec (total)
> TxSpeed:2,62 MB/sec (application)
> Error Msg:0
> Rx Msg:20000 messages
> Rx Speed:0,01 MB/sec (since 1st msg)
> Received:143,04 MB]
>
> [CLUSTERLOG] pool-2-thread-5 - INFO - 02/12/2008 15:23:49,278 -
> ThroughputInterceptor - ThroughputInterceptor Report[
> Tx Msg:20606 messages
> Sent:156,13 MB (total)
> Sent:156,13 MB (application)
> Time:61,55 seconds
> Tx Speed:2,54 MB/sec (total)
> TxSpeed:2,54 MB/sec (application)
> Error Msg:9
> Rx Msg:21000 messages
> Rx Speed:0,01 MB/sec (since 1st msg)
> Received:150,14 MB]
>
> and now we're receiving and sending message but also Error messages....
> and then:
>
> [CLUSTERLOG] NioReceiver - WARN - 02/12/2008 17:34:23,895 - NioReceiver -
> Channel key is registered, but has had no interest ops for the last 3000 ms.
> (canc
> elled:false):sun.nio.ch.SelectionKeyImpl@e161d6 last access:2008-02-12
> 17:34:17.957
> and finally:
>
> [CLUSTERLOG] 4564 - INFO - 02/12/2008 22:04:28,652 - TcpFailureDetector -
> Received
> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://
> 223.99.114.21:14022,223.99.114.21,14022, alive=43315450,id={10 -86 53 5 6
> -125 71 -7 -109 5 -36 4 48 -46 45 65 }, payload={}, command={}, domain={},
> ]] mess
> age. Will verify.
> [CLUSTERLOG] 4564 - INFO - 02/12/2008 22:04:29,654 - TcpFailureDetector -
> Verification complete. Member
> disappeared[org.apache.catalina.tribes.membership.Me
> mberImpl[tcp://223.99.114.21:14022,223.99.114.21,14022,
> alive=43316452,id={10 -86 53 5 6 -125 71 -7 -109 5 -36 4 48 -46 45 65 },
> payload={}, command={}, dom
> ain={}, ]]
> [CLUSTERLOG] 4564 - INFO - 02/12/2008 22:04:29,654 - SimpleTcpCluster -
> Received member
> disappeared:org.apache.catalina.tribes.membership.MemberImpl[tcp://2
> 23.99.114.21:14022,223.99.114.21,14022, alive=43316452,id={10 -86 53 5 6
> -125 71 -7 -109 5 -36 4 48 -46 45 65 }, payload={}, command={}, domain={}, ]
> [CLUSTERLOG] 4564 - ERROR - 02/12/2008 22:04:29,654 - SimpleTcpCluster -
> Unable to send message through cluster sender.
> org.apache.catalina.tribes.ChannelException: Operation has timed out(60000
> ms.).; Faulty members:tcp://223.99.114.21:14022;
> at
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:97)
> at
> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
> at
> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80)
> at
> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78)
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
> at
> org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor.sendMessage(ThroughputInterceptor.java:61)
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
> at
> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:60)
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
> at
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87)
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
> at
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
> at
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
> at
> org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:835)
> at
> org.apache.catalina.ha.tcp.SimpleTcpCluster.sendClusterDomain(SimpleTcpCluster.java:814)
> at
> org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:551)
> at
> org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:535)
> at
> org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:517)
> at
> org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:428)
> at
> org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:362)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:261)
> at
> org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
> at
> org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283)
> at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
> at
> org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
> at
> org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
> at
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:686)
> at java.lang.Thread.run(Thread.java:595)
> [CLUSTERLOG] 4575 - ERROR - 02/12/2008 22:04:34,920 - SimpleTcpCluster -
> Unable to send message through cluster sender.
> org.apache.catalina.tribes.ChannelException: No destination given; No faulty
> members identified.
> at
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:194)
> at
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
> ...
> the error messages are repeating...
>
>
> Log On Node 2:
>
> [CLUSTERLOG] main - WARN - 02/12/2008 10:02:35,175 - SimpleTcpCluster -
> Dynamic setProperty(className,value) has been disabled, please use explicit
> properties
> for the element you are trying to identify
> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:35,954 - SimpleTcpCluster -
> Cluster is about to start
> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:35,973 - ReceiverBase - Receiver
> Server Socket bound to:/223.99.114.21:14022
> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:36,025 - McastService - Setting
> cluster mcast soTimeout to 1000
> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:36,037 - McastService - Sleeping
> for 2000 milliseconds to establish cluster membership, start level:4
> [CLUSTERLOG] Thread-3 - INFO - 02/12/2008 10:02:36,913 - SimpleTcpCluster -
> Replication member
> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://22
> 3.99.114.20:14022,223.99.114.20,14022, alive=322404622,id={109 -10 62 -21 19
> 102 69 -42 -92 -123 -59 -15 37 16 50 -5 }, payload={}, command={},
> domain={}, ]
> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:38,039 - McastService - Done
> sleeping, membership established, start level:4
> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:38,044 - McastService - Sleeping
> for 2000 milliseconds to establish cluster membership, start level:8
> [CLUSTERLOG] pool-2-thread-1 - INFO - 02/12/2008 10:02:38,074 - BufferPool -
> Created a buffer pool with max size:104857600 bytes of
> type:org.apache.catalina.t
> ribes.io.BufferPool15Impl
> [CLUSTERLOG] pool-2-thread-3 - WARN - 02/12/2008 10:02:38,312 -
> ClusterListener - Context manager doesn't exist:
> [CLUSTERLOG] pool-2-thread-4 - WARN - 02/12/2008 10:02:38,816 -
> ClusterListener - Context manager doesn't exist:
> [CLUSTERLOG] pool-2-thread-5 - WARN - 02/12/2008 10:02:39,277 -
> ClusterListener - Context manager doesn't exist:
> [CLUSTERLOG] pool-2-thread-6 - WARN - 02/12/2008 10:02:39,337 -
> ClusterListener - Context manager doesn't exist:
> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:40,046 - McastService - Done
> sleeping, membership established, start level:8
> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:40,053 - JvmRouteBinderValve -
> JvmRouteBinderValve started
> [CLUSTERLOG] pool-2-thread-1 - WARN - 02/12/2008 10:02:42,816 -
> ClusterListener - Context manager doesn't exist:
> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:43,093 - DeltaManager - Register
> manager to cluster element Host with name localhost
> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:43,093 - DeltaManager - Starting
> clustering manager at
> [CLUSTERLOG] main - WARN - 02/12/2008 10:02:43,100 - DeltaManager - Manager
> [], requesting session state from
> org.apache.catalina.tribes.membership.MemberImpl
> [tcp://223.99.114.20:14022,223.99.114.20,14022, alive=322410633,id={109 -10
> 62 -21 19 102 69 -42 -92 -123 -59 -15 37 16 50 -5 }, payload={}, command={},
> domai
> n={}, ]. This operation will timeout if no session state has been received
> within 60 seconds.
> [CLUSTERLOG] pool-1-thread-1 - INFO - 02/12/2008 10:02:43,112 -
> ThroughputInterceptor - ThroughputInterceptor Report[
> Tx Msg:1 messages
> Sent:0,00 MB (total)
> Sent:0,00 MB (application)
> Time:0,01 seconds
> Tx Speed:0,04 MB/sec (total)
> TxSpeed:0,04 MB/sec (application)
> Error Msg:0
> Rx Msg:6 messages
> Rx Speed:0,00 MB/sec (since 1st msg)
> Received:0,01 MB]
>
> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:43,719 - DeltaManager - Manager
> []; session state send at 12.02.08 10:02 received in 624 ms.
> [CLUSTERLOG] Thread-31 - INFO - 02/12/2008 10:04:55,941 - TcpFailureDetector
> - Verification complete. Member
> disappeared[org.apache.catalina.tribes.membership
> .MemberImpl[tcp://223.99.114.20:14022,223.99.114.20,14022,
> alive=322543666,id={109 -10 62 -21 19 102 69 -42 -92 -123 -59 -15 37 16 50
> -5 }, payload={}, comman
> d={66 65 66 89 45 65 76 69 88 ...(9)}, domain={}, ]]
> [CLUSTERLOG] Thread-30 - INFO - 02/12/2008 10:04:55,941 - TcpFailureDetector
> - Verification complete. Member
> disappeared[org.apache.catalina.tribes.membership
> .MemberImpl[tcp://223.99.114.20:14022,223.99.114.20,14022,
> alive=322543666,id={109 -10 62 -21 19 102 69 -42 -92 -123 -59 -15 37 16 50
> -5 }, payload={}, comman
> d={66 65 66 89 45 65 76 69 88 ...(9)}, domain={}, ]]
> [CLUSTERLOG] Thread-31 - INFO - 02/12/2008 10:04:55,941 - SimpleTcpCluster -
> Received member
> disappeared:org.apache.catalina.tribes.membership.MemberImpl[tcp:
> //223.99.114.20:14022,223.99.114.20,14022, alive=322543666,id={109 -10 62
> -21 19 102 69 -42 -92 -123 -59 -15 37 16 50 -5 }, payload={}, command={66 65
> 66 89 4
> 5 65 76 69 88 ...(9)}, domain={}, ]
> [CLUSTERLOG] Thread-30 - INFO - 02/12/2008 10:04:55,941 - SimpleTcpCluster -
> Received member
> disappeared:org.apache.catalina.tribes.membership.MemberImpl[tcp:
> //223.99.114.20:14022,223.99.114.20,14022, alive=322543666,id={109 -10 62
> -21 19 102 69 -42 -92 -123 -59 -15 37 16 50 -5 }, payload={}, command={66 65
> 66 89 4
> 5 65 76 69 88 ...(9)}, domain={}, ]
> [CLUSTERLOG] Thread-36 - INFO - 02/12/2008 10:05:19,299 - SimpleTcpCluster -
> Replication member
> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://2
> 23.99.114.20:14022,223.99.114.20,14022, alive=2028,id={19 -115 -1 -101 8 91
> 66 -91 -112 105 -121 109 -82 11 56 50 }, payload={}, command={}, domain={},
> ]
> [CLUSTERLOG] pool-2-thread-4 - INFO - 02/12/2008 10:25:11,742 -
> ThroughputInterceptor - ThroughputInterceptor Report[
> Tx Msg:892 messages
> Sent:5,82 MB (total)
> Sent:5,82 MB (application)
> Time:1,33 seconds
> Tx Speed:4,39 MB/sec (total)
> TxSpeed:4,39 MB/sec (application)
> Error Msg:0
> Rx Msg:1000 messages
> Rx Speed:0,00 MB/sec (since 1st msg)
> Received:4,47 MB]
>
> ...
> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] - INFO -
> 02/12/2008 15:01:16,005 - DeltaManager - Manager [] expiring sessions upon
> shutdo
> wn
> [CLUSTERLOG] pool-2-thread-1 - WARN - 02/12/2008 15:01:20,014 -
> ClusterListener - Context manager doesn't exist:
> [CLUSTERLOG] pool-2-thread-1 - WARN - 02/12/2008 15:01:20,015 -
> ClusterListener - Context manager doesn't exist:
> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] - INFO -
> 02/12/2008 15:01:20,219 - DeltaManager - Register manager to cluster
> element Hos
> t with name localhost
> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] - INFO -
> 02/12/2008 15:01:20,220 - DeltaManager - Starting clustering manager at
> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] - WARN -
> 02/12/2008 15:01:20,220 - DeltaManager - Manager [], requesting session
> state fro
> m
> org.apache.catalina.tribes.membership.MemberImpl[tcp://223.99.114.20:14022,223.99.114.20,14022,
> alive=17762449,id={19 -115 -1 -101 8 91 66 -91 -112 105 -121
> 109 -82 11 56 50 }, payload={}, command={}, domain={}, ]. This operation
> will timeout if no session state has been received within 60 seconds.
> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] - INFO -
> 02/12/2008 15:01:24,093 - DeltaManager - Manager []; session state send at
> 12.02.
> 08 15:01 received in 3.872 ms.
> [CLUSTERLOG] pool-1-thread-1 - INFO - 02/12/2008 15:22:41,143 -
> ThroughputInterceptor - ThroughputInterceptor Report[
> Tx Msg:21000 messages
> Sent:150,30 MB (total)
> Sent:150,30 MB (application)
> Time:70,58 seconds
> Tx Speed:2,13 MB/sec (total)
> TxSpeed:2,13 MB/sec (application)
> Error Msg:0
> Rx Msg:20534 messages
> Rx Speed:0,01 MB/sec (since 1st msg)
> Received:156,92 MB]
>
> after this entry we're NOT receiving any messages (Rx Msg):
>
> [CLUSTERLOG] pool-1-thread-2 - INFO - 02/12/2008 15:40:02,612 -
> ThroughputInterceptor - ThroughputInterceptor Report[
> Tx Msg:22000 messages
> Sent:154,75 MB (total)
> Sent:154,75 MB (application)
> Time:72,92 seconds
> Tx Speed:2,12 MB/sec (total)
> TxSpeed:2,12 MB/sec (application)
> Error Msg:0
> Rx Msg:20534 messages
> Rx Speed:0,01 MB/sec (since 1st msg)
> Received:156,92 MB]
>
> [CLUSTERLOG] pool-1-thread-2 - INFO - 02/12/2008 15:54:15,184 -
> ThroughputInterceptor - ThroughputInterceptor Report[
> Tx Msg:23022 messages
> Sent:156,30 MB (total)
> Sent:156,30 MB (application)
> Time:74,46 seconds
> Tx Speed:2,10 MB/sec (total)
> TxSpeed:2,10 MB/sec (application)
> Error Msg:0
> Rx Msg:20534 messages
> Rx Speed:0,01 MB/sec (since 1st msg)
> Received:156,92 MB]
> ...
>
>
>
> Many thanks for your help!!!
>
>
>
>
>
>
>
>
>
>
>
>
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat Cluster Error -
Posted by Filip Hanik - Dev Lists <de...@hanik.com>.
gp456 wrote:
> thanks for your reply.
> first the fix you mentioned is not in the current tomcat release (6.0.16).
> Since I have to build/patch the tomcat for a productive environment what
> whould you recommend: patch the 6.0.16 sources or build tomcat from the
> newest trunk?
>
either or, they haven't changed that much
Filip
> We've ran another test and encountered a network error (packet got no ACK)
> on the multicast interfaces. The wireshark log (part of) is attached below.
> Perhaps you've an idea why this is happening!?
> Thanks for your help!
> Guido
>
> --------------------- wireshark log start --------------------
> No. Time Source Destination Protocol
> Info
> 37433 1543.747634 223.99.114.20 223.99.114.21 TCP
> 36615 > 14022 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=568786762 TSER=0 WS=2
>
> Frame 37433 (74 bytes on wire, 74 bytes captured)
> Arrival Time: Feb 12, 2008 22:04:28.652917000
> [Time delta from previous captured frame: 0.071229000 seconds]
> [Time delta from previous displayed frame: 1543.747634000 seconds]
> [Time since reference or first frame: 1543.747634000 seconds]
> Frame Number: 37433
> Frame Length: 74 bytes
> Capture Length: 74 bytes
> [Frame is marked: False]
> [Protocols in frame: eth:ip:tcp]
> [Coloring Rule Name: TCP SYN/FIN]
> [Coloring Rule String: tcp.flags & 0x02 || tcp.flags.fin == 1]
> Ethernet II, Src: Broadcom_11:2a:c1 (00:10:18:11:2a:c1), Dst:
> Broadcom_17:7f:8c (00:10:18:17:7f:8c)
> Destination: Broadcom_17:7f:8c (00:10:18:17:7f:8c)
> Address: Broadcom_17:7f:8c (00:10:18:17:7f:8c)
> .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
> .... ..0. .... .... .... .... = LG bit: Globally unique address
> (factory default)
> Source: Broadcom_11:2a:c1 (00:10:18:11:2a:c1)
> Address: Broadcom_11:2a:c1 (00:10:18:11:2a:c1)
> .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
> .... ..0. .... .... .... .... = LG bit: Globally unique address
> (factory default)
> Type: IP (0x0800)
> Internet Protocol, Src: 223.99.114.20 (223.99.114.20), Dst: 223.99.114.21
> (223.99.114.21)
> Version: 4
> Header length: 20 bytes
> Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
> 0000 00.. = Differentiated Services Codepoint: Default (0x00)
> .... ..0. = ECN-Capable Transport (ECT): 0
> .... ...0 = ECN-CE: 0
> Total Length: 60
> Identification: 0x926a (37482)
> Flags: 0x04 (Don't Fragment)
> 0... = Reserved bit: Not set
> .1.. = Don't fragment: Set
> ..0. = More fragments: Not set
> Fragment offset: 0
> Time to live: 64
> Protocol: TCP (0x06)
> Header checksum: 0x0561 [correct]
> [Good: True]
> [Bad : False]
> Source: 223.99.114.20 (223.99.114.20)
> Destination: 223.99.114.21 (223.99.114.21)
> Transmission Control Protocol, Src Port: 36615 (36615), Dst Port: 14022
> (14022), Seq: 0, Len: 0
> Source port: 36615 (36615)
> Destination port: 14022 (14022)
> Sequence number: 0 (relative sequence number)
> Header length: 40 bytes
> Flags: 0x02 (SYN)
> 0... .... = Congestion Window Reduced (CWR): Not set
> .0.. .... = ECN-Echo: Not set
> ..0. .... = Urgent: Not set
> ...0 .... = Acknowledgment: Not set
> .... 0... = Push: Not set
> .... .0.. = Reset: Not set
> .... ..1. = Syn: Set
> .... ...0 = Fin: Not set
> Window size: 5840
> Checksum: 0x7df2 [correct]
> [Good Checksum: True]
> [Bad Checksum: False]
> Options: (20 bytes)
> Maximum segment size: 1460 bytes
> SACK permitted
> Timestamps: TSval 568786762, TSecr 0
> NOP
> Window scale: 2 (multiply by 4)
>
> No. Time Source Destination Protocol
> Info
> 137308 1547.148341 223.99.114.20 223.99.114.21 TCP
> 36615 > 14022 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=568786762 TSER=0 WS=2
>
> Frame 137308 (74 bytes on wire, 74 bytes captured)
> Arrival Time: Feb 12, 2008 22:04:32.053624000
> [Time delta from previous captured frame: 0.071236000 seconds]
> [Time delta from previous displayed frame: 3.400707000 seconds]
> [Time since reference or first frame: 1547.148341000 seconds]
> Frame Number: 137308
> Frame Length: 74 bytes
> Capture Length: 74 bytes
> [Frame is marked: False]
> [Protocols in frame: eth:ip:tcp]
> [Coloring Rule Name: TCP SYN/FIN]
> [Coloring Rule String: tcp.flags & 0x02 || tcp.flags.fin == 1]
> Ethernet II, Src: Broadcom_11:2a:c1 (00:10:18:11:2a:c1), Dst:
> Broadcom_17:7f:8c (00:10:18:17:7f:8c)
> Destination: Broadcom_17:7f:8c (00:10:18:17:7f:8c)
> Address: Broadcom_17:7f:8c (00:10:18:17:7f:8c)
> .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
> .... ..0. .... .... .... .... = LG bit: Globally unique address
> (factory default)
> Source: Broadcom_11:2a:c1 (00:10:18:11:2a:c1)
> Address: Broadcom_11:2a:c1 (00:10:18:11:2a:c1)
> .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
> .... ..0. .... .... .... .... = LG bit: Globally unique address
> (factory default)
> Type: IP (0x0800)
> Internet Protocol, Src: 223.99.114.20 (223.99.114.20), Dst: 223.99.114.21
> (223.99.114.21)
> Version: 4
> Header length: 20 bytes
> Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
> 0000 00.. = Differentiated Services Codepoint: Default (0x00)
> .... ..0. = ECN-Capable Transport (ECT): 0
> .... ...0 = ECN-CE: 0
> Total Length: 60
> Identification: 0x926a (37482)
> Flags: 0x04 (Don't Fragment)
> 0... = Reserved bit: Not set
> .1.. = Don't fragment: Set
> ..0. = More fragments: Not set
> Fragment offset: 0
> Time to live: 64
> Protocol: TCP (0x06)
> Header checksum: 0x0561 [correct]
> [Good: True]
> [Bad : False]
> Source: 223.99.114.20 (223.99.114.20)
> Destination: 223.99.114.21 (223.99.114.21)
> Transmission Control Protocol, Src Port: 36615 (36615), Dst Port: 14022
> (14022), Seq: 0, Len: 0
> Source port: 36615 (36615)
> Destination port: 14022 (14022)
> Sequence number: 0 (relative sequence number)
> Header length: 40 bytes
> Flags: 0x02 (SYN)
> 0... .... = Congestion Window Reduced (CWR): Not set
> .0.. .... = ECN-Echo: Not set
> ..0. .... = Urgent: Not set
> ...0 .... = Acknowledgment: Not set
> .... 0... = Push: Not set
> .... .0.. = Reset: Not set
> .... ..1. = Syn: Set
> .... ...0 = Fin: Not set
> Window size: 5840
> Checksum: 0x7df2 [correct]
> [Good Checksum: True]
> [Bad Checksum: False]
> Options: (20 bytes)
> Maximum segment size: 1460 bytes
> SACK permitted
> Timestamps: TSval 568786762, TSecr 0
> NOP
> Window scale: 2 (multiply by 4)
>
> No. Time Source Destination Protocol
> Info
> 37451 1544.749177 223.99.114.20 223.99.114.21 TCP
> 36616 > 14022 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=568787763 TSER=0 WS=2
>
> Frame 37451 (74 bytes on wire, 74 bytes captured)
> Arrival Time: Feb 12, 2008 22:04:29.654460000
> [Time delta from previous captured frame: 0.076938000 seconds]
> [Time delta from previous displayed frame: 1544.749177000 seconds]
> [Time since reference or first frame: 1544.749177000 seconds]
> Frame Number: 37451
> Frame Length: 74 bytes
> Capture Length: 74 bytes
> [Frame is marked: False]
> [Protocols in frame: eth:ip:tcp]
> [Coloring Rule Name: TCP SYN/FIN]
> [Coloring Rule String: tcp.flags & 0x02 || tcp.flags.fin == 1]
> Ethernet II, Src: Broadcom_11:2a:c1 (00:10:18:11:2a:c1), Dst:
> Broadcom_17:7f:8c (00:10:18:17:7f:8c)
> Destination: Broadcom_17:7f:8c (00:10:18:17:7f:8c)
> Address: Broadcom_17:7f:8c (00:10:18:17:7f:8c)
> .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
> .... ..0. .... .... .... .... = LG bit: Globally unique address
> (factory default)
> Source: Broadcom_11:2a:c1 (00:10:18:11:2a:c1)
> Address: Broadcom_11:2a:c1 (00:10:18:11:2a:c1)
> .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
> .... ..0. .... .... .... .... = LG bit: Globally unique address
> (factory default)
> Type: IP (0x0800)
> Internet Protocol, Src: 223.99.114.20 (223.99.114.20), Dst: 223.99.114.21
> (223.99.114.21)
> Version: 4
> Header length: 20 bytes
> Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
> 0000 00.. = Differentiated Services Codepoint: Default (0x00)
> .... ..0. = ECN-Capable Transport (ECT): 0
> .... ...0 = ECN-CE: 0
> Total Length: 60
> Identification: 0xbb3b (47931)
> Flags: 0x04 (Don't Fragment)
> 0... = Reserved bit: Not set
> .1.. = Don't fragment: Set
> ..0. = More fragments: Not set
> Fragment offset: 0
> Time to live: 64
> Protocol: TCP (0x06)
> Header checksum: 0xdc8f [correct]
> [Good: True]
> [Bad : False]
> Source: 223.99.114.20 (223.99.114.20)
> Destination: 223.99.114.21 (223.99.114.21)
> Transmission Control Protocol, Src Port: 36616 (36616), Dst Port: 14022
> (14022), Seq: 0, Len: 0
> Source port: 36616 (36616)
> Destination port: 14022 (14022)
> Sequence number: 0 (relative sequence number)
> Header length: 40 bytes
> Flags: 0x02 (SYN)
> 0... .... = Congestion Window Reduced (CWR): Not set
> .0.. .... = ECN-Echo: Not set
> ..0. .... = Urgent: Not set
> ...0 .... = Acknowledgment: Not set
> .... 0... = Push: Not set
> .... .0.. = Reset: Not set
> .... ..1. = Syn: Set
> .... ...0 = Fin: Not set
> Window size: 5840
> Checksum: 0x2e75 [correct]
> [Good Checksum: True]
> [Bad Checksum: False]
> Options: (20 bytes)
> Maximum segment size: 1460 bytes
> SACK permitted
> Timestamps: TSval 568787763, TSecr 0
> NOP
> Window scale: 2 (multiply by 4)
>
> No. Time Source Destination Protocol
> Info
> 137326 1548.149884 223.99.114.20 223.99.114.21 TCP
> 36616 > 14022 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=568787763 TSER=0 WS=2
>
> Frame 137326 (74 bytes on wire, 74 bytes captured)
> Arrival Time: Feb 12, 2008 22:04:33.055167000
> [Time delta from previous captured frame: 0.077082000 seconds]
> [Time delta from previous displayed frame: 3.400707000 seconds]
> [Time since reference or first frame: 1548.149884000 seconds]
> Frame Number: 137326
> Frame Length: 74 bytes
> Capture Length: 74 bytes
> [Frame is marked: False]
> [Protocols in frame: eth:ip:tcp]
> [Coloring Rule Name: TCP SYN/FIN]
> [Coloring Rule String: tcp.flags & 0x02 || tcp.flags.fin == 1]
> Ethernet II, Src: Broadcom_11:2a:c1 (00:10:18:11:2a:c1), Dst:
> Broadcom_17:7f:8c (00:10:18:17:7f:8c)
> Destination: Broadcom_17:7f:8c (00:10:18:17:7f:8c)
> Address: Broadcom_17:7f:8c (00:10:18:17:7f:8c)
> .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
> .... ..0. .... .... .... .... = LG bit: Globally unique address
> (factory default)
> Source: Broadcom_11:2a:c1 (00:10:18:11:2a:c1)
> Address: Broadcom_11:2a:c1 (00:10:18:11:2a:c1)
> .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
> .... ..0. .... .... .... .... = LG bit: Globally unique address
> (factory default)
> Type: IP (0x0800)
> Internet Protocol, Src: 223.99.114.20 (223.99.114.20), Dst: 223.99.114.21
> (223.99.114.21)
> Version: 4
> Header length: 20 bytes
> Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
> 0000 00.. = Differentiated Services Codepoint: Default (0x00)
> .... ..0. = ECN-Capable Transport (ECT): 0
> .... ...0 = ECN-CE: 0
> Total Length: 60
> Identification: 0xbb3b (47931)
> Flags: 0x04 (Don't Fragment)
> 0... = Reserved bit: Not set
> .1.. = Don't fragment: Set
> ..0. = More fragments: Not set
> Fragment offset: 0
> Time to live: 64
> Protocol: TCP (0x06)
> Header checksum: 0xdc8f [correct]
> [Good: True]
> [Bad : False]
> Source: 223.99.114.20 (223.99.114.20)
> Destination: 223.99.114.21 (223.99.114.21)
> Transmission Control Protocol, Src Port: 36616 (36616), Dst Port: 14022
> (14022), Seq: 0, Len: 0
> Source port: 36616 (36616)
> Destination port: 14022 (14022)
> Sequence number: 0 (relative sequence number)
> Header length: 40 bytes
> Flags: 0x02 (SYN)
> 0... .... = Congestion Window Reduced (CWR): Not set
> .0.. .... = ECN-Echo: Not set
> ..0. .... = Urgent: Not set
> ...0 .... = Acknowledgment: Not set
> .... 0... = Push: Not set
> .... .0.. = Reset: Not set
> .... ..1. = Syn: Set
> .... ...0 = Fin: Not set
> Window size: 5840
> Checksum: 0x2e75 [correct]
> [Good Checksum: True]
> [Bad Checksum: False]
> Options: (20 bytes)
> Maximum segment size: 1460 bytes
> SACK permitted
> Timestamps: TSval 568787763, TSecr 0
> NOP
> Window scale: 2 (multiply by 4)
> --------------------- wireshark log end --------------------
>
>
> Filip Hanik - Dev Lists wrote:
>
>> [CLUSTERLOG] NioReceiver - WARN - 02/12/2008 17:34:23,895 - NioReceiver -
>> Channel key is registered, but has had no interest ops for the last 3000
>> ms.
>> (canc
>> elled:false):sun.nio.ch.SelectionKeyImpl@e161d6 last access:2008-02-12
>> 17:34:17.957
>> and finally:
>>
>> this message most likely indicates that the thread pool handling
>> incoming request is busy.
>> at this point do a thread dump to see the state of the threads.
>> There was a deadlock fix in relation to Raul's error, and was fixed in
>> this SVN commit
>> http://svn.apache.org/viewvc?view=rev&revision=618823
>>
>> it has been proposed to be merged back into the 6.0.x branch
>>
>> Filip
>>
>> gp456 wrote:
>>
>>> Hello,
>>>
>>> we have a similar problem as Raul Garcia has posted in
>>> http://www.nabble.com/Tomcat-6---Cluster-error.-to14921385.html#a14921385
>>> this message, but we're receiving the error of a disappearing member
>>> after
>>> appr. 12 hours!!
>>> Our load is also similar: 1 message/sec.
>>>
>>> We are using a hardware loadbalancer in front of two apache/tomcat pairs.
>>> The apache are connected via ajp
>>> to the local tomcat or (if the local tomcat is down) to the tomcat on the
>>> other node.
>>> Our config:
>>> Node1/2:
>>> Apache 2.0.59
>>> Tomcat 6.0.13
>>>
>>> Tomcat conf Node 1:
>>>
>>> ...
>>> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
>>> channelSendOptions="8">
>>>
>>> <Manager
>>> className="org.apache.catalina.ha.session.DeltaManager"
>>> expireSessionsOnShutdown="false"
>>> notifyListenersOnReplication="true"/>
>>>
>>> <Channel
>>> className="org.apache.catalina.tribes.group.GroupChannel">
>>> <Membership
>>> className="org.apache.catalina.tribes.membership.McastService"
>>> address="239.252.100.12"
>>> port="14022"
>>> frequency="1000"
>>> dropTime="30000"/>
>>> <Receiver
>>> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
>>> address="223.99.114.21"
>>> port="14022"
>>> autoBind="100"
>>> selectorTimeout="5000"
>>> maxThreads="6"/>
>>>
>>> <Sender
>>> className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
>>> <Transport
>>> className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
>>> timeout="60000" maxRetryAttempts="10" keepAliveCount="0" />
>>> </Sender>
>>> <Interceptor
>>> className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
>>> <Interceptor
>>> className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
>>> <Interceptor
>>> className="org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor"
>>> interval="1000" />
>>> </Channel>
>>>
>>> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
>>> filter=""/>
>>> <Valve
>>> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
>>>
>>> <!-- Deployer
>>> className="org.apache.catalina.ha.deploy.FarmWarDeployer"
>>> tempDir="/tmp/war-temp/"
>>> deployDir="/tmp/war-deploy/"
>>> watchDir="/tmp/war-listen/"
>>> watchEnabled="false"/ -->
>>>
>>> <ClusterListener
>>> className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
>>> <ClusterListener
>>> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
>>> </Cluster>
>>> ...
>>>
>>> Tomcat conf Node 2:
>>> ...
>>> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
>>> channelSendOptions="8">
>>>
>>> <Manager
>>> className="org.apache.catalina.ha.session.DeltaManager"
>>> expireSessionsOnShutdown="false"
>>> notifyListenersOnReplication="true"/>
>>>
>>> <Channel
>>> className="org.apache.catalina.tribes.group.GroupChannel">
>>> <Membership
>>> className="org.apache.catalina.tribes.membership.McastService"
>>> address="239.252.100.12"
>>> port="14022"
>>> frequency="1000"
>>> dropTime="30000"/>
>>> <Receiver
>>> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
>>> address="223.99.114.20"
>>> port="14022"
>>> autoBind="100"
>>> selectorTimeout="5000"
>>> maxThreads="6"/>
>>>
>>> <Sender
>>> className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
>>> <Transport
>>> className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
>>> timeout="60000" maxRetryAttempts="10" keepAliveCount="0" />
>>> </Sender>
>>> <Interceptor
>>> className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
>>> <Interceptor
>>> className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
>>> <Interceptor
>>> className="org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor"
>>> interval="1000" />
>>> </Channel>
>>>
>>> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
>>> filter=""/>
>>> <Valve
>>> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
>>>
>>> <!-- Deployer
>>> className="org.apache.catalina.ha.deploy.FarmWarDeployer"
>>> tempDir="/tmp/war-temp/"
>>> deployDir="/tmp/war-deploy/"
>>> watchDir="/tmp/war-listen/"
>>> watchEnabled="false"/ -->
>>>
>>> <ClusterListener
>>> className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
>>> <ClusterListener
>>> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
>>> </Cluster>
>>> ...
>>>
>>> Log On Node 1:
>>>
>>> ...
>>> [CLUSTERLOG] main - INFO - 02/12/2008 10:05:15,875 - McastService -
>>> Sleeping
>>> for 2000 milliseconds to establish cluster membership, start level:8
>>> [CLUSTERLOG] pool-2-thread-1 - INFO - 02/12/2008 10:05:15,908 -
>>> BufferPool -
>>> Created a buffer pool with max size:104857600 bytes of
>>> type:org.apache.catalina
>>> .tribes.io.BufferPool15Impl
>>> [CLUSTERLOG] main - INFO - 02/12/2008 10:05:17,877 - McastService - Done
>>> sleeping, membership established, start level:8
>>> [CLUSTERLOG] main - INFO - 02/12/2008 10:05:17,884 - JvmRouteBinderValve
>>> -
>>> JvmRouteBinderValve started
>>> [CLUSTERLOG] pool-2-thread-3 - WARN - 02/12/2008 10:05:19,328 -
>>> ClusterListener - Context manager doesn't exist:
>>> [CLUSTERLOG] main - INFO - 02/12/2008 10:05:20,015 - DeltaManager -
>>> Register
>>> manager to cluster element Host with name localhost
>>> [CLUSTERLOG] main - INFO - 02/12/2008 10:05:20,015 - DeltaManager -
>>> Starting
>>> clustering manager at
>>> [CLUSTERLOG] main - WARN - 02/12/2008 10:05:20,026 - DeltaManager -
>>> Manager
>>> [], requesting session state from
>>> org.apache.catalina.tribes.membership.MemberIm
>>> pl[tcp://223.99.114.21:14022,223.99.114.21,14022, alive=167423,id={10 -86
>>> 53
>>> 5 6 -125 71 -7 -109 5 -36 4 48 -46 45 65 }, payload={}, command={},
>>> domain={},
>>> ]. This operation will timeout if no session state has been received
>>> within
>>> 60 seconds.
>>> [CLUSTERLOG] pool-1-thread-1 - INFO - 02/12/2008 10:05:20,045 -
>>> ThroughputInterceptor - ThroughputInterceptor Report[
>>> Tx Msg:1 messages
>>> Sent:0,00 MB (total)
>>> Sent:0,00 MB (application)
>>> Time:0,02 seconds
>>> Tx Speed:0,03 MB/sec (total)
>>> TxSpeed:0,03 MB/sec (application)
>>> Error Msg:0
>>> Rx Msg:2 messages
>>> Rx Speed:0,00 MB/sec (since 1st msg)
>>> Received:0,00 MB]
>>> ...
>>> after some time we're getting this entry:
>>>
>>> [CLUSTERLOG] pool-2-thread-4 - WARN - 02/12/2008 15:00:52,044 -
>>> ClusterListener - Context manager doesn't exist:
>>> [CLUSTERLOG] pool-2-thread-5 - WARN - 02/12/2008 15:00:52,910 -
>>> ClusterListener - Context manager doesn't exist:
>>> [CLUSTERLOG] pool-2-thread-2 - WARN - 02/12/2008 15:00:54,299 -
>>> ClusterListener - Context manager doesn't exist:
>>> [CLUSTERLOG] pool-2-thread-6 - WARN - 02/12/2008 15:00:54,707 -
>>> ClusterListener - Context manager doesn't exist:
>>> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] -
>>> INFO -
>>> 02/12/2008 15:00:56,011 - DeltaManager - Register manager to cluster
>>> element H
>>> ost with name localhost
>>> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] -
>>> INFO -
>>> 02/12/2008 15:00:56,011 - DeltaManager - Starting clustering manager at
>>> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] -
>>> WARN -
>>> 02/12/2008 15:00:56,012 - DeltaManager - Manager [], requesting session
>>> state f
>>> rom
>>> org.apache.catalina.tribes.membership.MemberImpl[tcp://223.99.114.21:14022,223.99.114.21,14022,
>>> alive=17902419,id={10 -86 53 5 6 -125 71 -7 -109 5 -36 4
>>> 48 -46 45 65 }, payload={}, command={}, domain={}, ]. This operation
>>> will
>>> timeout if no session state has been received within 60 seconds.
>>> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] -
>>> INFO -
>>> 02/12/2008 15:01:02,245 - DeltaManager - Manager []; session state send
>>> at
>>> 12.0
>>> 2.08 15:00 received in 6.233 ms.
>>> [CLUSTERLOG] pool-2-thread-2 - INFO - 02/12/2008 15:01:13,178 -
>>> ThroughputInterceptor - ThroughputInterceptor Report[
>>> Tx Msg:20003 messages
>>> Sent:143,22 MB (total)
>>> Sent:143,22 MB (application)
>>> Time:54,66 seconds
>>> Tx Speed:2,62 MB/sec (total)
>>> TxSpeed:2,62 MB/sec (application)
>>> Error Msg:0
>>> Rx Msg:20000 messages
>>> Rx Speed:0,01 MB/sec (since 1st msg)
>>> Received:143,04 MB]
>>>
>>> [CLUSTERLOG] pool-2-thread-5 - INFO - 02/12/2008 15:23:49,278 -
>>> ThroughputInterceptor - ThroughputInterceptor Report[
>>> Tx Msg:20606 messages
>>> Sent:156,13 MB (total)
>>> Sent:156,13 MB (application)
>>> Time:61,55 seconds
>>> Tx Speed:2,54 MB/sec (total)
>>> TxSpeed:2,54 MB/sec (application)
>>> Error Msg:9
>>> Rx Msg:21000 messages
>>> Rx Speed:0,01 MB/sec (since 1st msg)
>>> Received:150,14 MB]
>>>
>>> and now we're receiving and sending message but also Error messages....
>>> and then:
>>>
>>> [CLUSTERLOG] NioReceiver - WARN - 02/12/2008 17:34:23,895 - NioReceiver -
>>> Channel key is registered, but has had no interest ops for the last 3000
>>> ms.
>>> (canc
>>> elled:false):sun.nio.ch.SelectionKeyImpl@e161d6 last access:2008-02-12
>>> 17:34:17.957
>>> and finally:
>>>
>>> [CLUSTERLOG] 4564 - INFO - 02/12/2008 22:04:28,652 - TcpFailureDetector -
>>> Received
>>> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://
>>> 223.99.114.21:14022,223.99.114.21,14022, alive=43315450,id={10 -86 53 5 6
>>> -125 71 -7 -109 5 -36 4 48 -46 45 65 }, payload={}, command={},
>>> domain={},
>>> ]] mess
>>> age. Will verify.
>>> [CLUSTERLOG] 4564 - INFO - 02/12/2008 22:04:29,654 - TcpFailureDetector -
>>> Verification complete. Member
>>> disappeared[org.apache.catalina.tribes.membership.Me
>>> mberImpl[tcp://223.99.114.21:14022,223.99.114.21,14022,
>>> alive=43316452,id={10 -86 53 5 6 -125 71 -7 -109 5 -36 4 48 -46 45 65 },
>>> payload={}, command={}, dom
>>> ain={}, ]]
>>> [CLUSTERLOG] 4564 - INFO - 02/12/2008 22:04:29,654 - SimpleTcpCluster -
>>> Received member
>>> disappeared:org.apache.catalina.tribes.membership.MemberImpl[tcp://2
>>> 23.99.114.21:14022,223.99.114.21,14022, alive=43316452,id={10 -86 53 5 6
>>> -125 71 -7 -109 5 -36 4 48 -46 45 65 }, payload={}, command={},
>>> domain={}, ]
>>> [CLUSTERLOG] 4564 - ERROR - 02/12/2008 22:04:29,654 - SimpleTcpCluster -
>>> Unable to send message through cluster sender.
>>> org.apache.catalina.tribes.ChannelException: Operation has timed
>>> out(60000
>>> ms.).; Faulty members:tcp://223.99.114.21:14022;
>>> at
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:97)
>>> at
>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
>>> at
>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80)
>>> at
>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78)
>>> at
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>>> at
>>> org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor.sendMessage(ThroughputInterceptor.java:61)
>>> at
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>>> at
>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:60)
>>> at
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>>> at
>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87)
>>> at
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>>> at
>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
>>> at
>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
>>> at
>>> org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:835)
>>> at
>>> org.apache.catalina.ha.tcp.SimpleTcpCluster.sendClusterDomain(SimpleTcpCluster.java:814)
>>> at
>>> org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:551)
>>> at
>>> org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:535)
>>> at
>>> org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:517)
>>> at
>>> org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:428)
>>> at
>>> org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:362)
>>> at
>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
>>> at
>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>>> at
>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:261)
>>> at
>>> org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
>>> at
>>> org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283)
>>> at
>>> org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
>>> at
>>> org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
>>> at
>>> org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
>>> at
>>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:686)
>>> at java.lang.Thread.run(Thread.java:595)
>>> [CLUSTERLOG] 4575 - ERROR - 02/12/2008 22:04:34,920 - SimpleTcpCluster -
>>> Unable to send message through cluster sender.
>>> org.apache.catalina.tribes.ChannelException: No destination given; No
>>> faulty
>>> members identified.
>>> at
>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:194)
>>> at
>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
>>> ...
>>> the error messages are repeating...
>>>
>>>
>>> Log On Node 2:
>>>
>>> [CLUSTERLOG] main - WARN - 02/12/2008 10:02:35,175 - SimpleTcpCluster -
>>> Dynamic setProperty(className,value) has been disabled, please use
>>> explicit
>>> properties
>>> for the element you are trying to identify
>>> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:35,954 - SimpleTcpCluster -
>>> Cluster is about to start
>>> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:35,973 - ReceiverBase -
>>> Receiver
>>> Server Socket bound to:/223.99.114.21:14022
>>> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:36,025 - McastService -
>>> Setting
>>> cluster mcast soTimeout to 1000
>>> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:36,037 - McastService -
>>> Sleeping
>>> for 2000 milliseconds to establish cluster membership, start level:4
>>> [CLUSTERLOG] Thread-3 - INFO - 02/12/2008 10:02:36,913 - SimpleTcpCluster
>>> -
>>> Replication member
>>> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://22
>>> 3.99.114.20:14022,223.99.114.20,14022, alive=322404622,id={109 -10 62 -21
>>> 19
>>> 102 69 -42 -92 -123 -59 -15 37 16 50 -5 }, payload={}, command={},
>>> domain={}, ]
>>> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:38,039 - McastService - Done
>>> sleeping, membership established, start level:4
>>> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:38,044 - McastService -
>>> Sleeping
>>> for 2000 milliseconds to establish cluster membership, start level:8
>>> [CLUSTERLOG] pool-2-thread-1 - INFO - 02/12/2008 10:02:38,074 -
>>> BufferPool -
>>> Created a buffer pool with max size:104857600 bytes of
>>> type:org.apache.catalina.t
>>> ribes.io.BufferPool15Impl
>>> [CLUSTERLOG] pool-2-thread-3 - WARN - 02/12/2008 10:02:38,312 -
>>> ClusterListener - Context manager doesn't exist:
>>> [CLUSTERLOG] pool-2-thread-4 - WARN - 02/12/2008 10:02:38,816 -
>>> ClusterListener - Context manager doesn't exist:
>>> [CLUSTERLOG] pool-2-thread-5 - WARN - 02/12/2008 10:02:39,277 -
>>> ClusterListener - Context manager doesn't exist:
>>> [CLUSTERLOG] pool-2-thread-6 - WARN - 02/12/2008 10:02:39,337 -
>>> ClusterListener - Context manager doesn't exist:
>>> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:40,046 - McastService - Done
>>> sleeping, membership established, start level:8
>>> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:40,053 - JvmRouteBinderValve
>>> -
>>> JvmRouteBinderValve started
>>> [CLUSTERLOG] pool-2-thread-1 - WARN - 02/12/2008 10:02:42,816 -
>>> ClusterListener - Context manager doesn't exist:
>>> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:43,093 - DeltaManager -
>>> Register
>>> manager to cluster element Host with name localhost
>>> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:43,093 - DeltaManager -
>>> Starting
>>> clustering manager at
>>> [CLUSTERLOG] main - WARN - 02/12/2008 10:02:43,100 - DeltaManager -
>>> Manager
>>> [], requesting session state from
>>> org.apache.catalina.tribes.membership.MemberImpl
>>> [tcp://223.99.114.20:14022,223.99.114.20,14022, alive=322410633,id={109
>>> -10
>>> 62 -21 19 102 69 -42 -92 -123 -59 -15 37 16 50 -5 }, payload={},
>>> command={},
>>> domai
>>> n={}, ]. This operation will timeout if no session state has been
>>> received
>>> within 60 seconds.
>>> [CLUSTERLOG] pool-1-thread-1 - INFO - 02/12/2008 10:02:43,112 -
>>> ThroughputInterceptor - ThroughputInterceptor Report[
>>> Tx Msg:1 messages
>>> Sent:0,00 MB (total)
>>> Sent:0,00 MB (application)
>>> Time:0,01 seconds
>>> Tx Speed:0,04 MB/sec (total)
>>> TxSpeed:0,04 MB/sec (application)
>>> Error Msg:0
>>> Rx Msg:6 messages
>>> Rx Speed:0,00 MB/sec (since 1st msg)
>>> Received:0,01 MB]
>>>
>>> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:43,719 - DeltaManager -
>>> Manager
>>> []; session state send at 12.02.08 10:02 received in 624 ms.
>>> [CLUSTERLOG] Thread-31 - INFO - 02/12/2008 10:04:55,941 -
>>> TcpFailureDetector
>>> - Verification complete. Member
>>> disappeared[org.apache.catalina.tribes.membership
>>> .MemberImpl[tcp://223.99.114.20:14022,223.99.114.20,14022,
>>> alive=322543666,id={109 -10 62 -21 19 102 69 -42 -92 -123 -59 -15 37 16
>>> 50
>>> -5 }, payload={}, comman
>>> d={66 65 66 89 45 65 76 69 88 ...(9)}, domain={}, ]]
>>> [CLUSTERLOG] Thread-30 - INFO - 02/12/2008 10:04:55,941 -
>>> TcpFailureDetector
>>> - Verification complete. Member
>>> disappeared[org.apache.catalina.tribes.membership
>>> .MemberImpl[tcp://223.99.114.20:14022,223.99.114.20,14022,
>>> alive=322543666,id={109 -10 62 -21 19 102 69 -42 -92 -123 -59 -15 37 16
>>> 50
>>> -5 }, payload={}, comman
>>> d={66 65 66 89 45 65 76 69 88 ...(9)}, domain={}, ]]
>>> [CLUSTERLOG] Thread-31 - INFO - 02/12/2008 10:04:55,941 -
>>> SimpleTcpCluster -
>>> Received member
>>> disappeared:org.apache.catalina.tribes.membership.MemberImpl[tcp:
>>> //223.99.114.20:14022,223.99.114.20,14022, alive=322543666,id={109 -10 62
>>> -21 19 102 69 -42 -92 -123 -59 -15 37 16 50 -5 }, payload={}, command={66
>>> 65
>>> 66 89 4
>>> 5 65 76 69 88 ...(9)}, domain={}, ]
>>> [CLUSTERLOG] Thread-30 - INFO - 02/12/2008 10:04:55,941 -
>>> SimpleTcpCluster -
>>> Received member
>>> disappeared:org.apache.catalina.tribes.membership.MemberImpl[tcp:
>>> //223.99.114.20:14022,223.99.114.20,14022, alive=322543666,id={109 -10 62
>>> -21 19 102 69 -42 -92 -123 -59 -15 37 16 50 -5 }, payload={}, command={66
>>> 65
>>> 66 89 4
>>> 5 65 76 69 88 ...(9)}, domain={}, ]
>>> [CLUSTERLOG] Thread-36 - INFO - 02/12/2008 10:05:19,299 -
>>> SimpleTcpCluster -
>>> Replication member
>>> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://2
>>> 23.99.114.20:14022,223.99.114.20,14022, alive=2028,id={19 -115 -1 -101 8
>>> 91
>>> 66 -91 -112 105 -121 109 -82 11 56 50 }, payload={}, command={},
>>> domain={},
>>> ]
>>> [CLUSTERLOG] pool-2-thread-4 - INFO - 02/12/2008 10:25:11,742 -
>>> ThroughputInterceptor - ThroughputInterceptor Report[
>>> Tx Msg:892 messages
>>> Sent:5,82 MB (total)
>>> Sent:5,82 MB (application)
>>> Time:1,33 seconds
>>> Tx Speed:4,39 MB/sec (total)
>>> TxSpeed:4,39 MB/sec (application)
>>> Error Msg:0
>>> Rx Msg:1000 messages
>>> Rx Speed:0,00 MB/sec (since 1st msg)
>>> Received:4,47 MB]
>>>
>>> ...
>>> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] -
>>> INFO -
>>> 02/12/2008 15:01:16,005 - DeltaManager - Manager [] expiring sessions
>>> upon
>>> shutdo
>>> wn
>>> [CLUSTERLOG] pool-2-thread-1 - WARN - 02/12/2008 15:01:20,014 -
>>> ClusterListener - Context manager doesn't exist:
>>> [CLUSTERLOG] pool-2-thread-1 - WARN - 02/12/2008 15:01:20,015 -
>>> ClusterListener - Context manager doesn't exist:
>>> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] -
>>> INFO -
>>> 02/12/2008 15:01:20,219 - DeltaManager - Register manager to cluster
>>> element Hos
>>> t with name localhost
>>> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] -
>>> INFO -
>>> 02/12/2008 15:01:20,220 - DeltaManager - Starting clustering manager at
>>> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] -
>>> WARN -
>>> 02/12/2008 15:01:20,220 - DeltaManager - Manager [], requesting session
>>> state fro
>>> m
>>> org.apache.catalina.tribes.membership.MemberImpl[tcp://223.99.114.20:14022,223.99.114.20,14022,
>>> alive=17762449,id={19 -115 -1 -101 8 91 66 -91 -112 105 -121
>>> 109 -82 11 56 50 }, payload={}, command={}, domain={}, ]. This operation
>>> will timeout if no session state has been received within 60 seconds.
>>> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] -
>>> INFO -
>>> 02/12/2008 15:01:24,093 - DeltaManager - Manager []; session state send
>>> at
>>> 12.02.
>>> 08 15:01 received in 3.872 ms.
>>> [CLUSTERLOG] pool-1-thread-1 - INFO - 02/12/2008 15:22:41,143 -
>>> ThroughputInterceptor - ThroughputInterceptor Report[
>>> Tx Msg:21000 messages
>>> Sent:150,30 MB (total)
>>> Sent:150,30 MB (application)
>>> Time:70,58 seconds
>>> Tx Speed:2,13 MB/sec (total)
>>> TxSpeed:2,13 MB/sec (application)
>>> Error Msg:0
>>> Rx Msg:20534 messages
>>> Rx Speed:0,01 MB/sec (since 1st msg)
>>> Received:156,92 MB]
>>>
>>> after this entry we're NOT receiving any messages (Rx Msg):
>>>
>>> [CLUSTERLOG] pool-1-thread-2 - INFO - 02/12/2008 15:40:02,612 -
>>> ThroughputInterceptor - ThroughputInterceptor Report[
>>> Tx Msg:22000 messages
>>> Sent:154,75 MB (total)
>>> Sent:154,75 MB (application)
>>> Time:72,92 seconds
>>> Tx Speed:2,12 MB/sec (total)
>>> TxSpeed:2,12 MB/sec (application)
>>> Error Msg:0
>>> Rx Msg:20534 messages
>>> Rx Speed:0,01 MB/sec (since 1st msg)
>>> Received:156,92 MB]
>>>
>>> [CLUSTERLOG] pool-1-thread-2 - INFO - 02/12/2008 15:54:15,184 -
>>> ThroughputInterceptor - ThroughputInterceptor Report[
>>> Tx Msg:23022 messages
>>> Sent:156,30 MB (total)
>>> Sent:156,30 MB (application)
>>> Time:74,46 seconds
>>> Tx Speed:2,10 MB/sec (total)
>>> TxSpeed:2,10 MB/sec (application)
>>> Error Msg:0
>>> Rx Msg:20534 messages
>>> Rx Speed:0,01 MB/sec (since 1st msg)
>>> Received:156,92 MB]
>>> ...
>>>
>>>
>>>
>>> Many thanks for your help!!!
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>> ---------------------------------------------------------------------
>> To start a new topic, e-mail: users@tomcat.apache.org
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>>
>>
>
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat Cluster Error -
Posted by gp456 <pa...@werum.de>.
thanks for your reply.
first the fix you mentioned is not in the current tomcat release (6.0.16).
Since I have to build/patch the tomcat for a productive environment what
whould you recommend: patch the 6.0.16 sources or build tomcat from the
newest trunk?
We've ran another test and encountered a network error (packet got no ACK)
on the multicast interfaces. The wireshark log (part of) is attached below.
Perhaps you've an idea why this is happening!?
Thanks for your help!
Guido
--------------------- wireshark log start --------------------
No. Time Source Destination Protocol
Info
37433 1543.747634 223.99.114.20 223.99.114.21 TCP
36615 > 14022 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=568786762 TSER=0 WS=2
Frame 37433 (74 bytes on wire, 74 bytes captured)
Arrival Time: Feb 12, 2008 22:04:28.652917000
[Time delta from previous captured frame: 0.071229000 seconds]
[Time delta from previous displayed frame: 1543.747634000 seconds]
[Time since reference or first frame: 1543.747634000 seconds]
Frame Number: 37433
Frame Length: 74 bytes
Capture Length: 74 bytes
[Frame is marked: False]
[Protocols in frame: eth:ip:tcp]
[Coloring Rule Name: TCP SYN/FIN]
[Coloring Rule String: tcp.flags & 0x02 || tcp.flags.fin == 1]
Ethernet II, Src: Broadcom_11:2a:c1 (00:10:18:11:2a:c1), Dst:
Broadcom_17:7f:8c (00:10:18:17:7f:8c)
Destination: Broadcom_17:7f:8c (00:10:18:17:7f:8c)
Address: Broadcom_17:7f:8c (00:10:18:17:7f:8c)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
.... ..0. .... .... .... .... = LG bit: Globally unique address
(factory default)
Source: Broadcom_11:2a:c1 (00:10:18:11:2a:c1)
Address: Broadcom_11:2a:c1 (00:10:18:11:2a:c1)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
.... ..0. .... .... .... .... = LG bit: Globally unique address
(factory default)
Type: IP (0x0800)
Internet Protocol, Src: 223.99.114.20 (223.99.114.20), Dst: 223.99.114.21
(223.99.114.21)
Version: 4
Header length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
0000 00.. = Differentiated Services Codepoint: Default (0x00)
.... ..0. = ECN-Capable Transport (ECT): 0
.... ...0 = ECN-CE: 0
Total Length: 60
Identification: 0x926a (37482)
Flags: 0x04 (Don't Fragment)
0... = Reserved bit: Not set
.1.. = Don't fragment: Set
..0. = More fragments: Not set
Fragment offset: 0
Time to live: 64
Protocol: TCP (0x06)
Header checksum: 0x0561 [correct]
[Good: True]
[Bad : False]
Source: 223.99.114.20 (223.99.114.20)
Destination: 223.99.114.21 (223.99.114.21)
Transmission Control Protocol, Src Port: 36615 (36615), Dst Port: 14022
(14022), Seq: 0, Len: 0
Source port: 36615 (36615)
Destination port: 14022 (14022)
Sequence number: 0 (relative sequence number)
Header length: 40 bytes
Flags: 0x02 (SYN)
0... .... = Congestion Window Reduced (CWR): Not set
.0.. .... = ECN-Echo: Not set
..0. .... = Urgent: Not set
...0 .... = Acknowledgment: Not set
.... 0... = Push: Not set
.... .0.. = Reset: Not set
.... ..1. = Syn: Set
.... ...0 = Fin: Not set
Window size: 5840
Checksum: 0x7df2 [correct]
[Good Checksum: True]
[Bad Checksum: False]
Options: (20 bytes)
Maximum segment size: 1460 bytes
SACK permitted
Timestamps: TSval 568786762, TSecr 0
NOP
Window scale: 2 (multiply by 4)
No. Time Source Destination Protocol
Info
137308 1547.148341 223.99.114.20 223.99.114.21 TCP
36615 > 14022 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=568786762 TSER=0 WS=2
Frame 137308 (74 bytes on wire, 74 bytes captured)
Arrival Time: Feb 12, 2008 22:04:32.053624000
[Time delta from previous captured frame: 0.071236000 seconds]
[Time delta from previous displayed frame: 3.400707000 seconds]
[Time since reference or first frame: 1547.148341000 seconds]
Frame Number: 137308
Frame Length: 74 bytes
Capture Length: 74 bytes
[Frame is marked: False]
[Protocols in frame: eth:ip:tcp]
[Coloring Rule Name: TCP SYN/FIN]
[Coloring Rule String: tcp.flags & 0x02 || tcp.flags.fin == 1]
Ethernet II, Src: Broadcom_11:2a:c1 (00:10:18:11:2a:c1), Dst:
Broadcom_17:7f:8c (00:10:18:17:7f:8c)
Destination: Broadcom_17:7f:8c (00:10:18:17:7f:8c)
Address: Broadcom_17:7f:8c (00:10:18:17:7f:8c)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
.... ..0. .... .... .... .... = LG bit: Globally unique address
(factory default)
Source: Broadcom_11:2a:c1 (00:10:18:11:2a:c1)
Address: Broadcom_11:2a:c1 (00:10:18:11:2a:c1)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
.... ..0. .... .... .... .... = LG bit: Globally unique address
(factory default)
Type: IP (0x0800)
Internet Protocol, Src: 223.99.114.20 (223.99.114.20), Dst: 223.99.114.21
(223.99.114.21)
Version: 4
Header length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
0000 00.. = Differentiated Services Codepoint: Default (0x00)
.... ..0. = ECN-Capable Transport (ECT): 0
.... ...0 = ECN-CE: 0
Total Length: 60
Identification: 0x926a (37482)
Flags: 0x04 (Don't Fragment)
0... = Reserved bit: Not set
.1.. = Don't fragment: Set
..0. = More fragments: Not set
Fragment offset: 0
Time to live: 64
Protocol: TCP (0x06)
Header checksum: 0x0561 [correct]
[Good: True]
[Bad : False]
Source: 223.99.114.20 (223.99.114.20)
Destination: 223.99.114.21 (223.99.114.21)
Transmission Control Protocol, Src Port: 36615 (36615), Dst Port: 14022
(14022), Seq: 0, Len: 0
Source port: 36615 (36615)
Destination port: 14022 (14022)
Sequence number: 0 (relative sequence number)
Header length: 40 bytes
Flags: 0x02 (SYN)
0... .... = Congestion Window Reduced (CWR): Not set
.0.. .... = ECN-Echo: Not set
..0. .... = Urgent: Not set
...0 .... = Acknowledgment: Not set
.... 0... = Push: Not set
.... .0.. = Reset: Not set
.... ..1. = Syn: Set
.... ...0 = Fin: Not set
Window size: 5840
Checksum: 0x7df2 [correct]
[Good Checksum: True]
[Bad Checksum: False]
Options: (20 bytes)
Maximum segment size: 1460 bytes
SACK permitted
Timestamps: TSval 568786762, TSecr 0
NOP
Window scale: 2 (multiply by 4)
No. Time Source Destination Protocol
Info
37451 1544.749177 223.99.114.20 223.99.114.21 TCP
36616 > 14022 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=568787763 TSER=0 WS=2
Frame 37451 (74 bytes on wire, 74 bytes captured)
Arrival Time: Feb 12, 2008 22:04:29.654460000
[Time delta from previous captured frame: 0.076938000 seconds]
[Time delta from previous displayed frame: 1544.749177000 seconds]
[Time since reference or first frame: 1544.749177000 seconds]
Frame Number: 37451
Frame Length: 74 bytes
Capture Length: 74 bytes
[Frame is marked: False]
[Protocols in frame: eth:ip:tcp]
[Coloring Rule Name: TCP SYN/FIN]
[Coloring Rule String: tcp.flags & 0x02 || tcp.flags.fin == 1]
Ethernet II, Src: Broadcom_11:2a:c1 (00:10:18:11:2a:c1), Dst:
Broadcom_17:7f:8c (00:10:18:17:7f:8c)
Destination: Broadcom_17:7f:8c (00:10:18:17:7f:8c)
Address: Broadcom_17:7f:8c (00:10:18:17:7f:8c)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
.... ..0. .... .... .... .... = LG bit: Globally unique address
(factory default)
Source: Broadcom_11:2a:c1 (00:10:18:11:2a:c1)
Address: Broadcom_11:2a:c1 (00:10:18:11:2a:c1)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
.... ..0. .... .... .... .... = LG bit: Globally unique address
(factory default)
Type: IP (0x0800)
Internet Protocol, Src: 223.99.114.20 (223.99.114.20), Dst: 223.99.114.21
(223.99.114.21)
Version: 4
Header length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
0000 00.. = Differentiated Services Codepoint: Default (0x00)
.... ..0. = ECN-Capable Transport (ECT): 0
.... ...0 = ECN-CE: 0
Total Length: 60
Identification: 0xbb3b (47931)
Flags: 0x04 (Don't Fragment)
0... = Reserved bit: Not set
.1.. = Don't fragment: Set
..0. = More fragments: Not set
Fragment offset: 0
Time to live: 64
Protocol: TCP (0x06)
Header checksum: 0xdc8f [correct]
[Good: True]
[Bad : False]
Source: 223.99.114.20 (223.99.114.20)
Destination: 223.99.114.21 (223.99.114.21)
Transmission Control Protocol, Src Port: 36616 (36616), Dst Port: 14022
(14022), Seq: 0, Len: 0
Source port: 36616 (36616)
Destination port: 14022 (14022)
Sequence number: 0 (relative sequence number)
Header length: 40 bytes
Flags: 0x02 (SYN)
0... .... = Congestion Window Reduced (CWR): Not set
.0.. .... = ECN-Echo: Not set
..0. .... = Urgent: Not set
...0 .... = Acknowledgment: Not set
.... 0... = Push: Not set
.... .0.. = Reset: Not set
.... ..1. = Syn: Set
.... ...0 = Fin: Not set
Window size: 5840
Checksum: 0x2e75 [correct]
[Good Checksum: True]
[Bad Checksum: False]
Options: (20 bytes)
Maximum segment size: 1460 bytes
SACK permitted
Timestamps: TSval 568787763, TSecr 0
NOP
Window scale: 2 (multiply by 4)
No. Time Source Destination Protocol
Info
137326 1548.149884 223.99.114.20 223.99.114.21 TCP
36616 > 14022 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=568787763 TSER=0 WS=2
Frame 137326 (74 bytes on wire, 74 bytes captured)
Arrival Time: Feb 12, 2008 22:04:33.055167000
[Time delta from previous captured frame: 0.077082000 seconds]
[Time delta from previous displayed frame: 3.400707000 seconds]
[Time since reference or first frame: 1548.149884000 seconds]
Frame Number: 137326
Frame Length: 74 bytes
Capture Length: 74 bytes
[Frame is marked: False]
[Protocols in frame: eth:ip:tcp]
[Coloring Rule Name: TCP SYN/FIN]
[Coloring Rule String: tcp.flags & 0x02 || tcp.flags.fin == 1]
Ethernet II, Src: Broadcom_11:2a:c1 (00:10:18:11:2a:c1), Dst:
Broadcom_17:7f:8c (00:10:18:17:7f:8c)
Destination: Broadcom_17:7f:8c (00:10:18:17:7f:8c)
Address: Broadcom_17:7f:8c (00:10:18:17:7f:8c)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
.... ..0. .... .... .... .... = LG bit: Globally unique address
(factory default)
Source: Broadcom_11:2a:c1 (00:10:18:11:2a:c1)
Address: Broadcom_11:2a:c1 (00:10:18:11:2a:c1)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
.... ..0. .... .... .... .... = LG bit: Globally unique address
(factory default)
Type: IP (0x0800)
Internet Protocol, Src: 223.99.114.20 (223.99.114.20), Dst: 223.99.114.21
(223.99.114.21)
Version: 4
Header length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
0000 00.. = Differentiated Services Codepoint: Default (0x00)
.... ..0. = ECN-Capable Transport (ECT): 0
.... ...0 = ECN-CE: 0
Total Length: 60
Identification: 0xbb3b (47931)
Flags: 0x04 (Don't Fragment)
0... = Reserved bit: Not set
.1.. = Don't fragment: Set
..0. = More fragments: Not set
Fragment offset: 0
Time to live: 64
Protocol: TCP (0x06)
Header checksum: 0xdc8f [correct]
[Good: True]
[Bad : False]
Source: 223.99.114.20 (223.99.114.20)
Destination: 223.99.114.21 (223.99.114.21)
Transmission Control Protocol, Src Port: 36616 (36616), Dst Port: 14022
(14022), Seq: 0, Len: 0
Source port: 36616 (36616)
Destination port: 14022 (14022)
Sequence number: 0 (relative sequence number)
Header length: 40 bytes
Flags: 0x02 (SYN)
0... .... = Congestion Window Reduced (CWR): Not set
.0.. .... = ECN-Echo: Not set
..0. .... = Urgent: Not set
...0 .... = Acknowledgment: Not set
.... 0... = Push: Not set
.... .0.. = Reset: Not set
.... ..1. = Syn: Set
.... ...0 = Fin: Not set
Window size: 5840
Checksum: 0x2e75 [correct]
[Good Checksum: True]
[Bad Checksum: False]
Options: (20 bytes)
Maximum segment size: 1460 bytes
SACK permitted
Timestamps: TSval 568787763, TSecr 0
NOP
Window scale: 2 (multiply by 4)
--------------------- wireshark log end --------------------
Filip Hanik - Dev Lists wrote:
>
> [CLUSTERLOG] NioReceiver - WARN - 02/12/2008 17:34:23,895 - NioReceiver -
> Channel key is registered, but has had no interest ops for the last 3000
> ms.
> (canc
> elled:false):sun.nio.ch.SelectionKeyImpl@e161d6 last access:2008-02-12
> 17:34:17.957
> and finally:
>
> this message most likely indicates that the thread pool handling
> incoming request is busy.
> at this point do a thread dump to see the state of the threads.
> There was a deadlock fix in relation to Raul's error, and was fixed in
> this SVN commit
> http://svn.apache.org/viewvc?view=rev&revision=618823
>
> it has been proposed to be merged back into the 6.0.x branch
>
> Filip
>
> gp456 wrote:
>> Hello,
>>
>> we have a similar problem as Raul Garcia has posted in
>> http://www.nabble.com/Tomcat-6---Cluster-error.-to14921385.html#a14921385
>> this message, but we're receiving the error of a disappearing member
>> after
>> appr. 12 hours!!
>> Our load is also similar: 1 message/sec.
>>
>> We are using a hardware loadbalancer in front of two apache/tomcat pairs.
>> The apache are connected via ajp
>> to the local tomcat or (if the local tomcat is down) to the tomcat on the
>> other node.
>> Our config:
>> Node1/2:
>> Apache 2.0.59
>> Tomcat 6.0.13
>>
>> Tomcat conf Node 1:
>>
>> ...
>> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
>> channelSendOptions="8">
>>
>> <Manager
>> className="org.apache.catalina.ha.session.DeltaManager"
>> expireSessionsOnShutdown="false"
>> notifyListenersOnReplication="true"/>
>>
>> <Channel
>> className="org.apache.catalina.tribes.group.GroupChannel">
>> <Membership
>> className="org.apache.catalina.tribes.membership.McastService"
>> address="239.252.100.12"
>> port="14022"
>> frequency="1000"
>> dropTime="30000"/>
>> <Receiver
>> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
>> address="223.99.114.21"
>> port="14022"
>> autoBind="100"
>> selectorTimeout="5000"
>> maxThreads="6"/>
>>
>> <Sender
>> className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
>> <Transport
>> className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
>> timeout="60000" maxRetryAttempts="10" keepAliveCount="0" />
>> </Sender>
>> <Interceptor
>> className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
>> <Interceptor
>> className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
>> <Interceptor
>> className="org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor"
>> interval="1000" />
>> </Channel>
>>
>> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
>> filter=""/>
>> <Valve
>> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
>>
>> <!-- Deployer
>> className="org.apache.catalina.ha.deploy.FarmWarDeployer"
>> tempDir="/tmp/war-temp/"
>> deployDir="/tmp/war-deploy/"
>> watchDir="/tmp/war-listen/"
>> watchEnabled="false"/ -->
>>
>> <ClusterListener
>> className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
>> <ClusterListener
>> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
>> </Cluster>
>> ...
>>
>> Tomcat conf Node 2:
>> ...
>> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
>> channelSendOptions="8">
>>
>> <Manager
>> className="org.apache.catalina.ha.session.DeltaManager"
>> expireSessionsOnShutdown="false"
>> notifyListenersOnReplication="true"/>
>>
>> <Channel
>> className="org.apache.catalina.tribes.group.GroupChannel">
>> <Membership
>> className="org.apache.catalina.tribes.membership.McastService"
>> address="239.252.100.12"
>> port="14022"
>> frequency="1000"
>> dropTime="30000"/>
>> <Receiver
>> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
>> address="223.99.114.20"
>> port="14022"
>> autoBind="100"
>> selectorTimeout="5000"
>> maxThreads="6"/>
>>
>> <Sender
>> className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
>> <Transport
>> className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
>> timeout="60000" maxRetryAttempts="10" keepAliveCount="0" />
>> </Sender>
>> <Interceptor
>> className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
>> <Interceptor
>> className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
>> <Interceptor
>> className="org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor"
>> interval="1000" />
>> </Channel>
>>
>> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
>> filter=""/>
>> <Valve
>> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
>>
>> <!-- Deployer
>> className="org.apache.catalina.ha.deploy.FarmWarDeployer"
>> tempDir="/tmp/war-temp/"
>> deployDir="/tmp/war-deploy/"
>> watchDir="/tmp/war-listen/"
>> watchEnabled="false"/ -->
>>
>> <ClusterListener
>> className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
>> <ClusterListener
>> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
>> </Cluster>
>> ...
>>
>> Log On Node 1:
>>
>> ...
>> [CLUSTERLOG] main - INFO - 02/12/2008 10:05:15,875 - McastService -
>> Sleeping
>> for 2000 milliseconds to establish cluster membership, start level:8
>> [CLUSTERLOG] pool-2-thread-1 - INFO - 02/12/2008 10:05:15,908 -
>> BufferPool -
>> Created a buffer pool with max size:104857600 bytes of
>> type:org.apache.catalina
>> .tribes.io.BufferPool15Impl
>> [CLUSTERLOG] main - INFO - 02/12/2008 10:05:17,877 - McastService - Done
>> sleeping, membership established, start level:8
>> [CLUSTERLOG] main - INFO - 02/12/2008 10:05:17,884 - JvmRouteBinderValve
>> -
>> JvmRouteBinderValve started
>> [CLUSTERLOG] pool-2-thread-3 - WARN - 02/12/2008 10:05:19,328 -
>> ClusterListener - Context manager doesn't exist:
>> [CLUSTERLOG] main - INFO - 02/12/2008 10:05:20,015 - DeltaManager -
>> Register
>> manager to cluster element Host with name localhost
>> [CLUSTERLOG] main - INFO - 02/12/2008 10:05:20,015 - DeltaManager -
>> Starting
>> clustering manager at
>> [CLUSTERLOG] main - WARN - 02/12/2008 10:05:20,026 - DeltaManager -
>> Manager
>> [], requesting session state from
>> org.apache.catalina.tribes.membership.MemberIm
>> pl[tcp://223.99.114.21:14022,223.99.114.21,14022, alive=167423,id={10 -86
>> 53
>> 5 6 -125 71 -7 -109 5 -36 4 48 -46 45 65 }, payload={}, command={},
>> domain={},
>> ]. This operation will timeout if no session state has been received
>> within
>> 60 seconds.
>> [CLUSTERLOG] pool-1-thread-1 - INFO - 02/12/2008 10:05:20,045 -
>> ThroughputInterceptor - ThroughputInterceptor Report[
>> Tx Msg:1 messages
>> Sent:0,00 MB (total)
>> Sent:0,00 MB (application)
>> Time:0,02 seconds
>> Tx Speed:0,03 MB/sec (total)
>> TxSpeed:0,03 MB/sec (application)
>> Error Msg:0
>> Rx Msg:2 messages
>> Rx Speed:0,00 MB/sec (since 1st msg)
>> Received:0,00 MB]
>> ...
>> after some time we're getting this entry:
>>
>> [CLUSTERLOG] pool-2-thread-4 - WARN - 02/12/2008 15:00:52,044 -
>> ClusterListener - Context manager doesn't exist:
>> [CLUSTERLOG] pool-2-thread-5 - WARN - 02/12/2008 15:00:52,910 -
>> ClusterListener - Context manager doesn't exist:
>> [CLUSTERLOG] pool-2-thread-2 - WARN - 02/12/2008 15:00:54,299 -
>> ClusterListener - Context manager doesn't exist:
>> [CLUSTERLOG] pool-2-thread-6 - WARN - 02/12/2008 15:00:54,707 -
>> ClusterListener - Context manager doesn't exist:
>> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] -
>> INFO -
>> 02/12/2008 15:00:56,011 - DeltaManager - Register manager to cluster
>> element H
>> ost with name localhost
>> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] -
>> INFO -
>> 02/12/2008 15:00:56,011 - DeltaManager - Starting clustering manager at
>> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] -
>> WARN -
>> 02/12/2008 15:00:56,012 - DeltaManager - Manager [], requesting session
>> state f
>> rom
>> org.apache.catalina.tribes.membership.MemberImpl[tcp://223.99.114.21:14022,223.99.114.21,14022,
>> alive=17902419,id={10 -86 53 5 6 -125 71 -7 -109 5 -36 4
>> 48 -46 45 65 }, payload={}, command={}, domain={}, ]. This operation
>> will
>> timeout if no session state has been received within 60 seconds.
>> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] -
>> INFO -
>> 02/12/2008 15:01:02,245 - DeltaManager - Manager []; session state send
>> at
>> 12.0
>> 2.08 15:00 received in 6.233 ms.
>> [CLUSTERLOG] pool-2-thread-2 - INFO - 02/12/2008 15:01:13,178 -
>> ThroughputInterceptor - ThroughputInterceptor Report[
>> Tx Msg:20003 messages
>> Sent:143,22 MB (total)
>> Sent:143,22 MB (application)
>> Time:54,66 seconds
>> Tx Speed:2,62 MB/sec (total)
>> TxSpeed:2,62 MB/sec (application)
>> Error Msg:0
>> Rx Msg:20000 messages
>> Rx Speed:0,01 MB/sec (since 1st msg)
>> Received:143,04 MB]
>>
>> [CLUSTERLOG] pool-2-thread-5 - INFO - 02/12/2008 15:23:49,278 -
>> ThroughputInterceptor - ThroughputInterceptor Report[
>> Tx Msg:20606 messages
>> Sent:156,13 MB (total)
>> Sent:156,13 MB (application)
>> Time:61,55 seconds
>> Tx Speed:2,54 MB/sec (total)
>> TxSpeed:2,54 MB/sec (application)
>> Error Msg:9
>> Rx Msg:21000 messages
>> Rx Speed:0,01 MB/sec (since 1st msg)
>> Received:150,14 MB]
>>
>> and now we're receiving and sending message but also Error messages....
>> and then:
>>
>> [CLUSTERLOG] NioReceiver - WARN - 02/12/2008 17:34:23,895 - NioReceiver -
>> Channel key is registered, but has had no interest ops for the last 3000
>> ms.
>> (canc
>> elled:false):sun.nio.ch.SelectionKeyImpl@e161d6 last access:2008-02-12
>> 17:34:17.957
>> and finally:
>>
>> [CLUSTERLOG] 4564 - INFO - 02/12/2008 22:04:28,652 - TcpFailureDetector -
>> Received
>> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://
>> 223.99.114.21:14022,223.99.114.21,14022, alive=43315450,id={10 -86 53 5 6
>> -125 71 -7 -109 5 -36 4 48 -46 45 65 }, payload={}, command={},
>> domain={},
>> ]] mess
>> age. Will verify.
>> [CLUSTERLOG] 4564 - INFO - 02/12/2008 22:04:29,654 - TcpFailureDetector -
>> Verification complete. Member
>> disappeared[org.apache.catalina.tribes.membership.Me
>> mberImpl[tcp://223.99.114.21:14022,223.99.114.21,14022,
>> alive=43316452,id={10 -86 53 5 6 -125 71 -7 -109 5 -36 4 48 -46 45 65 },
>> payload={}, command={}, dom
>> ain={}, ]]
>> [CLUSTERLOG] 4564 - INFO - 02/12/2008 22:04:29,654 - SimpleTcpCluster -
>> Received member
>> disappeared:org.apache.catalina.tribes.membership.MemberImpl[tcp://2
>> 23.99.114.21:14022,223.99.114.21,14022, alive=43316452,id={10 -86 53 5 6
>> -125 71 -7 -109 5 -36 4 48 -46 45 65 }, payload={}, command={},
>> domain={}, ]
>> [CLUSTERLOG] 4564 - ERROR - 02/12/2008 22:04:29,654 - SimpleTcpCluster -
>> Unable to send message through cluster sender.
>> org.apache.catalina.tribes.ChannelException: Operation has timed
>> out(60000
>> ms.).; Faulty members:tcp://223.99.114.21:14022;
>> at
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:97)
>> at
>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
>> at
>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80)
>> at
>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78)
>> at
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>> at
>> org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor.sendMessage(ThroughputInterceptor.java:61)
>> at
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>> at
>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:60)
>> at
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>> at
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87)
>> at
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>> at
>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
>> at
>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
>> at
>> org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:835)
>> at
>> org.apache.catalina.ha.tcp.SimpleTcpCluster.sendClusterDomain(SimpleTcpCluster.java:814)
>> at
>> org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:551)
>> at
>> org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:535)
>> at
>> org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:517)
>> at
>> org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:428)
>> at
>> org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:362)
>> at
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
>> at
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>> at
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:261)
>> at
>> org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
>> at
>> org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283)
>> at
>> org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
>> at
>> org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
>> at
>> org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
>> at
>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:686)
>> at java.lang.Thread.run(Thread.java:595)
>> [CLUSTERLOG] 4575 - ERROR - 02/12/2008 22:04:34,920 - SimpleTcpCluster -
>> Unable to send message through cluster sender.
>> org.apache.catalina.tribes.ChannelException: No destination given; No
>> faulty
>> members identified.
>> at
>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:194)
>> at
>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
>> ...
>> the error messages are repeating...
>>
>>
>> Log On Node 2:
>>
>> [CLUSTERLOG] main - WARN - 02/12/2008 10:02:35,175 - SimpleTcpCluster -
>> Dynamic setProperty(className,value) has been disabled, please use
>> explicit
>> properties
>> for the element you are trying to identify
>> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:35,954 - SimpleTcpCluster -
>> Cluster is about to start
>> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:35,973 - ReceiverBase -
>> Receiver
>> Server Socket bound to:/223.99.114.21:14022
>> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:36,025 - McastService -
>> Setting
>> cluster mcast soTimeout to 1000
>> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:36,037 - McastService -
>> Sleeping
>> for 2000 milliseconds to establish cluster membership, start level:4
>> [CLUSTERLOG] Thread-3 - INFO - 02/12/2008 10:02:36,913 - SimpleTcpCluster
>> -
>> Replication member
>> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://22
>> 3.99.114.20:14022,223.99.114.20,14022, alive=322404622,id={109 -10 62 -21
>> 19
>> 102 69 -42 -92 -123 -59 -15 37 16 50 -5 }, payload={}, command={},
>> domain={}, ]
>> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:38,039 - McastService - Done
>> sleeping, membership established, start level:4
>> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:38,044 - McastService -
>> Sleeping
>> for 2000 milliseconds to establish cluster membership, start level:8
>> [CLUSTERLOG] pool-2-thread-1 - INFO - 02/12/2008 10:02:38,074 -
>> BufferPool -
>> Created a buffer pool with max size:104857600 bytes of
>> type:org.apache.catalina.t
>> ribes.io.BufferPool15Impl
>> [CLUSTERLOG] pool-2-thread-3 - WARN - 02/12/2008 10:02:38,312 -
>> ClusterListener - Context manager doesn't exist:
>> [CLUSTERLOG] pool-2-thread-4 - WARN - 02/12/2008 10:02:38,816 -
>> ClusterListener - Context manager doesn't exist:
>> [CLUSTERLOG] pool-2-thread-5 - WARN - 02/12/2008 10:02:39,277 -
>> ClusterListener - Context manager doesn't exist:
>> [CLUSTERLOG] pool-2-thread-6 - WARN - 02/12/2008 10:02:39,337 -
>> ClusterListener - Context manager doesn't exist:
>> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:40,046 - McastService - Done
>> sleeping, membership established, start level:8
>> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:40,053 - JvmRouteBinderValve
>> -
>> JvmRouteBinderValve started
>> [CLUSTERLOG] pool-2-thread-1 - WARN - 02/12/2008 10:02:42,816 -
>> ClusterListener - Context manager doesn't exist:
>> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:43,093 - DeltaManager -
>> Register
>> manager to cluster element Host with name localhost
>> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:43,093 - DeltaManager -
>> Starting
>> clustering manager at
>> [CLUSTERLOG] main - WARN - 02/12/2008 10:02:43,100 - DeltaManager -
>> Manager
>> [], requesting session state from
>> org.apache.catalina.tribes.membership.MemberImpl
>> [tcp://223.99.114.20:14022,223.99.114.20,14022, alive=322410633,id={109
>> -10
>> 62 -21 19 102 69 -42 -92 -123 -59 -15 37 16 50 -5 }, payload={},
>> command={},
>> domai
>> n={}, ]. This operation will timeout if no session state has been
>> received
>> within 60 seconds.
>> [CLUSTERLOG] pool-1-thread-1 - INFO - 02/12/2008 10:02:43,112 -
>> ThroughputInterceptor - ThroughputInterceptor Report[
>> Tx Msg:1 messages
>> Sent:0,00 MB (total)
>> Sent:0,00 MB (application)
>> Time:0,01 seconds
>> Tx Speed:0,04 MB/sec (total)
>> TxSpeed:0,04 MB/sec (application)
>> Error Msg:0
>> Rx Msg:6 messages
>> Rx Speed:0,00 MB/sec (since 1st msg)
>> Received:0,01 MB]
>>
>> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:43,719 - DeltaManager -
>> Manager
>> []; session state send at 12.02.08 10:02 received in 624 ms.
>> [CLUSTERLOG] Thread-31 - INFO - 02/12/2008 10:04:55,941 -
>> TcpFailureDetector
>> - Verification complete. Member
>> disappeared[org.apache.catalina.tribes.membership
>> .MemberImpl[tcp://223.99.114.20:14022,223.99.114.20,14022,
>> alive=322543666,id={109 -10 62 -21 19 102 69 -42 -92 -123 -59 -15 37 16
>> 50
>> -5 }, payload={}, comman
>> d={66 65 66 89 45 65 76 69 88 ...(9)}, domain={}, ]]
>> [CLUSTERLOG] Thread-30 - INFO - 02/12/2008 10:04:55,941 -
>> TcpFailureDetector
>> - Verification complete. Member
>> disappeared[org.apache.catalina.tribes.membership
>> .MemberImpl[tcp://223.99.114.20:14022,223.99.114.20,14022,
>> alive=322543666,id={109 -10 62 -21 19 102 69 -42 -92 -123 -59 -15 37 16
>> 50
>> -5 }, payload={}, comman
>> d={66 65 66 89 45 65 76 69 88 ...(9)}, domain={}, ]]
>> [CLUSTERLOG] Thread-31 - INFO - 02/12/2008 10:04:55,941 -
>> SimpleTcpCluster -
>> Received member
>> disappeared:org.apache.catalina.tribes.membership.MemberImpl[tcp:
>> //223.99.114.20:14022,223.99.114.20,14022, alive=322543666,id={109 -10 62
>> -21 19 102 69 -42 -92 -123 -59 -15 37 16 50 -5 }, payload={}, command={66
>> 65
>> 66 89 4
>> 5 65 76 69 88 ...(9)}, domain={}, ]
>> [CLUSTERLOG] Thread-30 - INFO - 02/12/2008 10:04:55,941 -
>> SimpleTcpCluster -
>> Received member
>> disappeared:org.apache.catalina.tribes.membership.MemberImpl[tcp:
>> //223.99.114.20:14022,223.99.114.20,14022, alive=322543666,id={109 -10 62
>> -21 19 102 69 -42 -92 -123 -59 -15 37 16 50 -5 }, payload={}, command={66
>> 65
>> 66 89 4
>> 5 65 76 69 88 ...(9)}, domain={}, ]
>> [CLUSTERLOG] Thread-36 - INFO - 02/12/2008 10:05:19,299 -
>> SimpleTcpCluster -
>> Replication member
>> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://2
>> 23.99.114.20:14022,223.99.114.20,14022, alive=2028,id={19 -115 -1 -101 8
>> 91
>> 66 -91 -112 105 -121 109 -82 11 56 50 }, payload={}, command={},
>> domain={},
>> ]
>> [CLUSTERLOG] pool-2-thread-4 - INFO - 02/12/2008 10:25:11,742 -
>> ThroughputInterceptor - ThroughputInterceptor Report[
>> Tx Msg:892 messages
>> Sent:5,82 MB (total)
>> Sent:5,82 MB (application)
>> Time:1,33 seconds
>> Tx Speed:4,39 MB/sec (total)
>> TxSpeed:4,39 MB/sec (application)
>> Error Msg:0
>> Rx Msg:1000 messages
>> Rx Speed:0,00 MB/sec (since 1st msg)
>> Received:4,47 MB]
>>
>> ...
>> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] -
>> INFO -
>> 02/12/2008 15:01:16,005 - DeltaManager - Manager [] expiring sessions
>> upon
>> shutdo
>> wn
>> [CLUSTERLOG] pool-2-thread-1 - WARN - 02/12/2008 15:01:20,014 -
>> ClusterListener - Context manager doesn't exist:
>> [CLUSTERLOG] pool-2-thread-1 - WARN - 02/12/2008 15:01:20,015 -
>> ClusterListener - Context manager doesn't exist:
>> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] -
>> INFO -
>> 02/12/2008 15:01:20,219 - DeltaManager - Register manager to cluster
>> element Hos
>> t with name localhost
>> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] -
>> INFO -
>> 02/12/2008 15:01:20,220 - DeltaManager - Starting clustering manager at
>> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] -
>> WARN -
>> 02/12/2008 15:01:20,220 - DeltaManager - Manager [], requesting session
>> state fro
>> m
>> org.apache.catalina.tribes.membership.MemberImpl[tcp://223.99.114.20:14022,223.99.114.20,14022,
>> alive=17762449,id={19 -115 -1 -101 8 91 66 -91 -112 105 -121
>> 109 -82 11 56 50 }, payload={}, command={}, domain={}, ]. This operation
>> will timeout if no session state has been received within 60 seconds.
>> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] -
>> INFO -
>> 02/12/2008 15:01:24,093 - DeltaManager - Manager []; session state send
>> at
>> 12.02.
>> 08 15:01 received in 3.872 ms.
>> [CLUSTERLOG] pool-1-thread-1 - INFO - 02/12/2008 15:22:41,143 -
>> ThroughputInterceptor - ThroughputInterceptor Report[
>> Tx Msg:21000 messages
>> Sent:150,30 MB (total)
>> Sent:150,30 MB (application)
>> Time:70,58 seconds
>> Tx Speed:2,13 MB/sec (total)
>> TxSpeed:2,13 MB/sec (application)
>> Error Msg:0
>> Rx Msg:20534 messages
>> Rx Speed:0,01 MB/sec (since 1st msg)
>> Received:156,92 MB]
>>
>> after this entry we're NOT receiving any messages (Rx Msg):
>>
>> [CLUSTERLOG] pool-1-thread-2 - INFO - 02/12/2008 15:40:02,612 -
>> ThroughputInterceptor - ThroughputInterceptor Report[
>> Tx Msg:22000 messages
>> Sent:154,75 MB (total)
>> Sent:154,75 MB (application)
>> Time:72,92 seconds
>> Tx Speed:2,12 MB/sec (total)
>> TxSpeed:2,12 MB/sec (application)
>> Error Msg:0
>> Rx Msg:20534 messages
>> Rx Speed:0,01 MB/sec (since 1st msg)
>> Received:156,92 MB]
>>
>> [CLUSTERLOG] pool-1-thread-2 - INFO - 02/12/2008 15:54:15,184 -
>> ThroughputInterceptor - ThroughputInterceptor Report[
>> Tx Msg:23022 messages
>> Sent:156,30 MB (total)
>> Sent:156,30 MB (application)
>> Time:74,46 seconds
>> Tx Speed:2,10 MB/sec (total)
>> TxSpeed:2,10 MB/sec (application)
>> Error Msg:0
>> Rx Msg:20534 messages
>> Rx Speed:0,01 MB/sec (since 1st msg)
>> Received:156,92 MB]
>> ...
>>
>>
>>
>> Many thanks for your help!!!
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
--
View this message in context: http://www.nabble.com/Tomcat-Cluster-Error---Operation-has-timed-out-tp15457470p15610793.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat Cluster Error -
Posted by Filip Hanik - Dev Lists <de...@hanik.com>.
[CLUSTERLOG] NioReceiver - WARN - 02/12/2008 17:34:23,895 - NioReceiver -
Channel key is registered, but has had no interest ops for the last 3000 ms.
(canc
elled:false):sun.nio.ch.SelectionKeyImpl@e161d6 last access:2008-02-12
17:34:17.957
and finally:
this message most likely indicates that the thread pool handling
incoming request is busy.
at this point do a thread dump to see the state of the threads.
There was a deadlock fix in relation to Raul's error, and was fixed in
this SVN commit
http://svn.apache.org/viewvc?view=rev&revision=618823
it has been proposed to be merged back into the 6.0.x branch
Filip
gp456 wrote:
> Hello,
>
> we have a similar problem as Raul Garcia has posted in
> http://www.nabble.com/Tomcat-6---Cluster-error.-to14921385.html#a14921385
> this message, but we're receiving the error of a disappearing member after
> appr. 12 hours!!
> Our load is also similar: 1 message/sec.
>
> We are using a hardware loadbalancer in front of two apache/tomcat pairs.
> The apache are connected via ajp
> to the local tomcat or (if the local tomcat is down) to the tomcat on the
> other node.
> Our config:
> Node1/2:
> Apache 2.0.59
> Tomcat 6.0.13
>
> Tomcat conf Node 1:
>
> ...
> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
> channelSendOptions="8">
>
> <Manager className="org.apache.catalina.ha.session.DeltaManager"
> expireSessionsOnShutdown="false"
> notifyListenersOnReplication="true"/>
>
> <Channel
> className="org.apache.catalina.tribes.group.GroupChannel">
> <Membership
> className="org.apache.catalina.tribes.membership.McastService"
> address="239.252.100.12"
> port="14022"
> frequency="1000"
> dropTime="30000"/>
> <Receiver
> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
> address="223.99.114.21"
> port="14022"
> autoBind="100"
> selectorTimeout="5000"
> maxThreads="6"/>
>
> <Sender
> className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
> <Transport
> className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
> timeout="60000" maxRetryAttempts="10" keepAliveCount="0" />
> </Sender>
> <Interceptor
> className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
> <Interceptor
> className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
> <Interceptor
> className="org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor"
> interval="1000" />
> </Channel>
>
> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
> filter=""/>
> <Valve
> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
>
> <!-- Deployer
> className="org.apache.catalina.ha.deploy.FarmWarDeployer"
> tempDir="/tmp/war-temp/"
> deployDir="/tmp/war-deploy/"
> watchDir="/tmp/war-listen/"
> watchEnabled="false"/ -->
>
> <ClusterListener
> className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
> <ClusterListener
> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
> </Cluster>
> ...
>
> Tomcat conf Node 2:
> ...
> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
> channelSendOptions="8">
>
> <Manager className="org.apache.catalina.ha.session.DeltaManager"
> expireSessionsOnShutdown="false"
> notifyListenersOnReplication="true"/>
>
> <Channel
> className="org.apache.catalina.tribes.group.GroupChannel">
> <Membership
> className="org.apache.catalina.tribes.membership.McastService"
> address="239.252.100.12"
> port="14022"
> frequency="1000"
> dropTime="30000"/>
> <Receiver
> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
> address="223.99.114.20"
> port="14022"
> autoBind="100"
> selectorTimeout="5000"
> maxThreads="6"/>
>
> <Sender
> className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
> <Transport
> className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
> timeout="60000" maxRetryAttempts="10" keepAliveCount="0" />
> </Sender>
> <Interceptor
> className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
> <Interceptor
> className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
> <Interceptor
> className="org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor"
> interval="1000" />
> </Channel>
>
> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
> filter=""/>
> <Valve
> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
>
> <!-- Deployer
> className="org.apache.catalina.ha.deploy.FarmWarDeployer"
> tempDir="/tmp/war-temp/"
> deployDir="/tmp/war-deploy/"
> watchDir="/tmp/war-listen/"
> watchEnabled="false"/ -->
>
> <ClusterListener
> className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
> <ClusterListener
> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
> </Cluster>
> ...
>
> Log On Node 1:
>
> ...
> [CLUSTERLOG] main - INFO - 02/12/2008 10:05:15,875 - McastService - Sleeping
> for 2000 milliseconds to establish cluster membership, start level:8
> [CLUSTERLOG] pool-2-thread-1 - INFO - 02/12/2008 10:05:15,908 - BufferPool -
> Created a buffer pool with max size:104857600 bytes of
> type:org.apache.catalina
> .tribes.io.BufferPool15Impl
> [CLUSTERLOG] main - INFO - 02/12/2008 10:05:17,877 - McastService - Done
> sleeping, membership established, start level:8
> [CLUSTERLOG] main - INFO - 02/12/2008 10:05:17,884 - JvmRouteBinderValve -
> JvmRouteBinderValve started
> [CLUSTERLOG] pool-2-thread-3 - WARN - 02/12/2008 10:05:19,328 -
> ClusterListener - Context manager doesn't exist:
> [CLUSTERLOG] main - INFO - 02/12/2008 10:05:20,015 - DeltaManager - Register
> manager to cluster element Host with name localhost
> [CLUSTERLOG] main - INFO - 02/12/2008 10:05:20,015 - DeltaManager - Starting
> clustering manager at
> [CLUSTERLOG] main - WARN - 02/12/2008 10:05:20,026 - DeltaManager - Manager
> [], requesting session state from
> org.apache.catalina.tribes.membership.MemberIm
> pl[tcp://223.99.114.21:14022,223.99.114.21,14022, alive=167423,id={10 -86 53
> 5 6 -125 71 -7 -109 5 -36 4 48 -46 45 65 }, payload={}, command={},
> domain={},
> ]. This operation will timeout if no session state has been received within
> 60 seconds.
> [CLUSTERLOG] pool-1-thread-1 - INFO - 02/12/2008 10:05:20,045 -
> ThroughputInterceptor - ThroughputInterceptor Report[
> Tx Msg:1 messages
> Sent:0,00 MB (total)
> Sent:0,00 MB (application)
> Time:0,02 seconds
> Tx Speed:0,03 MB/sec (total)
> TxSpeed:0,03 MB/sec (application)
> Error Msg:0
> Rx Msg:2 messages
> Rx Speed:0,00 MB/sec (since 1st msg)
> Received:0,00 MB]
> ...
> after some time we're getting this entry:
>
> [CLUSTERLOG] pool-2-thread-4 - WARN - 02/12/2008 15:00:52,044 -
> ClusterListener - Context manager doesn't exist:
> [CLUSTERLOG] pool-2-thread-5 - WARN - 02/12/2008 15:00:52,910 -
> ClusterListener - Context manager doesn't exist:
> [CLUSTERLOG] pool-2-thread-2 - WARN - 02/12/2008 15:00:54,299 -
> ClusterListener - Context manager doesn't exist:
> [CLUSTERLOG] pool-2-thread-6 - WARN - 02/12/2008 15:00:54,707 -
> ClusterListener - Context manager doesn't exist:
> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] - INFO -
> 02/12/2008 15:00:56,011 - DeltaManager - Register manager to cluster
> element H
> ost with name localhost
> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] - INFO -
> 02/12/2008 15:00:56,011 - DeltaManager - Starting clustering manager at
> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] - WARN -
> 02/12/2008 15:00:56,012 - DeltaManager - Manager [], requesting session
> state f
> rom
> org.apache.catalina.tribes.membership.MemberImpl[tcp://223.99.114.21:14022,223.99.114.21,14022,
> alive=17902419,id={10 -86 53 5 6 -125 71 -7 -109 5 -36 4
> 48 -46 45 65 }, payload={}, command={}, domain={}, ]. This operation will
> timeout if no session state has been received within 60 seconds.
> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] - INFO -
> 02/12/2008 15:01:02,245 - DeltaManager - Manager []; session state send at
> 12.0
> 2.08 15:00 received in 6.233 ms.
> [CLUSTERLOG] pool-2-thread-2 - INFO - 02/12/2008 15:01:13,178 -
> ThroughputInterceptor - ThroughputInterceptor Report[
> Tx Msg:20003 messages
> Sent:143,22 MB (total)
> Sent:143,22 MB (application)
> Time:54,66 seconds
> Tx Speed:2,62 MB/sec (total)
> TxSpeed:2,62 MB/sec (application)
> Error Msg:0
> Rx Msg:20000 messages
> Rx Speed:0,01 MB/sec (since 1st msg)
> Received:143,04 MB]
>
> [CLUSTERLOG] pool-2-thread-5 - INFO - 02/12/2008 15:23:49,278 -
> ThroughputInterceptor - ThroughputInterceptor Report[
> Tx Msg:20606 messages
> Sent:156,13 MB (total)
> Sent:156,13 MB (application)
> Time:61,55 seconds
> Tx Speed:2,54 MB/sec (total)
> TxSpeed:2,54 MB/sec (application)
> Error Msg:9
> Rx Msg:21000 messages
> Rx Speed:0,01 MB/sec (since 1st msg)
> Received:150,14 MB]
>
> and now we're receiving and sending message but also Error messages....
> and then:
>
> [CLUSTERLOG] NioReceiver - WARN - 02/12/2008 17:34:23,895 - NioReceiver -
> Channel key is registered, but has had no interest ops for the last 3000 ms.
> (canc
> elled:false):sun.nio.ch.SelectionKeyImpl@e161d6 last access:2008-02-12
> 17:34:17.957
> and finally:
>
> [CLUSTERLOG] 4564 - INFO - 02/12/2008 22:04:28,652 - TcpFailureDetector -
> Received
> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://
> 223.99.114.21:14022,223.99.114.21,14022, alive=43315450,id={10 -86 53 5 6
> -125 71 -7 -109 5 -36 4 48 -46 45 65 }, payload={}, command={}, domain={},
> ]] mess
> age. Will verify.
> [CLUSTERLOG] 4564 - INFO - 02/12/2008 22:04:29,654 - TcpFailureDetector -
> Verification complete. Member
> disappeared[org.apache.catalina.tribes.membership.Me
> mberImpl[tcp://223.99.114.21:14022,223.99.114.21,14022,
> alive=43316452,id={10 -86 53 5 6 -125 71 -7 -109 5 -36 4 48 -46 45 65 },
> payload={}, command={}, dom
> ain={}, ]]
> [CLUSTERLOG] 4564 - INFO - 02/12/2008 22:04:29,654 - SimpleTcpCluster -
> Received member
> disappeared:org.apache.catalina.tribes.membership.MemberImpl[tcp://2
> 23.99.114.21:14022,223.99.114.21,14022, alive=43316452,id={10 -86 53 5 6
> -125 71 -7 -109 5 -36 4 48 -46 45 65 }, payload={}, command={}, domain={}, ]
> [CLUSTERLOG] 4564 - ERROR - 02/12/2008 22:04:29,654 - SimpleTcpCluster -
> Unable to send message through cluster sender.
> org.apache.catalina.tribes.ChannelException: Operation has timed out(60000
> ms.).; Faulty members:tcp://223.99.114.21:14022;
> at
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:97)
> at
> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
> at
> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80)
> at
> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78)
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
> at
> org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor.sendMessage(ThroughputInterceptor.java:61)
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
> at
> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:60)
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
> at
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87)
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
> at
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
> at
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
> at
> org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:835)
> at
> org.apache.catalina.ha.tcp.SimpleTcpCluster.sendClusterDomain(SimpleTcpCluster.java:814)
> at
> org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:551)
> at
> org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:535)
> at
> org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:517)
> at
> org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:428)
> at
> org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:362)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:261)
> at
> org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
> at
> org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283)
> at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
> at
> org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
> at
> org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
> at
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:686)
> at java.lang.Thread.run(Thread.java:595)
> [CLUSTERLOG] 4575 - ERROR - 02/12/2008 22:04:34,920 - SimpleTcpCluster -
> Unable to send message through cluster sender.
> org.apache.catalina.tribes.ChannelException: No destination given; No faulty
> members identified.
> at
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:194)
> at
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
> ...
> the error messages are repeating...
>
>
> Log On Node 2:
>
> [CLUSTERLOG] main - WARN - 02/12/2008 10:02:35,175 - SimpleTcpCluster -
> Dynamic setProperty(className,value) has been disabled, please use explicit
> properties
> for the element you are trying to identify
> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:35,954 - SimpleTcpCluster -
> Cluster is about to start
> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:35,973 - ReceiverBase - Receiver
> Server Socket bound to:/223.99.114.21:14022
> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:36,025 - McastService - Setting
> cluster mcast soTimeout to 1000
> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:36,037 - McastService - Sleeping
> for 2000 milliseconds to establish cluster membership, start level:4
> [CLUSTERLOG] Thread-3 - INFO - 02/12/2008 10:02:36,913 - SimpleTcpCluster -
> Replication member
> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://22
> 3.99.114.20:14022,223.99.114.20,14022, alive=322404622,id={109 -10 62 -21 19
> 102 69 -42 -92 -123 -59 -15 37 16 50 -5 }, payload={}, command={},
> domain={}, ]
> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:38,039 - McastService - Done
> sleeping, membership established, start level:4
> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:38,044 - McastService - Sleeping
> for 2000 milliseconds to establish cluster membership, start level:8
> [CLUSTERLOG] pool-2-thread-1 - INFO - 02/12/2008 10:02:38,074 - BufferPool -
> Created a buffer pool with max size:104857600 bytes of
> type:org.apache.catalina.t
> ribes.io.BufferPool15Impl
> [CLUSTERLOG] pool-2-thread-3 - WARN - 02/12/2008 10:02:38,312 -
> ClusterListener - Context manager doesn't exist:
> [CLUSTERLOG] pool-2-thread-4 - WARN - 02/12/2008 10:02:38,816 -
> ClusterListener - Context manager doesn't exist:
> [CLUSTERLOG] pool-2-thread-5 - WARN - 02/12/2008 10:02:39,277 -
> ClusterListener - Context manager doesn't exist:
> [CLUSTERLOG] pool-2-thread-6 - WARN - 02/12/2008 10:02:39,337 -
> ClusterListener - Context manager doesn't exist:
> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:40,046 - McastService - Done
> sleeping, membership established, start level:8
> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:40,053 - JvmRouteBinderValve -
> JvmRouteBinderValve started
> [CLUSTERLOG] pool-2-thread-1 - WARN - 02/12/2008 10:02:42,816 -
> ClusterListener - Context manager doesn't exist:
> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:43,093 - DeltaManager - Register
> manager to cluster element Host with name localhost
> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:43,093 - DeltaManager - Starting
> clustering manager at
> [CLUSTERLOG] main - WARN - 02/12/2008 10:02:43,100 - DeltaManager - Manager
> [], requesting session state from
> org.apache.catalina.tribes.membership.MemberImpl
> [tcp://223.99.114.20:14022,223.99.114.20,14022, alive=322410633,id={109 -10
> 62 -21 19 102 69 -42 -92 -123 -59 -15 37 16 50 -5 }, payload={}, command={},
> domai
> n={}, ]. This operation will timeout if no session state has been received
> within 60 seconds.
> [CLUSTERLOG] pool-1-thread-1 - INFO - 02/12/2008 10:02:43,112 -
> ThroughputInterceptor - ThroughputInterceptor Report[
> Tx Msg:1 messages
> Sent:0,00 MB (total)
> Sent:0,00 MB (application)
> Time:0,01 seconds
> Tx Speed:0,04 MB/sec (total)
> TxSpeed:0,04 MB/sec (application)
> Error Msg:0
> Rx Msg:6 messages
> Rx Speed:0,00 MB/sec (since 1st msg)
> Received:0,01 MB]
>
> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:43,719 - DeltaManager - Manager
> []; session state send at 12.02.08 10:02 received in 624 ms.
> [CLUSTERLOG] Thread-31 - INFO - 02/12/2008 10:04:55,941 - TcpFailureDetector
> - Verification complete. Member
> disappeared[org.apache.catalina.tribes.membership
> .MemberImpl[tcp://223.99.114.20:14022,223.99.114.20,14022,
> alive=322543666,id={109 -10 62 -21 19 102 69 -42 -92 -123 -59 -15 37 16 50
> -5 }, payload={}, comman
> d={66 65 66 89 45 65 76 69 88 ...(9)}, domain={}, ]]
> [CLUSTERLOG] Thread-30 - INFO - 02/12/2008 10:04:55,941 - TcpFailureDetector
> - Verification complete. Member
> disappeared[org.apache.catalina.tribes.membership
> .MemberImpl[tcp://223.99.114.20:14022,223.99.114.20,14022,
> alive=322543666,id={109 -10 62 -21 19 102 69 -42 -92 -123 -59 -15 37 16 50
> -5 }, payload={}, comman
> d={66 65 66 89 45 65 76 69 88 ...(9)}, domain={}, ]]
> [CLUSTERLOG] Thread-31 - INFO - 02/12/2008 10:04:55,941 - SimpleTcpCluster -
> Received member
> disappeared:org.apache.catalina.tribes.membership.MemberImpl[tcp:
> //223.99.114.20:14022,223.99.114.20,14022, alive=322543666,id={109 -10 62
> -21 19 102 69 -42 -92 -123 -59 -15 37 16 50 -5 }, payload={}, command={66 65
> 66 89 4
> 5 65 76 69 88 ...(9)}, domain={}, ]
> [CLUSTERLOG] Thread-30 - INFO - 02/12/2008 10:04:55,941 - SimpleTcpCluster -
> Received member
> disappeared:org.apache.catalina.tribes.membership.MemberImpl[tcp:
> //223.99.114.20:14022,223.99.114.20,14022, alive=322543666,id={109 -10 62
> -21 19 102 69 -42 -92 -123 -59 -15 37 16 50 -5 }, payload={}, command={66 65
> 66 89 4
> 5 65 76 69 88 ...(9)}, domain={}, ]
> [CLUSTERLOG] Thread-36 - INFO - 02/12/2008 10:05:19,299 - SimpleTcpCluster -
> Replication member
> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://2
> 23.99.114.20:14022,223.99.114.20,14022, alive=2028,id={19 -115 -1 -101 8 91
> 66 -91 -112 105 -121 109 -82 11 56 50 }, payload={}, command={}, domain={},
> ]
> [CLUSTERLOG] pool-2-thread-4 - INFO - 02/12/2008 10:25:11,742 -
> ThroughputInterceptor - ThroughputInterceptor Report[
> Tx Msg:892 messages
> Sent:5,82 MB (total)
> Sent:5,82 MB (application)
> Time:1,33 seconds
> Tx Speed:4,39 MB/sec (total)
> TxSpeed:4,39 MB/sec (application)
> Error Msg:0
> Rx Msg:1000 messages
> Rx Speed:0,00 MB/sec (since 1st msg)
> Received:4,47 MB]
>
> ...
> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] - INFO -
> 02/12/2008 15:01:16,005 - DeltaManager - Manager [] expiring sessions upon
> shutdo
> wn
> [CLUSTERLOG] pool-2-thread-1 - WARN - 02/12/2008 15:01:20,014 -
> ClusterListener - Context manager doesn't exist:
> [CLUSTERLOG] pool-2-thread-1 - WARN - 02/12/2008 15:01:20,015 -
> ClusterListener - Context manager doesn't exist:
> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] - INFO -
> 02/12/2008 15:01:20,219 - DeltaManager - Register manager to cluster
> element Hos
> t with name localhost
> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] - INFO -
> 02/12/2008 15:01:20,220 - DeltaManager - Starting clustering manager at
> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] - WARN -
> 02/12/2008 15:01:20,220 - DeltaManager - Manager [], requesting session
> state fro
> m
> org.apache.catalina.tribes.membership.MemberImpl[tcp://223.99.114.20:14022,223.99.114.20,14022,
> alive=17762449,id={19 -115 -1 -101 8 91 66 -91 -112 105 -121
> 109 -82 11 56 50 }, payload={}, command={}, domain={}, ]. This operation
> will timeout if no session state has been received within 60 seconds.
> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] - INFO -
> 02/12/2008 15:01:24,093 - DeltaManager - Manager []; session state send at
> 12.02.
> 08 15:01 received in 3.872 ms.
> [CLUSTERLOG] pool-1-thread-1 - INFO - 02/12/2008 15:22:41,143 -
> ThroughputInterceptor - ThroughputInterceptor Report[
> Tx Msg:21000 messages
> Sent:150,30 MB (total)
> Sent:150,30 MB (application)
> Time:70,58 seconds
> Tx Speed:2,13 MB/sec (total)
> TxSpeed:2,13 MB/sec (application)
> Error Msg:0
> Rx Msg:20534 messages
> Rx Speed:0,01 MB/sec (since 1st msg)
> Received:156,92 MB]
>
> after this entry we're NOT receiving any messages (Rx Msg):
>
> [CLUSTERLOG] pool-1-thread-2 - INFO - 02/12/2008 15:40:02,612 -
> ThroughputInterceptor - ThroughputInterceptor Report[
> Tx Msg:22000 messages
> Sent:154,75 MB (total)
> Sent:154,75 MB (application)
> Time:72,92 seconds
> Tx Speed:2,12 MB/sec (total)
> TxSpeed:2,12 MB/sec (application)
> Error Msg:0
> Rx Msg:20534 messages
> Rx Speed:0,01 MB/sec (since 1st msg)
> Received:156,92 MB]
>
> [CLUSTERLOG] pool-1-thread-2 - INFO - 02/12/2008 15:54:15,184 -
> ThroughputInterceptor - ThroughputInterceptor Report[
> Tx Msg:23022 messages
> Sent:156,30 MB (total)
> Sent:156,30 MB (application)
> Time:74,46 seconds
> Tx Speed:2,10 MB/sec (total)
> TxSpeed:2,10 MB/sec (application)
> Error Msg:0
> Rx Msg:20534 messages
> Rx Speed:0,01 MB/sec (since 1st msg)
> Received:156,92 MB]
> ...
>
>
>
> Many thanks for your help!!!
>
>
>
>
>
>
>
>
>
>
>
>
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org