You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by János <ja...@yale.edu> on 2008/07/10 21:12:34 UTC
Version 6 Clustering problem
Hi,
I am trying to establish a 3 member cluster on the same machine.
Here is my server.xml copied from a sample I found on this list.
<?xml version='1.0' encoding='utf-8'?>
<Server port="8005" shutdown="SHUTDOWN">
<Listener
className="org.apache.catalina.core.AprLifecycleListener"
SSLEngine="on" />
<Listener className="org.apache.catalina.core.JasperListener" />
<Listener
className="org.apache.catalina.mbeans.ServerLifecycleListener" />
<Listener
className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
/>
<GlobalNamingResources>
<Resource name="UserDatabase" auth="Container"
type="org.apache.catalina.UserDatabase"
description="User database that can be updated and saved"
factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
pathname="conf/tomcat-users.xml" />
</GlobalNamingResources>
<Service name="Catalina">
<Connector port="8080" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8443" />
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
<Engine name="Catalina" defaultHost="localhost" jvmRoute="node1">
<Cluster
className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
<Realm className="org.apache.catalina.realm.UserDatabaseRealm"
resourceName="UserDatabase"/>
<Host name="localhost" appBase="webapps"
unpackWARs="true" autoDeploy="true"
xmlValidation="false" xmlNamespaceAware="false">
<Context path="" docBase="ROOT"
className="org.apache.catalina.ha.context.ReplicatedContext"/>
</Host>
</Engine>
</Service>
</Server>
The other two server.xml-s are the same, except the shutdown ports
are 8105, 8205, http ports are 8180, 8280 and https ports are 8543,
8643.
As soon as I am starting the second instance I receive the following
into the logs.
In node1 logs:
bml0042:/usr/local/tomcat/node1/logs janos$ cat catalina.2008-07-10.log
Jul 10, 2008 1:03:24 PM org.apache.catalina.core.AprLifecycleListener
init
INFO: The APR based Apache Tomcat Native library which allows optimal
performance in production environments was not found on the
java.library.path: .:/Library/Java/Extensions:/System/Library/Java/
Extensions:/usr/lib/java
Jul 10, 2008 1:03:24 PM org.apache.coyote.http11.Http11Protocol init
INFO: Initializing Coyote HTTP/1.1 on http-8080
Jul 10, 2008 1:03:24 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 1187 ms
Jul 10, 2008 1:03:24 PM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
Jul 10, 2008 1:03:24 PM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.16
Jul 10, 2008 1:03:24 PM org.apache.catalina.ha.tcp.SimpleTcpCluster
start
INFO: Cluster is about to start
Jul 10, 2008 1:03:24 PM
org.apache.catalina.tribes.transport.ReceiverBase bind
INFO: Receiver Server Socket bound to:/10.84.2.42:4000
Jul 10, 2008 1:03:24 PM
org.apache.catalina.tribes.membership.McastServiceImpl setupSocket
INFO: Setting cluster mcast soTimeout to 500
Jul 10, 2008 1:03:24 PM
org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
INFO: Sleeping for 1000 milliseconds to establish cluster membership,
start level:4
Jul 10, 2008 1:03:25 PM
org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
INFO: Done sleeping, membership established, start level:4
Jul 10, 2008 1:03:25 PM
org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
INFO: Sleeping for 1000 milliseconds to establish cluster membership,
start level:8
Jul 10, 2008 1:03:26 PM
org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
INFO: Done sleeping, membership established, start level:8
Jul 10, 2008 1:03:26 PM
org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
INFO: Initializing AbstractReplicatedMap with context name:
Jul 10, 2008 1:03:27 PM org.apache.catalina.core.StandardContext
addApplicationListener
INFO: The listener "listeners.ContextListener" is already configured
for this context. The duplicate definition has been ignored.
Jul 10, 2008 1:03:27 PM org.apache.catalina.core.StandardContext
addApplicationListener
INFO: The listener "listeners.SessionListener" is already configured
for this context. The duplicate definition has been ignored.
Jul 10, 2008 1:03:27 PM org.apache.catalina.ha.session.DeltaManager
start
INFO: Register manager /examples to cluster element Engine with name
Catalina
Jul 10, 2008 1:03:27 PM org.apache.catalina.ha.session.DeltaManager
start
INFO: Starting clustering manager at /examples
Jul 10, 2008 1:03:27 PM org.apache.catalina.ha.session.DeltaManager
getAllClusterSessions
INFO: Manager [localhost#/examples]: skipping state transfer. No
members active in cluster group.
Jul 10, 2008 1:03:28 PM
org.apache.catalina.ha.session.JvmRouteBinderValve start
INFO: JvmRouteBinderValve started
Jul 10, 2008 1:03:28 PM org.apache.coyote.http11.Http11Protocol start
INFO: Starting Coyote HTTP/1.1 on http-8080
Jul 10, 2008 1:03:28 PM org.apache.jk.common.ChannelSocket init
INFO: JK: ajp13 listening on /0.0.0.0:8009
Jul 10, 2008 1:03:28 PM org.apache.jk.server.JkMain start
INFO: Jk running ID=0 time=0/45 config=null
Jul 10, 2008 1:03:28 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 3616 ms
Jul 10, 2008 1:06:41 PM
org.apache.catalina.tribes.transport.nio.NioReceiver listen
SEVERE: Unable to process request in NioReceiver
java.net.SocketException: Invalid argument
at sun.nio.ch.Net.setIntOption0(Native Method)
at sun.nio.ch.Net.setIntOption(Net.java:152)
at sun.nio.ch.SocketChannelImpl$1.setInt
(SocketChannelImpl.java:372)
at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
at sun.nio.ch.SocketOptsImpl$IP.typeOfService
(SocketOptsImpl.java:249)
at sun.nio.ch.OptionAdaptor.setTrafficClass
(OptionAdaptor.java:158)
at sun.nio.ch.SocketAdaptor.setTrafficClass
(SocketAdaptor.java:330)
at
org.apache.catalina.tribes.transport.nio.NioReceiver.listen
(NioReceiver.java:273)
at org.apache.catalina.tribes.transport.nio.NioReceiver.run
(NioReceiver.java:353)
at java.lang.Thread.run(Thread.java:613)
Jul 10, 2008 1:06:41 PM org.apache.catalina.ha.tcp.SimpleTcpCluster
memberAdded
INFO: Replication member
added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,
2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46 127
-68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={}, command={},
domain={}, ]
Jul 10, 2008 1:06:41 PM org.apache.catalina.tribes.io.BufferPool
getBufferPool
INFO: Created a buffer pool with max size:104857600 bytes of
type:org.apache.catalina.tribes.io.BufferPool15Impl
Jul 10, 2008 1:06:41 PM
org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
WARNING: Member send is failing for:tcp://{10, 84, 2, 42}:4001 ;
Setting to suspect and retrying.
Jul 10, 2008 1:06:41 PM
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Received memberDisappeared
[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84, 2,
42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46 127 -68
65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={}, command={},
domain={}, ]] message. Will verify.
Jul 10, 2008 1:06:41 PM
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Verification complete. Member still alive
[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84, 2,
42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46 127 -68
65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={}, command={},
domain={}, ]]
Jul 10, 2008 1:06:41 PM
org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
SEVERE: Unable to send AbstractReplicatedMap.ping message
org.apache.catalina.tribes.ChannelException: Send failed, attempt:2
max:1; Faulty members:tcp://{10, 84, 2, 42}:4001;
at
org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop
(ParallelNioSender.java:172)
at
org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage
(ParallelNioSender.java:78)
at
org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessag
e(PooledParallelSender.java:53)
at
org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(
ReplicationTransmitter.java:80)
at
org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage
(ChannelCoordinator.java:78)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
(ChannelInterceptorBase.java:75)
at
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMes
sage(TcpFailureDetector.java:87)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
(ChannelInterceptorBase.java:75)
at
org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor
.sendMessage(MessageDispatchInterceptor.java:73)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
(ChannelInterceptorBase.java:75)
at org.apache.catalina.tribes.group.GroupChannel.send
(GroupChannel.java:216)
at org.apache.catalina.tribes.group.GroupChannel.send
(GroupChannel.java:175)
at org.apache.catalina.tribes.group.RpcChannel.send
(RpcChannel.java:89)
at
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping
(AbstractReplicatedMap.java:253)
at
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
(AbstractReplicatedMap.java:793)
at org.apache.catalina.tribes.group.GroupChannel.heartbeat
(GroupChannel.java:153)
at org.apache.catalina.tribes.group.GroupChannel
$HeartbeatThread.run(GroupChannel.java:661)
Caused by: java.net.SocketException: Invalid argument
at sun.nio.ch.Net.setIntOption0(Native Method)
at sun.nio.ch.Net.setIntOption(Net.java:152)
at sun.nio.ch.SocketChannelImpl$1.setInt
(SocketChannelImpl.java:372)
at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
at sun.nio.ch.SocketOptsImpl$IP.typeOfService
(SocketOptsImpl.java:249)
at sun.nio.ch.OptionAdaptor.setTrafficClass
(OptionAdaptor.java:158)
at sun.nio.ch.SocketAdaptor.setTrafficClass
(SocketAdaptor.java:330)
at
org.apache.catalina.tribes.transport.nio.NioSender.completeConnect
(NioSender.java:147)
at org.apache.catalina.tribes.transport.nio.NioSender.process
(NioSender.java:89)
at
org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop
(ParallelNioSender.java:130)
... 16 more
Jul 10, 2008 1:06:41 PM
org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
WARNING: Not retrying send for:tcp://{10, 84, 2, 42}:4001; Sender is
disconnected.
Jul 10, 2008 1:06:41 PM
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Received memberDisappeared
[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84, 2,
42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46 127 -68
65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={}, command={},
domain={}, ]] message. Will verify.
Jul 10, 2008 1:06:42 PM
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Verification complete. Member still alive
[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84, 2,
42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46 127 -68
65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={}, command={},
domain={}, ]]
Jul 10, 2008 1:06:42 PM
org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
SEVERE: Unable to send AbstractReplicatedMap.ping message
org.apache.catalina.tribes.ChannelException: Send failed, and sender
is disconnected. Not retrying.; Faulty members:tcp://{10, 84, 2, 42}:
4001;
at
org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop
(ParallelNioSender.java:156)
at
org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage
(ParallelNioSender.java:78)
at
org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessag
e(PooledParallelSender.java:53)
at
org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(
ReplicationTransmitter.java:80)
at
org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage
(ChannelCoordinator.java:78)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
(ChannelInterceptorBase.java:75)
at
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMes
sage(TcpFailureDetector.java:87)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
(ChannelInterceptorBase.java:75)
at
org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor
.sendMessage(MessageDispatchInterceptor.java:73)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
(ChannelInterceptorBase.java:75)
at org.apache.catalina.tribes.group.GroupChannel.send
(GroupChannel.java:216)
at org.apache.catalina.tribes.group.GroupChannel.send
(GroupChannel.java:175)
at org.apache.catalina.tribes.group.RpcChannel.send
(RpcChannel.java:89)
at
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping
(AbstractReplicatedMap.java:253)
at
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
(AbstractReplicatedMap.java:793)
at org.apache.catalina.tribes.group.GroupChannel.heartbeat
(GroupChannel.java:158)
at org.apache.catalina.tribes.group.GroupChannel
$HeartbeatThread.run(GroupChannel.java:661)
Caused by: java.net.SocketException: Invalid argument
at sun.nio.ch.Net.setIntOption0(Native Method)
at sun.nio.ch.Net.setIntOption(Net.java:152)
at sun.nio.ch.SocketChannelImpl$1.setInt
(SocketChannelImpl.java:372)
at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
at sun.nio.ch.SocketOptsImpl$IP.typeOfService
(SocketOptsImpl.java:249)
at sun.nio.ch.OptionAdaptor.setTrafficClass
(OptionAdaptor.java:158)
at sun.nio.ch.SocketAdaptor.setTrafficClass
(SocketAdaptor.java:330)
at
org.apache.catalina.tribes.transport.nio.NioSender.completeConnect
(NioSender.java:147)
at org.apache.catalina.tribes.transport.nio.NioSender.process
(NioSender.java:89)
at
org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop
(ParallelNioSender.java:130)
... 16 more
and so on forever.
In node2 logs I see the following:
bml0042:/usr/local/tomcat/node2/logs janos$ cat catalina.2008-07-10.log
Jul 10, 2008 1:06:40 PM org.apache.catalina.core.AprLifecycleListener
init
INFO: The APR based Apache Tomcat Native library which allows optimal
performance in production environments was not found on the
java.library.path: .:/Library/Java/Extensions:/System/Library/Java/
Extensions:/usr/lib/java
Jul 10, 2008 1:06:40 PM org.apache.coyote.http11.Http11Protocol init
INFO: Initializing Coyote HTTP/1.1 on http-8180
Jul 10, 2008 1:06:40 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 1148 ms
Jul 10, 2008 1:06:40 PM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
Jul 10, 2008 1:06:40 PM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.16
Jul 10, 2008 1:06:40 PM org.apache.catalina.ha.tcp.SimpleTcpCluster
start
INFO: Cluster is about to start
Jul 10, 2008 1:06:40 PM
org.apache.catalina.tribes.transport.ReceiverBase bind
INFO: Receiver Server Socket bound to:/10.84.2.42:4001
Jul 10, 2008 1:06:40 PM
org.apache.catalina.tribes.membership.McastServiceImpl setupSocket
INFO: Setting cluster mcast soTimeout to 500
Jul 10, 2008 1:06:40 PM
org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
INFO: Sleeping for 1000 milliseconds to establish cluster membership,
start level:4
Jul 10, 2008 1:06:41 PM org.apache.catalina.ha.tcp.SimpleTcpCluster
memberAdded
INFO: Replication member
added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,
2, 42}:4000,{10, 84, 2, 42},4000, alive=196161,id={-25 34 -32 116 42
-104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command={},
domain={}, ]
Jul 10, 2008 1:06:41 PM
org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
INFO: Done sleeping, membership established, start level:4
Jul 10, 2008 1:06:41 PM
org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
INFO: Sleeping for 1000 milliseconds to establish cluster membership,
start level:8
Jul 10, 2008 1:06:41 PM
org.apache.catalina.tribes.transport.nio.NioReceiver listen
SEVERE: Unable to process request in NioReceiver
java.net.SocketException: Invalid argument
at sun.nio.ch.Net.setIntOption0(Native Method)
at sun.nio.ch.Net.setIntOption(Net.java:152)
at sun.nio.ch.SocketChannelImpl$1.setInt
(SocketChannelImpl.java:372)
at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
at sun.nio.ch.SocketOptsImpl$IP.typeOfService
(SocketOptsImpl.java:249)
at sun.nio.ch.OptionAdaptor.setTrafficClass
(OptionAdaptor.java:158)
at sun.nio.ch.SocketAdaptor.setTrafficClass
(SocketAdaptor.java:330)
at
org.apache.catalina.tribes.transport.nio.NioReceiver.listen
(NioReceiver.java:273)
at org.apache.catalina.tribes.transport.nio.NioReceiver.run
(NioReceiver.java:353)
at java.lang.Thread.run(Thread.java:613)
Jul 10, 2008 1:06:42 PM
org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
INFO: Done sleeping, membership established, start level:8
Jul 10, 2008 1:06:42 PM
org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
INFO: Initializing AbstractReplicatedMap with context name:
Jul 10, 2008 1:06:42 PM org.apache.catalina.tribes.io.BufferPool
getBufferPool
INFO: Created a buffer pool with max size:104857600 bytes of
type:org.apache.catalina.tribes.io.BufferPool15Impl
Jul 10, 2008 1:06:42 PM
org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
WARNING: Member send is failing for:tcp://{10, 84, 2, 42}:4000 ;
Setting to suspect and retrying.
Jul 10, 2008 1:06:42 PM
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Received memberDisappeared
[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84, 2,
42}:4000,{10, 84, 2, 42},4000, alive=197662,id={-25 34 -32 116 42
-104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command={},
domain={}, ]] message. Will verify.
Jul 10, 2008 1:06:42 PM
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Verification complete. Member still alive
[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84, 2,
42}:4000,{10, 84, 2, 42},4000, alive=197662,id={-25 34 -32 116 42
-104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command={},
domain={}, ]]
Jul 10, 2008 1:06:42 PM
org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
WARNING: Unable to send map start message.
Jul 10, 2008 1:06:42 PM
org.apache.catalina.ha.context.ReplicatedContext start
SEVERE: Unable to start ReplicatedContext
java.lang.RuntimeException: Unable to start replicated map.
at
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.init
(AbstractReplicatedMap.java:230)
at
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.<init>
(AbstractReplicatedMap.java:171)
at org.apache.catalina.tribes.tipis.ReplicatedMap.<init>
(ReplicatedMap.java:90)
at org.apache.catalina.ha.context.ReplicatedContext.start
(ReplicatedContext.java:64)
at org.apache.catalina.core.ContainerBase.start
(ContainerBase.java:1045)
at org.apache.catalina.core.StandardHost.start
(StandardHost.java:719)
at org.apache.catalina.core.ContainerBase.start
(ContainerBase.java:1045)
at org.apache.catalina.core.StandardEngine.start
(StandardEngine.java:443)
at org.apache.catalina.core.StandardService.start
(StandardService.java:516)
at org.apache.catalina.core.StandardServer.start
(StandardServer.java:710)
at org.apache.catalina.startup.Catalina.start(Catalina.java:
578)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.catalina.startup.Bootstrap.start
(Bootstrap.java:288)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:
413)
Caused by: org.apache.catalina.tribes.ChannelException: Send failed,
attempt:2 max:1; Faulty members:tcp://{10, 84, 2, 42}:4000;
at
org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop
(ParallelNioSender.java:172)
at
org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage
(ParallelNioSender.java:78)
at
org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessag
e(PooledParallelSender.java:53)
at
org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(
ReplicationTransmitter.java:80)
at
org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage
(ChannelCoordinator.java:78)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
(ChannelInterceptorBase.java:75)
at
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMes
sage(TcpFailureDetector.java:87)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
(ChannelInterceptorBase.java:75)
at
org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor
.sendMessage(MessageDispatchInterceptor.java:73)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
(ChannelInterceptorBase.java:75)
at org.apache.catalina.tribes.group.GroupChannel.send
(GroupChannel.java:216)
at org.apache.catalina.tribes.group.GroupChannel.send
(GroupChannel.java:175)
at org.apache.catalina.tribes.group.RpcChannel.send
(RpcChannel.java:89)
at
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.broadcast
(AbstractReplicatedMap.java:300)
at
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.init
(AbstractReplicatedMap.java:223)
... 16 more
Caused by: java.net.SocketException: Invalid argument
at sun.nio.ch.Net.setIntOption0(Native Method)
at sun.nio.ch.Net.setIntOption(Net.java:152)
at sun.nio.ch.SocketChannelImpl$1.setInt
(SocketChannelImpl.java:372)
at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
at sun.nio.ch.SocketOptsImpl$IP.typeOfService
(SocketOptsImpl.java:249)
at sun.nio.ch.OptionAdaptor.setTrafficClass
(OptionAdaptor.java:158)
at sun.nio.ch.SocketAdaptor.setTrafficClass
(SocketAdaptor.java:330)
at
org.apache.catalina.tribes.transport.nio.NioSender.completeConnect
(NioSender.java:147)
at org.apache.catalina.tribes.transport.nio.NioSender.process
(NioSender.java:89)
at
org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop
(ParallelNioSender.java:130)
... 30 more
Jul 10, 2008 1:06:42 PM org.apache.catalina.startup.Catalina start
SEVERE: Catalina.start:
LifecycleException: Failed to start ReplicatedContext:
java.lang.RuntimeException: Unable to start replicated map.
at org.apache.catalina.ha.context.ReplicatedContext.start
(ReplicatedContext.java:73)
at org.apache.catalina.core.ContainerBase.start
(ContainerBase.java:1045)
at org.apache.catalina.core.StandardHost.start
(StandardHost.java:719)
at org.apache.catalina.core.ContainerBase.start
(ContainerBase.java:1045)
at org.apache.catalina.core.StandardEngine.start
(StandardEngine.java:443)
at org.apache.catalina.core.StandardService.start
(StandardService.java:516)
at org.apache.catalina.core.StandardServer.start
(StandardServer.java:710)
at org.apache.catalina.startup.Catalina.start(Catalina.java:
578)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.catalina.startup.Bootstrap.start
(Bootstrap.java:288)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:
413)
Jul 10, 2008 1:06:42 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 2329 ms
Jul 10, 2008 1:06:47 PM
org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
WARNING: Not retrying send for:tcp://{10, 84, 2, 42}:4000; Sender is
disconnected.
Jul 10, 2008 1:06:47 PM
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Received memberDisappeared
[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84, 2,
42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32 116 42
-104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command={},
domain={}, ]] message. Will verify.
Jul 10, 2008 1:06:47 PM
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Verification complete. Member still alive
[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84, 2,
42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32 116 42
-104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command={},
domain={}, ]]
Jul 10, 2008 1:06:47 PM
org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
SEVERE: Unable to send AbstractReplicatedMap.ping message
org.apache.catalina.tribes.ChannelException: Send failed, and sender
is disconnected. Not retrying.; Faulty members:tcp://{10, 84, 2, 42}:
4000;
at
org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop
(ParallelNioSender.java:156)
at
org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage
(ParallelNioSender.java:78)
at
org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessag
e(PooledParallelSender.java:53)
at
org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(
ReplicationTransmitter.java:80)
at
org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage
(ChannelCoordinator.java:78)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
(ChannelInterceptorBase.java:75)
at
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMes
sage(TcpFailureDetector.java:87)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
(ChannelInterceptorBase.java:75)
at
org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor
.sendMessage(MessageDispatchInterceptor.java:73)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
(ChannelInterceptorBase.java:75)
at org.apache.catalina.tribes.group.GroupChannel.send
(GroupChannel.java:216)
at org.apache.catalina.tribes.group.GroupChannel.send
(GroupChannel.java:175)
at org.apache.catalina.tribes.group.RpcChannel.send
(RpcChannel.java:89)
at
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping
(AbstractReplicatedMap.java:253)
at
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
(AbstractReplicatedMap.java:793)
at org.apache.catalina.tribes.group.GroupChannel.heartbeat
(GroupChannel.java:153)
at org.apache.catalina.tribes.group.GroupChannel
$HeartbeatThread.run(GroupChannel.java:661)
Caused by: java.net.SocketException: Invalid argument
at sun.nio.ch.Net.setIntOption0(Native Method)
at sun.nio.ch.Net.setIntOption(Net.java:152)
at sun.nio.ch.SocketChannelImpl$1.setInt
(SocketChannelImpl.java:372)
at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
at sun.nio.ch.SocketOptsImpl$IP.typeOfService
(SocketOptsImpl.java:249)
at sun.nio.ch.OptionAdaptor.setTrafficClass
(OptionAdaptor.java:158)
at sun.nio.ch.SocketAdaptor.setTrafficClass
(SocketAdaptor.java:330)
at
org.apache.catalina.tribes.transport.nio.NioSender.completeConnect
(NioSender.java:147)
at org.apache.catalina.tribes.transport.nio.NioSender.process
(NioSender.java:89)
at
org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop
(ParallelNioSender.java:130)
... 16 more
Jul 10, 2008 1:06:47 PM
org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
WARNING: Not retrying send for:tcp://{10, 84, 2, 42}:4000; Sender is
disconnected.
Jul 10, 2008 1:06:47 PM
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Received memberDisappeared
[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84, 2,
42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32 116 42
-104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command={},
domain={}, ]] message. Will verify.
Jul 10, 2008 1:06:47 PM
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Verification complete. Member still alive
[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84, 2,
42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32 116 42
-104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command={},
domain={}, ]]
Jul 10, 2008 1:06:47 PM
org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
SEVERE: Unable to send AbstractReplicatedMap.ping message
org.apache.catalina.tribes.ChannelException: Send failed, and sender
is disconnected. Not retrying.; Faulty members:tcp://{10, 84, 2, 42}:
4000;
at
org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop
(ParallelNioSender.java:156)
at
org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage
(ParallelNioSender.java:78)
at
org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessag
e(PooledParallelSender.java:53)
at
org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(
ReplicationTransmitter.java:80)
at
org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage
(ChannelCoordinator.java:78)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
(ChannelInterceptorBase.java:75)
at
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMes
sage(TcpFailureDetector.java:87)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
(ChannelInterceptorBase.java:75)
at
org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor
.sendMessage(MessageDispatchInterceptor.java:73)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
(ChannelInterceptorBase.java:75)
at org.apache.catalina.tribes.group.GroupChannel.send
(GroupChannel.java:216)
at org.apache.catalina.tribes.group.GroupChannel.send
(GroupChannel.java:175)
at org.apache.catalina.tribes.group.RpcChannel.send
(RpcChannel.java:89)
at
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping
(AbstractReplicatedMap.java:253)
at
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
(AbstractReplicatedMap.java:793)
at org.apache.catalina.tribes.group.GroupChannel.heartbeat
(GroupChannel.java:158)
at org.apache.catalina.tribes.group.GroupChannel
$HeartbeatThread.run(GroupChannel.java:661)
Caused by: java.net.SocketException: Invalid argument
at sun.nio.ch.Net.setIntOption0(Native Method)
at sun.nio.ch.Net.setIntOption(Net.java:152)
at sun.nio.ch.SocketChannelImpl$1.setInt
(SocketChannelImpl.java:372)
at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
at sun.nio.ch.SocketOptsImpl$IP.typeOfService
(SocketOptsImpl.java:249)
at sun.nio.ch.OptionAdaptor.setTrafficClass
(OptionAdaptor.java:158)
at sun.nio.ch.SocketAdaptor.setTrafficClass
(SocketAdaptor.java:330)
at
org.apache.catalina.tribes.transport.nio.NioSender.completeConnect
(NioSender.java:147)
at org.apache.catalina.tribes.transport.nio.NioSender.process
(NioSender.java:89)
at
org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop
(ParallelNioSender.java:130)
... 16 more
and so on. Top shows that both Java process is Up to 86%, the
machine is never idle and all memory is consumed. The machine is a
PowerPc G6 from Apple with OSX 10.4.11.
Tomcat version is: Apache Tomcat/6.0.16
JVM version is: 1.5.0_13-b05-241
I suspect the heartbeat pings have a problem, I just do not know what
to do about them.
Thanks ahead,
János
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Version 6 Clustering problem
Posted by Filip Hanik - Dev Lists <de...@hanik.com>.
János wrote:
> Filip,
>
> You are right. I called the session example wrongly via the
> "bml0042.yalepath.org" host, but that was not configured. As soon as
> I called it via the "localhost" host - which was
glad you got it working
> configured - it worked fine. Here is the log from node3:
>
> bml0042:/usr/local/tomcat janos$ cat node3/logs/localhost.2008-07-14.log
> Jul 14, 2008 2:22:49 PM org.apache.catalina.core.ApplicationContext log
> INFO: SessionListener:
> sessionCreated('B3BA285CF8FABC0BD374C237C22B2B4E.node1')
> Jul 14, 2008 2:23:19 PM org.apache.catalina.core.ApplicationContext log
> INFO: SessionListener:
> attributeAdded('B3BA285CF8FABC0BD374C237C22B2B4E.node1', 'els?', 'egy')
> Jul 14, 2008 2:23:49 PM org.apache.catalina.core.ApplicationContext log
> INFO: SessionListener:
> attributeAdded('B3BA285CF8FABC0BD374C237C22B2B4E.node1', 'm?sodik',
> 'Kett?')
>
> Now I have a working tomcat cluster, I JUST have to make it to work
> with apache via jk_mod :)
>
> How a normal roadrunner, like me, should know about the
> -Djava.net.preferIPv4Stack=true system property and its dramatical
> impact ?
the option doesn't really impact your system
Filip
>
> Thanks a lot,
>
> János
>
> On Jul 14, 2008, at 10:42 AM, Filip Hanik - Dev Lists wrote:
>
>> write your own webapp, to make sure you actually replicate
>> serializable data
>>
>> Filip
>>
>> János wrote:
>>> Hi Filip,
>>>
>>> I added it in the catalina.sh file on all three nodes and I am not
>>> seeing the error. I also disabled IP6 in Network Preferences.
>>>
>>> I added the distrubuted="true" to the context.xml file on all three
>>> nodes and added the <distributable/> tag to the
>>> webapps/examples/WEB-INF/web.xml on all 3 nodes and tried the
>>> SessionExample servlet
>>>
>>> http://bml0042.yalepath.org:8080/examples/servlets/servlet/SessionExample
>>>
>>>
>>> to see if I have a working cluster. I am following the book at
>>>
>>> http://proquest.safaribooksonline.com:80/9780596101060/tomcat_6_clustering_implementation
>>>
>>>
>>> and it says that I should see messages similar to this one in the
>>> nodes' logs"
>>>
>>> Sep 27, 2008 5:39:38 PM org.apache.catalina.core.ApplicationContext log
>>> INFO: SessionListener: contextInitialized( )
>>> Sep 27, 2008 5:40:07 PM org.apache.catalina.core.ApplicationContext log
>>> INFO: SessionListener:
>>> sessionCreated('F4B2D7191C1F335FFAAC93DA461CA95F.tc1')
>>> Sep 27, 2008 5:40:19 PM org.apache.catalina.core.ApplicationContext log
>>> INFO: SessionListener:
>>> attributeAdded('F4B2D7191C1F335FFAAC93DA461CA95F.tc1', 'test',
>>>
>>>
>>> But I am not seeing them. This is the resent log from node1:
>>>
>>> Jul 11, 2008 3:28:01 PM
>>> org.apache.catalina.core.AprLifecycleListener init
>>> INFO: The APR based Apache Tomcat Native library which allows
>>> optimal performance in production environments was not found on the
>>> java.library.path:
>>> .:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java
>>>
>>> Jul 11, 2008 3:28:01 PM org.apache.coyote.http11.Http11Protocol init
>>> INFO: Initializing Coyote HTTP/1.1 on http-8080
>>> Jul 11, 2008 3:28:01 PM org.apache.catalina.startup.Catalina load
>>> INFO: Initialization processed in 1098 ms
>>> Jul 11, 2008 3:28:01 PM org.apache.catalina.core.StandardService start
>>> INFO: Starting service Catalina
>>> Jul 11, 2008 3:28:01 PM org.apache.catalina.core.StandardEngine start
>>> INFO: Starting Servlet Engine: Apache Tomcat/6.0.16
>>> Jul 11, 2008 3:28:01 PM org.apache.catalina.ha.tcp.SimpleTcpCluster
>>> start
>>> INFO: Cluster is about to start
>>> Jul 11, 2008 3:28:01 PM
>>> org.apache.catalina.tribes.transport.ReceiverBase bind
>>> INFO: Receiver Server Socket bound to:/10.84.2.42:4000
>>> Jul 11, 2008 3:28:01 PM
>>> org.apache.catalina.tribes.membership.McastServiceImpl setupSocket
>>> INFO: Setting cluster mcast soTimeout to 500
>>> Jul 11, 2008 3:28:01 PM
>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>> INFO: Sleeping for 1000 milliseconds to establish cluster
>>> membership, start level:4
>>> Jul 11, 2008 3:28:02 PM
>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>> INFO: Done sleeping, membership established, start level:4
>>> Jul 11, 2008 3:28:02 PM
>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>> INFO: Sleeping for 1000 milliseconds to establish cluster
>>> membership, start level:8
>>> Jul 11, 2008 3:28:03 PM
>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>> INFO: Done sleeping, membership established, start level:8
>>> Jul 11, 2008 3:28:03 PM
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
>>> INFO: Initializing AbstractReplicatedMap with context name:
>>> Jul 11, 2008 3:28:04 PM org.apache.catalina.core.StandardContext
>>> addApplicationListener
>>> INFO: The listener "listeners.ContextListener" is already configured
>>> for this context. The duplicate definition has been ignored.
>>> Jul 11, 2008 3:28:04 PM org.apache.catalina.core.StandardContext
>>> addApplicationListener
>>> INFO: The listener "listeners.SessionListener" is already configured
>>> for this context. The duplicate definition has been ignored.
>>> Jul 11, 2008 3:28:04 PM org.apache.catalina.ha.session.DeltaManager
>>> start
>>> INFO: Register manager /examples to cluster element Engine with name
>>> Catalina
>>> Jul 11, 2008 3:28:04 PM org.apache.catalina.ha.session.DeltaManager
>>> start
>>> INFO: Starting clustering manager at /examples
>>> Jul 11, 2008 3:28:04 PM org.apache.catalina.ha.session.DeltaManager
>>> getAllClusterSessions
>>> INFO: Manager [localhost#/examples]: skipping state transfer. No
>>> members active in cluster group.
>>> Jul 11, 2008 3:28:04 PM
>>> org.apache.catalina.ha.session.JvmRouteBinderValve start
>>> INFO: JvmRouteBinderValve started
>>> Jul 11, 2008 3:28:04 PM org.apache.coyote.http11.Http11Protocol start
>>> INFO: Starting Coyote HTTP/1.1 on http-8080
>>> Jul 11, 2008 3:28:05 PM org.apache.jk.common.ChannelSocket init
>>> INFO: JK: ajp13 listening on /0.0.0.0:8009
>>> Jul 11, 2008 3:28:05 PM org.apache.jk.server.JkMain start
>>> INFO: Jk running ID=0 time=0/45 config=null
>>> Jul 11, 2008 3:28:05 PM org.apache.catalina.startup.Catalina start
>>> INFO: Server startup in 3601 ms
>>> Jul 11, 2008 3:28:37 PM org.apache.catalina.tribes.io.BufferPool
>>> getBufferPool
>>> INFO: Created a buffer pool with max size:104857600 bytes of
>>> type:org.apache.catalina.tribes.io.BufferPool15Impl
>>> Jul 11, 2008 3:28:37 PM org.apache.catalina.ha.tcp.SimpleTcpCluster
>>> memberAdded
>>> INFO: Replication member
>>> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
>>> 84, 2, 42}:4001,{10, 84, 2, 42},4001, alive=1020,id={60 103 -111 -34
>>> 98 62 71 -116 -104 76 -100 115 -77 15 -52 -92 }, payload={},
>>> command={}, domain={}, ]
>>> Jul 11, 2008 3:29:08 PM org.apache.catalina.ha.tcp.SimpleTcpCluster
>>> memberAdded
>>> INFO: Replication member
>>> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
>>> 84, 2, 42}:4002,{10, 84, 2, 42},4002, alive=1019,id={27 24 -73 72 48
>>> -6 70 32 -90 57 51 106 -94 -54 126 -52 }, payload={}, command={},
>>> domain={}, ]
>>>
>>>
>>>
>>> Similarly on the other two nodes. What am I missing ?? Should I
>>> add the <Context distributed=True" /> to the server.xml files ?
>>>
>>> Thanks ahead,
>>>
>>> János
>>> On Jul 10, 2008, at 6:03 PM, Filip Hanik - Dev Lists wrote:
>>>
>>>> try setting the
>>>> |-Djava.net.preferIPv4Stack=true
>>>>
>>>> option for your tomcat launch
>>>>
>>>> Filip
>>>> |
>>>> János wrote:
>>>>> Hi,
>>>>>
>>>>> I am trying to establish a 3 member cluster on the same machine.
>>>>> Here is my server.xml copied from a sample I found on this list.
>>>>>
>>>>> <?xml version='1.0' encoding='utf-8'?>
>>>>> <Server port="8005" shutdown="SHUTDOWN">
>>>>> <Listener
>>>>> className="org.apache.catalina.core.AprLifecycleListener"
>>>>> SSLEngine="on" />
>>>>> <Listener className="org.apache.catalina.core.JasperListener" />
>>>>> <Listener
>>>>> className="org.apache.catalina.mbeans.ServerLifecycleListener" />
>>>>> <Listener
>>>>> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
>>>>> />
>>>>>
>>>>> <GlobalNamingResources>
>>>>> <Resource name="UserDatabase" auth="Container"
>>>>> type="org.apache.catalina.UserDatabase"
>>>>> description="User database that can be updated and
>>>>> saved"
>>>>>
>>>>> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
>>>>> pathname="conf/tomcat-users.xml" />
>>>>> </GlobalNamingResources>
>>>>>
>>>>> <Service name="Catalina">
>>>>>
>>>>> <Connector port="8080" protocol="HTTP/1.1"
>>>>> connectionTimeout="20000"
>>>>> redirectPort="8443" />
>>>>> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
>>>>>
>>>>> <Engine name="Catalina" defaultHost="localhost" jvmRoute="node1">
>>>>>
>>>>> <Cluster
>>>>> className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
>>>>> <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
>>>>> resourceName="UserDatabase"/>
>>>>>
>>>>> <Host name="localhost" appBase="webapps"
>>>>> unpackWARs="true" autoDeploy="true"
>>>>> xmlValidation="false" xmlNamespaceAware="false">
>>>>> <Context path="" docBase="ROOT"
>>>>> className="org.apache.catalina.ha.context.ReplicatedContext"/>
>>>>> </Host>
>>>>> </Engine>
>>>>> </Service>
>>>>> </Server>
>>>>>
>>>>> The other two server.xml-s are the same, except the shutdown ports
>>>>> are 8105, 8205, http ports are 8180, 8280 and https ports are
>>>>> 8543, 8643.
>>>>> As soon as I am starting the second instance I receive the
>>>>> following into the logs.
>>>>>
>>>>> In node1 logs:
>>>>>
>>>>> bml0042:/usr/local/tomcat/node1/logs janos$ cat
>>>>> catalina.2008-07-10.log
>>>>> Jul 10, 2008 1:03:24 PM
>>>>> org.apache.catalina.core.AprLifecycleListener init
>>>>> INFO: The APR based Apache Tomcat Native library which allows
>>>>> optimal performance in production environments was not found on
>>>>> the java.library.path:
>>>>> .:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java
>>>>>
>>>>> Jul 10, 2008 1:03:24 PM org.apache.coyote.http11.Http11Protocol init
>>>>> INFO: Initializing Coyote HTTP/1.1 on http-8080
>>>>> Jul 10, 2008 1:03:24 PM org.apache.catalina.startup.Catalina load
>>>>> INFO: Initialization processed in 1187 ms
>>>>> Jul 10, 2008 1:03:24 PM org.apache.catalina.core.StandardService
>>>>> start
>>>>> INFO: Starting service Catalina
>>>>> Jul 10, 2008 1:03:24 PM org.apache.catalina.core.StandardEngine start
>>>>> INFO: Starting Servlet Engine: Apache Tomcat/6.0.16
>>>>> Jul 10, 2008 1:03:24 PM
>>>>> org.apache.catalina.ha.tcp.SimpleTcpCluster start
>>>>> INFO: Cluster is about to start
>>>>> Jul 10, 2008 1:03:24 PM
>>>>> org.apache.catalina.tribes.transport.ReceiverBase bind
>>>>> INFO: Receiver Server Socket bound to:/10.84.2.42:4000
>>>>> Jul 10, 2008 1:03:24 PM
>>>>> org.apache.catalina.tribes.membership.McastServiceImpl setupSocket
>>>>> INFO: Setting cluster mcast soTimeout to 500
>>>>> Jul 10, 2008 1:03:24 PM
>>>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>>>> INFO: Sleeping for 1000 milliseconds to establish cluster
>>>>> membership, start level:4
>>>>> Jul 10, 2008 1:03:25 PM
>>>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>>>> INFO: Done sleeping, membership established, start level:4
>>>>> Jul 10, 2008 1:03:25 PM
>>>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>>>> INFO: Sleeping for 1000 milliseconds to establish cluster
>>>>> membership, start level:8
>>>>> Jul 10, 2008 1:03:26 PM
>>>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>>>> INFO: Done sleeping, membership established, start level:8
>>>>> Jul 10, 2008 1:03:26 PM
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
>>>>> INFO: Initializing AbstractReplicatedMap with context name:
>>>>> Jul 10, 2008 1:03:27 PM org.apache.catalina.core.StandardContext
>>>>> addApplicationListener
>>>>> INFO: The listener "listeners.ContextListener" is already
>>>>> configured for this context. The duplicate definition has been
>>>>> ignored.
>>>>> Jul 10, 2008 1:03:27 PM org.apache.catalina.core.StandardContext
>>>>> addApplicationListener
>>>>> INFO: The listener "listeners.SessionListener" is already
>>>>> configured for this context. The duplicate definition has been
>>>>> ignored.
>>>>> Jul 10, 2008 1:03:27 PM
>>>>> org.apache.catalina.ha.session.DeltaManager start
>>>>> INFO: Register manager /examples to cluster element Engine with
>>>>> name Catalina
>>>>> Jul 10, 2008 1:03:27 PM
>>>>> org.apache.catalina.ha.session.DeltaManager start
>>>>> INFO: Starting clustering manager at /examples
>>>>> Jul 10, 2008 1:03:27 PM
>>>>> org.apache.catalina.ha.session.DeltaManager getAllClusterSessions
>>>>> INFO: Manager [localhost#/examples]: skipping state transfer. No
>>>>> members active in cluster group.
>>>>> Jul 10, 2008 1:03:28 PM
>>>>> org.apache.catalina.ha.session.JvmRouteBinderValve start
>>>>> INFO: JvmRouteBinderValve started
>>>>> Jul 10, 2008 1:03:28 PM org.apache.coyote.http11.Http11Protocol start
>>>>> INFO: Starting Coyote HTTP/1.1 on http-8080
>>>>> Jul 10, 2008 1:03:28 PM org.apache.jk.common.ChannelSocket init
>>>>> INFO: JK: ajp13 listening on /0.0.0.0:8009
>>>>> Jul 10, 2008 1:03:28 PM org.apache.jk.server.JkMain start
>>>>> INFO: Jk running ID=0 time=0/45 config=null
>>>>> Jul 10, 2008 1:03:28 PM org.apache.catalina.startup.Catalina start
>>>>> INFO: Server startup in 3616 ms
>>>>> Jul 10, 2008 1:06:41 PM
>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver listen
>>>>> SEVERE: Unable to process request in NioReceiver
>>>>> java.net.SocketException: Invalid argument
>>>>> at sun.nio.ch.Net.setIntOption0(Native Method)
>>>>> at sun.nio.ch.Net.setIntOption(Net.java:152)
>>>>> at
>>>>> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
>>>>> at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>>>> at
>>>>> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
>>>>> at
>>>>> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
>>>>> at
>>>>> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
>>>>> at
>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:273)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioReceiver.java:353)
>>>>>
>>>>> at java.lang.Thread.run(Thread.java:613)
>>>>> Jul 10, 2008 1:06:41 PM
>>>>> org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
>>>>> INFO: Replication member
>>>>> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
>>>>> 84, 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46
>>>>> 127 -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={},
>>>>> command={}, domain={}, ]
>>>>> Jul 10, 2008 1:06:41 PM org.apache.catalina.tribes.io.BufferPool
>>>>> getBufferPool
>>>>> INFO: Created a buffer pool with max size:104857600 bytes of
>>>>> type:org.apache.catalina.tribes.io.BufferPool15Impl
>>>>> Jul 10, 2008 1:06:41 PM
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>>>>> WARNING: Member send is failing for:tcp://{10, 84, 2, 42}:4001 ;
>>>>> Setting to suspect and retrying.
>>>>> Jul 10, 2008 1:06:41 PM
>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>>>>> memberDisappeared
>>>>> INFO: Received
>>>>> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
>>>>> 84, 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46
>>>>> 127 -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={},
>>>>> command={}, domain={}, ]] message. Will verify.
>>>>> Jul 10, 2008 1:06:41 PM
>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>>>>> memberDisappeared
>>>>> INFO: Verification complete. Member still
>>>>> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
>>>>> 84, 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46
>>>>> 127 -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={},
>>>>> command={}, domain={}, ]]
>>>>> Jul 10, 2008 1:06:41 PM
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
>>>>> SEVERE: Unable to send AbstractReplicatedMap.ping message
>>>>> org.apache.catalina.tribes.ChannelException: Send failed,
>>>>> attempt:2 max:1; Faulty members:tcp://{10, 84, 2, 42}:4001;
>>>>> at
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:172)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:78)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:73)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:89)
>>>>> at
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:253)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:793)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:153)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:661)
>>>>>
>>>>> Caused by: java.net.SocketException: Invalid argument
>>>>> at sun.nio.ch.Net.setIntOption0(Native Method)
>>>>> at sun.nio.ch.Net.setIntOption(Net.java:152)
>>>>> at
>>>>> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
>>>>> at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>>>> at
>>>>> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
>>>>> at
>>>>> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
>>>>> at
>>>>> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
>>>>> at
>>>>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect(NioSender.java:147)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.transport.nio.NioSender.process(NioSender.java:89)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:130)
>>>>>
>>>>> ... 16 more
>>>>> Jul 10, 2008 1:06:41 PM
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>>>>> WARNING: Not retrying send for:tcp://{10, 84, 2, 42}:4001; Sender
>>>>> is disconnected.
>>>>> Jul 10, 2008 1:06:41 PM
>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>>>>> memberDisappeared
>>>>> INFO: Received
>>>>> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
>>>>> 84, 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46
>>>>> 127 -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={},
>>>>> command={}, domain={}, ]] message. Will verify.
>>>>> Jul 10, 2008 1:06:42 PM
>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>>>>> memberDisappeared
>>>>> INFO: Verification complete. Member still
>>>>> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
>>>>> 84, 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46
>>>>> 127 -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={},
>>>>> command={}, domain={}, ]]
>>>>> Jul 10, 2008 1:06:42 PM
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
>>>>> SEVERE: Unable to send AbstractReplicatedMap.ping message
>>>>> org.apache.catalina.tribes.ChannelException: Send failed, and
>>>>> sender is disconnected. Not retrying.; Faulty members:tcp://{10,
>>>>> 84, 2, 42}:4001;
>>>>> at
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:156)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:78)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:73)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:89)
>>>>> at
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:253)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:793)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:158)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:661)
>>>>>
>>>>> Caused by: java.net.SocketException: Invalid argument
>>>>> at sun.nio.ch.Net.setIntOption0(Native Method)
>>>>> at sun.nio.ch.Net.setIntOption(Net.java:152)
>>>>> at
>>>>> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
>>>>> at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>>>> at
>>>>> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
>>>>> at
>>>>> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
>>>>> at
>>>>> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
>>>>> at
>>>>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect(NioSender.java:147)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.transport.nio.NioSender.process(NioSender.java:89)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:130)
>>>>>
>>>>> ... 16 more
>>>>>
>>>>>
>>>>> and so on forever.
>>>>>
>>>>> In node2 logs I see the following:
>>>>>
>>>>> bml0042:/usr/local/tomcat/node2/logs janos$ cat
>>>>> catalina.2008-07-10.log
>>>>> Jul 10, 2008 1:06:40 PM
>>>>> org.apache.catalina.core.AprLifecycleListener init
>>>>> INFO: The APR based Apache Tomcat Native library which allows
>>>>> optimal performance in production environments was not found on
>>>>> the java.library.path:
>>>>> .:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java
>>>>>
>>>>> Jul 10, 2008 1:06:40 PM org.apache.coyote.http11.Http11Protocol init
>>>>> INFO: Initializing Coyote HTTP/1.1 on http-8180
>>>>> Jul 10, 2008 1:06:40 PM org.apache.catalina.startup.Catalina load
>>>>> INFO: Initialization processed in 1148 ms
>>>>> Jul 10, 2008 1:06:40 PM org.apache.catalina.core.StandardService
>>>>> start
>>>>> INFO: Starting service Catalina
>>>>> Jul 10, 2008 1:06:40 PM org.apache.catalina.core.StandardEngine start
>>>>> INFO: Starting Servlet Engine: Apache Tomcat/6.0.16
>>>>> Jul 10, 2008 1:06:40 PM
>>>>> org.apache.catalina.ha.tcp.SimpleTcpCluster start
>>>>> INFO: Cluster is about to start
>>>>> Jul 10, 2008 1:06:40 PM
>>>>> org.apache.catalina.tribes.transport.ReceiverBase bind
>>>>> INFO: Receiver Server Socket bound to:/10.84.2.42:4001
>>>>> Jul 10, 2008 1:06:40 PM
>>>>> org.apache.catalina.tribes.membership.McastServiceImpl setupSocket
>>>>> INFO: Setting cluster mcast soTimeout to 500
>>>>> Jul 10, 2008 1:06:40 PM
>>>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>>>> INFO: Sleeping for 1000 milliseconds to establish cluster
>>>>> membership, start level:4
>>>>> Jul 10, 2008 1:06:41 PM
>>>>> org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
>>>>> INFO: Replication member
>>>>> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
>>>>> 84, 2, 42}:4000,{10, 84, 2, 42},4000, alive=196161,id={-25 34 -32
>>>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={},
>>>>> command={}, domain={}, ]
>>>>> Jul 10, 2008 1:06:41 PM
>>>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>>>> INFO: Done sleeping, membership established, start level:4
>>>>> Jul 10, 2008 1:06:41 PM
>>>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>>>> INFO: Sleeping for 1000 milliseconds to establish cluster
>>>>> membership, start level:8
>>>>> Jul 10, 2008 1:06:41 PM
>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver listen
>>>>> SEVERE: Unable to process request in NioReceiver
>>>>> java.net.SocketException: Invalid argument
>>>>> at sun.nio.ch.Net.setIntOption0(Native Method)
>>>>> at sun.nio.ch.Net.setIntOption(Net.java:152)
>>>>> at
>>>>> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
>>>>> at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>>>> at
>>>>> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
>>>>> at
>>>>> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
>>>>> at
>>>>> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
>>>>> at
>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:273)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioReceiver.java:353)
>>>>>
>>>>> at java.lang.Thread.run(Thread.java:613)
>>>>> Jul 10, 2008 1:06:42 PM
>>>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>>>> INFO: Done sleeping, membership established, start level:8
>>>>> Jul 10, 2008 1:06:42 PM
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
>>>>> INFO: Initializing AbstractReplicatedMap with context name:
>>>>> Jul 10, 2008 1:06:42 PM org.apache.catalina.tribes.io.BufferPool
>>>>> getBufferPool
>>>>> INFO: Created a buffer pool with max size:104857600 bytes of
>>>>> type:org.apache.catalina.tribes.io.BufferPool15Impl
>>>>> Jul 10, 2008 1:06:42 PM
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>>>>> WARNING: Member send is failing for:tcp://{10, 84, 2, 42}:4000 ;
>>>>> Setting to suspect and retrying.
>>>>> Jul 10, 2008 1:06:42 PM
>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>>>>> memberDisappeared
>>>>> INFO: Received
>>>>> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
>>>>> 84, 2, 42}:4000,{10, 84, 2, 42},4000, alive=197662,id={-25 34 -32
>>>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={},
>>>>> command={}, domain={}, ]] message. Will verify.
>>>>> Jul 10, 2008 1:06:42 PM
>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>>>>> memberDisappeared
>>>>> INFO: Verification complete. Member still
>>>>> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
>>>>> 84, 2, 42}:4000,{10, 84, 2, 42},4000, alive=197662,id={-25 34 -32
>>>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={},
>>>>> command={}, domain={}, ]]
>>>>> Jul 10, 2008 1:06:42 PM
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
>>>>> WARNING: Unable to send map start message.
>>>>> Jul 10, 2008 1:06:42 PM
>>>>> org.apache.catalina.ha.context.ReplicatedContext start
>>>>> SEVERE: Unable to start ReplicatedContext
>>>>> java.lang.RuntimeException: Unable to start replicated map.
>>>>> at
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.init(AbstractReplicatedMap.java:230)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.<init>(AbstractReplicatedMap.java:171)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.tipis.ReplicatedMap.<init>(ReplicatedMap.java:90)
>>>>>
>>>>> at
>>>>> org.apache.catalina.ha.context.ReplicatedContext.start(ReplicatedContext.java:64)
>>>>>
>>>>> at
>>>>> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
>>>>> at
>>>>> org.apache.catalina.core.StandardHost.start(StandardHost.java:719)
>>>>> at
>>>>> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
>>>>> at
>>>>> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443)
>>>>>
>>>>> at
>>>>> org.apache.catalina.core.StandardService.start(StandardService.java:516)
>>>>>
>>>>> at
>>>>> org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
>>>>>
>>>>> at
>>>>> org.apache.catalina.startup.Catalina.start(Catalina.java:578)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>>> Method)
>>>>> at
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>
>>>>> at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>
>>>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>>>> at
>>>>> org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
>>>>> at
>>>>> org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
>>>>> Caused by: org.apache.catalina.tribes.ChannelException: Send
>>>>> failed, attempt:2 max:1; Faulty members:tcp://{10, 84, 2, 42}:4000;
>>>>> at
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:172)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:78)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:73)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:89)
>>>>> at
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.broadcast(AbstractReplicatedMap.java:300)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.init(AbstractReplicatedMap.java:223)
>>>>>
>>>>> ... 16 more
>>>>> Caused by: java.net.SocketException: Invalid argument
>>>>> at sun.nio.ch.Net.setIntOption0(Native Method)
>>>>> at sun.nio.ch.Net.setIntOption(Net.java:152)
>>>>> at
>>>>> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
>>>>> at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>>>> at
>>>>> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
>>>>> at
>>>>> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
>>>>> at
>>>>> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
>>>>> at
>>>>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect(NioSender.java:147)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.transport.nio.NioSender.process(NioSender.java:89)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:130)
>>>>>
>>>>> ... 30 more
>>>>> Jul 10, 2008 1:06:42 PM org.apache.catalina.startup.Catalina start
>>>>> SEVERE: Catalina.start:
>>>>> LifecycleException: Failed to start ReplicatedContext:
>>>>> java.lang.RuntimeException: Unable to start replicated map.
>>>>> at
>>>>> org.apache.catalina.ha.context.ReplicatedContext.start(ReplicatedContext.java:73)
>>>>>
>>>>> at
>>>>> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
>>>>> at
>>>>> org.apache.catalina.core.StandardHost.start(StandardHost.java:719)
>>>>> at
>>>>> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
>>>>> at
>>>>> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443)
>>>>>
>>>>> at
>>>>> org.apache.catalina.core.StandardService.start(StandardService.java:516)
>>>>>
>>>>> at
>>>>> org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
>>>>>
>>>>> at
>>>>> org.apache.catalina.startup.Catalina.start(Catalina.java:578)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>>> Method)
>>>>> at
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>
>>>>> at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>
>>>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>>>> at
>>>>> org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
>>>>> at
>>>>> org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
>>>>> Jul 10, 2008 1:06:42 PM org.apache.catalina.startup.Catalina start
>>>>> INFO: Server startup in 2329 ms
>>>>> Jul 10, 2008 1:06:47 PM
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>>>>> WARNING: Not retrying send for:tcp://{10, 84, 2, 42}:4000; Sender
>>>>> is disconnected.
>>>>> Jul 10, 2008 1:06:47 PM
>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>>>>> memberDisappeared
>>>>> INFO: Received
>>>>> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
>>>>> 84, 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32
>>>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={},
>>>>> command={}, domain={}, ]] message. Will verify.
>>>>> Jul 10, 2008 1:06:47 PM
>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>>>>> memberDisappeared
>>>>> INFO: Verification complete. Member still
>>>>> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
>>>>> 84, 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32
>>>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={},
>>>>> command={}, domain={}, ]]
>>>>> Jul 10, 2008 1:06:47 PM
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
>>>>> SEVERE: Unable to send AbstractReplicatedMap.ping message
>>>>> org.apache.catalina.tribes.ChannelException: Send failed, and
>>>>> sender is disconnected. Not retrying.; Faulty members:tcp://{10,
>>>>> 84, 2, 42}:4000;
>>>>> at
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:156)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:78)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:73)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:89)
>>>>> at
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:253)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:793)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:153)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:661)
>>>>>
>>>>> Caused by: java.net.SocketException: Invalid argument
>>>>> at sun.nio.ch.Net.setIntOption0(Native Method)
>>>>> at sun.nio.ch.Net.setIntOption(Net.java:152)
>>>>> at
>>>>> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
>>>>> at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>>>> at
>>>>> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
>>>>> at
>>>>> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
>>>>> at
>>>>> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
>>>>> at
>>>>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect(NioSender.java:147)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.transport.nio.NioSender.process(NioSender.java:89)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:130)
>>>>>
>>>>> ... 16 more
>>>>> Jul 10, 2008 1:06:47 PM
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>>>>> WARNING: Not retrying send for:tcp://{10, 84, 2, 42}:4000; Sender
>>>>> is disconnected.
>>>>> Jul 10, 2008 1:06:47 PM
>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>>>>> memberDisappeared
>>>>> INFO: Received
>>>>> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
>>>>> 84, 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32
>>>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={},
>>>>> command={}, domain={}, ]] message. Will verify.
>>>>> Jul 10, 2008 1:06:47 PM
>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>>>>> memberDisappeared
>>>>> INFO: Verification complete. Member still
>>>>> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
>>>>> 84, 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32
>>>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={},
>>>>> command={}, domain={}, ]]
>>>>> Jul 10, 2008 1:06:47 PM
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
>>>>> SEVERE: Unable to send AbstractReplicatedMap.ping message
>>>>> org.apache.catalina.tribes.ChannelException: Send failed, and
>>>>> sender is disconnected. Not retrying.; Faulty members:tcp://{10,
>>>>> 84, 2, 42}:4000;
>>>>> at
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:156)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:78)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:73)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:89)
>>>>> at
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:253)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:793)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:158)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:661)
>>>>>
>>>>> Caused by: java.net.SocketException: Invalid argument
>>>>> at sun.nio.ch.Net.setIntOption0(Native Method)
>>>>> at sun.nio.ch.Net.setIntOption(Net.java:152)
>>>>> at
>>>>> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
>>>>> at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>>>> at
>>>>> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
>>>>> at
>>>>> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
>>>>> at
>>>>> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
>>>>> at
>>>>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect(NioSender.java:147)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.transport.nio.NioSender.process(NioSender.java:89)
>>>>>
>>>>> at
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:130)
>>>>>
>>>>> ... 16 more
>>>>>
>>>>>
>>>>> and so on. Top shows that both Java process is Up to 86%, the
>>>>> machine is never idle and all memory is consumed. The machine is
>>>>> a PowerPc G6 from Apple with OSX 10.4.11.
>>>>> Tomcat version is: Apache Tomcat/6.0.16
>>>>> JVM version is: 1.5.0_13-b05-241
>>>>> I suspect the heartbeat pings have a problem, I just do not know
>>>>> what to do about them.
>>>>>
>>>>> Thanks ahead,
>>>>>
>>>>> János
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To start a new topic, e-mail: users@tomcat.apache.org
>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To start a new topic, e-mail: users@tomcat.apache.org
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To start a new topic, e-mail: users@tomcat.apache.org
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To start a new topic, e-mail: users@tomcat.apache.org
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Version 6 Clustering problem
Posted by János <ja...@yale.edu>.
Filip,
You are right. I called the session example wrongly via the
"bml0042.yalepath.org" host, but that was not configured. As soon as
I called it via the "localhost" host - which was configured - it
worked fine. Here is the log from node3:
bml0042:/usr/local/tomcat janos$ cat node3/logs/localhost.2008-07-14.log
Jul 14, 2008 2:22:49 PM org.apache.catalina.core.ApplicationContext log
INFO: SessionListener: sessionCreated
('B3BA285CF8FABC0BD374C237C22B2B4E.node1')
Jul 14, 2008 2:23:19 PM org.apache.catalina.core.ApplicationContext log
INFO: SessionListener: attributeAdded
('B3BA285CF8FABC0BD374C237C22B2B4E.node1', 'els?', 'egy')
Jul 14, 2008 2:23:49 PM org.apache.catalina.core.ApplicationContext log
INFO: SessionListener: attributeAdded
('B3BA285CF8FABC0BD374C237C22B2B4E.node1', 'm?sodik', 'Kett?')
Now I have a working tomcat cluster, I JUST have to make it to work
with apache via jk_mod :)
How a normal roadrunner, like me, should know about the -
Djava.net.preferIPv4Stack=true system property and its dramatical
impact ?
Thanks a lot,
János
On Jul 14, 2008, at 10:42 AM, Filip Hanik - Dev Lists wrote:
> write your own webapp, to make sure you actually replicate
> serializable data
>
> Filip
>
> János wrote:
>> Hi Filip,
>>
>> I added it in the catalina.sh file on all three nodes and I am not
>> seeing the error. I also disabled IP6 in Network Preferences.
>>
>> I added the distrubuted="true" to the context.xml file on all
>> three nodes and added the <distributable/> tag to the webapps/
>> examples/WEB-INF/web.xml on all 3 nodes and tried the
>> SessionExample servlet
>>
>> http://bml0042.yalepath.org:8080/examples/servlets/servlet/
>> SessionExample
>>
>> to see if I have a working cluster. I am following the book at
>>
>> http://proquest.safaribooksonline.com:80/9780596101060/
>> tomcat_6_clustering_implementation
>>
>> and it says that I should see messages similar to this one in the
>> nodes' logs"
>>
>> Sep 27, 2008 5:39:38 PM
>> org.apache.catalina.core.ApplicationContext log
>> INFO: SessionListener: contextInitialized( )
>> Sep 27, 2008 5:40:07 PM
>> org.apache.catalina.core.ApplicationContext log
>> INFO: SessionListener: sessionCreated
>> ('F4B2D7191C1F335FFAAC93DA461CA95F.tc1')
>> Sep 27, 2008 5:40:19 PM
>> org.apache.catalina.core.ApplicationContext log
>> INFO: SessionListener: attributeAdded
>> ('F4B2D7191C1F335FFAAC93DA461CA95F.tc1', 'test',
>>
>>
>> But I am not seeing them. This is the resent log from node1:
>>
>> Jul 11, 2008 3:28:01 PM
>> org.apache.catalina.core.AprLifecycleListener init
>> INFO: The APR based Apache Tomcat Native library which allows
>> optimal performance in production environments was not found on
>> the java.library.path: .:/Library/Java/Extensions:/System/Library/
>> Java/Extensions:/usr/lib/java
>> Jul 11, 2008 3:28:01 PM org.apache.coyote.http11.Http11Protocol init
>> INFO: Initializing Coyote HTTP/1.1 on http-8080
>> Jul 11, 2008 3:28:01 PM org.apache.catalina.startup.Catalina load
>> INFO: Initialization processed in 1098 ms
>> Jul 11, 2008 3:28:01 PM org.apache.catalina.core.StandardService
>> start
>> INFO: Starting service Catalina
>> Jul 11, 2008 3:28:01 PM org.apache.catalina.core.StandardEngine start
>> INFO: Starting Servlet Engine: Apache Tomcat/6.0.16
>> Jul 11, 2008 3:28:01 PM
>> org.apache.catalina.ha.tcp.SimpleTcpCluster start
>> INFO: Cluster is about to start
>> Jul 11, 2008 3:28:01 PM
>> org.apache.catalina.tribes.transport.ReceiverBase bind
>> INFO: Receiver Server Socket bound to:/10.84.2.42:4000
>> Jul 11, 2008 3:28:01 PM
>> org.apache.catalina.tribes.membership.McastServiceImpl setupSocket
>> INFO: Setting cluster mcast soTimeout to 500
>> Jul 11, 2008 3:28:01 PM
>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>> INFO: Sleeping for 1000 milliseconds to establish cluster
>> membership, start level:4
>> Jul 11, 2008 3:28:02 PM
>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>> INFO: Done sleeping, membership established, start level:4
>> Jul 11, 2008 3:28:02 PM
>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>> INFO: Sleeping for 1000 milliseconds to establish cluster
>> membership, start level:8
>> Jul 11, 2008 3:28:03 PM
>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>> INFO: Done sleeping, membership established, start level:8
>> Jul 11, 2008 3:28:03 PM
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
>> INFO: Initializing AbstractReplicatedMap with context name:
>> Jul 11, 2008 3:28:04 PM org.apache.catalina.core.StandardContext
>> addApplicationListener
>> INFO: The listener "listeners.ContextListener" is already
>> configured for this context. The duplicate definition has been
>> ignored.
>> Jul 11, 2008 3:28:04 PM org.apache.catalina.core.StandardContext
>> addApplicationListener
>> INFO: The listener "listeners.SessionListener" is already
>> configured for this context. The duplicate definition has been
>> ignored.
>> Jul 11, 2008 3:28:04 PM
>> org.apache.catalina.ha.session.DeltaManager start
>> INFO: Register manager /examples to cluster element Engine with
>> name Catalina
>> Jul 11, 2008 3:28:04 PM
>> org.apache.catalina.ha.session.DeltaManager start
>> INFO: Starting clustering manager at /examples
>> Jul 11, 2008 3:28:04 PM
>> org.apache.catalina.ha.session.DeltaManager getAllClusterSessions
>> INFO: Manager [localhost#/examples]: skipping state transfer. No
>> members active in cluster group.
>> Jul 11, 2008 3:28:04 PM
>> org.apache.catalina.ha.session.JvmRouteBinderValve start
>> INFO: JvmRouteBinderValve started
>> Jul 11, 2008 3:28:04 PM org.apache.coyote.http11.Http11Protocol start
>> INFO: Starting Coyote HTTP/1.1 on http-8080
>> Jul 11, 2008 3:28:05 PM org.apache.jk.common.ChannelSocket init
>> INFO: JK: ajp13 listening on /0.0.0.0:8009
>> Jul 11, 2008 3:28:05 PM org.apache.jk.server.JkMain start
>> INFO: Jk running ID=0 time=0/45 config=null
>> Jul 11, 2008 3:28:05 PM org.apache.catalina.startup.Catalina start
>> INFO: Server startup in 3601 ms
>> Jul 11, 2008 3:28:37 PM org.apache.catalina.tribes.io.BufferPool
>> getBufferPool
>> INFO: Created a buffer pool with max size:104857600 bytes of
>> type:org.apache.catalina.tribes.io.BufferPool15Impl
>> Jul 11, 2008 3:28:37 PM
>> org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
>> INFO: Replication member
>> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
>> 84, 2, 42}:4001,{10, 84, 2, 42},4001, alive=1020,id={60 103 -111
>> -34 98 62 71 -116 -104 76 -100 115 -77 15 -52 -92 }, payload={},
>> command={}, domain={}, ]
>> Jul 11, 2008 3:29:08 PM
>> org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
>> INFO: Replication member
>> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
>> 84, 2, 42}:4002,{10, 84, 2, 42},4002, alive=1019,id={27 24 -73 72
>> 48 -6 70 32 -90 57 51 106 -94 -54 126 -52 }, payload={}, command=
>> {}, domain={}, ]
>>
>>
>>
>> Similarly on the other two nodes. What am I missing ?? Should I
>> add the <Context distributed=True" /> to the server.xml files ?
>>
>> Thanks ahead,
>>
>> János
>> On Jul 10, 2008, at 6:03 PM, Filip Hanik - Dev Lists wrote:
>>
>>> try setting the
>>> |-Djava.net.preferIPv4Stack=true
>>>
>>> option for your tomcat launch
>>>
>>> Filip
>>> |
>>> János wrote:
>>>> Hi,
>>>>
>>>> I am trying to establish a 3 member cluster on the same
>>>> machine. Here is my server.xml copied from a sample I found on
>>>> this list.
>>>>
>>>> <?xml version='1.0' encoding='utf-8'?>
>>>> <Server port="8005" shutdown="SHUTDOWN">
>>>> <Listener
>>>> className="org.apache.catalina.core.AprLifecycleListener"
>>>> SSLEngine="on" />
>>>> <Listener className="org.apache.catalina.core.JasperListener" />
>>>> <Listener
>>>> className="org.apache.catalina.mbeans.ServerLifecycleListener" />
>>>> <Listener
>>>> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListe
>>>> ner" />
>>>>
>>>> <GlobalNamingResources>
>>>> <Resource name="UserDatabase" auth="Container"
>>>> type="org.apache.catalina.UserDatabase"
>>>> description="User database that can be updated and
>>>> saved"
>>>>
>>>> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
>>>> pathname="conf/tomcat-users.xml" />
>>>> </GlobalNamingResources>
>>>>
>>>> <Service name="Catalina">
>>>>
>>>> <Connector port="8080" protocol="HTTP/1.1"
>>>> connectionTimeout="20000"
>>>> redirectPort="8443" />
>>>> <Connector port="8009" protocol="AJP/1.3"
>>>> redirectPort="8443" />
>>>>
>>>> <Engine name="Catalina" defaultHost="localhost"
>>>> jvmRoute="node1">
>>>>
>>>> <Cluster
>>>> className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
>>>> <Realm
>>>> className="org.apache.catalina.realm.UserDatabaseRealm"
>>>> resourceName="UserDatabase"/>
>>>>
>>>> <Host name="localhost" appBase="webapps"
>>>> unpackWARs="true" autoDeploy="true"
>>>> xmlValidation="false" xmlNamespaceAware="false">
>>>> <Context path="" docBase="ROOT"
>>>> className="org.apache.catalina.ha.context.ReplicatedContext"/>
>>>> </Host>
>>>> </Engine>
>>>> </Service>
>>>> </Server>
>>>>
>>>> The other two server.xml-s are the same, except the shutdown
>>>> ports are 8105, 8205, http ports are 8180, 8280 and https ports
>>>> are 8543, 8643.
>>>> As soon as I am starting the second instance I receive the
>>>> following into the logs.
>>>>
>>>> In node1 logs:
>>>>
>>>> bml0042:/usr/local/tomcat/node1/logs janos$ cat catalina.
>>>> 2008-07-10.log
>>>> Jul 10, 2008 1:03:24 PM
>>>> org.apache.catalina.core.AprLifecycleListener init
>>>> INFO: The APR based Apache Tomcat Native library which allows
>>>> optimal performance in production environments was not found on
>>>> the java.library.path: .:/Library/Java/Extensions:/System/
>>>> Library/Java/Extensions:/usr/lib/java
>>>> Jul 10, 2008 1:03:24 PM org.apache.coyote.http11.Http11Protocol
>>>> init
>>>> INFO: Initializing Coyote HTTP/1.1 on http-8080
>>>> Jul 10, 2008 1:03:24 PM org.apache.catalina.startup.Catalina load
>>>> INFO: Initialization processed in 1187 ms
>>>> Jul 10, 2008 1:03:24 PM org.apache.catalina.core.StandardService
>>>> start
>>>> INFO: Starting service Catalina
>>>> Jul 10, 2008 1:03:24 PM org.apache.catalina.core.StandardEngine
>>>> start
>>>> INFO: Starting Servlet Engine: Apache Tomcat/6.0.16
>>>> Jul 10, 2008 1:03:24 PM
>>>> org.apache.catalina.ha.tcp.SimpleTcpCluster start
>>>> INFO: Cluster is about to start
>>>> Jul 10, 2008 1:03:24 PM
>>>> org.apache.catalina.tribes.transport.ReceiverBase bind
>>>> INFO: Receiver Server Socket bound to:/10.84.2.42:4000
>>>> Jul 10, 2008 1:03:24 PM
>>>> org.apache.catalina.tribes.membership.McastServiceImpl setupSocket
>>>> INFO: Setting cluster mcast soTimeout to 500
>>>> Jul 10, 2008 1:03:24 PM
>>>> org.apache.catalina.tribes.membership.McastServiceImpl
>>>> waitForMembers
>>>> INFO: Sleeping for 1000 milliseconds to establish cluster
>>>> membership, start level:4
>>>> Jul 10, 2008 1:03:25 PM
>>>> org.apache.catalina.tribes.membership.McastServiceImpl
>>>> waitForMembers
>>>> INFO: Done sleeping, membership established, start level:4
>>>> Jul 10, 2008 1:03:25 PM
>>>> org.apache.catalina.tribes.membership.McastServiceImpl
>>>> waitForMembers
>>>> INFO: Sleeping for 1000 milliseconds to establish cluster
>>>> membership, start level:8
>>>> Jul 10, 2008 1:03:26 PM
>>>> org.apache.catalina.tribes.membership.McastServiceImpl
>>>> waitForMembers
>>>> INFO: Done sleeping, membership established, start level:8
>>>> Jul 10, 2008 1:03:26 PM
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
>>>> INFO: Initializing AbstractReplicatedMap with context name:
>>>> Jul 10, 2008 1:03:27 PM org.apache.catalina.core.StandardContext
>>>> addApplicationListener
>>>> INFO: The listener "listeners.ContextListener" is already
>>>> configured for this context. The duplicate definition has been
>>>> ignored.
>>>> Jul 10, 2008 1:03:27 PM org.apache.catalina.core.StandardContext
>>>> addApplicationListener
>>>> INFO: The listener "listeners.SessionListener" is already
>>>> configured for this context. The duplicate definition has been
>>>> ignored.
>>>> Jul 10, 2008 1:03:27 PM
>>>> org.apache.catalina.ha.session.DeltaManager start
>>>> INFO: Register manager /examples to cluster element Engine with
>>>> name Catalina
>>>> Jul 10, 2008 1:03:27 PM
>>>> org.apache.catalina.ha.session.DeltaManager start
>>>> INFO: Starting clustering manager at /examples
>>>> Jul 10, 2008 1:03:27 PM
>>>> org.apache.catalina.ha.session.DeltaManager getAllClusterSessions
>>>> INFO: Manager [localhost#/examples]: skipping state transfer. No
>>>> members active in cluster group.
>>>> Jul 10, 2008 1:03:28 PM
>>>> org.apache.catalina.ha.session.JvmRouteBinderValve start
>>>> INFO: JvmRouteBinderValve started
>>>> Jul 10, 2008 1:03:28 PM org.apache.coyote.http11.Http11Protocol
>>>> start
>>>> INFO: Starting Coyote HTTP/1.1 on http-8080
>>>> Jul 10, 2008 1:03:28 PM org.apache.jk.common.ChannelSocket init
>>>> INFO: JK: ajp13 listening on /0.0.0.0:8009
>>>> Jul 10, 2008 1:03:28 PM org.apache.jk.server.JkMain start
>>>> INFO: Jk running ID=0 time=0/45 config=null
>>>> Jul 10, 2008 1:03:28 PM org.apache.catalina.startup.Catalina start
>>>> INFO: Server startup in 3616 ms
>>>> Jul 10, 2008 1:06:41 PM
>>>> org.apache.catalina.tribes.transport.nio.NioReceiver listen
>>>> SEVERE: Unable to process request in NioReceiver
>>>> java.net.SocketException: Invalid argument
>>>> at sun.nio.ch.Net.setIntOption0(Native Method)
>>>> at sun.nio.ch.Net.setIntOption(Net.java:152)
>>>> at sun.nio.ch.SocketChannelImpl$1.setInt
>>>> (SocketChannelImpl.java:372)
>>>> at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>>> at sun.nio.ch.SocketOptsImpl$IP.typeOfService
>>>> (SocketOptsImpl.java:249)
>>>> at sun.nio.ch.OptionAdaptor.setTrafficClass
>>>> (OptionAdaptor.java:158)
>>>> at sun.nio.ch.SocketAdaptor.setTrafficClass
>>>> (SocketAdaptor.java:330)
>>>> at
>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen
>>>> (NioReceiver.java:273)
>>>> at
>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.run
>>>> (NioReceiver.java:353)
>>>> at java.lang.Thread.run(Thread.java:613)
>>>> Jul 10, 2008 1:06:41 PM
>>>> org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
>>>> INFO: Replication member
>>>> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://
>>>> {10, 84, 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24
>>>> -65 -46 127 -68 65 119 -107 -12 -61 -24 -12 -122 10 56 },
>>>> payload={}, command={}, domain={}, ]
>>>> Jul 10, 2008 1:06:41 PM org.apache.catalina.tribes.io.BufferPool
>>>> getBufferPool
>>>> INFO: Created a buffer pool with max size:104857600 bytes of
>>>> type:org.apache.catalina.tribes.io.BufferPool15Impl
>>>> Jul 10, 2008 1:06:41 PM
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>>>> WARNING: Member send is failing for:tcp://{10, 84, 2, 42}:4001 ;
>>>> Setting to suspect and retrying.
>>>> Jul 10, 2008 1:06:41 PM
>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>>>> memberDisappeared
>>>> INFO: Received memberDisappeared
>>>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,
>>>> 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46
>>>> 127 -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={},
>>>> command={}, domain={}, ]] message. Will verify.
>>>> Jul 10, 2008 1:06:41 PM
>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>>>> memberDisappeared
>>>> INFO: Verification complete. Member still alive
>>>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,
>>>> 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46
>>>> 127 -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={},
>>>> command={}, domain={}, ]]
>>>> Jul 10, 2008 1:06:41 PM
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
>>>> SEVERE: Unable to send AbstractReplicatedMap.ping message
>>>> org.apache.catalina.tribes.ChannelException: Send failed,
>>>> attempt:2 max:1; Faulty members:tcp://{10, 84, 2, 42}:4001;
>>>> at
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop
>>>> (ParallelNioSender.java:172)
>>>> at
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMess
>>>> age(ParallelNioSender.java:78)
>>>> at
>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendM
>>>> essage(PooledParallelSender.java:53)
>>>> at
>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMes
>>>> sage(ReplicationTransmitter.java:80)
>>>> at
>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage
>>>> (ChannelCoordinator.java:78)
>>>> at
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
>>>> (ChannelInterceptorBase.java:75)
>>>> at
>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.se
>>>> ndMessage(TcpFailureDetector.java:87)
>>>> at
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
>>>> (ChannelInterceptorBase.java:75)
>>>> at
>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterc
>>>> eptor.sendMessage(MessageDispatchInterceptor.java:73)
>>>> at
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
>>>> (ChannelInterceptorBase.java:75)
>>>> at org.apache.catalina.tribes.group.GroupChannel.send
>>>> (GroupChannel.java:216)
>>>> at org.apache.catalina.tribes.group.GroupChannel.send
>>>> (GroupChannel.java:175)
>>>> at org.apache.catalina.tribes.group.RpcChannel.send
>>>> (RpcChannel.java:89)
>>>> at
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping
>>>> (AbstractReplicatedMap.java:253)
>>>> at
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
>>>> (AbstractReplicatedMap.java:793)
>>>> at
>>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat
>>>> (GroupChannel.java:153)
>>>> at org.apache.catalina.tribes.group.GroupChannel
>>>> $HeartbeatThread.run(GroupChannel.java:661)
>>>> Caused by: java.net.SocketException: Invalid argument
>>>> at sun.nio.ch.Net.setIntOption0(Native Method)
>>>> at sun.nio.ch.Net.setIntOption(Net.java:152)
>>>> at sun.nio.ch.SocketChannelImpl$1.setInt
>>>> (SocketChannelImpl.java:372)
>>>> at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>>> at sun.nio.ch.SocketOptsImpl$IP.typeOfService
>>>> (SocketOptsImpl.java:249)
>>>> at sun.nio.ch.OptionAdaptor.setTrafficClass
>>>> (OptionAdaptor.java:158)
>>>> at sun.nio.ch.SocketAdaptor.setTrafficClass
>>>> (SocketAdaptor.java:330)
>>>> at
>>>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect(
>>>> NioSender.java:147)
>>>> at
>>>> org.apache.catalina.tribes.transport.nio.NioSender.process
>>>> (NioSender.java:89)
>>>> at
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop
>>>> (ParallelNioSender.java:130)
>>>> ... 16 more
>>>> Jul 10, 2008 1:06:41 PM
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>>>> WARNING: Not retrying send for:tcp://{10, 84, 2, 42}:4001;
>>>> Sender is disconnected.
>>>> Jul 10, 2008 1:06:41 PM
>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>>>> memberDisappeared
>>>> INFO: Received memberDisappeared
>>>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,
>>>> 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46
>>>> 127 -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={},
>>>> command={}, domain={}, ]] message. Will verify.
>>>> Jul 10, 2008 1:06:42 PM
>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>>>> memberDisappeared
>>>> INFO: Verification complete. Member still alive
>>>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,
>>>> 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46
>>>> 127 -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={},
>>>> command={}, domain={}, ]]
>>>> Jul 10, 2008 1:06:42 PM
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
>>>> SEVERE: Unable to send AbstractReplicatedMap.ping message
>>>> org.apache.catalina.tribes.ChannelException: Send failed, and
>>>> sender is disconnected. Not retrying.; Faulty members:tcp://{10,
>>>> 84, 2, 42}:4001;
>>>> at
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop
>>>> (ParallelNioSender.java:156)
>>>> at
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMess
>>>> age(ParallelNioSender.java:78)
>>>> at
>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendM
>>>> essage(PooledParallelSender.java:53)
>>>> at
>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMes
>>>> sage(ReplicationTransmitter.java:80)
>>>> at
>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage
>>>> (ChannelCoordinator.java:78)
>>>> at
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
>>>> (ChannelInterceptorBase.java:75)
>>>> at
>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.se
>>>> ndMessage(TcpFailureDetector.java:87)
>>>> at
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
>>>> (ChannelInterceptorBase.java:75)
>>>> at
>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterc
>>>> eptor.sendMessage(MessageDispatchInterceptor.java:73)
>>>> at
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
>>>> (ChannelInterceptorBase.java:75)
>>>> at org.apache.catalina.tribes.group.GroupChannel.send
>>>> (GroupChannel.java:216)
>>>> at org.apache.catalina.tribes.group.GroupChannel.send
>>>> (GroupChannel.java:175)
>>>> at org.apache.catalina.tribes.group.RpcChannel.send
>>>> (RpcChannel.java:89)
>>>> at
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping
>>>> (AbstractReplicatedMap.java:253)
>>>> at
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
>>>> (AbstractReplicatedMap.java:793)
>>>> at
>>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat
>>>> (GroupChannel.java:158)
>>>> at org.apache.catalina.tribes.group.GroupChannel
>>>> $HeartbeatThread.run(GroupChannel.java:661)
>>>> Caused by: java.net.SocketException: Invalid argument
>>>> at sun.nio.ch.Net.setIntOption0(Native Method)
>>>> at sun.nio.ch.Net.setIntOption(Net.java:152)
>>>> at sun.nio.ch.SocketChannelImpl$1.setInt
>>>> (SocketChannelImpl.java:372)
>>>> at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>>> at sun.nio.ch.SocketOptsImpl$IP.typeOfService
>>>> (SocketOptsImpl.java:249)
>>>> at sun.nio.ch.OptionAdaptor.setTrafficClass
>>>> (OptionAdaptor.java:158)
>>>> at sun.nio.ch.SocketAdaptor.setTrafficClass
>>>> (SocketAdaptor.java:330)
>>>> at
>>>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect(
>>>> NioSender.java:147)
>>>> at
>>>> org.apache.catalina.tribes.transport.nio.NioSender.process
>>>> (NioSender.java:89)
>>>> at
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop
>>>> (ParallelNioSender.java:130)
>>>> ... 16 more
>>>>
>>>>
>>>> and so on forever.
>>>>
>>>> In node2 logs I see the following:
>>>>
>>>> bml0042:/usr/local/tomcat/node2/logs janos$ cat catalina.
>>>> 2008-07-10.log
>>>> Jul 10, 2008 1:06:40 PM
>>>> org.apache.catalina.core.AprLifecycleListener init
>>>> INFO: The APR based Apache Tomcat Native library which allows
>>>> optimal performance in production environments was not found on
>>>> the java.library.path: .:/Library/Java/Extensions:/System/
>>>> Library/Java/Extensions:/usr/lib/java
>>>> Jul 10, 2008 1:06:40 PM org.apache.coyote.http11.Http11Protocol
>>>> init
>>>> INFO: Initializing Coyote HTTP/1.1 on http-8180
>>>> Jul 10, 2008 1:06:40 PM org.apache.catalina.startup.Catalina load
>>>> INFO: Initialization processed in 1148 ms
>>>> Jul 10, 2008 1:06:40 PM org.apache.catalina.core.StandardService
>>>> start
>>>> INFO: Starting service Catalina
>>>> Jul 10, 2008 1:06:40 PM org.apache.catalina.core.StandardEngine
>>>> start
>>>> INFO: Starting Servlet Engine: Apache Tomcat/6.0.16
>>>> Jul 10, 2008 1:06:40 PM
>>>> org.apache.catalina.ha.tcp.SimpleTcpCluster start
>>>> INFO: Cluster is about to start
>>>> Jul 10, 2008 1:06:40 PM
>>>> org.apache.catalina.tribes.transport.ReceiverBase bind
>>>> INFO: Receiver Server Socket bound to:/10.84.2.42:4001
>>>> Jul 10, 2008 1:06:40 PM
>>>> org.apache.catalina.tribes.membership.McastServiceImpl setupSocket
>>>> INFO: Setting cluster mcast soTimeout to 500
>>>> Jul 10, 2008 1:06:40 PM
>>>> org.apache.catalina.tribes.membership.McastServiceImpl
>>>> waitForMembers
>>>> INFO: Sleeping for 1000 milliseconds to establish cluster
>>>> membership, start level:4
>>>> Jul 10, 2008 1:06:41 PM
>>>> org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
>>>> INFO: Replication member
>>>> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://
>>>> {10, 84, 2, 42}:4000,{10, 84, 2, 42},4000, alive=196161,id={-25
>>>> 34 -32 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 },
>>>> payload={}, command={}, domain={}, ]
>>>> Jul 10, 2008 1:06:41 PM
>>>> org.apache.catalina.tribes.membership.McastServiceImpl
>>>> waitForMembers
>>>> INFO: Done sleeping, membership established, start level:4
>>>> Jul 10, 2008 1:06:41 PM
>>>> org.apache.catalina.tribes.membership.McastServiceImpl
>>>> waitForMembers
>>>> INFO: Sleeping for 1000 milliseconds to establish cluster
>>>> membership, start level:8
>>>> Jul 10, 2008 1:06:41 PM
>>>> org.apache.catalina.tribes.transport.nio.NioReceiver listen
>>>> SEVERE: Unable to process request in NioReceiver
>>>> java.net.SocketException: Invalid argument
>>>> at sun.nio.ch.Net.setIntOption0(Native Method)
>>>> at sun.nio.ch.Net.setIntOption(Net.java:152)
>>>> at sun.nio.ch.SocketChannelImpl$1.setInt
>>>> (SocketChannelImpl.java:372)
>>>> at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>>> at sun.nio.ch.SocketOptsImpl$IP.typeOfService
>>>> (SocketOptsImpl.java:249)
>>>> at sun.nio.ch.OptionAdaptor.setTrafficClass
>>>> (OptionAdaptor.java:158)
>>>> at sun.nio.ch.SocketAdaptor.setTrafficClass
>>>> (SocketAdaptor.java:330)
>>>> at
>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen
>>>> (NioReceiver.java:273)
>>>> at
>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.run
>>>> (NioReceiver.java:353)
>>>> at java.lang.Thread.run(Thread.java:613)
>>>> Jul 10, 2008 1:06:42 PM
>>>> org.apache.catalina.tribes.membership.McastServiceImpl
>>>> waitForMembers
>>>> INFO: Done sleeping, membership established, start level:8
>>>> Jul 10, 2008 1:06:42 PM
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
>>>> INFO: Initializing AbstractReplicatedMap with context name:
>>>> Jul 10, 2008 1:06:42 PM org.apache.catalina.tribes.io.BufferPool
>>>> getBufferPool
>>>> INFO: Created a buffer pool with max size:104857600 bytes of
>>>> type:org.apache.catalina.tribes.io.BufferPool15Impl
>>>> Jul 10, 2008 1:06:42 PM
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>>>> WARNING: Member send is failing for:tcp://{10, 84, 2, 42}:4000 ;
>>>> Setting to suspect and retrying.
>>>> Jul 10, 2008 1:06:42 PM
>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>>>> memberDisappeared
>>>> INFO: Received memberDisappeared
>>>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,
>>>> 2, 42}:4000,{10, 84, 2, 42},4000, alive=197662,id={-25 34 -32
>>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={},
>>>> command={}, domain={}, ]] message. Will verify.
>>>> Jul 10, 2008 1:06:42 PM
>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>>>> memberDisappeared
>>>> INFO: Verification complete. Member still alive
>>>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,
>>>> 2, 42}:4000,{10, 84, 2, 42},4000, alive=197662,id={-25 34 -32
>>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={},
>>>> command={}, domain={}, ]]
>>>> Jul 10, 2008 1:06:42 PM
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
>>>> WARNING: Unable to send map start message.
>>>> Jul 10, 2008 1:06:42 PM
>>>> org.apache.catalina.ha.context.ReplicatedContext start
>>>> SEVERE: Unable to start ReplicatedContext
>>>> java.lang.RuntimeException: Unable to start replicated map.
>>>> at
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.init
>>>> (AbstractReplicatedMap.java:230)
>>>> at
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.<init>
>>>> (AbstractReplicatedMap.java:171)
>>>> at org.apache.catalina.tribes.tipis.ReplicatedMap.<init>
>>>> (ReplicatedMap.java:90)
>>>> at org.apache.catalina.ha.context.ReplicatedContext.start
>>>> (ReplicatedContext.java:64)
>>>> at org.apache.catalina.core.ContainerBase.start
>>>> (ContainerBase.java:1045)
>>>> at org.apache.catalina.core.StandardHost.start
>>>> (StandardHost.java:719)
>>>> at org.apache.catalina.core.ContainerBase.start
>>>> (ContainerBase.java:1045)
>>>> at org.apache.catalina.core.StandardEngine.start
>>>> (StandardEngine.java:443)
>>>> at org.apache.catalina.core.StandardService.start
>>>> (StandardService.java:516)
>>>> at org.apache.catalina.core.StandardServer.start
>>>> (StandardServer.java:710)
>>>> at org.apache.catalina.startup.Catalina.start
>>>> (Catalina.java:578)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke
>>>> (NativeMethodAccessorImpl.java:39)
>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke
>>>> (DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>>> at org.apache.catalina.startup.Bootstrap.start
>>>> (Bootstrap.java:288)
>>>> at org.apache.catalina.startup.Bootstrap.main
>>>> (Bootstrap.java:413)
>>>> Caused by: org.apache.catalina.tribes.ChannelException: Send
>>>> failed, attempt:2 max:1; Faulty members:tcp://{10, 84, 2, 42}:4000;
>>>> at
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop
>>>> (ParallelNioSender.java:172)
>>>> at
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMess
>>>> age(ParallelNioSender.java:78)
>>>> at
>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendM
>>>> essage(PooledParallelSender.java:53)
>>>> at
>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMes
>>>> sage(ReplicationTransmitter.java:80)
>>>> at
>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage
>>>> (ChannelCoordinator.java:78)
>>>> at
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
>>>> (ChannelInterceptorBase.java:75)
>>>> at
>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.se
>>>> ndMessage(TcpFailureDetector.java:87)
>>>> at
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
>>>> (ChannelInterceptorBase.java:75)
>>>> at
>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterc
>>>> eptor.sendMessage(MessageDispatchInterceptor.java:73)
>>>> at
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
>>>> (ChannelInterceptorBase.java:75)
>>>> at org.apache.catalina.tribes.group.GroupChannel.send
>>>> (GroupChannel.java:216)
>>>> at org.apache.catalina.tribes.group.GroupChannel.send
>>>> (GroupChannel.java:175)
>>>> at org.apache.catalina.tribes.group.RpcChannel.send
>>>> (RpcChannel.java:89)
>>>> at
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.broadcast
>>>> (AbstractReplicatedMap.java:300)
>>>> at
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.init
>>>> (AbstractReplicatedMap.java:223)
>>>> ... 16 more
>>>> Caused by: java.net.SocketException: Invalid argument
>>>> at sun.nio.ch.Net.setIntOption0(Native Method)
>>>> at sun.nio.ch.Net.setIntOption(Net.java:152)
>>>> at sun.nio.ch.SocketChannelImpl$1.setInt
>>>> (SocketChannelImpl.java:372)
>>>> at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>>> at sun.nio.ch.SocketOptsImpl$IP.typeOfService
>>>> (SocketOptsImpl.java:249)
>>>> at sun.nio.ch.OptionAdaptor.setTrafficClass
>>>> (OptionAdaptor.java:158)
>>>> at sun.nio.ch.SocketAdaptor.setTrafficClass
>>>> (SocketAdaptor.java:330)
>>>> at
>>>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect(
>>>> NioSender.java:147)
>>>> at
>>>> org.apache.catalina.tribes.transport.nio.NioSender.process
>>>> (NioSender.java:89)
>>>> at
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop
>>>> (ParallelNioSender.java:130)
>>>> ... 30 more
>>>> Jul 10, 2008 1:06:42 PM org.apache.catalina.startup.Catalina start
>>>> SEVERE: Catalina.start:
>>>> LifecycleException: Failed to start ReplicatedContext:
>>>> java.lang.RuntimeException: Unable to start replicated map.
>>>> at org.apache.catalina.ha.context.ReplicatedContext.start
>>>> (ReplicatedContext.java:73)
>>>> at org.apache.catalina.core.ContainerBase.start
>>>> (ContainerBase.java:1045)
>>>> at org.apache.catalina.core.StandardHost.start
>>>> (StandardHost.java:719)
>>>> at org.apache.catalina.core.ContainerBase.start
>>>> (ContainerBase.java:1045)
>>>> at org.apache.catalina.core.StandardEngine.start
>>>> (StandardEngine.java:443)
>>>> at org.apache.catalina.core.StandardService.start
>>>> (StandardService.java:516)
>>>> at org.apache.catalina.core.StandardServer.start
>>>> (StandardServer.java:710)
>>>> at org.apache.catalina.startup.Catalina.start
>>>> (Catalina.java:578)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke
>>>> (NativeMethodAccessorImpl.java:39)
>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke
>>>> (DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>>> at org.apache.catalina.startup.Bootstrap.start
>>>> (Bootstrap.java:288)
>>>> at org.apache.catalina.startup.Bootstrap.main
>>>> (Bootstrap.java:413)
>>>> Jul 10, 2008 1:06:42 PM org.apache.catalina.startup.Catalina start
>>>> INFO: Server startup in 2329 ms
>>>> Jul 10, 2008 1:06:47 PM
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>>>> WARNING: Not retrying send for:tcp://{10, 84, 2, 42}:4000;
>>>> Sender is disconnected.
>>>> Jul 10, 2008 1:06:47 PM
>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>>>> memberDisappeared
>>>> INFO: Received memberDisappeared
>>>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,
>>>> 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32
>>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={},
>>>> command={}, domain={}, ]] message. Will verify.
>>>> Jul 10, 2008 1:06:47 PM
>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>>>> memberDisappeared
>>>> INFO: Verification complete. Member still alive
>>>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,
>>>> 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32
>>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={},
>>>> command={}, domain={}, ]]
>>>> Jul 10, 2008 1:06:47 PM
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
>>>> SEVERE: Unable to send AbstractReplicatedMap.ping message
>>>> org.apache.catalina.tribes.ChannelException: Send failed, and
>>>> sender is disconnected. Not retrying.; Faulty members:tcp://{10,
>>>> 84, 2, 42}:4000;
>>>> at
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop
>>>> (ParallelNioSender.java:156)
>>>> at
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMess
>>>> age(ParallelNioSender.java:78)
>>>> at
>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendM
>>>> essage(PooledParallelSender.java:53)
>>>> at
>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMes
>>>> sage(ReplicationTransmitter.java:80)
>>>> at
>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage
>>>> (ChannelCoordinator.java:78)
>>>> at
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
>>>> (ChannelInterceptorBase.java:75)
>>>> at
>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.se
>>>> ndMessage(TcpFailureDetector.java:87)
>>>> at
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
>>>> (ChannelInterceptorBase.java:75)
>>>> at
>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterc
>>>> eptor.sendMessage(MessageDispatchInterceptor.java:73)
>>>> at
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
>>>> (ChannelInterceptorBase.java:75)
>>>> at org.apache.catalina.tribes.group.GroupChannel.send
>>>> (GroupChannel.java:216)
>>>> at org.apache.catalina.tribes.group.GroupChannel.send
>>>> (GroupChannel.java:175)
>>>> at org.apache.catalina.tribes.group.RpcChannel.send
>>>> (RpcChannel.java:89)
>>>> at
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping
>>>> (AbstractReplicatedMap.java:253)
>>>> at
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
>>>> (AbstractReplicatedMap.java:793)
>>>> at
>>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat
>>>> (GroupChannel.java:153)
>>>> at org.apache.catalina.tribes.group.GroupChannel
>>>> $HeartbeatThread.run(GroupChannel.java:661)
>>>> Caused by: java.net.SocketException: Invalid argument
>>>> at sun.nio.ch.Net.setIntOption0(Native Method)
>>>> at sun.nio.ch.Net.setIntOption(Net.java:152)
>>>> at sun.nio.ch.SocketChannelImpl$1.setInt
>>>> (SocketChannelImpl.java:372)
>>>> at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>>> at sun.nio.ch.SocketOptsImpl$IP.typeOfService
>>>> (SocketOptsImpl.java:249)
>>>> at sun.nio.ch.OptionAdaptor.setTrafficClass
>>>> (OptionAdaptor.java:158)
>>>> at sun.nio.ch.SocketAdaptor.setTrafficClass
>>>> (SocketAdaptor.java:330)
>>>> at
>>>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect(
>>>> NioSender.java:147)
>>>> at
>>>> org.apache.catalina.tribes.transport.nio.NioSender.process
>>>> (NioSender.java:89)
>>>> at
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop
>>>> (ParallelNioSender.java:130)
>>>> ... 16 more
>>>> Jul 10, 2008 1:06:47 PM
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>>>> WARNING: Not retrying send for:tcp://{10, 84, 2, 42}:4000;
>>>> Sender is disconnected.
>>>> Jul 10, 2008 1:06:47 PM
>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>>>> memberDisappeared
>>>> INFO: Received memberDisappeared
>>>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,
>>>> 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32
>>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={},
>>>> command={}, domain={}, ]] message. Will verify.
>>>> Jul 10, 2008 1:06:47 PM
>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>>>> memberDisappeared
>>>> INFO: Verification complete. Member still alive
>>>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,
>>>> 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32
>>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={},
>>>> command={}, domain={}, ]]
>>>> Jul 10, 2008 1:06:47 PM
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
>>>> SEVERE: Unable to send AbstractReplicatedMap.ping message
>>>> org.apache.catalina.tribes.ChannelException: Send failed, and
>>>> sender is disconnected. Not retrying.; Faulty members:tcp://{10,
>>>> 84, 2, 42}:4000;
>>>> at
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop
>>>> (ParallelNioSender.java:156)
>>>> at
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMess
>>>> age(ParallelNioSender.java:78)
>>>> at
>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendM
>>>> essage(PooledParallelSender.java:53)
>>>> at
>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMes
>>>> sage(ReplicationTransmitter.java:80)
>>>> at
>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage
>>>> (ChannelCoordinator.java:78)
>>>> at
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
>>>> (ChannelInterceptorBase.java:75)
>>>> at
>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.se
>>>> ndMessage(TcpFailureDetector.java:87)
>>>> at
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
>>>> (ChannelInterceptorBase.java:75)
>>>> at
>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterc
>>>> eptor.sendMessage(MessageDispatchInterceptor.java:73)
>>>> at
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
>>>> (ChannelInterceptorBase.java:75)
>>>> at org.apache.catalina.tribes.group.GroupChannel.send
>>>> (GroupChannel.java:216)
>>>> at org.apache.catalina.tribes.group.GroupChannel.send
>>>> (GroupChannel.java:175)
>>>> at org.apache.catalina.tribes.group.RpcChannel.send
>>>> (RpcChannel.java:89)
>>>> at
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping
>>>> (AbstractReplicatedMap.java:253)
>>>> at
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
>>>> (AbstractReplicatedMap.java:793)
>>>> at
>>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat
>>>> (GroupChannel.java:158)
>>>> at org.apache.catalina.tribes.group.GroupChannel
>>>> $HeartbeatThread.run(GroupChannel.java:661)
>>>> Caused by: java.net.SocketException: Invalid argument
>>>> at sun.nio.ch.Net.setIntOption0(Native Method)
>>>> at sun.nio.ch.Net.setIntOption(Net.java:152)
>>>> at sun.nio.ch.SocketChannelImpl$1.setInt
>>>> (SocketChannelImpl.java:372)
>>>> at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>>> at sun.nio.ch.SocketOptsImpl$IP.typeOfService
>>>> (SocketOptsImpl.java:249)
>>>> at sun.nio.ch.OptionAdaptor.setTrafficClass
>>>> (OptionAdaptor.java:158)
>>>> at sun.nio.ch.SocketAdaptor.setTrafficClass
>>>> (SocketAdaptor.java:330)
>>>> at
>>>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect(
>>>> NioSender.java:147)
>>>> at
>>>> org.apache.catalina.tribes.transport.nio.NioSender.process
>>>> (NioSender.java:89)
>>>> at
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop
>>>> (ParallelNioSender.java:130)
>>>> ... 16 more
>>>>
>>>>
>>>> and so on. Top shows that both Java process is Up to 86%, the
>>>> machine is never idle and all memory is consumed. The machine
>>>> is a PowerPc G6 from Apple with OSX 10.4.11.
>>>> Tomcat version is: Apache Tomcat/6.0.16
>>>> JVM version is: 1.5.0_13-b05-241
>>>> I suspect the heartbeat pings have a problem, I just do not know
>>>> what to do about them.
>>>>
>>>> Thanks ahead,
>>>>
>>>> János
>>>>
>>>>
>>>>
>>>> -------------------------------------------------------------------
>>>> --
>>>> To start a new topic, e-mail: users@tomcat.apache.org
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>>
>>>>
>>>
>>>
>>> --------------------------------------------------------------------
>>> -
>>> To start a new topic, e-mail: users@tomcat.apache.org
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To start a new topic, e-mail: users@tomcat.apache.org
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>>
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Version 6 Clustering problem
Posted by Filip Hanik - Dev Lists <de...@hanik.com>.
write your own webapp, to make sure you actually replicate serializable data
Filip
János wrote:
> Hi Filip,
>
> I added it in the catalina.sh file on all three nodes and I am not
> seeing the error. I also disabled IP6 in Network Preferences.
>
> I added the distrubuted="true" to the context.xml file on all three
> nodes and added the <distributable/> tag to the
> webapps/examples/WEB-INF/web.xml on all 3 nodes and tried the
> SessionExample servlet
>
> http://bml0042.yalepath.org:8080/examples/servlets/servlet/SessionExample
>
> to see if I have a working cluster. I am following the book at
>
> http://proquest.safaribooksonline.com:80/9780596101060/tomcat_6_clustering_implementation
>
>
> and it says that I should see messages similar to this one in the
> nodes' logs"
>
> Sep 27, 2008 5:39:38 PM org.apache.catalina.core.ApplicationContext log
> INFO: SessionListener: contextInitialized( )
> Sep 27, 2008 5:40:07 PM org.apache.catalina.core.ApplicationContext log
> INFO: SessionListener:
> sessionCreated('F4B2D7191C1F335FFAAC93DA461CA95F.tc1')
> Sep 27, 2008 5:40:19 PM org.apache.catalina.core.ApplicationContext log
> INFO: SessionListener:
> attributeAdded('F4B2D7191C1F335FFAAC93DA461CA95F.tc1', 'test',
>
>
> But I am not seeing them. This is the resent log from node1:
>
> Jul 11, 2008 3:28:01 PM org.apache.catalina.core.AprLifecycleListener
> init
> INFO: The APR based Apache Tomcat Native library which allows optimal
> performance in production environments was not found on the
> java.library.path:
> .:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java
> Jul 11, 2008 3:28:01 PM org.apache.coyote.http11.Http11Protocol init
> INFO: Initializing Coyote HTTP/1.1 on http-8080
> Jul 11, 2008 3:28:01 PM org.apache.catalina.startup.Catalina load
> INFO: Initialization processed in 1098 ms
> Jul 11, 2008 3:28:01 PM org.apache.catalina.core.StandardService start
> INFO: Starting service Catalina
> Jul 11, 2008 3:28:01 PM org.apache.catalina.core.StandardEngine start
> INFO: Starting Servlet Engine: Apache Tomcat/6.0.16
> Jul 11, 2008 3:28:01 PM org.apache.catalina.ha.tcp.SimpleTcpCluster start
> INFO: Cluster is about to start
> Jul 11, 2008 3:28:01 PM
> org.apache.catalina.tribes.transport.ReceiverBase bind
> INFO: Receiver Server Socket bound to:/10.84.2.42:4000
> Jul 11, 2008 3:28:01 PM
> org.apache.catalina.tribes.membership.McastServiceImpl setupSocket
> INFO: Setting cluster mcast soTimeout to 500
> Jul 11, 2008 3:28:01 PM
> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
> INFO: Sleeping for 1000 milliseconds to establish cluster membership,
> start level:4
> Jul 11, 2008 3:28:02 PM
> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
> INFO: Done sleeping, membership established, start level:4
> Jul 11, 2008 3:28:02 PM
> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
> INFO: Sleeping for 1000 milliseconds to establish cluster membership,
> start level:8
> Jul 11, 2008 3:28:03 PM
> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
> INFO: Done sleeping, membership established, start level:8
> Jul 11, 2008 3:28:03 PM
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
> INFO: Initializing AbstractReplicatedMap with context name:
> Jul 11, 2008 3:28:04 PM org.apache.catalina.core.StandardContext
> addApplicationListener
> INFO: The listener "listeners.ContextListener" is already configured
> for this context. The duplicate definition has been ignored.
> Jul 11, 2008 3:28:04 PM org.apache.catalina.core.StandardContext
> addApplicationListener
> INFO: The listener "listeners.SessionListener" is already configured
> for this context. The duplicate definition has been ignored.
> Jul 11, 2008 3:28:04 PM org.apache.catalina.ha.session.DeltaManager start
> INFO: Register manager /examples to cluster element Engine with name
> Catalina
> Jul 11, 2008 3:28:04 PM org.apache.catalina.ha.session.DeltaManager start
> INFO: Starting clustering manager at /examples
> Jul 11, 2008 3:28:04 PM org.apache.catalina.ha.session.DeltaManager
> getAllClusterSessions
> INFO: Manager [localhost#/examples]: skipping state transfer. No
> members active in cluster group.
> Jul 11, 2008 3:28:04 PM
> org.apache.catalina.ha.session.JvmRouteBinderValve start
> INFO: JvmRouteBinderValve started
> Jul 11, 2008 3:28:04 PM org.apache.coyote.http11.Http11Protocol start
> INFO: Starting Coyote HTTP/1.1 on http-8080
> Jul 11, 2008 3:28:05 PM org.apache.jk.common.ChannelSocket init
> INFO: JK: ajp13 listening on /0.0.0.0:8009
> Jul 11, 2008 3:28:05 PM org.apache.jk.server.JkMain start
> INFO: Jk running ID=0 time=0/45 config=null
> Jul 11, 2008 3:28:05 PM org.apache.catalina.startup.Catalina start
> INFO: Server startup in 3601 ms
> Jul 11, 2008 3:28:37 PM org.apache.catalina.tribes.io.BufferPool
> getBufferPool
> INFO: Created a buffer pool with max size:104857600 bytes of
> type:org.apache.catalina.tribes.io.BufferPool15Impl
> Jul 11, 2008 3:28:37 PM org.apache.catalina.ha.tcp.SimpleTcpCluster
> memberAdded
> INFO: Replication member
> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,
> 2, 42}:4001,{10, 84, 2, 42},4001, alive=1020,id={60 103 -111 -34 98 62
> 71 -116 -104 76 -100 115 -77 15 -52 -92 }, payload={}, command={},
> domain={}, ]
> Jul 11, 2008 3:29:08 PM org.apache.catalina.ha.tcp.SimpleTcpCluster
> memberAdded
> INFO: Replication member
> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,
> 2, 42}:4002,{10, 84, 2, 42},4002, alive=1019,id={27 24 -73 72 48 -6 70
> 32 -90 57 51 106 -94 -54 126 -52 }, payload={}, command={}, domain={}, ]
>
>
>
> Similarly on the other two nodes. What am I missing ?? Should I add
> the <Context distributed=True" /> to the server.xml files ?
>
> Thanks ahead,
>
> János
> On Jul 10, 2008, at 6:03 PM, Filip Hanik - Dev Lists wrote:
>
>> try setting the
>> |-Djava.net.preferIPv4Stack=true
>>
>> option for your tomcat launch
>>
>> Filip
>> |
>> János wrote:
>>> Hi,
>>>
>>> I am trying to establish a 3 member cluster on the same machine.
>>> Here is my server.xml copied from a sample I found on this list.
>>>
>>> <?xml version='1.0' encoding='utf-8'?>
>>> <Server port="8005" shutdown="SHUTDOWN">
>>> <Listener
>>> className="org.apache.catalina.core.AprLifecycleListener"
>>> SSLEngine="on" />
>>> <Listener className="org.apache.catalina.core.JasperListener" />
>>> <Listener
>>> className="org.apache.catalina.mbeans.ServerLifecycleListener" />
>>> <Listener
>>> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
>>> />
>>>
>>> <GlobalNamingResources>
>>> <Resource name="UserDatabase" auth="Container"
>>> type="org.apache.catalina.UserDatabase"
>>> description="User database that can be updated and saved"
>>>
>>> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
>>> pathname="conf/tomcat-users.xml" />
>>> </GlobalNamingResources>
>>>
>>> <Service name="Catalina">
>>>
>>> <Connector port="8080" protocol="HTTP/1.1"
>>> connectionTimeout="20000"
>>> redirectPort="8443" />
>>> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
>>>
>>> <Engine name="Catalina" defaultHost="localhost" jvmRoute="node1">
>>>
>>> <Cluster
>>> className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
>>> <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
>>> resourceName="UserDatabase"/>
>>>
>>> <Host name="localhost" appBase="webapps"
>>> unpackWARs="true" autoDeploy="true"
>>> xmlValidation="false" xmlNamespaceAware="false">
>>> <Context path="" docBase="ROOT"
>>> className="org.apache.catalina.ha.context.ReplicatedContext"/>
>>> </Host>
>>> </Engine>
>>> </Service>
>>> </Server>
>>>
>>> The other two server.xml-s are the same, except the shutdown ports
>>> are 8105, 8205, http ports are 8180, 8280 and https ports are 8543,
>>> 8643.
>>> As soon as I am starting the second instance I receive the following
>>> into the logs.
>>>
>>> In node1 logs:
>>>
>>> bml0042:/usr/local/tomcat/node1/logs janos$ cat catalina.2008-07-10.log
>>> Jul 10, 2008 1:03:24 PM
>>> org.apache.catalina.core.AprLifecycleListener init
>>> INFO: The APR based Apache Tomcat Native library which allows
>>> optimal performance in production environments was not found on the
>>> java.library.path:
>>> .:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java
>>>
>>> Jul 10, 2008 1:03:24 PM org.apache.coyote.http11.Http11Protocol init
>>> INFO: Initializing Coyote HTTP/1.1 on http-8080
>>> Jul 10, 2008 1:03:24 PM org.apache.catalina.startup.Catalina load
>>> INFO: Initialization processed in 1187 ms
>>> Jul 10, 2008 1:03:24 PM org.apache.catalina.core.StandardService start
>>> INFO: Starting service Catalina
>>> Jul 10, 2008 1:03:24 PM org.apache.catalina.core.StandardEngine start
>>> INFO: Starting Servlet Engine: Apache Tomcat/6.0.16
>>> Jul 10, 2008 1:03:24 PM org.apache.catalina.ha.tcp.SimpleTcpCluster
>>> start
>>> INFO: Cluster is about to start
>>> Jul 10, 2008 1:03:24 PM
>>> org.apache.catalina.tribes.transport.ReceiverBase bind
>>> INFO: Receiver Server Socket bound to:/10.84.2.42:4000
>>> Jul 10, 2008 1:03:24 PM
>>> org.apache.catalina.tribes.membership.McastServiceImpl setupSocket
>>> INFO: Setting cluster mcast soTimeout to 500
>>> Jul 10, 2008 1:03:24 PM
>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>> INFO: Sleeping for 1000 milliseconds to establish cluster
>>> membership, start level:4
>>> Jul 10, 2008 1:03:25 PM
>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>> INFO: Done sleeping, membership established, start level:4
>>> Jul 10, 2008 1:03:25 PM
>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>> INFO: Sleeping for 1000 milliseconds to establish cluster
>>> membership, start level:8
>>> Jul 10, 2008 1:03:26 PM
>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>> INFO: Done sleeping, membership established, start level:8
>>> Jul 10, 2008 1:03:26 PM
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
>>> INFO: Initializing AbstractReplicatedMap with context name:
>>> Jul 10, 2008 1:03:27 PM org.apache.catalina.core.StandardContext
>>> addApplicationListener
>>> INFO: The listener "listeners.ContextListener" is already configured
>>> for this context. The duplicate definition has been ignored.
>>> Jul 10, 2008 1:03:27 PM org.apache.catalina.core.StandardContext
>>> addApplicationListener
>>> INFO: The listener "listeners.SessionListener" is already configured
>>> for this context. The duplicate definition has been ignored.
>>> Jul 10, 2008 1:03:27 PM org.apache.catalina.ha.session.DeltaManager
>>> start
>>> INFO: Register manager /examples to cluster element Engine with name
>>> Catalina
>>> Jul 10, 2008 1:03:27 PM org.apache.catalina.ha.session.DeltaManager
>>> start
>>> INFO: Starting clustering manager at /examples
>>> Jul 10, 2008 1:03:27 PM org.apache.catalina.ha.session.DeltaManager
>>> getAllClusterSessions
>>> INFO: Manager [localhost#/examples]: skipping state transfer. No
>>> members active in cluster group.
>>> Jul 10, 2008 1:03:28 PM
>>> org.apache.catalina.ha.session.JvmRouteBinderValve start
>>> INFO: JvmRouteBinderValve started
>>> Jul 10, 2008 1:03:28 PM org.apache.coyote.http11.Http11Protocol start
>>> INFO: Starting Coyote HTTP/1.1 on http-8080
>>> Jul 10, 2008 1:03:28 PM org.apache.jk.common.ChannelSocket init
>>> INFO: JK: ajp13 listening on /0.0.0.0:8009
>>> Jul 10, 2008 1:03:28 PM org.apache.jk.server.JkMain start
>>> INFO: Jk running ID=0 time=0/45 config=null
>>> Jul 10, 2008 1:03:28 PM org.apache.catalina.startup.Catalina start
>>> INFO: Server startup in 3616 ms
>>> Jul 10, 2008 1:06:41 PM
>>> org.apache.catalina.tribes.transport.nio.NioReceiver listen
>>> SEVERE: Unable to process request in NioReceiver
>>> java.net.SocketException: Invalid argument
>>> at sun.nio.ch.Net.setIntOption0(Native Method)
>>> at sun.nio.ch.Net.setIntOption(Net.java:152)
>>> at
>>> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
>>> at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>> at
>>> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
>>> at
>>> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
>>> at
>>> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
>>> at
>>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:273)
>>>
>>> at
>>> org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioReceiver.java:353)
>>>
>>> at java.lang.Thread.run(Thread.java:613)
>>> Jul 10, 2008 1:06:41 PM org.apache.catalina.ha.tcp.SimpleTcpCluster
>>> memberAdded
>>> INFO: Replication member
>>> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
>>> 84, 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46
>>> 127 -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={},
>>> command={}, domain={}, ]
>>> Jul 10, 2008 1:06:41 PM org.apache.catalina.tribes.io.BufferPool
>>> getBufferPool
>>> INFO: Created a buffer pool with max size:104857600 bytes of
>>> type:org.apache.catalina.tribes.io.BufferPool15Impl
>>> Jul 10, 2008 1:06:41 PM
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>>> WARNING: Member send is failing for:tcp://{10, 84, 2, 42}:4001 ;
>>> Setting to suspect and retrying.
>>> Jul 10, 2008 1:06:41 PM
>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>>> memberDisappeared
>>> INFO: Received
>>> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
>>> 84, 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46
>>> 127 -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={},
>>> command={}, domain={}, ]] message. Will verify.
>>> Jul 10, 2008 1:06:41 PM
>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>>> memberDisappeared
>>> INFO: Verification complete. Member still
>>> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
>>> 84, 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46
>>> 127 -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={},
>>> command={}, domain={}, ]]
>>> Jul 10, 2008 1:06:41 PM
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
>>> SEVERE: Unable to send AbstractReplicatedMap.ping message
>>> org.apache.catalina.tribes.ChannelException: Send failed, attempt:2
>>> max:1; Faulty members:tcp://{10, 84, 2, 42}:4001;
>>> at
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:172)
>>>
>>> at
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:78)
>>>
>>> at
>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53)
>>>
>>> at
>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80)
>>>
>>> at
>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78)
>>>
>>> at
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>>>
>>> at
>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87)
>>>
>>> at
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>>>
>>> at
>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:73)
>>>
>>> at
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>>>
>>> at
>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
>>>
>>> at
>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
>>>
>>> at
>>> org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:89)
>>> at
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:253)
>>>
>>> at
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:793)
>>>
>>> at
>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:153)
>>>
>>> at
>>> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:661)
>>>
>>> Caused by: java.net.SocketException: Invalid argument
>>> at sun.nio.ch.Net.setIntOption0(Native Method)
>>> at sun.nio.ch.Net.setIntOption(Net.java:152)
>>> at
>>> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
>>> at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>> at
>>> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
>>> at
>>> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
>>> at
>>> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
>>> at
>>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect(NioSender.java:147)
>>>
>>> at
>>> org.apache.catalina.tribes.transport.nio.NioSender.process(NioSender.java:89)
>>>
>>> at
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:130)
>>>
>>> ... 16 more
>>> Jul 10, 2008 1:06:41 PM
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>>> WARNING: Not retrying send for:tcp://{10, 84, 2, 42}:4001; Sender is
>>> disconnected.
>>> Jul 10, 2008 1:06:41 PM
>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>>> memberDisappeared
>>> INFO: Received
>>> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
>>> 84, 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46
>>> 127 -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={},
>>> command={}, domain={}, ]] message. Will verify.
>>> Jul 10, 2008 1:06:42 PM
>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>>> memberDisappeared
>>> INFO: Verification complete. Member still
>>> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
>>> 84, 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46
>>> 127 -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={},
>>> command={}, domain={}, ]]
>>> Jul 10, 2008 1:06:42 PM
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
>>> SEVERE: Unable to send AbstractReplicatedMap.ping message
>>> org.apache.catalina.tribes.ChannelException: Send failed, and sender
>>> is disconnected. Not retrying.; Faulty members:tcp://{10, 84, 2,
>>> 42}:4001;
>>> at
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:156)
>>>
>>> at
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:78)
>>>
>>> at
>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53)
>>>
>>> at
>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80)
>>>
>>> at
>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78)
>>>
>>> at
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>>>
>>> at
>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87)
>>>
>>> at
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>>>
>>> at
>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:73)
>>>
>>> at
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>>>
>>> at
>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
>>>
>>> at
>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
>>>
>>> at
>>> org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:89)
>>> at
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:253)
>>>
>>> at
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:793)
>>>
>>> at
>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:158)
>>>
>>> at
>>> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:661)
>>>
>>> Caused by: java.net.SocketException: Invalid argument
>>> at sun.nio.ch.Net.setIntOption0(Native Method)
>>> at sun.nio.ch.Net.setIntOption(Net.java:152)
>>> at
>>> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
>>> at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>> at
>>> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
>>> at
>>> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
>>> at
>>> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
>>> at
>>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect(NioSender.java:147)
>>>
>>> at
>>> org.apache.catalina.tribes.transport.nio.NioSender.process(NioSender.java:89)
>>>
>>> at
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:130)
>>>
>>> ... 16 more
>>>
>>>
>>> and so on forever.
>>>
>>> In node2 logs I see the following:
>>>
>>> bml0042:/usr/local/tomcat/node2/logs janos$ cat catalina.2008-07-10.log
>>> Jul 10, 2008 1:06:40 PM
>>> org.apache.catalina.core.AprLifecycleListener init
>>> INFO: The APR based Apache Tomcat Native library which allows
>>> optimal performance in production environments was not found on the
>>> java.library.path:
>>> .:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java
>>>
>>> Jul 10, 2008 1:06:40 PM org.apache.coyote.http11.Http11Protocol init
>>> INFO: Initializing Coyote HTTP/1.1 on http-8180
>>> Jul 10, 2008 1:06:40 PM org.apache.catalina.startup.Catalina load
>>> INFO: Initialization processed in 1148 ms
>>> Jul 10, 2008 1:06:40 PM org.apache.catalina.core.StandardService start
>>> INFO: Starting service Catalina
>>> Jul 10, 2008 1:06:40 PM org.apache.catalina.core.StandardEngine start
>>> INFO: Starting Servlet Engine: Apache Tomcat/6.0.16
>>> Jul 10, 2008 1:06:40 PM org.apache.catalina.ha.tcp.SimpleTcpCluster
>>> start
>>> INFO: Cluster is about to start
>>> Jul 10, 2008 1:06:40 PM
>>> org.apache.catalina.tribes.transport.ReceiverBase bind
>>> INFO: Receiver Server Socket bound to:/10.84.2.42:4001
>>> Jul 10, 2008 1:06:40 PM
>>> org.apache.catalina.tribes.membership.McastServiceImpl setupSocket
>>> INFO: Setting cluster mcast soTimeout to 500
>>> Jul 10, 2008 1:06:40 PM
>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>> INFO: Sleeping for 1000 milliseconds to establish cluster
>>> membership, start level:4
>>> Jul 10, 2008 1:06:41 PM org.apache.catalina.ha.tcp.SimpleTcpCluster
>>> memberAdded
>>> INFO: Replication member
>>> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
>>> 84, 2, 42}:4000,{10, 84, 2, 42},4000, alive=196161,id={-25 34 -32
>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={},
>>> command={}, domain={}, ]
>>> Jul 10, 2008 1:06:41 PM
>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>> INFO: Done sleeping, membership established, start level:4
>>> Jul 10, 2008 1:06:41 PM
>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>> INFO: Sleeping for 1000 milliseconds to establish cluster
>>> membership, start level:8
>>> Jul 10, 2008 1:06:41 PM
>>> org.apache.catalina.tribes.transport.nio.NioReceiver listen
>>> SEVERE: Unable to process request in NioReceiver
>>> java.net.SocketException: Invalid argument
>>> at sun.nio.ch.Net.setIntOption0(Native Method)
>>> at sun.nio.ch.Net.setIntOption(Net.java:152)
>>> at
>>> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
>>> at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>> at
>>> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
>>> at
>>> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
>>> at
>>> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
>>> at
>>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:273)
>>>
>>> at
>>> org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioReceiver.java:353)
>>>
>>> at java.lang.Thread.run(Thread.java:613)
>>> Jul 10, 2008 1:06:42 PM
>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>> INFO: Done sleeping, membership established, start level:8
>>> Jul 10, 2008 1:06:42 PM
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
>>> INFO: Initializing AbstractReplicatedMap with context name:
>>> Jul 10, 2008 1:06:42 PM org.apache.catalina.tribes.io.BufferPool
>>> getBufferPool
>>> INFO: Created a buffer pool with max size:104857600 bytes of
>>> type:org.apache.catalina.tribes.io.BufferPool15Impl
>>> Jul 10, 2008 1:06:42 PM
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>>> WARNING: Member send is failing for:tcp://{10, 84, 2, 42}:4000 ;
>>> Setting to suspect and retrying.
>>> Jul 10, 2008 1:06:42 PM
>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>>> memberDisappeared
>>> INFO: Received
>>> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
>>> 84, 2, 42}:4000,{10, 84, 2, 42},4000, alive=197662,id={-25 34 -32
>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={},
>>> command={}, domain={}, ]] message. Will verify.
>>> Jul 10, 2008 1:06:42 PM
>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>>> memberDisappeared
>>> INFO: Verification complete. Member still
>>> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
>>> 84, 2, 42}:4000,{10, 84, 2, 42},4000, alive=197662,id={-25 34 -32
>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={},
>>> command={}, domain={}, ]]
>>> Jul 10, 2008 1:06:42 PM
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
>>> WARNING: Unable to send map start message.
>>> Jul 10, 2008 1:06:42 PM
>>> org.apache.catalina.ha.context.ReplicatedContext start
>>> SEVERE: Unable to start ReplicatedContext
>>> java.lang.RuntimeException: Unable to start replicated map.
>>> at
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.init(AbstractReplicatedMap.java:230)
>>>
>>> at
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.<init>(AbstractReplicatedMap.java:171)
>>>
>>> at
>>> org.apache.catalina.tribes.tipis.ReplicatedMap.<init>(ReplicatedMap.java:90)
>>>
>>> at
>>> org.apache.catalina.ha.context.ReplicatedContext.start(ReplicatedContext.java:64)
>>>
>>> at
>>> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
>>> at
>>> org.apache.catalina.core.StandardHost.start(StandardHost.java:719)
>>> at
>>> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
>>> at
>>> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443)
>>> at
>>> org.apache.catalina.core.StandardService.start(StandardService.java:516)
>>>
>>> at
>>> org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
>>> at
>>> org.apache.catalina.startup.Catalina.start(Catalina.java:578)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>
>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>> at
>>> org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
>>> at
>>> org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
>>> Caused by: org.apache.catalina.tribes.ChannelException: Send failed,
>>> attempt:2 max:1; Faulty members:tcp://{10, 84, 2, 42}:4000;
>>> at
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:172)
>>>
>>> at
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:78)
>>>
>>> at
>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53)
>>>
>>> at
>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80)
>>>
>>> at
>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78)
>>>
>>> at
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>>>
>>> at
>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87)
>>>
>>> at
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>>>
>>> at
>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:73)
>>>
>>> at
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>>>
>>> at
>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
>>>
>>> at
>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
>>>
>>> at
>>> org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:89)
>>> at
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.broadcast(AbstractReplicatedMap.java:300)
>>>
>>> at
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.init(AbstractReplicatedMap.java:223)
>>>
>>> ... 16 more
>>> Caused by: java.net.SocketException: Invalid argument
>>> at sun.nio.ch.Net.setIntOption0(Native Method)
>>> at sun.nio.ch.Net.setIntOption(Net.java:152)
>>> at
>>> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
>>> at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>> at
>>> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
>>> at
>>> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
>>> at
>>> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
>>> at
>>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect(NioSender.java:147)
>>>
>>> at
>>> org.apache.catalina.tribes.transport.nio.NioSender.process(NioSender.java:89)
>>>
>>> at
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:130)
>>>
>>> ... 30 more
>>> Jul 10, 2008 1:06:42 PM org.apache.catalina.startup.Catalina start
>>> SEVERE: Catalina.start:
>>> LifecycleException: Failed to start ReplicatedContext:
>>> java.lang.RuntimeException: Unable to start replicated map.
>>> at
>>> org.apache.catalina.ha.context.ReplicatedContext.start(ReplicatedContext.java:73)
>>>
>>> at
>>> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
>>> at
>>> org.apache.catalina.core.StandardHost.start(StandardHost.java:719)
>>> at
>>> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
>>> at
>>> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443)
>>> at
>>> org.apache.catalina.core.StandardService.start(StandardService.java:516)
>>>
>>> at
>>> org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
>>> at
>>> org.apache.catalina.startup.Catalina.start(Catalina.java:578)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>
>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>> at
>>> org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
>>> at
>>> org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
>>> Jul 10, 2008 1:06:42 PM org.apache.catalina.startup.Catalina start
>>> INFO: Server startup in 2329 ms
>>> Jul 10, 2008 1:06:47 PM
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>>> WARNING: Not retrying send for:tcp://{10, 84, 2, 42}:4000; Sender is
>>> disconnected.
>>> Jul 10, 2008 1:06:47 PM
>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>>> memberDisappeared
>>> INFO: Received
>>> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
>>> 84, 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32
>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={},
>>> command={}, domain={}, ]] message. Will verify.
>>> Jul 10, 2008 1:06:47 PM
>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>>> memberDisappeared
>>> INFO: Verification complete. Member still
>>> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
>>> 84, 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32
>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={},
>>> command={}, domain={}, ]]
>>> Jul 10, 2008 1:06:47 PM
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
>>> SEVERE: Unable to send AbstractReplicatedMap.ping message
>>> org.apache.catalina.tribes.ChannelException: Send failed, and sender
>>> is disconnected. Not retrying.; Faulty members:tcp://{10, 84, 2,
>>> 42}:4000;
>>> at
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:156)
>>>
>>> at
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:78)
>>>
>>> at
>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53)
>>>
>>> at
>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80)
>>>
>>> at
>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78)
>>>
>>> at
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>>>
>>> at
>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87)
>>>
>>> at
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>>>
>>> at
>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:73)
>>>
>>> at
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>>>
>>> at
>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
>>>
>>> at
>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
>>>
>>> at
>>> org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:89)
>>> at
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:253)
>>>
>>> at
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:793)
>>>
>>> at
>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:153)
>>>
>>> at
>>> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:661)
>>>
>>> Caused by: java.net.SocketException: Invalid argument
>>> at sun.nio.ch.Net.setIntOption0(Native Method)
>>> at sun.nio.ch.Net.setIntOption(Net.java:152)
>>> at
>>> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
>>> at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>> at
>>> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
>>> at
>>> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
>>> at
>>> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
>>> at
>>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect(NioSender.java:147)
>>>
>>> at
>>> org.apache.catalina.tribes.transport.nio.NioSender.process(NioSender.java:89)
>>>
>>> at
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:130)
>>>
>>> ... 16 more
>>> Jul 10, 2008 1:06:47 PM
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>>> WARNING: Not retrying send for:tcp://{10, 84, 2, 42}:4000; Sender is
>>> disconnected.
>>> Jul 10, 2008 1:06:47 PM
>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>>> memberDisappeared
>>> INFO: Received
>>> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
>>> 84, 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32
>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={},
>>> command={}, domain={}, ]] message. Will verify.
>>> Jul 10, 2008 1:06:47 PM
>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>>> memberDisappeared
>>> INFO: Verification complete. Member still
>>> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
>>> 84, 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32
>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={},
>>> command={}, domain={}, ]]
>>> Jul 10, 2008 1:06:47 PM
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
>>> SEVERE: Unable to send AbstractReplicatedMap.ping message
>>> org.apache.catalina.tribes.ChannelException: Send failed, and sender
>>> is disconnected. Not retrying.; Faulty members:tcp://{10, 84, 2,
>>> 42}:4000;
>>> at
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:156)
>>>
>>> at
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:78)
>>>
>>> at
>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53)
>>>
>>> at
>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80)
>>>
>>> at
>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78)
>>>
>>> at
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>>>
>>> at
>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87)
>>>
>>> at
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>>>
>>> at
>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:73)
>>>
>>> at
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>>>
>>> at
>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
>>>
>>> at
>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
>>>
>>> at
>>> org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:89)
>>> at
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:253)
>>>
>>> at
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:793)
>>>
>>> at
>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:158)
>>>
>>> at
>>> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:661)
>>>
>>> Caused by: java.net.SocketException: Invalid argument
>>> at sun.nio.ch.Net.setIntOption0(Native Method)
>>> at sun.nio.ch.Net.setIntOption(Net.java:152)
>>> at
>>> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
>>> at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>> at
>>> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
>>> at
>>> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
>>> at
>>> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
>>> at
>>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect(NioSender.java:147)
>>>
>>> at
>>> org.apache.catalina.tribes.transport.nio.NioSender.process(NioSender.java:89)
>>>
>>> at
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:130)
>>>
>>> ... 16 more
>>>
>>>
>>> and so on. Top shows that both Java process is Up to 86%, the
>>> machine is never idle and all memory is consumed. The machine is a
>>> PowerPc G6 from Apple with OSX 10.4.11.
>>> Tomcat version is: Apache Tomcat/6.0.16
>>> JVM version is: 1.5.0_13-b05-241
>>> I suspect the heartbeat pings have a problem, I just do not know
>>> what to do about them.
>>>
>>> Thanks ahead,
>>>
>>> János
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To start a new topic, e-mail: users@tomcat.apache.org
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To start a new topic, e-mail: users@tomcat.apache.org
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Version 6 Clustering problem
Posted by János <ja...@yale.edu>.
Hi Filip,
I added it in the catalina.sh file on all three nodes and I am not
seeing the error. I also disabled IP6 in Network Preferences.
I added the distrubuted="true" to the context.xml file on all three
nodes and added the <distributable/> tag to the webapps/examples/WEB-
INF/web.xml on all 3 nodes and tried the SessionExample servlet
http://bml0042.yalepath.org:8080/examples/servlets/servlet/
SessionExample
to see if I have a working cluster. I am following the book at
http://proquest.safaribooksonline.com:80/9780596101060/
tomcat_6_clustering_implementation
and it says that I should see messages similar to this one in the
nodes' logs"
Sep 27, 2008 5:39:38 PM org.apache.catalina.core.ApplicationContext log
INFO: SessionListener: contextInitialized( )
Sep 27, 2008 5:40:07 PM org.apache.catalina.core.ApplicationContext log
INFO: SessionListener: sessionCreated
('F4B2D7191C1F335FFAAC93DA461CA95F.tc1')
Sep 27, 2008 5:40:19 PM org.apache.catalina.core.ApplicationContext log
INFO: SessionListener: attributeAdded
('F4B2D7191C1F335FFAAC93DA461CA95F.tc1', 'test',
But I am not seeing them. This is the resent log from node1:
Jul 11, 2008 3:28:01 PM org.apache.catalina.core.AprLifecycleListener
init
INFO: The APR based Apache Tomcat Native library which allows optimal
performance in production environments was not found on the
java.library.path: .:/Library/Java/Extensions:/System/Library/Java/
Extensions:/usr/lib/java
Jul 11, 2008 3:28:01 PM org.apache.coyote.http11.Http11Protocol init
INFO: Initializing Coyote HTTP/1.1 on http-8080
Jul 11, 2008 3:28:01 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 1098 ms
Jul 11, 2008 3:28:01 PM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
Jul 11, 2008 3:28:01 PM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.16
Jul 11, 2008 3:28:01 PM org.apache.catalina.ha.tcp.SimpleTcpCluster
start
INFO: Cluster is about to start
Jul 11, 2008 3:28:01 PM
org.apache.catalina.tribes.transport.ReceiverBase bind
INFO: Receiver Server Socket bound to:/10.84.2.42:4000
Jul 11, 2008 3:28:01 PM
org.apache.catalina.tribes.membership.McastServiceImpl setupSocket
INFO: Setting cluster mcast soTimeout to 500
Jul 11, 2008 3:28:01 PM
org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
INFO: Sleeping for 1000 milliseconds to establish cluster membership,
start level:4
Jul 11, 2008 3:28:02 PM
org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
INFO: Done sleeping, membership established, start level:4
Jul 11, 2008 3:28:02 PM
org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
INFO: Sleeping for 1000 milliseconds to establish cluster membership,
start level:8
Jul 11, 2008 3:28:03 PM
org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
INFO: Done sleeping, membership established, start level:8
Jul 11, 2008 3:28:03 PM
org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
INFO: Initializing AbstractReplicatedMap with context name:
Jul 11, 2008 3:28:04 PM org.apache.catalina.core.StandardContext
addApplicationListener
INFO: The listener "listeners.ContextListener" is already configured
for this context. The duplicate definition has been ignored.
Jul 11, 2008 3:28:04 PM org.apache.catalina.core.StandardContext
addApplicationListener
INFO: The listener "listeners.SessionListener" is already configured
for this context. The duplicate definition has been ignored.
Jul 11, 2008 3:28:04 PM org.apache.catalina.ha.session.DeltaManager
start
INFO: Register manager /examples to cluster element Engine with name
Catalina
Jul 11, 2008 3:28:04 PM org.apache.catalina.ha.session.DeltaManager
start
INFO: Starting clustering manager at /examples
Jul 11, 2008 3:28:04 PM org.apache.catalina.ha.session.DeltaManager
getAllClusterSessions
INFO: Manager [localhost#/examples]: skipping state transfer. No
members active in cluster group.
Jul 11, 2008 3:28:04 PM
org.apache.catalina.ha.session.JvmRouteBinderValve start
INFO: JvmRouteBinderValve started
Jul 11, 2008 3:28:04 PM org.apache.coyote.http11.Http11Protocol start
INFO: Starting Coyote HTTP/1.1 on http-8080
Jul 11, 2008 3:28:05 PM org.apache.jk.common.ChannelSocket init
INFO: JK: ajp13 listening on /0.0.0.0:8009
Jul 11, 2008 3:28:05 PM org.apache.jk.server.JkMain start
INFO: Jk running ID=0 time=0/45 config=null
Jul 11, 2008 3:28:05 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 3601 ms
Jul 11, 2008 3:28:37 PM org.apache.catalina.tribes.io.BufferPool
getBufferPool
INFO: Created a buffer pool with max size:104857600 bytes of
type:org.apache.catalina.tribes.io.BufferPool15Impl
Jul 11, 2008 3:28:37 PM org.apache.catalina.ha.tcp.SimpleTcpCluster
memberAdded
INFO: Replication member
added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,
2, 42}:4001,{10, 84, 2, 42},4001, alive=1020,id={60 103 -111 -34 98
62 71 -116 -104 76 -100 115 -77 15 -52 -92 }, payload={}, command={},
domain={}, ]
Jul 11, 2008 3:29:08 PM org.apache.catalina.ha.tcp.SimpleTcpCluster
memberAdded
INFO: Replication member
added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,
2, 42}:4002,{10, 84, 2, 42},4002, alive=1019,id={27 24 -73 72 48 -6
70 32 -90 57 51 106 -94 -54 126 -52 }, payload={}, command={}, domain=
{}, ]
Similarly on the other two nodes. What am I missing ?? Should I add
the <Context distributed=True" /> to the server.xml files ?
Thanks ahead,
János
On Jul 10, 2008, at 6:03 PM, Filip Hanik - Dev Lists wrote:
> try setting the
> |-Djava.net.preferIPv4Stack=true
>
> option for your tomcat launch
>
> Filip
> |
> János wrote:
>> Hi,
>>
>> I am trying to establish a 3 member cluster on the same machine.
>> Here is my server.xml copied from a sample I found on this list.
>>
>> <?xml version='1.0' encoding='utf-8'?>
>> <Server port="8005" shutdown="SHUTDOWN">
>> <Listener
>> className="org.apache.catalina.core.AprLifecycleListener"
>> SSLEngine="on" />
>> <Listener className="org.apache.catalina.core.JasperListener" />
>> <Listener
>> className="org.apache.catalina.mbeans.ServerLifecycleListener" />
>> <Listener
>> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListene
>> r" />
>>
>> <GlobalNamingResources>
>> <Resource name="UserDatabase" auth="Container"
>> type="org.apache.catalina.UserDatabase"
>> description="User database that can be updated and
>> saved"
>>
>> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
>> pathname="conf/tomcat-users.xml" />
>> </GlobalNamingResources>
>>
>> <Service name="Catalina">
>>
>> <Connector port="8080" protocol="HTTP/1.1"
>> connectionTimeout="20000"
>> redirectPort="8443" />
>> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
>>
>> <Engine name="Catalina" defaultHost="localhost" jvmRoute="node1">
>>
>> <Cluster
>> className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
>> <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
>> resourceName="UserDatabase"/>
>>
>> <Host name="localhost" appBase="webapps"
>> unpackWARs="true" autoDeploy="true"
>> xmlValidation="false" xmlNamespaceAware="false">
>> <Context path="" docBase="ROOT"
>> className="org.apache.catalina.ha.context.ReplicatedContext"/>
>> </Host>
>> </Engine>
>> </Service>
>> </Server>
>>
>> The other two server.xml-s are the same, except the shutdown ports
>> are 8105, 8205, http ports are 8180, 8280 and https ports are
>> 8543, 8643.
>> As soon as I am starting the second instance I receive the
>> following into the logs.
>>
>> In node1 logs:
>>
>> bml0042:/usr/local/tomcat/node1/logs janos$ cat catalina.
>> 2008-07-10.log
>> Jul 10, 2008 1:03:24 PM
>> org.apache.catalina.core.AprLifecycleListener init
>> INFO: The APR based Apache Tomcat Native library which allows
>> optimal performance in production environments was not found on
>> the java.library.path: .:/Library/Java/Extensions:/System/Library/
>> Java/Extensions:/usr/lib/java
>> Jul 10, 2008 1:03:24 PM org.apache.coyote.http11.Http11Protocol init
>> INFO: Initializing Coyote HTTP/1.1 on http-8080
>> Jul 10, 2008 1:03:24 PM org.apache.catalina.startup.Catalina load
>> INFO: Initialization processed in 1187 ms
>> Jul 10, 2008 1:03:24 PM org.apache.catalina.core.StandardService
>> start
>> INFO: Starting service Catalina
>> Jul 10, 2008 1:03:24 PM org.apache.catalina.core.StandardEngine start
>> INFO: Starting Servlet Engine: Apache Tomcat/6.0.16
>> Jul 10, 2008 1:03:24 PM
>> org.apache.catalina.ha.tcp.SimpleTcpCluster start
>> INFO: Cluster is about to start
>> Jul 10, 2008 1:03:24 PM
>> org.apache.catalina.tribes.transport.ReceiverBase bind
>> INFO: Receiver Server Socket bound to:/10.84.2.42:4000
>> Jul 10, 2008 1:03:24 PM
>> org.apache.catalina.tribes.membership.McastServiceImpl setupSocket
>> INFO: Setting cluster mcast soTimeout to 500
>> Jul 10, 2008 1:03:24 PM
>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>> INFO: Sleeping for 1000 milliseconds to establish cluster
>> membership, start level:4
>> Jul 10, 2008 1:03:25 PM
>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>> INFO: Done sleeping, membership established, start level:4
>> Jul 10, 2008 1:03:25 PM
>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>> INFO: Sleeping for 1000 milliseconds to establish cluster
>> membership, start level:8
>> Jul 10, 2008 1:03:26 PM
>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>> INFO: Done sleeping, membership established, start level:8
>> Jul 10, 2008 1:03:26 PM
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
>> INFO: Initializing AbstractReplicatedMap with context name:
>> Jul 10, 2008 1:03:27 PM org.apache.catalina.core.StandardContext
>> addApplicationListener
>> INFO: The listener "listeners.ContextListener" is already
>> configured for this context. The duplicate definition has been
>> ignored.
>> Jul 10, 2008 1:03:27 PM org.apache.catalina.core.StandardContext
>> addApplicationListener
>> INFO: The listener "listeners.SessionListener" is already
>> configured for this context. The duplicate definition has been
>> ignored.
>> Jul 10, 2008 1:03:27 PM
>> org.apache.catalina.ha.session.DeltaManager start
>> INFO: Register manager /examples to cluster element Engine with
>> name Catalina
>> Jul 10, 2008 1:03:27 PM
>> org.apache.catalina.ha.session.DeltaManager start
>> INFO: Starting clustering manager at /examples
>> Jul 10, 2008 1:03:27 PM
>> org.apache.catalina.ha.session.DeltaManager getAllClusterSessions
>> INFO: Manager [localhost#/examples]: skipping state transfer. No
>> members active in cluster group.
>> Jul 10, 2008 1:03:28 PM
>> org.apache.catalina.ha.session.JvmRouteBinderValve start
>> INFO: JvmRouteBinderValve started
>> Jul 10, 2008 1:03:28 PM org.apache.coyote.http11.Http11Protocol start
>> INFO: Starting Coyote HTTP/1.1 on http-8080
>> Jul 10, 2008 1:03:28 PM org.apache.jk.common.ChannelSocket init
>> INFO: JK: ajp13 listening on /0.0.0.0:8009
>> Jul 10, 2008 1:03:28 PM org.apache.jk.server.JkMain start
>> INFO: Jk running ID=0 time=0/45 config=null
>> Jul 10, 2008 1:03:28 PM org.apache.catalina.startup.Catalina start
>> INFO: Server startup in 3616 ms
>> Jul 10, 2008 1:06:41 PM
>> org.apache.catalina.tribes.transport.nio.NioReceiver listen
>> SEVERE: Unable to process request in NioReceiver
>> java.net.SocketException: Invalid argument
>> at sun.nio.ch.Net.setIntOption0(Native Method)
>> at sun.nio.ch.Net.setIntOption(Net.java:152)
>> at sun.nio.ch.SocketChannelImpl$1.setInt
>> (SocketChannelImpl.java:372)
>> at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>> at sun.nio.ch.SocketOptsImpl$IP.typeOfService
>> (SocketOptsImpl.java:249)
>> at sun.nio.ch.OptionAdaptor.setTrafficClass
>> (OptionAdaptor.java:158)
>> at sun.nio.ch.SocketAdaptor.setTrafficClass
>> (SocketAdaptor.java:330)
>> at
>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen
>> (NioReceiver.java:273)
>> at org.apache.catalina.tribes.transport.nio.NioReceiver.run
>> (NioReceiver.java:353)
>> at java.lang.Thread.run(Thread.java:613)
>> Jul 10, 2008 1:06:41 PM
>> org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
>> INFO: Replication member
>> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
>> 84, 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46
>> 127 -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={},
>> command={}, domain={}, ]
>> Jul 10, 2008 1:06:41 PM org.apache.catalina.tribes.io.BufferPool
>> getBufferPool
>> INFO: Created a buffer pool with max size:104857600 bytes of
>> type:org.apache.catalina.tribes.io.BufferPool15Impl
>> Jul 10, 2008 1:06:41 PM
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>> WARNING: Member send is failing for:tcp://{10, 84, 2, 42}:4001 ;
>> Setting to suspect and retrying.
>> Jul 10, 2008 1:06:41 PM
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>> memberDisappeared
>> INFO: Received memberDisappeared
>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,
>> 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46 127
>> -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={}, command=
>> {}, domain={}, ]] message. Will verify.
>> Jul 10, 2008 1:06:41 PM
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>> memberDisappeared
>> INFO: Verification complete. Member still alive
>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,
>> 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46 127
>> -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={}, command=
>> {}, domain={}, ]]
>> Jul 10, 2008 1:06:41 PM
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
>> SEVERE: Unable to send AbstractReplicatedMap.ping message
>> org.apache.catalina.tribes.ChannelException: Send failed, attempt:
>> 2 max:1; Faulty members:tcp://{10, 84, 2, 42}:4001;
>> at
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop
>> (ParallelNioSender.java:172)
>> at
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessag
>> e(ParallelNioSender.java:78)
>> at
>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMes
>> sage(PooledParallelSender.java:53)
>> at
>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessa
>> ge(ReplicationTransmitter.java:80)
>> at
>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage
>> (ChannelCoordinator.java:78)
>> at
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
>> (ChannelInterceptorBase.java:75)
>> at
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.send
>> Message(TcpFailureDetector.java:87)
>> at
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
>> (ChannelInterceptorBase.java:75)
>> at
>> org.apache.catalina.tribes.group.interceptors.MessageDispatchIntercep
>> tor.sendMessage(MessageDispatchInterceptor.java:73)
>> at
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
>> (ChannelInterceptorBase.java:75)
>> at org.apache.catalina.tribes.group.GroupChannel.send
>> (GroupChannel.java:216)
>> at org.apache.catalina.tribes.group.GroupChannel.send
>> (GroupChannel.java:175)
>> at org.apache.catalina.tribes.group.RpcChannel.send
>> (RpcChannel.java:89)
>> at
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping
>> (AbstractReplicatedMap.java:253)
>> at
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
>> (AbstractReplicatedMap.java:793)
>> at org.apache.catalina.tribes.group.GroupChannel.heartbeat
>> (GroupChannel.java:153)
>> at org.apache.catalina.tribes.group.GroupChannel
>> $HeartbeatThread.run(GroupChannel.java:661)
>> Caused by: java.net.SocketException: Invalid argument
>> at sun.nio.ch.Net.setIntOption0(Native Method)
>> at sun.nio.ch.Net.setIntOption(Net.java:152)
>> at sun.nio.ch.SocketChannelImpl$1.setInt
>> (SocketChannelImpl.java:372)
>> at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>> at sun.nio.ch.SocketOptsImpl$IP.typeOfService
>> (SocketOptsImpl.java:249)
>> at sun.nio.ch.OptionAdaptor.setTrafficClass
>> (OptionAdaptor.java:158)
>> at sun.nio.ch.SocketAdaptor.setTrafficClass
>> (SocketAdaptor.java:330)
>> at
>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect
>> (NioSender.java:147)
>> at
>> org.apache.catalina.tribes.transport.nio.NioSender.process
>> (NioSender.java:89)
>> at
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop
>> (ParallelNioSender.java:130)
>> ... 16 more
>> Jul 10, 2008 1:06:41 PM
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>> WARNING: Not retrying send for:tcp://{10, 84, 2, 42}:4001; Sender
>> is disconnected.
>> Jul 10, 2008 1:06:41 PM
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>> memberDisappeared
>> INFO: Received memberDisappeared
>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,
>> 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46 127
>> -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={}, command=
>> {}, domain={}, ]] message. Will verify.
>> Jul 10, 2008 1:06:42 PM
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>> memberDisappeared
>> INFO: Verification complete. Member still alive
>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,
>> 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46 127
>> -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={}, command=
>> {}, domain={}, ]]
>> Jul 10, 2008 1:06:42 PM
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
>> SEVERE: Unable to send AbstractReplicatedMap.ping message
>> org.apache.catalina.tribes.ChannelException: Send failed, and
>> sender is disconnected. Not retrying.; Faulty members:tcp://{10,
>> 84, 2, 42}:4001;
>> at
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop
>> (ParallelNioSender.java:156)
>> at
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessag
>> e(ParallelNioSender.java:78)
>> at
>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMes
>> sage(PooledParallelSender.java:53)
>> at
>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessa
>> ge(ReplicationTransmitter.java:80)
>> at
>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage
>> (ChannelCoordinator.java:78)
>> at
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
>> (ChannelInterceptorBase.java:75)
>> at
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.send
>> Message(TcpFailureDetector.java:87)
>> at
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
>> (ChannelInterceptorBase.java:75)
>> at
>> org.apache.catalina.tribes.group.interceptors.MessageDispatchIntercep
>> tor.sendMessage(MessageDispatchInterceptor.java:73)
>> at
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
>> (ChannelInterceptorBase.java:75)
>> at org.apache.catalina.tribes.group.GroupChannel.send
>> (GroupChannel.java:216)
>> at org.apache.catalina.tribes.group.GroupChannel.send
>> (GroupChannel.java:175)
>> at org.apache.catalina.tribes.group.RpcChannel.send
>> (RpcChannel.java:89)
>> at
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping
>> (AbstractReplicatedMap.java:253)
>> at
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
>> (AbstractReplicatedMap.java:793)
>> at org.apache.catalina.tribes.group.GroupChannel.heartbeat
>> (GroupChannel.java:158)
>> at org.apache.catalina.tribes.group.GroupChannel
>> $HeartbeatThread.run(GroupChannel.java:661)
>> Caused by: java.net.SocketException: Invalid argument
>> at sun.nio.ch.Net.setIntOption0(Native Method)
>> at sun.nio.ch.Net.setIntOption(Net.java:152)
>> at sun.nio.ch.SocketChannelImpl$1.setInt
>> (SocketChannelImpl.java:372)
>> at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>> at sun.nio.ch.SocketOptsImpl$IP.typeOfService
>> (SocketOptsImpl.java:249)
>> at sun.nio.ch.OptionAdaptor.setTrafficClass
>> (OptionAdaptor.java:158)
>> at sun.nio.ch.SocketAdaptor.setTrafficClass
>> (SocketAdaptor.java:330)
>> at
>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect
>> (NioSender.java:147)
>> at
>> org.apache.catalina.tribes.transport.nio.NioSender.process
>> (NioSender.java:89)
>> at
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop
>> (ParallelNioSender.java:130)
>> ... 16 more
>>
>>
>> and so on forever.
>>
>> In node2 logs I see the following:
>>
>> bml0042:/usr/local/tomcat/node2/logs janos$ cat catalina.
>> 2008-07-10.log
>> Jul 10, 2008 1:06:40 PM
>> org.apache.catalina.core.AprLifecycleListener init
>> INFO: The APR based Apache Tomcat Native library which allows
>> optimal performance in production environments was not found on
>> the java.library.path: .:/Library/Java/Extensions:/System/Library/
>> Java/Extensions:/usr/lib/java
>> Jul 10, 2008 1:06:40 PM org.apache.coyote.http11.Http11Protocol init
>> INFO: Initializing Coyote HTTP/1.1 on http-8180
>> Jul 10, 2008 1:06:40 PM org.apache.catalina.startup.Catalina load
>> INFO: Initialization processed in 1148 ms
>> Jul 10, 2008 1:06:40 PM org.apache.catalina.core.StandardService
>> start
>> INFO: Starting service Catalina
>> Jul 10, 2008 1:06:40 PM org.apache.catalina.core.StandardEngine start
>> INFO: Starting Servlet Engine: Apache Tomcat/6.0.16
>> Jul 10, 2008 1:06:40 PM
>> org.apache.catalina.ha.tcp.SimpleTcpCluster start
>> INFO: Cluster is about to start
>> Jul 10, 2008 1:06:40 PM
>> org.apache.catalina.tribes.transport.ReceiverBase bind
>> INFO: Receiver Server Socket bound to:/10.84.2.42:4001
>> Jul 10, 2008 1:06:40 PM
>> org.apache.catalina.tribes.membership.McastServiceImpl setupSocket
>> INFO: Setting cluster mcast soTimeout to 500
>> Jul 10, 2008 1:06:40 PM
>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>> INFO: Sleeping for 1000 milliseconds to establish cluster
>> membership, start level:4
>> Jul 10, 2008 1:06:41 PM
>> org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
>> INFO: Replication member
>> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
>> 84, 2, 42}:4000,{10, 84, 2, 42},4000, alive=196161,id={-25 34 -32
>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={},
>> command={}, domain={}, ]
>> Jul 10, 2008 1:06:41 PM
>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>> INFO: Done sleeping, membership established, start level:4
>> Jul 10, 2008 1:06:41 PM
>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>> INFO: Sleeping for 1000 milliseconds to establish cluster
>> membership, start level:8
>> Jul 10, 2008 1:06:41 PM
>> org.apache.catalina.tribes.transport.nio.NioReceiver listen
>> SEVERE: Unable to process request in NioReceiver
>> java.net.SocketException: Invalid argument
>> at sun.nio.ch.Net.setIntOption0(Native Method)
>> at sun.nio.ch.Net.setIntOption(Net.java:152)
>> at sun.nio.ch.SocketChannelImpl$1.setInt
>> (SocketChannelImpl.java:372)
>> at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>> at sun.nio.ch.SocketOptsImpl$IP.typeOfService
>> (SocketOptsImpl.java:249)
>> at sun.nio.ch.OptionAdaptor.setTrafficClass
>> (OptionAdaptor.java:158)
>> at sun.nio.ch.SocketAdaptor.setTrafficClass
>> (SocketAdaptor.java:330)
>> at
>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen
>> (NioReceiver.java:273)
>> at org.apache.catalina.tribes.transport.nio.NioReceiver.run
>> (NioReceiver.java:353)
>> at java.lang.Thread.run(Thread.java:613)
>> Jul 10, 2008 1:06:42 PM
>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>> INFO: Done sleeping, membership established, start level:8
>> Jul 10, 2008 1:06:42 PM
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
>> INFO: Initializing AbstractReplicatedMap with context name:
>> Jul 10, 2008 1:06:42 PM org.apache.catalina.tribes.io.BufferPool
>> getBufferPool
>> INFO: Created a buffer pool with max size:104857600 bytes of
>> type:org.apache.catalina.tribes.io.BufferPool15Impl
>> Jul 10, 2008 1:06:42 PM
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>> WARNING: Member send is failing for:tcp://{10, 84, 2, 42}:4000 ;
>> Setting to suspect and retrying.
>> Jul 10, 2008 1:06:42 PM
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>> memberDisappeared
>> INFO: Received memberDisappeared
>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,
>> 2, 42}:4000,{10, 84, 2, 42},4000, alive=197662,id={-25 34 -32 116
>> 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command=
>> {}, domain={}, ]] message. Will verify.
>> Jul 10, 2008 1:06:42 PM
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>> memberDisappeared
>> INFO: Verification complete. Member still alive
>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,
>> 2, 42}:4000,{10, 84, 2, 42},4000, alive=197662,id={-25 34 -32 116
>> 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command=
>> {}, domain={}, ]]
>> Jul 10, 2008 1:06:42 PM
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
>> WARNING: Unable to send map start message.
>> Jul 10, 2008 1:06:42 PM
>> org.apache.catalina.ha.context.ReplicatedContext start
>> SEVERE: Unable to start ReplicatedContext
>> java.lang.RuntimeException: Unable to start replicated map.
>> at
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.init
>> (AbstractReplicatedMap.java:230)
>> at
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.<init>
>> (AbstractReplicatedMap.java:171)
>> at org.apache.catalina.tribes.tipis.ReplicatedMap.<init>
>> (ReplicatedMap.java:90)
>> at org.apache.catalina.ha.context.ReplicatedContext.start
>> (ReplicatedContext.java:64)
>> at org.apache.catalina.core.ContainerBase.start
>> (ContainerBase.java:1045)
>> at org.apache.catalina.core.StandardHost.start
>> (StandardHost.java:719)
>> at org.apache.catalina.core.ContainerBase.start
>> (ContainerBase.java:1045)
>> at org.apache.catalina.core.StandardEngine.start
>> (StandardEngine.java:443)
>> at org.apache.catalina.core.StandardService.start
>> (StandardService.java:516)
>> at org.apache.catalina.core.StandardServer.start
>> (StandardServer.java:710)
>> at org.apache.catalina.startup.Catalina.start
>> (Catalina.java:578)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke
>> (NativeMethodAccessorImpl.java:39)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke
>> (DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:585)
>> at org.apache.catalina.startup.Bootstrap.start
>> (Bootstrap.java:288)
>> at org.apache.catalina.startup.Bootstrap.main
>> (Bootstrap.java:413)
>> Caused by: org.apache.catalina.tribes.ChannelException: Send
>> failed, attempt:2 max:1; Faulty members:tcp://{10, 84, 2, 42}:4000;
>> at
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop
>> (ParallelNioSender.java:172)
>> at
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessag
>> e(ParallelNioSender.java:78)
>> at
>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMes
>> sage(PooledParallelSender.java:53)
>> at
>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessa
>> ge(ReplicationTransmitter.java:80)
>> at
>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage
>> (ChannelCoordinator.java:78)
>> at
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
>> (ChannelInterceptorBase.java:75)
>> at
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.send
>> Message(TcpFailureDetector.java:87)
>> at
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
>> (ChannelInterceptorBase.java:75)
>> at
>> org.apache.catalina.tribes.group.interceptors.MessageDispatchIntercep
>> tor.sendMessage(MessageDispatchInterceptor.java:73)
>> at
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
>> (ChannelInterceptorBase.java:75)
>> at org.apache.catalina.tribes.group.GroupChannel.send
>> (GroupChannel.java:216)
>> at org.apache.catalina.tribes.group.GroupChannel.send
>> (GroupChannel.java:175)
>> at org.apache.catalina.tribes.group.RpcChannel.send
>> (RpcChannel.java:89)
>> at
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.broadcast
>> (AbstractReplicatedMap.java:300)
>> at
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.init
>> (AbstractReplicatedMap.java:223)
>> ... 16 more
>> Caused by: java.net.SocketException: Invalid argument
>> at sun.nio.ch.Net.setIntOption0(Native Method)
>> at sun.nio.ch.Net.setIntOption(Net.java:152)
>> at sun.nio.ch.SocketChannelImpl$1.setInt
>> (SocketChannelImpl.java:372)
>> at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>> at sun.nio.ch.SocketOptsImpl$IP.typeOfService
>> (SocketOptsImpl.java:249)
>> at sun.nio.ch.OptionAdaptor.setTrafficClass
>> (OptionAdaptor.java:158)
>> at sun.nio.ch.SocketAdaptor.setTrafficClass
>> (SocketAdaptor.java:330)
>> at
>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect
>> (NioSender.java:147)
>> at
>> org.apache.catalina.tribes.transport.nio.NioSender.process
>> (NioSender.java:89)
>> at
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop
>> (ParallelNioSender.java:130)
>> ... 30 more
>> Jul 10, 2008 1:06:42 PM org.apache.catalina.startup.Catalina start
>> SEVERE: Catalina.start:
>> LifecycleException: Failed to start ReplicatedContext:
>> java.lang.RuntimeException: Unable to start replicated map.
>> at org.apache.catalina.ha.context.ReplicatedContext.start
>> (ReplicatedContext.java:73)
>> at org.apache.catalina.core.ContainerBase.start
>> (ContainerBase.java:1045)
>> at org.apache.catalina.core.StandardHost.start
>> (StandardHost.java:719)
>> at org.apache.catalina.core.ContainerBase.start
>> (ContainerBase.java:1045)
>> at org.apache.catalina.core.StandardEngine.start
>> (StandardEngine.java:443)
>> at org.apache.catalina.core.StandardService.start
>> (StandardService.java:516)
>> at org.apache.catalina.core.StandardServer.start
>> (StandardServer.java:710)
>> at org.apache.catalina.startup.Catalina.start
>> (Catalina.java:578)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke
>> (NativeMethodAccessorImpl.java:39)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke
>> (DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:585)
>> at org.apache.catalina.startup.Bootstrap.start
>> (Bootstrap.java:288)
>> at org.apache.catalina.startup.Bootstrap.main
>> (Bootstrap.java:413)
>> Jul 10, 2008 1:06:42 PM org.apache.catalina.startup.Catalina start
>> INFO: Server startup in 2329 ms
>> Jul 10, 2008 1:06:47 PM
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>> WARNING: Not retrying send for:tcp://{10, 84, 2, 42}:4000; Sender
>> is disconnected.
>> Jul 10, 2008 1:06:47 PM
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>> memberDisappeared
>> INFO: Received memberDisappeared
>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,
>> 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32 116
>> 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command=
>> {}, domain={}, ]] message. Will verify.
>> Jul 10, 2008 1:06:47 PM
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>> memberDisappeared
>> INFO: Verification complete. Member still alive
>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,
>> 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32 116
>> 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command=
>> {}, domain={}, ]]
>> Jul 10, 2008 1:06:47 PM
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
>> SEVERE: Unable to send AbstractReplicatedMap.ping message
>> org.apache.catalina.tribes.ChannelException: Send failed, and
>> sender is disconnected. Not retrying.; Faulty members:tcp://{10,
>> 84, 2, 42}:4000;
>> at
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop
>> (ParallelNioSender.java:156)
>> at
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessag
>> e(ParallelNioSender.java:78)
>> at
>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMes
>> sage(PooledParallelSender.java:53)
>> at
>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessa
>> ge(ReplicationTransmitter.java:80)
>> at
>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage
>> (ChannelCoordinator.java:78)
>> at
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
>> (ChannelInterceptorBase.java:75)
>> at
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.send
>> Message(TcpFailureDetector.java:87)
>> at
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
>> (ChannelInterceptorBase.java:75)
>> at
>> org.apache.catalina.tribes.group.interceptors.MessageDispatchIntercep
>> tor.sendMessage(MessageDispatchInterceptor.java:73)
>> at
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
>> (ChannelInterceptorBase.java:75)
>> at org.apache.catalina.tribes.group.GroupChannel.send
>> (GroupChannel.java:216)
>> at org.apache.catalina.tribes.group.GroupChannel.send
>> (GroupChannel.java:175)
>> at org.apache.catalina.tribes.group.RpcChannel.send
>> (RpcChannel.java:89)
>> at
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping
>> (AbstractReplicatedMap.java:253)
>> at
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
>> (AbstractReplicatedMap.java:793)
>> at org.apache.catalina.tribes.group.GroupChannel.heartbeat
>> (GroupChannel.java:153)
>> at org.apache.catalina.tribes.group.GroupChannel
>> $HeartbeatThread.run(GroupChannel.java:661)
>> Caused by: java.net.SocketException: Invalid argument
>> at sun.nio.ch.Net.setIntOption0(Native Method)
>> at sun.nio.ch.Net.setIntOption(Net.java:152)
>> at sun.nio.ch.SocketChannelImpl$1.setInt
>> (SocketChannelImpl.java:372)
>> at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>> at sun.nio.ch.SocketOptsImpl$IP.typeOfService
>> (SocketOptsImpl.java:249)
>> at sun.nio.ch.OptionAdaptor.setTrafficClass
>> (OptionAdaptor.java:158)
>> at sun.nio.ch.SocketAdaptor.setTrafficClass
>> (SocketAdaptor.java:330)
>> at
>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect
>> (NioSender.java:147)
>> at
>> org.apache.catalina.tribes.transport.nio.NioSender.process
>> (NioSender.java:89)
>> at
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop
>> (ParallelNioSender.java:130)
>> ... 16 more
>> Jul 10, 2008 1:06:47 PM
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>> WARNING: Not retrying send for:tcp://{10, 84, 2, 42}:4000; Sender
>> is disconnected.
>> Jul 10, 2008 1:06:47 PM
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>> memberDisappeared
>> INFO: Received memberDisappeared
>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,
>> 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32 116
>> 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command=
>> {}, domain={}, ]] message. Will verify.
>> Jul 10, 2008 1:06:47 PM
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>> memberDisappeared
>> INFO: Verification complete. Member still alive
>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,
>> 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32 116
>> 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command=
>> {}, domain={}, ]]
>> Jul 10, 2008 1:06:47 PM
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
>> SEVERE: Unable to send AbstractReplicatedMap.ping message
>> org.apache.catalina.tribes.ChannelException: Send failed, and
>> sender is disconnected. Not retrying.; Faulty members:tcp://{10,
>> 84, 2, 42}:4000;
>> at
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop
>> (ParallelNioSender.java:156)
>> at
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessag
>> e(ParallelNioSender.java:78)
>> at
>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMes
>> sage(PooledParallelSender.java:53)
>> at
>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessa
>> ge(ReplicationTransmitter.java:80)
>> at
>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage
>> (ChannelCoordinator.java:78)
>> at
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
>> (ChannelInterceptorBase.java:75)
>> at
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.send
>> Message(TcpFailureDetector.java:87)
>> at
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
>> (ChannelInterceptorBase.java:75)
>> at
>> org.apache.catalina.tribes.group.interceptors.MessageDispatchIntercep
>> tor.sendMessage(MessageDispatchInterceptor.java:73)
>> at
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage
>> (ChannelInterceptorBase.java:75)
>> at org.apache.catalina.tribes.group.GroupChannel.send
>> (GroupChannel.java:216)
>> at org.apache.catalina.tribes.group.GroupChannel.send
>> (GroupChannel.java:175)
>> at org.apache.catalina.tribes.group.RpcChannel.send
>> (RpcChannel.java:89)
>> at
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping
>> (AbstractReplicatedMap.java:253)
>> at
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
>> (AbstractReplicatedMap.java:793)
>> at org.apache.catalina.tribes.group.GroupChannel.heartbeat
>> (GroupChannel.java:158)
>> at org.apache.catalina.tribes.group.GroupChannel
>> $HeartbeatThread.run(GroupChannel.java:661)
>> Caused by: java.net.SocketException: Invalid argument
>> at sun.nio.ch.Net.setIntOption0(Native Method)
>> at sun.nio.ch.Net.setIntOption(Net.java:152)
>> at sun.nio.ch.SocketChannelImpl$1.setInt
>> (SocketChannelImpl.java:372)
>> at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>> at sun.nio.ch.SocketOptsImpl$IP.typeOfService
>> (SocketOptsImpl.java:249)
>> at sun.nio.ch.OptionAdaptor.setTrafficClass
>> (OptionAdaptor.java:158)
>> at sun.nio.ch.SocketAdaptor.setTrafficClass
>> (SocketAdaptor.java:330)
>> at
>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect
>> (NioSender.java:147)
>> at
>> org.apache.catalina.tribes.transport.nio.NioSender.process
>> (NioSender.java:89)
>> at
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop
>> (ParallelNioSender.java:130)
>> ... 16 more
>>
>>
>> and so on. Top shows that both Java process is Up to 86%, the
>> machine is never idle and all memory is consumed. The machine is
>> a PowerPc G6 from Apple with OSX 10.4.11.
>> Tomcat version is: Apache Tomcat/6.0.16
>> JVM version is: 1.5.0_13-b05-241
>> I suspect the heartbeat pings have a problem, I just do not know
>> what to do about them.
>>
>> Thanks ahead,
>>
>> János
>>
>>
>>
>> ---------------------------------------------------------------------
>> To start a new topic, e-mail: users@tomcat.apache.org
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>>
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Version 6 Clustering problem
Posted by Filip Hanik - Dev Lists <de...@hanik.com>.
try setting the
|-Djava.net.preferIPv4Stack=true
option for your tomcat launch
Filip
|
János wrote:
> Hi,
>
> I am trying to establish a 3 member cluster on the same machine. Here
> is my server.xml copied from a sample I found on this list.
>
> <?xml version='1.0' encoding='utf-8'?>
> <Server port="8005" shutdown="SHUTDOWN">
> <Listener className="org.apache.catalina.core.AprLifecycleListener"
> SSLEngine="on" />
> <Listener className="org.apache.catalina.core.JasperListener" />
> <Listener
> className="org.apache.catalina.mbeans.ServerLifecycleListener" />
> <Listener
> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
> />
>
> <GlobalNamingResources>
> <Resource name="UserDatabase" auth="Container"
> type="org.apache.catalina.UserDatabase"
> description="User database that can be updated and saved"
>
> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
> pathname="conf/tomcat-users.xml" />
> </GlobalNamingResources>
>
> <Service name="Catalina">
>
> <Connector port="8080" protocol="HTTP/1.1"
> connectionTimeout="20000"
> redirectPort="8443" />
> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
>
> <Engine name="Catalina" defaultHost="localhost" jvmRoute="node1">
>
> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
> <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
> resourceName="UserDatabase"/>
>
> <Host name="localhost" appBase="webapps"
> unpackWARs="true" autoDeploy="true"
> xmlValidation="false" xmlNamespaceAware="false">
> <Context path="" docBase="ROOT"
> className="org.apache.catalina.ha.context.ReplicatedContext"/>
> </Host>
> </Engine>
> </Service>
> </Server>
>
> The other two server.xml-s are the same, except the shutdown ports
> are 8105, 8205, http ports are 8180, 8280 and https ports are 8543,
> 8643.
> As soon as I am starting the second instance I receive the following
> into the logs.
>
> In node1 logs:
>
> bml0042:/usr/local/tomcat/node1/logs janos$ cat catalina.2008-07-10.log
> Jul 10, 2008 1:03:24 PM org.apache.catalina.core.AprLifecycleListener
> init
> INFO: The APR based Apache Tomcat Native library which allows optimal
> performance in production environments was not found on the
> java.library.path:
> .:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java
> Jul 10, 2008 1:03:24 PM org.apache.coyote.http11.Http11Protocol init
> INFO: Initializing Coyote HTTP/1.1 on http-8080
> Jul 10, 2008 1:03:24 PM org.apache.catalina.startup.Catalina load
> INFO: Initialization processed in 1187 ms
> Jul 10, 2008 1:03:24 PM org.apache.catalina.core.StandardService start
> INFO: Starting service Catalina
> Jul 10, 2008 1:03:24 PM org.apache.catalina.core.StandardEngine start
> INFO: Starting Servlet Engine: Apache Tomcat/6.0.16
> Jul 10, 2008 1:03:24 PM org.apache.catalina.ha.tcp.SimpleTcpCluster start
> INFO: Cluster is about to start
> Jul 10, 2008 1:03:24 PM
> org.apache.catalina.tribes.transport.ReceiverBase bind
> INFO: Receiver Server Socket bound to:/10.84.2.42:4000
> Jul 10, 2008 1:03:24 PM
> org.apache.catalina.tribes.membership.McastServiceImpl setupSocket
> INFO: Setting cluster mcast soTimeout to 500
> Jul 10, 2008 1:03:24 PM
> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
> INFO: Sleeping for 1000 milliseconds to establish cluster membership,
> start level:4
> Jul 10, 2008 1:03:25 PM
> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
> INFO: Done sleeping, membership established, start level:4
> Jul 10, 2008 1:03:25 PM
> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
> INFO: Sleeping for 1000 milliseconds to establish cluster membership,
> start level:8
> Jul 10, 2008 1:03:26 PM
> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
> INFO: Done sleeping, membership established, start level:8
> Jul 10, 2008 1:03:26 PM
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
> INFO: Initializing AbstractReplicatedMap with context name:
> Jul 10, 2008 1:03:27 PM org.apache.catalina.core.StandardContext
> addApplicationListener
> INFO: The listener "listeners.ContextListener" is already configured
> for this context. The duplicate definition has been ignored.
> Jul 10, 2008 1:03:27 PM org.apache.catalina.core.StandardContext
> addApplicationListener
> INFO: The listener "listeners.SessionListener" is already configured
> for this context. The duplicate definition has been ignored.
> Jul 10, 2008 1:03:27 PM org.apache.catalina.ha.session.DeltaManager start
> INFO: Register manager /examples to cluster element Engine with name
> Catalina
> Jul 10, 2008 1:03:27 PM org.apache.catalina.ha.session.DeltaManager start
> INFO: Starting clustering manager at /examples
> Jul 10, 2008 1:03:27 PM org.apache.catalina.ha.session.DeltaManager
> getAllClusterSessions
> INFO: Manager [localhost#/examples]: skipping state transfer. No
> members active in cluster group.
> Jul 10, 2008 1:03:28 PM
> org.apache.catalina.ha.session.JvmRouteBinderValve start
> INFO: JvmRouteBinderValve started
> Jul 10, 2008 1:03:28 PM org.apache.coyote.http11.Http11Protocol start
> INFO: Starting Coyote HTTP/1.1 on http-8080
> Jul 10, 2008 1:03:28 PM org.apache.jk.common.ChannelSocket init
> INFO: JK: ajp13 listening on /0.0.0.0:8009
> Jul 10, 2008 1:03:28 PM org.apache.jk.server.JkMain start
> INFO: Jk running ID=0 time=0/45 config=null
> Jul 10, 2008 1:03:28 PM org.apache.catalina.startup.Catalina start
> INFO: Server startup in 3616 ms
> Jul 10, 2008 1:06:41 PM
> org.apache.catalina.tribes.transport.nio.NioReceiver listen
> SEVERE: Unable to process request in NioReceiver
> java.net.SocketException: Invalid argument
> at sun.nio.ch.Net.setIntOption0(Native Method)
> at sun.nio.ch.Net.setIntOption(Net.java:152)
> at
> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
> at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
> at
> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
> at
> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
> at
> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
> at
> org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:273)
>
> at
> org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioReceiver.java:353)
>
> at java.lang.Thread.run(Thread.java:613)
> Jul 10, 2008 1:06:41 PM org.apache.catalina.ha.tcp.SimpleTcpCluster
> memberAdded
> INFO: Replication member
> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,
> 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46 127 -68
> 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={}, command={},
> domain={}, ]
> Jul 10, 2008 1:06:41 PM org.apache.catalina.tribes.io.BufferPool
> getBufferPool
> INFO: Created a buffer pool with max size:104857600 bytes of
> type:org.apache.catalina.tribes.io.BufferPool15Impl
> Jul 10, 2008 1:06:41 PM
> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
> WARNING: Member send is failing for:tcp://{10, 84, 2, 42}:4001 ;
> Setting to suspect and retrying.
> Jul 10, 2008 1:06:41 PM
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
> memberDisappeared
> INFO: Received
> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
> 84, 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46 127
> -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={}, command={},
> domain={}, ]] message. Will verify.
> Jul 10, 2008 1:06:41 PM
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
> memberDisappeared
> INFO: Verification complete. Member still
> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,
> 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46 127 -68
> 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={}, command={},
> domain={}, ]]
> Jul 10, 2008 1:06:41 PM
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
> SEVERE: Unable to send AbstractReplicatedMap.ping message
> org.apache.catalina.tribes.ChannelException: Send failed, attempt:2
> max:1; Faulty members:tcp://{10, 84, 2, 42}:4001;
> at
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:172)
>
> at
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:78)
>
> at
> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53)
>
> at
> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80)
>
> at
> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78)
>
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>
> at
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87)
>
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>
> at
> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:73)
>
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>
> at
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
> at
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
> at
> org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:89)
> at
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:253)
>
> at
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:793)
>
> at
> org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:153)
>
> at
> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:661)
>
> Caused by: java.net.SocketException: Invalid argument
> at sun.nio.ch.Net.setIntOption0(Native Method)
> at sun.nio.ch.Net.setIntOption(Net.java:152)
> at
> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
> at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
> at
> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
> at
> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
> at
> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
> at
> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect(NioSender.java:147)
>
> at
> org.apache.catalina.tribes.transport.nio.NioSender.process(NioSender.java:89)
>
> at
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:130)
>
> ... 16 more
> Jul 10, 2008 1:06:41 PM
> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
> WARNING: Not retrying send for:tcp://{10, 84, 2, 42}:4001; Sender is
> disconnected.
> Jul 10, 2008 1:06:41 PM
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
> memberDisappeared
> INFO: Received
> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
> 84, 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46 127
> -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={}, command={},
> domain={}, ]] message. Will verify.
> Jul 10, 2008 1:06:42 PM
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
> memberDisappeared
> INFO: Verification complete. Member still
> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,
> 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46 127 -68
> 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={}, command={},
> domain={}, ]]
> Jul 10, 2008 1:06:42 PM
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
> SEVERE: Unable to send AbstractReplicatedMap.ping message
> org.apache.catalina.tribes.ChannelException: Send failed, and sender
> is disconnected. Not retrying.; Faulty members:tcp://{10, 84, 2,
> 42}:4001;
> at
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:156)
>
> at
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:78)
>
> at
> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53)
>
> at
> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80)
>
> at
> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78)
>
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>
> at
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87)
>
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>
> at
> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:73)
>
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>
> at
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
> at
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
> at
> org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:89)
> at
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:253)
>
> at
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:793)
>
> at
> org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:158)
>
> at
> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:661)
>
> Caused by: java.net.SocketException: Invalid argument
> at sun.nio.ch.Net.setIntOption0(Native Method)
> at sun.nio.ch.Net.setIntOption(Net.java:152)
> at
> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
> at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
> at
> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
> at
> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
> at
> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
> at
> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect(NioSender.java:147)
>
> at
> org.apache.catalina.tribes.transport.nio.NioSender.process(NioSender.java:89)
>
> at
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:130)
>
> ... 16 more
>
>
> and so on forever.
>
> In node2 logs I see the following:
>
> bml0042:/usr/local/tomcat/node2/logs janos$ cat catalina.2008-07-10.log
> Jul 10, 2008 1:06:40 PM org.apache.catalina.core.AprLifecycleListener
> init
> INFO: The APR based Apache Tomcat Native library which allows optimal
> performance in production environments was not found on the
> java.library.path:
> .:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java
> Jul 10, 2008 1:06:40 PM org.apache.coyote.http11.Http11Protocol init
> INFO: Initializing Coyote HTTP/1.1 on http-8180
> Jul 10, 2008 1:06:40 PM org.apache.catalina.startup.Catalina load
> INFO: Initialization processed in 1148 ms
> Jul 10, 2008 1:06:40 PM org.apache.catalina.core.StandardService start
> INFO: Starting service Catalina
> Jul 10, 2008 1:06:40 PM org.apache.catalina.core.StandardEngine start
> INFO: Starting Servlet Engine: Apache Tomcat/6.0.16
> Jul 10, 2008 1:06:40 PM org.apache.catalina.ha.tcp.SimpleTcpCluster start
> INFO: Cluster is about to start
> Jul 10, 2008 1:06:40 PM
> org.apache.catalina.tribes.transport.ReceiverBase bind
> INFO: Receiver Server Socket bound to:/10.84.2.42:4001
> Jul 10, 2008 1:06:40 PM
> org.apache.catalina.tribes.membership.McastServiceImpl setupSocket
> INFO: Setting cluster mcast soTimeout to 500
> Jul 10, 2008 1:06:40 PM
> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
> INFO: Sleeping for 1000 milliseconds to establish cluster membership,
> start level:4
> Jul 10, 2008 1:06:41 PM org.apache.catalina.ha.tcp.SimpleTcpCluster
> memberAdded
> INFO: Replication member
> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,
> 2, 42}:4000,{10, 84, 2, 42},4000, alive=196161,id={-25 34 -32 116 42
> -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command={},
> domain={}, ]
> Jul 10, 2008 1:06:41 PM
> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
> INFO: Done sleeping, membership established, start level:4
> Jul 10, 2008 1:06:41 PM
> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
> INFO: Sleeping for 1000 milliseconds to establish cluster membership,
> start level:8
> Jul 10, 2008 1:06:41 PM
> org.apache.catalina.tribes.transport.nio.NioReceiver listen
> SEVERE: Unable to process request in NioReceiver
> java.net.SocketException: Invalid argument
> at sun.nio.ch.Net.setIntOption0(Native Method)
> at sun.nio.ch.Net.setIntOption(Net.java:152)
> at
> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
> at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
> at
> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
> at
> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
> at
> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
> at
> org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:273)
>
> at
> org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioReceiver.java:353)
>
> at java.lang.Thread.run(Thread.java:613)
> Jul 10, 2008 1:06:42 PM
> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
> INFO: Done sleeping, membership established, start level:8
> Jul 10, 2008 1:06:42 PM
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
> INFO: Initializing AbstractReplicatedMap with context name:
> Jul 10, 2008 1:06:42 PM org.apache.catalina.tribes.io.BufferPool
> getBufferPool
> INFO: Created a buffer pool with max size:104857600 bytes of
> type:org.apache.catalina.tribes.io.BufferPool15Impl
> Jul 10, 2008 1:06:42 PM
> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
> WARNING: Member send is failing for:tcp://{10, 84, 2, 42}:4000 ;
> Setting to suspect and retrying.
> Jul 10, 2008 1:06:42 PM
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
> memberDisappeared
> INFO: Received
> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
> 84, 2, 42}:4000,{10, 84, 2, 42},4000, alive=197662,id={-25 34 -32 116
> 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command={},
> domain={}, ]] message. Will verify.
> Jul 10, 2008 1:06:42 PM
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
> memberDisappeared
> INFO: Verification complete. Member still
> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,
> 2, 42}:4000,{10, 84, 2, 42},4000, alive=197662,id={-25 34 -32 116 42
> -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command={},
> domain={}, ]]
> Jul 10, 2008 1:06:42 PM
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
> WARNING: Unable to send map start message.
> Jul 10, 2008 1:06:42 PM
> org.apache.catalina.ha.context.ReplicatedContext start
> SEVERE: Unable to start ReplicatedContext
> java.lang.RuntimeException: Unable to start replicated map.
> at
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.init(AbstractReplicatedMap.java:230)
>
> at
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.<init>(AbstractReplicatedMap.java:171)
>
> at
> org.apache.catalina.tribes.tipis.ReplicatedMap.<init>(ReplicatedMap.java:90)
>
> at
> org.apache.catalina.ha.context.ReplicatedContext.start(ReplicatedContext.java:64)
>
> at
> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
> at
> org.apache.catalina.core.StandardHost.start(StandardHost.java:719)
> at
> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
> at
> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443)
> at
> org.apache.catalina.core.StandardService.start(StandardService.java:516)
> at
> org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
> at org.apache.catalina.startup.Catalina.start(Catalina.java:578)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:585)
> at
> org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
> Caused by: org.apache.catalina.tribes.ChannelException: Send failed,
> attempt:2 max:1; Faulty members:tcp://{10, 84, 2, 42}:4000;
> at
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:172)
>
> at
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:78)
>
> at
> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53)
>
> at
> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80)
>
> at
> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78)
>
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>
> at
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87)
>
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>
> at
> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:73)
>
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>
> at
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
> at
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
> at
> org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:89)
> at
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.broadcast(AbstractReplicatedMap.java:300)
>
> at
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.init(AbstractReplicatedMap.java:223)
>
> ... 16 more
> Caused by: java.net.SocketException: Invalid argument
> at sun.nio.ch.Net.setIntOption0(Native Method)
> at sun.nio.ch.Net.setIntOption(Net.java:152)
> at
> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
> at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
> at
> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
> at
> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
> at
> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
> at
> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect(NioSender.java:147)
>
> at
> org.apache.catalina.tribes.transport.nio.NioSender.process(NioSender.java:89)
>
> at
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:130)
>
> ... 30 more
> Jul 10, 2008 1:06:42 PM org.apache.catalina.startup.Catalina start
> SEVERE: Catalina.start:
> LifecycleException: Failed to start ReplicatedContext:
> java.lang.RuntimeException: Unable to start replicated map.
> at
> org.apache.catalina.ha.context.ReplicatedContext.start(ReplicatedContext.java:73)
>
> at
> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
> at
> org.apache.catalina.core.StandardHost.start(StandardHost.java:719)
> at
> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
> at
> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443)
> at
> org.apache.catalina.core.StandardService.start(StandardService.java:516)
> at
> org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
> at org.apache.catalina.startup.Catalina.start(Catalina.java:578)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:585)
> at
> org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
> Jul 10, 2008 1:06:42 PM org.apache.catalina.startup.Catalina start
> INFO: Server startup in 2329 ms
> Jul 10, 2008 1:06:47 PM
> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
> WARNING: Not retrying send for:tcp://{10, 84, 2, 42}:4000; Sender is
> disconnected.
> Jul 10, 2008 1:06:47 PM
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
> memberDisappeared
> INFO: Received
> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
> 84, 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32 116
> 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command={},
> domain={}, ]] message. Will verify.
> Jul 10, 2008 1:06:47 PM
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
> memberDisappeared
> INFO: Verification complete. Member still
> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,
> 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32 116 42
> -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command={},
> domain={}, ]]
> Jul 10, 2008 1:06:47 PM
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
> SEVERE: Unable to send AbstractReplicatedMap.ping message
> org.apache.catalina.tribes.ChannelException: Send failed, and sender
> is disconnected. Not retrying.; Faulty members:tcp://{10, 84, 2,
> 42}:4000;
> at
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:156)
>
> at
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:78)
>
> at
> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53)
>
> at
> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80)
>
> at
> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78)
>
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>
> at
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87)
>
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>
> at
> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:73)
>
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>
> at
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
> at
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
> at
> org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:89)
> at
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:253)
>
> at
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:793)
>
> at
> org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:153)
>
> at
> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:661)
>
> Caused by: java.net.SocketException: Invalid argument
> at sun.nio.ch.Net.setIntOption0(Native Method)
> at sun.nio.ch.Net.setIntOption(Net.java:152)
> at
> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
> at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
> at
> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
> at
> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
> at
> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
> at
> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect(NioSender.java:147)
>
> at
> org.apache.catalina.tribes.transport.nio.NioSender.process(NioSender.java:89)
>
> at
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:130)
>
> ... 16 more
> Jul 10, 2008 1:06:47 PM
> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
> WARNING: Not retrying send for:tcp://{10, 84, 2, 42}:4000; Sender is
> disconnected.
> Jul 10, 2008 1:06:47 PM
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
> memberDisappeared
> INFO: Received
> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
> 84, 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32 116
> 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command={},
> domain={}, ]] message. Will verify.
> Jul 10, 2008 1:06:47 PM
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
> memberDisappeared
> INFO: Verification complete. Member still
> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,
> 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32 116 42
> -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command={},
> domain={}, ]]
> Jul 10, 2008 1:06:47 PM
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
> SEVERE: Unable to send AbstractReplicatedMap.ping message
> org.apache.catalina.tribes.ChannelException: Send failed, and sender
> is disconnected. Not retrying.; Faulty members:tcp://{10, 84, 2,
> 42}:4000;
> at
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:156)
>
> at
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:78)
>
> at
> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53)
>
> at
> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80)
>
> at
> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78)
>
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>
> at
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87)
>
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>
> at
> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:73)
>
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>
> at
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
> at
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
> at
> org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:89)
> at
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:253)
>
> at
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:793)
>
> at
> org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:158)
>
> at
> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:661)
>
> Caused by: java.net.SocketException: Invalid argument
> at sun.nio.ch.Net.setIntOption0(Native Method)
> at sun.nio.ch.Net.setIntOption(Net.java:152)
> at
> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
> at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
> at
> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
> at
> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
> at
> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
> at
> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect(NioSender.java:147)
>
> at
> org.apache.catalina.tribes.transport.nio.NioSender.process(NioSender.java:89)
>
> at
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:130)
>
> ... 16 more
>
>
> and so on. Top shows that both Java process is Up to 86%, the machine
> is never idle and all memory is consumed. The machine is a PowerPc G6
> from Apple with OSX 10.4.11.
> Tomcat version is: Apache Tomcat/6.0.16
> JVM version is: 1.5.0_13-b05-241
> I suspect the heartbeat pings have a problem, I just do not know what
> to do about them.
>
> Thanks ahead,
>
> János
>
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org