You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by János <ja...@yale.edu> on 2008/07/10 21:12:34 UTC

Version 6 Clustering problem

Hi,

I am trying to establish a 3 member cluster on the same machine.   
Here is my server.xml copied from a sample I found on this list.

<?xml version='1.0' encoding='utf-8'?>
<Server port="8005" shutdown="SHUTDOWN">
   <Listener  
className="org.apache.catalina.core.AprLifecycleListener"  
SSLEngine="on" />
   <Listener className="org.apache.catalina.core.JasperListener" />
   <Listener  
className="org.apache.catalina.mbeans.ServerLifecycleListener" />
   <Listener  
className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"  
/>

   <GlobalNamingResources>
     <Resource name="UserDatabase" auth="Container"
               type="org.apache.catalina.UserDatabase"
               description="User database that can be updated and saved"
                
factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
               pathname="conf/tomcat-users.xml" />
   </GlobalNamingResources>

   <Service name="Catalina">

     <Connector port="8080" protocol="HTTP/1.1"
                connectionTimeout="20000"
                redirectPort="8443" />
     <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />

     <Engine name="Catalina" defaultHost="localhost" jvmRoute="node1">

       <Cluster  
className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
       <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
              resourceName="UserDatabase"/>

       <Host name="localhost"  appBase="webapps"
             unpackWARs="true" autoDeploy="true"
             xmlValidation="false" xmlNamespaceAware="false">
         <Context path="" docBase="ROOT"  
className="org.apache.catalina.ha.context.ReplicatedContext"/>
       </Host>
     </Engine>
   </Service>
</Server>

The other two server.xml-s are the same, except the shutdown ports  
are  8105, 8205, http ports are 8180, 8280 and https ports are 8543,  
8643.
As soon as I am starting the second instance I receive the following  
into the logs.

In node1 logs:

bml0042:/usr/local/tomcat/node1/logs janos$ cat catalina.2008-07-10.log
Jul 10, 2008 1:03:24 PM org.apache.catalina.core.AprLifecycleListener  
init
INFO: The APR based Apache Tomcat Native library which allows optimal  
performance in production environments was not found on the  
java.library.path: .:/Library/Java/Extensions:/System/Library/Java/ 
Extensions:/usr/lib/java
Jul 10, 2008 1:03:24 PM org.apache.coyote.http11.Http11Protocol init
INFO: Initializing Coyote HTTP/1.1 on http-8080
Jul 10, 2008 1:03:24 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 1187 ms
Jul 10, 2008 1:03:24 PM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
Jul 10, 2008 1:03:24 PM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.16
Jul 10, 2008 1:03:24 PM org.apache.catalina.ha.tcp.SimpleTcpCluster  
start
INFO: Cluster is about to start
Jul 10, 2008 1:03:24 PM  
org.apache.catalina.tribes.transport.ReceiverBase bind
INFO: Receiver Server Socket bound to:/10.84.2.42:4000
Jul 10, 2008 1:03:24 PM  
org.apache.catalina.tribes.membership.McastServiceImpl setupSocket
INFO: Setting cluster mcast soTimeout to 500
Jul 10, 2008 1:03:24 PM  
org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
INFO: Sleeping for 1000 milliseconds to establish cluster membership,  
start level:4
Jul 10, 2008 1:03:25 PM  
org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
INFO: Done sleeping, membership established, start level:4
Jul 10, 2008 1:03:25 PM  
org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
INFO: Sleeping for 1000 milliseconds to establish cluster membership,  
start level:8
Jul 10, 2008 1:03:26 PM  
org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
INFO: Done sleeping, membership established, start level:8
Jul 10, 2008 1:03:26 PM  
org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
INFO: Initializing AbstractReplicatedMap with context name:
Jul 10, 2008 1:03:27 PM org.apache.catalina.core.StandardContext  
addApplicationListener
INFO: The listener "listeners.ContextListener" is already configured  
for this context. The duplicate definition has been ignored.
Jul 10, 2008 1:03:27 PM org.apache.catalina.core.StandardContext  
addApplicationListener
INFO: The listener "listeners.SessionListener" is already configured  
for this context. The duplicate definition has been ignored.
Jul 10, 2008 1:03:27 PM org.apache.catalina.ha.session.DeltaManager  
start
INFO: Register manager /examples to cluster element Engine with name  
Catalina
Jul 10, 2008 1:03:27 PM org.apache.catalina.ha.session.DeltaManager  
start
INFO: Starting clustering manager at /examples
Jul 10, 2008 1:03:27 PM org.apache.catalina.ha.session.DeltaManager  
getAllClusterSessions
INFO: Manager [localhost#/examples]: skipping state transfer. No  
members active in cluster group.
Jul 10, 2008 1:03:28 PM  
org.apache.catalina.ha.session.JvmRouteBinderValve start
INFO: JvmRouteBinderValve started
Jul 10, 2008 1:03:28 PM org.apache.coyote.http11.Http11Protocol start
INFO: Starting Coyote HTTP/1.1 on http-8080
Jul 10, 2008 1:03:28 PM org.apache.jk.common.ChannelSocket init
INFO: JK: ajp13 listening on /0.0.0.0:8009
Jul 10, 2008 1:03:28 PM org.apache.jk.server.JkMain start
INFO: Jk running ID=0 time=0/45  config=null
Jul 10, 2008 1:03:28 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 3616 ms
Jul 10, 2008 1:06:41 PM  
org.apache.catalina.tribes.transport.nio.NioReceiver listen
SEVERE: Unable to process request in NioReceiver
java.net.SocketException: Invalid argument
         at sun.nio.ch.Net.setIntOption0(Native Method)
         at sun.nio.ch.Net.setIntOption(Net.java:152)
         at sun.nio.ch.SocketChannelImpl$1.setInt 
(SocketChannelImpl.java:372)
         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
         at sun.nio.ch.SocketOptsImpl$IP.typeOfService 
(SocketOptsImpl.java:249)
         at sun.nio.ch.OptionAdaptor.setTrafficClass 
(OptionAdaptor.java:158)
         at sun.nio.ch.SocketAdaptor.setTrafficClass 
(SocketAdaptor.java:330)
         at  
org.apache.catalina.tribes.transport.nio.NioReceiver.listen 
(NioReceiver.java:273)
         at org.apache.catalina.tribes.transport.nio.NioReceiver.run 
(NioReceiver.java:353)
         at java.lang.Thread.run(Thread.java:613)
Jul 10, 2008 1:06:41 PM org.apache.catalina.ha.tcp.SimpleTcpCluster  
memberAdded
INFO: Replication member  
added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,  
2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46 127  
-68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={}, command={},  
domain={}, ]
Jul 10, 2008 1:06:41 PM org.apache.catalina.tribes.io.BufferPool  
getBufferPool
INFO: Created a buffer pool with max size:104857600 bytes of  
type:org.apache.catalina.tribes.io.BufferPool15Impl
Jul 10, 2008 1:06:41 PM  
org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
WARNING: Member send is failing for:tcp://{10, 84, 2, 42}:4001 ;  
Setting to suspect and retrying.
Jul 10, 2008 1:06:41 PM  
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector  
memberDisappeared
INFO: Received memberDisappeared 
[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84, 2,  
42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46 127 -68  
65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={}, command={},  
domain={}, ]] message. Will verify.
Jul 10, 2008 1:06:41 PM  
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector  
memberDisappeared
INFO: Verification complete. Member still alive 
[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84, 2,  
42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46 127 -68  
65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={}, command={},  
domain={}, ]]
Jul 10, 2008 1:06:41 PM  
org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
SEVERE: Unable to send AbstractReplicatedMap.ping message
org.apache.catalina.tribes.ChannelException: Send failed, attempt:2  
max:1; Faulty members:tcp://{10, 84, 2, 42}:4001;
         at  
org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop 
(ParallelNioSender.java:172)
         at  
org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage 
(ParallelNioSender.java:78)
         at  
org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessag 
e(PooledParallelSender.java:53)
         at  
org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage( 
ReplicationTransmitter.java:80)
         at  
org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage 
(ChannelCoordinator.java:78)
         at  
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
(ChannelInterceptorBase.java:75)
         at  
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMes 
sage(TcpFailureDetector.java:87)
         at  
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
(ChannelInterceptorBase.java:75)
         at  
org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor 
.sendMessage(MessageDispatchInterceptor.java:73)
         at  
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
(ChannelInterceptorBase.java:75)
         at org.apache.catalina.tribes.group.GroupChannel.send 
(GroupChannel.java:216)
         at org.apache.catalina.tribes.group.GroupChannel.send 
(GroupChannel.java:175)
         at org.apache.catalina.tribes.group.RpcChannel.send 
(RpcChannel.java:89)
         at  
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping 
(AbstractReplicatedMap.java:253)
         at  
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat 
(AbstractReplicatedMap.java:793)
         at org.apache.catalina.tribes.group.GroupChannel.heartbeat 
(GroupChannel.java:153)
         at org.apache.catalina.tribes.group.GroupChannel 
$HeartbeatThread.run(GroupChannel.java:661)
Caused by: java.net.SocketException: Invalid argument
         at sun.nio.ch.Net.setIntOption0(Native Method)
         at sun.nio.ch.Net.setIntOption(Net.java:152)
         at sun.nio.ch.SocketChannelImpl$1.setInt 
(SocketChannelImpl.java:372)
         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
         at sun.nio.ch.SocketOptsImpl$IP.typeOfService 
(SocketOptsImpl.java:249)
         at sun.nio.ch.OptionAdaptor.setTrafficClass 
(OptionAdaptor.java:158)
         at sun.nio.ch.SocketAdaptor.setTrafficClass 
(SocketAdaptor.java:330)
         at  
org.apache.catalina.tribes.transport.nio.NioSender.completeConnect 
(NioSender.java:147)
         at org.apache.catalina.tribes.transport.nio.NioSender.process 
(NioSender.java:89)
         at  
org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop 
(ParallelNioSender.java:130)
         ... 16 more
Jul 10, 2008 1:06:41 PM  
org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
WARNING: Not retrying send for:tcp://{10, 84, 2, 42}:4001; Sender is  
disconnected.
Jul 10, 2008 1:06:41 PM  
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector  
memberDisappeared
INFO: Received memberDisappeared 
[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84, 2,  
42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46 127 -68  
65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={}, command={},  
domain={}, ]] message. Will verify.
Jul 10, 2008 1:06:42 PM  
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector  
memberDisappeared
INFO: Verification complete. Member still alive 
[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84, 2,  
42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46 127 -68  
65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={}, command={},  
domain={}, ]]
Jul 10, 2008 1:06:42 PM  
org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
SEVERE: Unable to send AbstractReplicatedMap.ping message
org.apache.catalina.tribes.ChannelException: Send failed, and sender  
is disconnected. Not retrying.; Faulty members:tcp://{10, 84, 2, 42}: 
4001;
         at  
org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop 
(ParallelNioSender.java:156)
         at  
org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage 
(ParallelNioSender.java:78)
         at  
org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessag 
e(PooledParallelSender.java:53)
         at  
org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage( 
ReplicationTransmitter.java:80)
         at  
org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage 
(ChannelCoordinator.java:78)
         at  
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
(ChannelInterceptorBase.java:75)
         at  
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMes 
sage(TcpFailureDetector.java:87)
         at  
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
(ChannelInterceptorBase.java:75)
         at  
org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor 
.sendMessage(MessageDispatchInterceptor.java:73)
         at  
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
(ChannelInterceptorBase.java:75)
         at org.apache.catalina.tribes.group.GroupChannel.send 
(GroupChannel.java:216)
         at org.apache.catalina.tribes.group.GroupChannel.send 
(GroupChannel.java:175)
         at org.apache.catalina.tribes.group.RpcChannel.send 
(RpcChannel.java:89)
         at  
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping 
(AbstractReplicatedMap.java:253)
         at  
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat 
(AbstractReplicatedMap.java:793)
         at org.apache.catalina.tribes.group.GroupChannel.heartbeat 
(GroupChannel.java:158)
         at org.apache.catalina.tribes.group.GroupChannel 
$HeartbeatThread.run(GroupChannel.java:661)
Caused by: java.net.SocketException: Invalid argument
         at sun.nio.ch.Net.setIntOption0(Native Method)
         at sun.nio.ch.Net.setIntOption(Net.java:152)
         at sun.nio.ch.SocketChannelImpl$1.setInt 
(SocketChannelImpl.java:372)
         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
         at sun.nio.ch.SocketOptsImpl$IP.typeOfService 
(SocketOptsImpl.java:249)
         at sun.nio.ch.OptionAdaptor.setTrafficClass 
(OptionAdaptor.java:158)
         at sun.nio.ch.SocketAdaptor.setTrafficClass 
(SocketAdaptor.java:330)
         at  
org.apache.catalina.tribes.transport.nio.NioSender.completeConnect 
(NioSender.java:147)
         at org.apache.catalina.tribes.transport.nio.NioSender.process 
(NioSender.java:89)
         at  
org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop 
(ParallelNioSender.java:130)
         ... 16 more


and so on forever.

In node2 logs I see the following:

bml0042:/usr/local/tomcat/node2/logs janos$ cat catalina.2008-07-10.log
Jul 10, 2008 1:06:40 PM org.apache.catalina.core.AprLifecycleListener  
init
INFO: The APR based Apache Tomcat Native library which allows optimal  
performance in production environments was not found on the  
java.library.path: .:/Library/Java/Extensions:/System/Library/Java/ 
Extensions:/usr/lib/java
Jul 10, 2008 1:06:40 PM org.apache.coyote.http11.Http11Protocol init
INFO: Initializing Coyote HTTP/1.1 on http-8180
Jul 10, 2008 1:06:40 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 1148 ms
Jul 10, 2008 1:06:40 PM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
Jul 10, 2008 1:06:40 PM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.16
Jul 10, 2008 1:06:40 PM org.apache.catalina.ha.tcp.SimpleTcpCluster  
start
INFO: Cluster is about to start
Jul 10, 2008 1:06:40 PM  
org.apache.catalina.tribes.transport.ReceiverBase bind
INFO: Receiver Server Socket bound to:/10.84.2.42:4001
Jul 10, 2008 1:06:40 PM  
org.apache.catalina.tribes.membership.McastServiceImpl setupSocket
INFO: Setting cluster mcast soTimeout to 500
Jul 10, 2008 1:06:40 PM  
org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
INFO: Sleeping for 1000 milliseconds to establish cluster membership,  
start level:4
Jul 10, 2008 1:06:41 PM org.apache.catalina.ha.tcp.SimpleTcpCluster  
memberAdded
INFO: Replication member  
added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,  
2, 42}:4000,{10, 84, 2, 42},4000, alive=196161,id={-25 34 -32 116 42  
-104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command={},  
domain={}, ]
Jul 10, 2008 1:06:41 PM  
org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
INFO: Done sleeping, membership established, start level:4
Jul 10, 2008 1:06:41 PM  
org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
INFO: Sleeping for 1000 milliseconds to establish cluster membership,  
start level:8
Jul 10, 2008 1:06:41 PM  
org.apache.catalina.tribes.transport.nio.NioReceiver listen
SEVERE: Unable to process request in NioReceiver
java.net.SocketException: Invalid argument
         at sun.nio.ch.Net.setIntOption0(Native Method)
         at sun.nio.ch.Net.setIntOption(Net.java:152)
         at sun.nio.ch.SocketChannelImpl$1.setInt 
(SocketChannelImpl.java:372)
         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
         at sun.nio.ch.SocketOptsImpl$IP.typeOfService 
(SocketOptsImpl.java:249)
         at sun.nio.ch.OptionAdaptor.setTrafficClass 
(OptionAdaptor.java:158)
         at sun.nio.ch.SocketAdaptor.setTrafficClass 
(SocketAdaptor.java:330)
         at  
org.apache.catalina.tribes.transport.nio.NioReceiver.listen 
(NioReceiver.java:273)
         at org.apache.catalina.tribes.transport.nio.NioReceiver.run 
(NioReceiver.java:353)
         at java.lang.Thread.run(Thread.java:613)
Jul 10, 2008 1:06:42 PM  
org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
INFO: Done sleeping, membership established, start level:8
Jul 10, 2008 1:06:42 PM  
org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
INFO: Initializing AbstractReplicatedMap with context name:
Jul 10, 2008 1:06:42 PM org.apache.catalina.tribes.io.BufferPool  
getBufferPool
INFO: Created a buffer pool with max size:104857600 bytes of  
type:org.apache.catalina.tribes.io.BufferPool15Impl
Jul 10, 2008 1:06:42 PM  
org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
WARNING: Member send is failing for:tcp://{10, 84, 2, 42}:4000 ;  
Setting to suspect and retrying.
Jul 10, 2008 1:06:42 PM  
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector  
memberDisappeared
INFO: Received memberDisappeared 
[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84, 2,  
42}:4000,{10, 84, 2, 42},4000, alive=197662,id={-25 34 -32 116 42  
-104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command={},  
domain={}, ]] message. Will verify.
Jul 10, 2008 1:06:42 PM  
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector  
memberDisappeared
INFO: Verification complete. Member still alive 
[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84, 2,  
42}:4000,{10, 84, 2, 42},4000, alive=197662,id={-25 34 -32 116 42  
-104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command={},  
domain={}, ]]
Jul 10, 2008 1:06:42 PM  
org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
WARNING: Unable to send map start message.
Jul 10, 2008 1:06:42 PM  
org.apache.catalina.ha.context.ReplicatedContext start
SEVERE: Unable to start ReplicatedContext
java.lang.RuntimeException: Unable to start replicated map.
         at  
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.init 
(AbstractReplicatedMap.java:230)
         at  
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.<init> 
(AbstractReplicatedMap.java:171)
         at org.apache.catalina.tribes.tipis.ReplicatedMap.<init> 
(ReplicatedMap.java:90)
         at org.apache.catalina.ha.context.ReplicatedContext.start 
(ReplicatedContext.java:64)
         at org.apache.catalina.core.ContainerBase.start 
(ContainerBase.java:1045)
         at org.apache.catalina.core.StandardHost.start 
(StandardHost.java:719)
         at org.apache.catalina.core.ContainerBase.start 
(ContainerBase.java:1045)
         at org.apache.catalina.core.StandardEngine.start 
(StandardEngine.java:443)
         at org.apache.catalina.core.StandardService.start 
(StandardService.java:516)
         at org.apache.catalina.core.StandardServer.start 
(StandardServer.java:710)
         at org.apache.catalina.startup.Catalina.start(Catalina.java: 
578)
         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
         at sun.reflect.NativeMethodAccessorImpl.invoke 
(NativeMethodAccessorImpl.java:39)
         at sun.reflect.DelegatingMethodAccessorImpl.invoke 
(DelegatingMethodAccessorImpl.java:25)
         at java.lang.reflect.Method.invoke(Method.java:585)
         at org.apache.catalina.startup.Bootstrap.start 
(Bootstrap.java:288)
         at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java: 
413)
Caused by: org.apache.catalina.tribes.ChannelException: Send failed,  
attempt:2 max:1; Faulty members:tcp://{10, 84, 2, 42}:4000;
         at  
org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop 
(ParallelNioSender.java:172)
         at  
org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage 
(ParallelNioSender.java:78)
         at  
org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessag 
e(PooledParallelSender.java:53)
         at  
org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage( 
ReplicationTransmitter.java:80)
         at  
org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage 
(ChannelCoordinator.java:78)
         at  
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
(ChannelInterceptorBase.java:75)
         at  
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMes 
sage(TcpFailureDetector.java:87)
         at  
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
(ChannelInterceptorBase.java:75)
         at  
org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor 
.sendMessage(MessageDispatchInterceptor.java:73)
         at  
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
(ChannelInterceptorBase.java:75)
         at org.apache.catalina.tribes.group.GroupChannel.send 
(GroupChannel.java:216)
         at org.apache.catalina.tribes.group.GroupChannel.send 
(GroupChannel.java:175)
         at org.apache.catalina.tribes.group.RpcChannel.send 
(RpcChannel.java:89)
         at  
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.broadcast 
(AbstractReplicatedMap.java:300)
         at  
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.init 
(AbstractReplicatedMap.java:223)
         ... 16 more
Caused by: java.net.SocketException: Invalid argument
         at sun.nio.ch.Net.setIntOption0(Native Method)
         at sun.nio.ch.Net.setIntOption(Net.java:152)
         at sun.nio.ch.SocketChannelImpl$1.setInt 
(SocketChannelImpl.java:372)
         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
         at sun.nio.ch.SocketOptsImpl$IP.typeOfService 
(SocketOptsImpl.java:249)
         at sun.nio.ch.OptionAdaptor.setTrafficClass 
(OptionAdaptor.java:158)
         at sun.nio.ch.SocketAdaptor.setTrafficClass 
(SocketAdaptor.java:330)
         at  
org.apache.catalina.tribes.transport.nio.NioSender.completeConnect 
(NioSender.java:147)
         at org.apache.catalina.tribes.transport.nio.NioSender.process 
(NioSender.java:89)
         at  
org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop 
(ParallelNioSender.java:130)
         ... 30 more
Jul 10, 2008 1:06:42 PM org.apache.catalina.startup.Catalina start
SEVERE: Catalina.start:
LifecycleException:  Failed to start ReplicatedContext:   
java.lang.RuntimeException: Unable to start replicated map.
         at org.apache.catalina.ha.context.ReplicatedContext.start 
(ReplicatedContext.java:73)
         at org.apache.catalina.core.ContainerBase.start 
(ContainerBase.java:1045)
         at org.apache.catalina.core.StandardHost.start 
(StandardHost.java:719)
         at org.apache.catalina.core.ContainerBase.start 
(ContainerBase.java:1045)
         at org.apache.catalina.core.StandardEngine.start 
(StandardEngine.java:443)
         at org.apache.catalina.core.StandardService.start 
(StandardService.java:516)
         at org.apache.catalina.core.StandardServer.start 
(StandardServer.java:710)
         at org.apache.catalina.startup.Catalina.start(Catalina.java: 
578)
         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
         at sun.reflect.NativeMethodAccessorImpl.invoke 
(NativeMethodAccessorImpl.java:39)
         at sun.reflect.DelegatingMethodAccessorImpl.invoke 
(DelegatingMethodAccessorImpl.java:25)
         at java.lang.reflect.Method.invoke(Method.java:585)
         at org.apache.catalina.startup.Bootstrap.start 
(Bootstrap.java:288)
         at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java: 
413)
Jul 10, 2008 1:06:42 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 2329 ms
Jul 10, 2008 1:06:47 PM  
org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
WARNING: Not retrying send for:tcp://{10, 84, 2, 42}:4000; Sender is  
disconnected.
Jul 10, 2008 1:06:47 PM  
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector  
memberDisappeared
INFO: Received memberDisappeared 
[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84, 2,  
42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32 116 42  
-104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command={},  
domain={}, ]] message. Will verify.
Jul 10, 2008 1:06:47 PM  
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector  
memberDisappeared
INFO: Verification complete. Member still alive 
[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84, 2,  
42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32 116 42  
-104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command={},  
domain={}, ]]
Jul 10, 2008 1:06:47 PM  
org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
SEVERE: Unable to send AbstractReplicatedMap.ping message
org.apache.catalina.tribes.ChannelException: Send failed, and sender  
is disconnected. Not retrying.; Faulty members:tcp://{10, 84, 2, 42}: 
4000;
         at  
org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop 
(ParallelNioSender.java:156)
         at  
org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage 
(ParallelNioSender.java:78)
         at  
org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessag 
e(PooledParallelSender.java:53)
         at  
org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage( 
ReplicationTransmitter.java:80)
         at  
org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage 
(ChannelCoordinator.java:78)
         at  
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
(ChannelInterceptorBase.java:75)
         at  
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMes 
sage(TcpFailureDetector.java:87)
         at  
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
(ChannelInterceptorBase.java:75)
         at  
org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor 
.sendMessage(MessageDispatchInterceptor.java:73)
         at  
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
(ChannelInterceptorBase.java:75)
         at org.apache.catalina.tribes.group.GroupChannel.send 
(GroupChannel.java:216)
         at org.apache.catalina.tribes.group.GroupChannel.send 
(GroupChannel.java:175)
         at org.apache.catalina.tribes.group.RpcChannel.send 
(RpcChannel.java:89)
         at  
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping 
(AbstractReplicatedMap.java:253)
         at  
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat 
(AbstractReplicatedMap.java:793)
         at org.apache.catalina.tribes.group.GroupChannel.heartbeat 
(GroupChannel.java:153)
         at org.apache.catalina.tribes.group.GroupChannel 
$HeartbeatThread.run(GroupChannel.java:661)
Caused by: java.net.SocketException: Invalid argument
         at sun.nio.ch.Net.setIntOption0(Native Method)
         at sun.nio.ch.Net.setIntOption(Net.java:152)
         at sun.nio.ch.SocketChannelImpl$1.setInt 
(SocketChannelImpl.java:372)
         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
         at sun.nio.ch.SocketOptsImpl$IP.typeOfService 
(SocketOptsImpl.java:249)
         at sun.nio.ch.OptionAdaptor.setTrafficClass 
(OptionAdaptor.java:158)
         at sun.nio.ch.SocketAdaptor.setTrafficClass 
(SocketAdaptor.java:330)
         at  
org.apache.catalina.tribes.transport.nio.NioSender.completeConnect 
(NioSender.java:147)
         at org.apache.catalina.tribes.transport.nio.NioSender.process 
(NioSender.java:89)
         at  
org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop 
(ParallelNioSender.java:130)
         ... 16 more
Jul 10, 2008 1:06:47 PM  
org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
WARNING: Not retrying send for:tcp://{10, 84, 2, 42}:4000; Sender is  
disconnected.
Jul 10, 2008 1:06:47 PM  
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector  
memberDisappeared
INFO: Received memberDisappeared 
[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84, 2,  
42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32 116 42  
-104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command={},  
domain={}, ]] message. Will verify.
Jul 10, 2008 1:06:47 PM  
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector  
memberDisappeared
INFO: Verification complete. Member still alive 
[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84, 2,  
42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32 116 42  
-104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command={},  
domain={}, ]]
Jul 10, 2008 1:06:47 PM  
org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
SEVERE: Unable to send AbstractReplicatedMap.ping message
org.apache.catalina.tribes.ChannelException: Send failed, and sender  
is disconnected. Not retrying.; Faulty members:tcp://{10, 84, 2, 42}: 
4000;
         at  
org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop 
(ParallelNioSender.java:156)
         at  
org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage 
(ParallelNioSender.java:78)
         at  
org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessag 
e(PooledParallelSender.java:53)
         at  
org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage( 
ReplicationTransmitter.java:80)
         at  
org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage 
(ChannelCoordinator.java:78)
         at  
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
(ChannelInterceptorBase.java:75)
         at  
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMes 
sage(TcpFailureDetector.java:87)
         at  
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
(ChannelInterceptorBase.java:75)
         at  
org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor 
.sendMessage(MessageDispatchInterceptor.java:73)
         at  
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
(ChannelInterceptorBase.java:75)
         at org.apache.catalina.tribes.group.GroupChannel.send 
(GroupChannel.java:216)
         at org.apache.catalina.tribes.group.GroupChannel.send 
(GroupChannel.java:175)
         at org.apache.catalina.tribes.group.RpcChannel.send 
(RpcChannel.java:89)
         at  
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping 
(AbstractReplicatedMap.java:253)
         at  
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat 
(AbstractReplicatedMap.java:793)
         at org.apache.catalina.tribes.group.GroupChannel.heartbeat 
(GroupChannel.java:158)
         at org.apache.catalina.tribes.group.GroupChannel 
$HeartbeatThread.run(GroupChannel.java:661)
Caused by: java.net.SocketException: Invalid argument
         at sun.nio.ch.Net.setIntOption0(Native Method)
         at sun.nio.ch.Net.setIntOption(Net.java:152)
         at sun.nio.ch.SocketChannelImpl$1.setInt 
(SocketChannelImpl.java:372)
         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
         at sun.nio.ch.SocketOptsImpl$IP.typeOfService 
(SocketOptsImpl.java:249)
         at sun.nio.ch.OptionAdaptor.setTrafficClass 
(OptionAdaptor.java:158)
         at sun.nio.ch.SocketAdaptor.setTrafficClass 
(SocketAdaptor.java:330)
         at  
org.apache.catalina.tribes.transport.nio.NioSender.completeConnect 
(NioSender.java:147)
         at org.apache.catalina.tribes.transport.nio.NioSender.process 
(NioSender.java:89)
         at  
org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop 
(ParallelNioSender.java:130)
         ... 16 more


and so on.  Top shows that both Java process is Up to 86%, the  
machine is never idle and all memory is consumed.  The machine is a  
PowerPc G6 from Apple with OSX 10.4.11.
Tomcat version is: Apache Tomcat/6.0.16
JVM version is: 1.5.0_13-b05-241
I suspect the heartbeat pings have a problem, I just do not know what  
to do about them.

Thanks ahead,

János



---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: Version 6 Clustering problem

Posted by Filip Hanik - Dev Lists <de...@hanik.com>.
János wrote:
> Filip,
>
> You are right.  I called the session example wrongly via the 
> "bml0042.yalepath.org" host, but that was not configured.  As soon as 
> I called it via the "localhost" host - which was
glad you got it working
> configured - it worked fine.  Here is the log from node3:
>
> bml0042:/usr/local/tomcat janos$ cat node3/logs/localhost.2008-07-14.log
> Jul 14, 2008 2:22:49 PM org.apache.catalina.core.ApplicationContext log
> INFO: SessionListener: 
> sessionCreated('B3BA285CF8FABC0BD374C237C22B2B4E.node1')
> Jul 14, 2008 2:23:19 PM org.apache.catalina.core.ApplicationContext log
> INFO: SessionListener: 
> attributeAdded('B3BA285CF8FABC0BD374C237C22B2B4E.node1', 'els?', 'egy')
> Jul 14, 2008 2:23:49 PM org.apache.catalina.core.ApplicationContext log
> INFO: SessionListener: 
> attributeAdded('B3BA285CF8FABC0BD374C237C22B2B4E.node1', 'm?sodik', 
> 'Kett?')
>
> Now I have a working tomcat cluster, I JUST have to make it to work 
> with apache via jk_mod :)
>
> How a normal roadrunner, like me, should know about the 
> -Djava.net.preferIPv4Stack=true system property and its dramatical 
> impact ?
the option doesn't really impact your system

Filip
>
> Thanks a lot,
>
> János
>
> On Jul 14, 2008, at 10:42 AM, Filip Hanik - Dev Lists wrote:
>
>> write your own webapp, to make sure you actually replicate 
>> serializable data
>>
>> Filip
>>
>> János wrote:
>>> Hi Filip,
>>>
>>> I added it in the catalina.sh file on all three nodes and I am not 
>>> seeing the error.  I also disabled IP6 in Network Preferences.
>>>
>>> I added the distrubuted="true" to the context.xml file on all three 
>>> nodes and added the <distributable/> tag to the 
>>> webapps/examples/WEB-INF/web.xml on all 3 nodes and tried the 
>>> SessionExample servlet
>>>
>>> http://bml0042.yalepath.org:8080/examples/servlets/servlet/SessionExample 
>>>
>>>
>>> to see if I have a working cluster.  I am following the book at
>>>
>>> http://proquest.safaribooksonline.com:80/9780596101060/tomcat_6_clustering_implementation 
>>>
>>>
>>> and it says that I should see messages similar to this one in the 
>>> nodes' logs"
>>>
>>> Sep 27, 2008 5:39:38 PM org.apache.catalina.core.ApplicationContext log
>>> INFO: SessionListener: contextInitialized(  )
>>> Sep 27, 2008 5:40:07 PM org.apache.catalina.core.ApplicationContext log
>>> INFO: SessionListener: 
>>> sessionCreated('F4B2D7191C1F335FFAAC93DA461CA95F.tc1')
>>> Sep 27, 2008 5:40:19 PM org.apache.catalina.core.ApplicationContext log
>>> INFO: SessionListener: 
>>> attributeAdded('F4B2D7191C1F335FFAAC93DA461CA95F.tc1', 'test',
>>>
>>>
>>> But I am not seeing them.  This is the resent log from node1:
>>>
>>> Jul 11, 2008 3:28:01 PM 
>>> org.apache.catalina.core.AprLifecycleListener init
>>> INFO: The APR based Apache Tomcat Native library which allows 
>>> optimal performance in production environments was not found on the 
>>> java.library.path: 
>>> .:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java 
>>>
>>> Jul 11, 2008 3:28:01 PM org.apache.coyote.http11.Http11Protocol init
>>> INFO: Initializing Coyote HTTP/1.1 on http-8080
>>> Jul 11, 2008 3:28:01 PM org.apache.catalina.startup.Catalina load
>>> INFO: Initialization processed in 1098 ms
>>> Jul 11, 2008 3:28:01 PM org.apache.catalina.core.StandardService start
>>> INFO: Starting service Catalina
>>> Jul 11, 2008 3:28:01 PM org.apache.catalina.core.StandardEngine start
>>> INFO: Starting Servlet Engine: Apache Tomcat/6.0.16
>>> Jul 11, 2008 3:28:01 PM org.apache.catalina.ha.tcp.SimpleTcpCluster 
>>> start
>>> INFO: Cluster is about to start
>>> Jul 11, 2008 3:28:01 PM 
>>> org.apache.catalina.tribes.transport.ReceiverBase bind
>>> INFO: Receiver Server Socket bound to:/10.84.2.42:4000
>>> Jul 11, 2008 3:28:01 PM 
>>> org.apache.catalina.tribes.membership.McastServiceImpl setupSocket
>>> INFO: Setting cluster mcast soTimeout to 500
>>> Jul 11, 2008 3:28:01 PM 
>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>> INFO: Sleeping for 1000 milliseconds to establish cluster 
>>> membership, start level:4
>>> Jul 11, 2008 3:28:02 PM 
>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>> INFO: Done sleeping, membership established, start level:4
>>> Jul 11, 2008 3:28:02 PM 
>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>> INFO: Sleeping for 1000 milliseconds to establish cluster 
>>> membership, start level:8
>>> Jul 11, 2008 3:28:03 PM 
>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>> INFO: Done sleeping, membership established, start level:8
>>> Jul 11, 2008 3:28:03 PM 
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
>>> INFO: Initializing AbstractReplicatedMap with context name:
>>> Jul 11, 2008 3:28:04 PM org.apache.catalina.core.StandardContext 
>>> addApplicationListener
>>> INFO: The listener "listeners.ContextListener" is already configured 
>>> for this context. The duplicate definition has been ignored.
>>> Jul 11, 2008 3:28:04 PM org.apache.catalina.core.StandardContext 
>>> addApplicationListener
>>> INFO: The listener "listeners.SessionListener" is already configured 
>>> for this context. The duplicate definition has been ignored.
>>> Jul 11, 2008 3:28:04 PM org.apache.catalina.ha.session.DeltaManager 
>>> start
>>> INFO: Register manager /examples to cluster element Engine with name 
>>> Catalina
>>> Jul 11, 2008 3:28:04 PM org.apache.catalina.ha.session.DeltaManager 
>>> start
>>> INFO: Starting clustering manager at /examples
>>> Jul 11, 2008 3:28:04 PM org.apache.catalina.ha.session.DeltaManager 
>>> getAllClusterSessions
>>> INFO: Manager [localhost#/examples]: skipping state transfer. No 
>>> members active in cluster group.
>>> Jul 11, 2008 3:28:04 PM 
>>> org.apache.catalina.ha.session.JvmRouteBinderValve start
>>> INFO: JvmRouteBinderValve started
>>> Jul 11, 2008 3:28:04 PM org.apache.coyote.http11.Http11Protocol start
>>> INFO: Starting Coyote HTTP/1.1 on http-8080
>>> Jul 11, 2008 3:28:05 PM org.apache.jk.common.ChannelSocket init
>>> INFO: JK: ajp13 listening on /0.0.0.0:8009
>>> Jul 11, 2008 3:28:05 PM org.apache.jk.server.JkMain start
>>> INFO: Jk running ID=0 time=0/45  config=null
>>> Jul 11, 2008 3:28:05 PM org.apache.catalina.startup.Catalina start
>>> INFO: Server startup in 3601 ms
>>> Jul 11, 2008 3:28:37 PM org.apache.catalina.tribes.io.BufferPool 
>>> getBufferPool
>>> INFO: Created a buffer pool with max size:104857600 bytes of 
>>> type:org.apache.catalina.tribes.io.BufferPool15Impl
>>> Jul 11, 2008 3:28:37 PM org.apache.catalina.ha.tcp.SimpleTcpCluster 
>>> memberAdded
>>> INFO: Replication member 
>>> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 
>>> 84, 2, 42}:4001,{10, 84, 2, 42},4001, alive=1020,id={60 103 -111 -34 
>>> 98 62 71 -116 -104 76 -100 115 -77 15 -52 -92 }, payload={}, 
>>> command={}, domain={}, ]
>>> Jul 11, 2008 3:29:08 PM org.apache.catalina.ha.tcp.SimpleTcpCluster 
>>> memberAdded
>>> INFO: Replication member 
>>> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 
>>> 84, 2, 42}:4002,{10, 84, 2, 42},4002, alive=1019,id={27 24 -73 72 48 
>>> -6 70 32 -90 57 51 106 -94 -54 126 -52 }, payload={}, command={}, 
>>> domain={}, ]
>>>
>>>
>>>
>>> Similarly on the other two nodes.  What am I missing ??  Should I 
>>> add the <Context distributed=True" /> to the server.xml files ?
>>>
>>> Thanks ahead,
>>>
>>> János
>>> On Jul 10, 2008, at 6:03 PM, Filip Hanik - Dev Lists wrote:
>>>
>>>> try setting the
>>>> |-Djava.net.preferIPv4Stack=true
>>>>
>>>> option for your tomcat launch
>>>>
>>>> Filip
>>>> |
>>>> János wrote:
>>>>> Hi,
>>>>>
>>>>> I am trying to establish a 3 member cluster on the same machine.  
>>>>> Here is my server.xml copied from a sample I found on this list.
>>>>>
>>>>> <?xml version='1.0' encoding='utf-8'?>
>>>>> <Server port="8005" shutdown="SHUTDOWN">
>>>>>   <Listener 
>>>>> className="org.apache.catalina.core.AprLifecycleListener" 
>>>>> SSLEngine="on" />
>>>>>   <Listener className="org.apache.catalina.core.JasperListener" />
>>>>>   <Listener 
>>>>> className="org.apache.catalina.mbeans.ServerLifecycleListener" />
>>>>>   <Listener 
>>>>> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" 
>>>>> />
>>>>>
>>>>>   <GlobalNamingResources>
>>>>>     <Resource name="UserDatabase" auth="Container"
>>>>>               type="org.apache.catalina.UserDatabase"
>>>>>               description="User database that can be updated and 
>>>>> saved"
>>>>>               
>>>>> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
>>>>>               pathname="conf/tomcat-users.xml" />
>>>>>   </GlobalNamingResources>
>>>>>
>>>>>   <Service name="Catalina">
>>>>>
>>>>>     <Connector port="8080" protocol="HTTP/1.1"
>>>>>                connectionTimeout="20000"
>>>>>                redirectPort="8443" />
>>>>>     <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
>>>>>
>>>>>     <Engine name="Catalina" defaultHost="localhost" jvmRoute="node1">
>>>>>
>>>>>       <Cluster 
>>>>> className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
>>>>>       <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
>>>>>              resourceName="UserDatabase"/>
>>>>>
>>>>>       <Host name="localhost"  appBase="webapps"
>>>>>             unpackWARs="true" autoDeploy="true"
>>>>>             xmlValidation="false" xmlNamespaceAware="false">
>>>>>         <Context path="" docBase="ROOT" 
>>>>> className="org.apache.catalina.ha.context.ReplicatedContext"/>
>>>>>       </Host>
>>>>>     </Engine>
>>>>>   </Service>
>>>>> </Server>
>>>>>
>>>>> The other two server.xml-s are the same, except the shutdown ports 
>>>>> are  8105, 8205, http ports are 8180, 8280 and https ports are 
>>>>> 8543, 8643.
>>>>> As soon as I am starting the second instance I receive the 
>>>>> following into the logs.
>>>>>
>>>>> In node1 logs:
>>>>>
>>>>> bml0042:/usr/local/tomcat/node1/logs janos$ cat 
>>>>> catalina.2008-07-10.log
>>>>> Jul 10, 2008 1:03:24 PM 
>>>>> org.apache.catalina.core.AprLifecycleListener init
>>>>> INFO: The APR based Apache Tomcat Native library which allows 
>>>>> optimal performance in production environments was not found on 
>>>>> the java.library.path: 
>>>>> .:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java 
>>>>>
>>>>> Jul 10, 2008 1:03:24 PM org.apache.coyote.http11.Http11Protocol init
>>>>> INFO: Initializing Coyote HTTP/1.1 on http-8080
>>>>> Jul 10, 2008 1:03:24 PM org.apache.catalina.startup.Catalina load
>>>>> INFO: Initialization processed in 1187 ms
>>>>> Jul 10, 2008 1:03:24 PM org.apache.catalina.core.StandardService 
>>>>> start
>>>>> INFO: Starting service Catalina
>>>>> Jul 10, 2008 1:03:24 PM org.apache.catalina.core.StandardEngine start
>>>>> INFO: Starting Servlet Engine: Apache Tomcat/6.0.16
>>>>> Jul 10, 2008 1:03:24 PM 
>>>>> org.apache.catalina.ha.tcp.SimpleTcpCluster start
>>>>> INFO: Cluster is about to start
>>>>> Jul 10, 2008 1:03:24 PM 
>>>>> org.apache.catalina.tribes.transport.ReceiverBase bind
>>>>> INFO: Receiver Server Socket bound to:/10.84.2.42:4000
>>>>> Jul 10, 2008 1:03:24 PM 
>>>>> org.apache.catalina.tribes.membership.McastServiceImpl setupSocket
>>>>> INFO: Setting cluster mcast soTimeout to 500
>>>>> Jul 10, 2008 1:03:24 PM 
>>>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>>>> INFO: Sleeping for 1000 milliseconds to establish cluster 
>>>>> membership, start level:4
>>>>> Jul 10, 2008 1:03:25 PM 
>>>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>>>> INFO: Done sleeping, membership established, start level:4
>>>>> Jul 10, 2008 1:03:25 PM 
>>>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>>>> INFO: Sleeping for 1000 milliseconds to establish cluster 
>>>>> membership, start level:8
>>>>> Jul 10, 2008 1:03:26 PM 
>>>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>>>> INFO: Done sleeping, membership established, start level:8
>>>>> Jul 10, 2008 1:03:26 PM 
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
>>>>> INFO: Initializing AbstractReplicatedMap with context name:
>>>>> Jul 10, 2008 1:03:27 PM org.apache.catalina.core.StandardContext 
>>>>> addApplicationListener
>>>>> INFO: The listener "listeners.ContextListener" is already 
>>>>> configured for this context. The duplicate definition has been 
>>>>> ignored.
>>>>> Jul 10, 2008 1:03:27 PM org.apache.catalina.core.StandardContext 
>>>>> addApplicationListener
>>>>> INFO: The listener "listeners.SessionListener" is already 
>>>>> configured for this context. The duplicate definition has been 
>>>>> ignored.
>>>>> Jul 10, 2008 1:03:27 PM 
>>>>> org.apache.catalina.ha.session.DeltaManager start
>>>>> INFO: Register manager /examples to cluster element Engine with 
>>>>> name Catalina
>>>>> Jul 10, 2008 1:03:27 PM 
>>>>> org.apache.catalina.ha.session.DeltaManager start
>>>>> INFO: Starting clustering manager at /examples
>>>>> Jul 10, 2008 1:03:27 PM 
>>>>> org.apache.catalina.ha.session.DeltaManager getAllClusterSessions
>>>>> INFO: Manager [localhost#/examples]: skipping state transfer. No 
>>>>> members active in cluster group.
>>>>> Jul 10, 2008 1:03:28 PM 
>>>>> org.apache.catalina.ha.session.JvmRouteBinderValve start
>>>>> INFO: JvmRouteBinderValve started
>>>>> Jul 10, 2008 1:03:28 PM org.apache.coyote.http11.Http11Protocol start
>>>>> INFO: Starting Coyote HTTP/1.1 on http-8080
>>>>> Jul 10, 2008 1:03:28 PM org.apache.jk.common.ChannelSocket init
>>>>> INFO: JK: ajp13 listening on /0.0.0.0:8009
>>>>> Jul 10, 2008 1:03:28 PM org.apache.jk.server.JkMain start
>>>>> INFO: Jk running ID=0 time=0/45  config=null
>>>>> Jul 10, 2008 1:03:28 PM org.apache.catalina.startup.Catalina start
>>>>> INFO: Server startup in 3616 ms
>>>>> Jul 10, 2008 1:06:41 PM 
>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver listen
>>>>> SEVERE: Unable to process request in NioReceiver
>>>>> java.net.SocketException: Invalid argument
>>>>>         at sun.nio.ch.Net.setIntOption0(Native Method)
>>>>>         at sun.nio.ch.Net.setIntOption(Net.java:152)
>>>>>         at 
>>>>> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
>>>>>         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>>>>         at 
>>>>> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
>>>>>         at 
>>>>> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
>>>>>         at 
>>>>> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
>>>>>         at 
>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:273) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioReceiver.java:353) 
>>>>>
>>>>>         at java.lang.Thread.run(Thread.java:613)
>>>>> Jul 10, 2008 1:06:41 PM 
>>>>> org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
>>>>> INFO: Replication member 
>>>>> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 
>>>>> 84, 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46 
>>>>> 127 -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={}, 
>>>>> command={}, domain={}, ]
>>>>> Jul 10, 2008 1:06:41 PM org.apache.catalina.tribes.io.BufferPool 
>>>>> getBufferPool
>>>>> INFO: Created a buffer pool with max size:104857600 bytes of 
>>>>> type:org.apache.catalina.tribes.io.BufferPool15Impl
>>>>> Jul 10, 2008 1:06:41 PM 
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>>>>> WARNING: Member send is failing for:tcp://{10, 84, 2, 42}:4001 ; 
>>>>> Setting to suspect and retrying.
>>>>> Jul 10, 2008 1:06:41 PM 
>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector 
>>>>> memberDisappeared
>>>>> INFO: Received 
>>>>> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 
>>>>> 84, 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46 
>>>>> 127 -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={}, 
>>>>> command={}, domain={}, ]] message. Will verify.
>>>>> Jul 10, 2008 1:06:41 PM 
>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector 
>>>>> memberDisappeared
>>>>> INFO: Verification complete. Member still 
>>>>> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 
>>>>> 84, 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46 
>>>>> 127 -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={}, 
>>>>> command={}, domain={}, ]]
>>>>> Jul 10, 2008 1:06:41 PM 
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
>>>>> SEVERE: Unable to send AbstractReplicatedMap.ping message
>>>>> org.apache.catalina.tribes.ChannelException: Send failed, 
>>>>> attempt:2 max:1; Faulty members:tcp://{10, 84, 2, 42}:4001;
>>>>>         at 
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:172) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:78) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:73) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:89)
>>>>>         at 
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:253) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:793) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:153) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:661) 
>>>>>
>>>>> Caused by: java.net.SocketException: Invalid argument
>>>>>         at sun.nio.ch.Net.setIntOption0(Native Method)
>>>>>         at sun.nio.ch.Net.setIntOption(Net.java:152)
>>>>>         at 
>>>>> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
>>>>>         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>>>>         at 
>>>>> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
>>>>>         at 
>>>>> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
>>>>>         at 
>>>>> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
>>>>>         at 
>>>>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect(NioSender.java:147) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.transport.nio.NioSender.process(NioSender.java:89) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:130) 
>>>>>
>>>>>         ... 16 more
>>>>> Jul 10, 2008 1:06:41 PM 
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>>>>> WARNING: Not retrying send for:tcp://{10, 84, 2, 42}:4001; Sender 
>>>>> is disconnected.
>>>>> Jul 10, 2008 1:06:41 PM 
>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector 
>>>>> memberDisappeared
>>>>> INFO: Received 
>>>>> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 
>>>>> 84, 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46 
>>>>> 127 -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={}, 
>>>>> command={}, domain={}, ]] message. Will verify.
>>>>> Jul 10, 2008 1:06:42 PM 
>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector 
>>>>> memberDisappeared
>>>>> INFO: Verification complete. Member still 
>>>>> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 
>>>>> 84, 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46 
>>>>> 127 -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={}, 
>>>>> command={}, domain={}, ]]
>>>>> Jul 10, 2008 1:06:42 PM 
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
>>>>> SEVERE: Unable to send AbstractReplicatedMap.ping message
>>>>> org.apache.catalina.tribes.ChannelException: Send failed, and 
>>>>> sender is disconnected. Not retrying.; Faulty members:tcp://{10, 
>>>>> 84, 2, 42}:4001;
>>>>>         at 
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:156) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:78) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:73) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:89)
>>>>>         at 
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:253) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:793) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:158) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:661) 
>>>>>
>>>>> Caused by: java.net.SocketException: Invalid argument
>>>>>         at sun.nio.ch.Net.setIntOption0(Native Method)
>>>>>         at sun.nio.ch.Net.setIntOption(Net.java:152)
>>>>>         at 
>>>>> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
>>>>>         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>>>>         at 
>>>>> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
>>>>>         at 
>>>>> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
>>>>>         at 
>>>>> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
>>>>>         at 
>>>>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect(NioSender.java:147) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.transport.nio.NioSender.process(NioSender.java:89) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:130) 
>>>>>
>>>>>         ... 16 more
>>>>>
>>>>>
>>>>> and so on forever.
>>>>>
>>>>> In node2 logs I see the following:
>>>>>
>>>>> bml0042:/usr/local/tomcat/node2/logs janos$ cat 
>>>>> catalina.2008-07-10.log
>>>>> Jul 10, 2008 1:06:40 PM 
>>>>> org.apache.catalina.core.AprLifecycleListener init
>>>>> INFO: The APR based Apache Tomcat Native library which allows 
>>>>> optimal performance in production environments was not found on 
>>>>> the java.library.path: 
>>>>> .:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java 
>>>>>
>>>>> Jul 10, 2008 1:06:40 PM org.apache.coyote.http11.Http11Protocol init
>>>>> INFO: Initializing Coyote HTTP/1.1 on http-8180
>>>>> Jul 10, 2008 1:06:40 PM org.apache.catalina.startup.Catalina load
>>>>> INFO: Initialization processed in 1148 ms
>>>>> Jul 10, 2008 1:06:40 PM org.apache.catalina.core.StandardService 
>>>>> start
>>>>> INFO: Starting service Catalina
>>>>> Jul 10, 2008 1:06:40 PM org.apache.catalina.core.StandardEngine start
>>>>> INFO: Starting Servlet Engine: Apache Tomcat/6.0.16
>>>>> Jul 10, 2008 1:06:40 PM 
>>>>> org.apache.catalina.ha.tcp.SimpleTcpCluster start
>>>>> INFO: Cluster is about to start
>>>>> Jul 10, 2008 1:06:40 PM 
>>>>> org.apache.catalina.tribes.transport.ReceiverBase bind
>>>>> INFO: Receiver Server Socket bound to:/10.84.2.42:4001
>>>>> Jul 10, 2008 1:06:40 PM 
>>>>> org.apache.catalina.tribes.membership.McastServiceImpl setupSocket
>>>>> INFO: Setting cluster mcast soTimeout to 500
>>>>> Jul 10, 2008 1:06:40 PM 
>>>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>>>> INFO: Sleeping for 1000 milliseconds to establish cluster 
>>>>> membership, start level:4
>>>>> Jul 10, 2008 1:06:41 PM 
>>>>> org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
>>>>> INFO: Replication member 
>>>>> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 
>>>>> 84, 2, 42}:4000,{10, 84, 2, 42},4000, alive=196161,id={-25 34 -32 
>>>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, 
>>>>> command={}, domain={}, ]
>>>>> Jul 10, 2008 1:06:41 PM 
>>>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>>>> INFO: Done sleeping, membership established, start level:4
>>>>> Jul 10, 2008 1:06:41 PM 
>>>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>>>> INFO: Sleeping for 1000 milliseconds to establish cluster 
>>>>> membership, start level:8
>>>>> Jul 10, 2008 1:06:41 PM 
>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver listen
>>>>> SEVERE: Unable to process request in NioReceiver
>>>>> java.net.SocketException: Invalid argument
>>>>>         at sun.nio.ch.Net.setIntOption0(Native Method)
>>>>>         at sun.nio.ch.Net.setIntOption(Net.java:152)
>>>>>         at 
>>>>> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
>>>>>         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>>>>         at 
>>>>> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
>>>>>         at 
>>>>> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
>>>>>         at 
>>>>> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
>>>>>         at 
>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:273) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioReceiver.java:353) 
>>>>>
>>>>>         at java.lang.Thread.run(Thread.java:613)
>>>>> Jul 10, 2008 1:06:42 PM 
>>>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>>>> INFO: Done sleeping, membership established, start level:8
>>>>> Jul 10, 2008 1:06:42 PM 
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
>>>>> INFO: Initializing AbstractReplicatedMap with context name:
>>>>> Jul 10, 2008 1:06:42 PM org.apache.catalina.tribes.io.BufferPool 
>>>>> getBufferPool
>>>>> INFO: Created a buffer pool with max size:104857600 bytes of 
>>>>> type:org.apache.catalina.tribes.io.BufferPool15Impl
>>>>> Jul 10, 2008 1:06:42 PM 
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>>>>> WARNING: Member send is failing for:tcp://{10, 84, 2, 42}:4000 ; 
>>>>> Setting to suspect and retrying.
>>>>> Jul 10, 2008 1:06:42 PM 
>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector 
>>>>> memberDisappeared
>>>>> INFO: Received 
>>>>> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 
>>>>> 84, 2, 42}:4000,{10, 84, 2, 42},4000, alive=197662,id={-25 34 -32 
>>>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, 
>>>>> command={}, domain={}, ]] message. Will verify.
>>>>> Jul 10, 2008 1:06:42 PM 
>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector 
>>>>> memberDisappeared
>>>>> INFO: Verification complete. Member still 
>>>>> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 
>>>>> 84, 2, 42}:4000,{10, 84, 2, 42},4000, alive=197662,id={-25 34 -32 
>>>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, 
>>>>> command={}, domain={}, ]]
>>>>> Jul 10, 2008 1:06:42 PM 
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
>>>>> WARNING: Unable to send map start message.
>>>>> Jul 10, 2008 1:06:42 PM 
>>>>> org.apache.catalina.ha.context.ReplicatedContext start
>>>>> SEVERE: Unable to start ReplicatedContext
>>>>> java.lang.RuntimeException: Unable to start replicated map.
>>>>>         at 
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.init(AbstractReplicatedMap.java:230) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.<init>(AbstractReplicatedMap.java:171) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.tipis.ReplicatedMap.<init>(ReplicatedMap.java:90) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.ha.context.ReplicatedContext.start(ReplicatedContext.java:64) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
>>>>>         at 
>>>>> org.apache.catalina.core.StandardHost.start(StandardHost.java:719)
>>>>>         at 
>>>>> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
>>>>>         at 
>>>>> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.core.StandardService.start(StandardService.java:516) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.core.StandardServer.start(StandardServer.java:710) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.startup.Catalina.start(Catalina.java:578)
>>>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
>>>>> Method)
>>>>>         at 
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
>>>>>
>>>>>         at 
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
>>>>>
>>>>>         at java.lang.reflect.Method.invoke(Method.java:585)
>>>>>         at 
>>>>> org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
>>>>>         at 
>>>>> org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
>>>>> Caused by: org.apache.catalina.tribes.ChannelException: Send 
>>>>> failed, attempt:2 max:1; Faulty members:tcp://{10, 84, 2, 42}:4000;
>>>>>         at 
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:172) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:78) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:73) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:89)
>>>>>         at 
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.broadcast(AbstractReplicatedMap.java:300) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.init(AbstractReplicatedMap.java:223) 
>>>>>
>>>>>         ... 16 more
>>>>> Caused by: java.net.SocketException: Invalid argument
>>>>>         at sun.nio.ch.Net.setIntOption0(Native Method)
>>>>>         at sun.nio.ch.Net.setIntOption(Net.java:152)
>>>>>         at 
>>>>> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
>>>>>         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>>>>         at 
>>>>> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
>>>>>         at 
>>>>> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
>>>>>         at 
>>>>> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
>>>>>         at 
>>>>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect(NioSender.java:147) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.transport.nio.NioSender.process(NioSender.java:89) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:130) 
>>>>>
>>>>>         ... 30 more
>>>>> Jul 10, 2008 1:06:42 PM org.apache.catalina.startup.Catalina start
>>>>> SEVERE: Catalina.start:
>>>>> LifecycleException:  Failed to start ReplicatedContext:  
>>>>> java.lang.RuntimeException: Unable to start replicated map.
>>>>>         at 
>>>>> org.apache.catalina.ha.context.ReplicatedContext.start(ReplicatedContext.java:73) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
>>>>>         at 
>>>>> org.apache.catalina.core.StandardHost.start(StandardHost.java:719)
>>>>>         at 
>>>>> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
>>>>>         at 
>>>>> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.core.StandardService.start(StandardService.java:516) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.core.StandardServer.start(StandardServer.java:710) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.startup.Catalina.start(Catalina.java:578)
>>>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
>>>>> Method)
>>>>>         at 
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
>>>>>
>>>>>         at 
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
>>>>>
>>>>>         at java.lang.reflect.Method.invoke(Method.java:585)
>>>>>         at 
>>>>> org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
>>>>>         at 
>>>>> org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
>>>>> Jul 10, 2008 1:06:42 PM org.apache.catalina.startup.Catalina start
>>>>> INFO: Server startup in 2329 ms
>>>>> Jul 10, 2008 1:06:47 PM 
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>>>>> WARNING: Not retrying send for:tcp://{10, 84, 2, 42}:4000; Sender 
>>>>> is disconnected.
>>>>> Jul 10, 2008 1:06:47 PM 
>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector 
>>>>> memberDisappeared
>>>>> INFO: Received 
>>>>> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 
>>>>> 84, 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32 
>>>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, 
>>>>> command={}, domain={}, ]] message. Will verify.
>>>>> Jul 10, 2008 1:06:47 PM 
>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector 
>>>>> memberDisappeared
>>>>> INFO: Verification complete. Member still 
>>>>> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 
>>>>> 84, 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32 
>>>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, 
>>>>> command={}, domain={}, ]]
>>>>> Jul 10, 2008 1:06:47 PM 
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
>>>>> SEVERE: Unable to send AbstractReplicatedMap.ping message
>>>>> org.apache.catalina.tribes.ChannelException: Send failed, and 
>>>>> sender is disconnected. Not retrying.; Faulty members:tcp://{10, 
>>>>> 84, 2, 42}:4000;
>>>>>         at 
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:156) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:78) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:73) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:89)
>>>>>         at 
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:253) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:793) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:153) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:661) 
>>>>>
>>>>> Caused by: java.net.SocketException: Invalid argument
>>>>>         at sun.nio.ch.Net.setIntOption0(Native Method)
>>>>>         at sun.nio.ch.Net.setIntOption(Net.java:152)
>>>>>         at 
>>>>> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
>>>>>         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>>>>         at 
>>>>> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
>>>>>         at 
>>>>> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
>>>>>         at 
>>>>> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
>>>>>         at 
>>>>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect(NioSender.java:147) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.transport.nio.NioSender.process(NioSender.java:89) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:130) 
>>>>>
>>>>>         ... 16 more
>>>>> Jul 10, 2008 1:06:47 PM 
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>>>>> WARNING: Not retrying send for:tcp://{10, 84, 2, 42}:4000; Sender 
>>>>> is disconnected.
>>>>> Jul 10, 2008 1:06:47 PM 
>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector 
>>>>> memberDisappeared
>>>>> INFO: Received 
>>>>> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 
>>>>> 84, 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32 
>>>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, 
>>>>> command={}, domain={}, ]] message. Will verify.
>>>>> Jul 10, 2008 1:06:47 PM 
>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector 
>>>>> memberDisappeared
>>>>> INFO: Verification complete. Member still 
>>>>> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 
>>>>> 84, 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32 
>>>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, 
>>>>> command={}, domain={}, ]]
>>>>> Jul 10, 2008 1:06:47 PM 
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
>>>>> SEVERE: Unable to send AbstractReplicatedMap.ping message
>>>>> org.apache.catalina.tribes.ChannelException: Send failed, and 
>>>>> sender is disconnected. Not retrying.; Faulty members:tcp://{10, 
>>>>> 84, 2, 42}:4000;
>>>>>         at 
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:156) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:78) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:73) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:89)
>>>>>         at 
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:253) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:793) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:158) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:661) 
>>>>>
>>>>> Caused by: java.net.SocketException: Invalid argument
>>>>>         at sun.nio.ch.Net.setIntOption0(Native Method)
>>>>>         at sun.nio.ch.Net.setIntOption(Net.java:152)
>>>>>         at 
>>>>> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
>>>>>         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>>>>         at 
>>>>> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
>>>>>         at 
>>>>> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
>>>>>         at 
>>>>> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
>>>>>         at 
>>>>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect(NioSender.java:147) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.transport.nio.NioSender.process(NioSender.java:89) 
>>>>>
>>>>>         at 
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:130) 
>>>>>
>>>>>         ... 16 more
>>>>>
>>>>>
>>>>> and so on.  Top shows that both Java process is Up to 86%, the 
>>>>> machine is never idle and all memory is consumed.  The machine is 
>>>>> a PowerPc G6 from Apple with OSX 10.4.11.
>>>>> Tomcat version is: Apache Tomcat/6.0.16
>>>>> JVM version is: 1.5.0_13-b05-241
>>>>> I suspect the heartbeat pings have a problem, I just do not know 
>>>>> what to do about them.
>>>>>
>>>>> Thanks ahead,
>>>>>
>>>>> János
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To start a new topic, e-mail: users@tomcat.apache.org
>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To start a new topic, e-mail: users@tomcat.apache.org
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To start a new topic, e-mail: users@tomcat.apache.org
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To start a new topic, e-mail: users@tomcat.apache.org
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>


---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: Version 6 Clustering problem

Posted by János <ja...@yale.edu>.
Filip,

You are right.  I called the session example wrongly via the  
"bml0042.yalepath.org" host, but that was not configured.  As soon as  
I called it via the "localhost" host - which was configured - it  
worked fine.  Here is the log from node3:

bml0042:/usr/local/tomcat janos$ cat node3/logs/localhost.2008-07-14.log
Jul 14, 2008 2:22:49 PM org.apache.catalina.core.ApplicationContext log
INFO: SessionListener: sessionCreated 
('B3BA285CF8FABC0BD374C237C22B2B4E.node1')
Jul 14, 2008 2:23:19 PM org.apache.catalina.core.ApplicationContext log
INFO: SessionListener: attributeAdded 
('B3BA285CF8FABC0BD374C237C22B2B4E.node1', 'els?', 'egy')
Jul 14, 2008 2:23:49 PM org.apache.catalina.core.ApplicationContext log
INFO: SessionListener: attributeAdded 
('B3BA285CF8FABC0BD374C237C22B2B4E.node1', 'm?sodik', 'Kett?')

Now I have a working tomcat cluster, I JUST have to make it to work  
with apache via jk_mod :)

How a normal roadrunner, like me, should know about the - 
Djava.net.preferIPv4Stack=true system property and its dramatical  
impact ?

Thanks a lot,

János

On Jul 14, 2008, at 10:42 AM, Filip Hanik - Dev Lists wrote:

> write your own webapp, to make sure you actually replicate  
> serializable data
>
> Filip
>
> János wrote:
>> Hi Filip,
>>
>> I added it in the catalina.sh file on all three nodes and I am not  
>> seeing the error.  I also disabled IP6 in Network Preferences.
>>
>> I added the distrubuted="true" to the context.xml file on all  
>> three nodes and added the <distributable/> tag to the webapps/ 
>> examples/WEB-INF/web.xml on all 3 nodes and tried the  
>> SessionExample servlet
>>
>> http://bml0042.yalepath.org:8080/examples/servlets/servlet/ 
>> SessionExample
>>
>> to see if I have a working cluster.  I am following the book at
>>
>> http://proquest.safaribooksonline.com:80/9780596101060/ 
>> tomcat_6_clustering_implementation
>>
>> and it says that I should see messages similar to this one in the  
>> nodes' logs"
>>
>> Sep 27, 2008 5:39:38 PM  
>> org.apache.catalina.core.ApplicationContext log
>> INFO: SessionListener: contextInitialized(  )
>> Sep 27, 2008 5:40:07 PM  
>> org.apache.catalina.core.ApplicationContext log
>> INFO: SessionListener: sessionCreated 
>> ('F4B2D7191C1F335FFAAC93DA461CA95F.tc1')
>> Sep 27, 2008 5:40:19 PM  
>> org.apache.catalina.core.ApplicationContext log
>> INFO: SessionListener: attributeAdded 
>> ('F4B2D7191C1F335FFAAC93DA461CA95F.tc1', 'test',
>>
>>
>> But I am not seeing them.  This is the resent log from node1:
>>
>> Jul 11, 2008 3:28:01 PM  
>> org.apache.catalina.core.AprLifecycleListener init
>> INFO: The APR based Apache Tomcat Native library which allows  
>> optimal performance in production environments was not found on  
>> the java.library.path: .:/Library/Java/Extensions:/System/Library/ 
>> Java/Extensions:/usr/lib/java
>> Jul 11, 2008 3:28:01 PM org.apache.coyote.http11.Http11Protocol init
>> INFO: Initializing Coyote HTTP/1.1 on http-8080
>> Jul 11, 2008 3:28:01 PM org.apache.catalina.startup.Catalina load
>> INFO: Initialization processed in 1098 ms
>> Jul 11, 2008 3:28:01 PM org.apache.catalina.core.StandardService  
>> start
>> INFO: Starting service Catalina
>> Jul 11, 2008 3:28:01 PM org.apache.catalina.core.StandardEngine start
>> INFO: Starting Servlet Engine: Apache Tomcat/6.0.16
>> Jul 11, 2008 3:28:01 PM  
>> org.apache.catalina.ha.tcp.SimpleTcpCluster start
>> INFO: Cluster is about to start
>> Jul 11, 2008 3:28:01 PM  
>> org.apache.catalina.tribes.transport.ReceiverBase bind
>> INFO: Receiver Server Socket bound to:/10.84.2.42:4000
>> Jul 11, 2008 3:28:01 PM  
>> org.apache.catalina.tribes.membership.McastServiceImpl setupSocket
>> INFO: Setting cluster mcast soTimeout to 500
>> Jul 11, 2008 3:28:01 PM  
>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>> INFO: Sleeping for 1000 milliseconds to establish cluster  
>> membership, start level:4
>> Jul 11, 2008 3:28:02 PM  
>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>> INFO: Done sleeping, membership established, start level:4
>> Jul 11, 2008 3:28:02 PM  
>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>> INFO: Sleeping for 1000 milliseconds to establish cluster  
>> membership, start level:8
>> Jul 11, 2008 3:28:03 PM  
>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>> INFO: Done sleeping, membership established, start level:8
>> Jul 11, 2008 3:28:03 PM  
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
>> INFO: Initializing AbstractReplicatedMap with context name:
>> Jul 11, 2008 3:28:04 PM org.apache.catalina.core.StandardContext  
>> addApplicationListener
>> INFO: The listener "listeners.ContextListener" is already  
>> configured for this context. The duplicate definition has been  
>> ignored.
>> Jul 11, 2008 3:28:04 PM org.apache.catalina.core.StandardContext  
>> addApplicationListener
>> INFO: The listener "listeners.SessionListener" is already  
>> configured for this context. The duplicate definition has been  
>> ignored.
>> Jul 11, 2008 3:28:04 PM  
>> org.apache.catalina.ha.session.DeltaManager start
>> INFO: Register manager /examples to cluster element Engine with  
>> name Catalina
>> Jul 11, 2008 3:28:04 PM  
>> org.apache.catalina.ha.session.DeltaManager start
>> INFO: Starting clustering manager at /examples
>> Jul 11, 2008 3:28:04 PM  
>> org.apache.catalina.ha.session.DeltaManager getAllClusterSessions
>> INFO: Manager [localhost#/examples]: skipping state transfer. No  
>> members active in cluster group.
>> Jul 11, 2008 3:28:04 PM  
>> org.apache.catalina.ha.session.JvmRouteBinderValve start
>> INFO: JvmRouteBinderValve started
>> Jul 11, 2008 3:28:04 PM org.apache.coyote.http11.Http11Protocol start
>> INFO: Starting Coyote HTTP/1.1 on http-8080
>> Jul 11, 2008 3:28:05 PM org.apache.jk.common.ChannelSocket init
>> INFO: JK: ajp13 listening on /0.0.0.0:8009
>> Jul 11, 2008 3:28:05 PM org.apache.jk.server.JkMain start
>> INFO: Jk running ID=0 time=0/45  config=null
>> Jul 11, 2008 3:28:05 PM org.apache.catalina.startup.Catalina start
>> INFO: Server startup in 3601 ms
>> Jul 11, 2008 3:28:37 PM org.apache.catalina.tribes.io.BufferPool  
>> getBufferPool
>> INFO: Created a buffer pool with max size:104857600 bytes of  
>> type:org.apache.catalina.tribes.io.BufferPool15Impl
>> Jul 11, 2008 3:28:37 PM  
>> org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
>> INFO: Replication member  
>> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,  
>> 84, 2, 42}:4001,{10, 84, 2, 42},4001, alive=1020,id={60 103 -111  
>> -34 98 62 71 -116 -104 76 -100 115 -77 15 -52 -92 }, payload={},  
>> command={}, domain={}, ]
>> Jul 11, 2008 3:29:08 PM  
>> org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
>> INFO: Replication member  
>> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,  
>> 84, 2, 42}:4002,{10, 84, 2, 42},4002, alive=1019,id={27 24 -73 72  
>> 48 -6 70 32 -90 57 51 106 -94 -54 126 -52 }, payload={}, command= 
>> {}, domain={}, ]
>>
>>
>>
>> Similarly on the other two nodes.  What am I missing ??  Should I  
>> add the <Context distributed=True" /> to the server.xml files ?
>>
>> Thanks ahead,
>>
>> János
>> On Jul 10, 2008, at 6:03 PM, Filip Hanik - Dev Lists wrote:
>>
>>> try setting the
>>> |-Djava.net.preferIPv4Stack=true
>>>
>>> option for your tomcat launch
>>>
>>> Filip
>>> |
>>> János wrote:
>>>> Hi,
>>>>
>>>> I am trying to establish a 3 member cluster on the same  
>>>> machine.  Here is my server.xml copied from a sample I found on  
>>>> this list.
>>>>
>>>> <?xml version='1.0' encoding='utf-8'?>
>>>> <Server port="8005" shutdown="SHUTDOWN">
>>>>   <Listener  
>>>> className="org.apache.catalina.core.AprLifecycleListener"  
>>>> SSLEngine="on" />
>>>>   <Listener className="org.apache.catalina.core.JasperListener" />
>>>>   <Listener  
>>>> className="org.apache.catalina.mbeans.ServerLifecycleListener" />
>>>>   <Listener  
>>>> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListe 
>>>> ner" />
>>>>
>>>>   <GlobalNamingResources>
>>>>     <Resource name="UserDatabase" auth="Container"
>>>>               type="org.apache.catalina.UserDatabase"
>>>>               description="User database that can be updated and  
>>>> saved"
>>>>                
>>>> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
>>>>               pathname="conf/tomcat-users.xml" />
>>>>   </GlobalNamingResources>
>>>>
>>>>   <Service name="Catalina">
>>>>
>>>>     <Connector port="8080" protocol="HTTP/1.1"
>>>>                connectionTimeout="20000"
>>>>                redirectPort="8443" />
>>>>     <Connector port="8009" protocol="AJP/1.3"  
>>>> redirectPort="8443" />
>>>>
>>>>     <Engine name="Catalina" defaultHost="localhost"  
>>>> jvmRoute="node1">
>>>>
>>>>       <Cluster  
>>>> className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
>>>>       <Realm  
>>>> className="org.apache.catalina.realm.UserDatabaseRealm"
>>>>              resourceName="UserDatabase"/>
>>>>
>>>>       <Host name="localhost"  appBase="webapps"
>>>>             unpackWARs="true" autoDeploy="true"
>>>>             xmlValidation="false" xmlNamespaceAware="false">
>>>>         <Context path="" docBase="ROOT"  
>>>> className="org.apache.catalina.ha.context.ReplicatedContext"/>
>>>>       </Host>
>>>>     </Engine>
>>>>   </Service>
>>>> </Server>
>>>>
>>>> The other two server.xml-s are the same, except the shutdown  
>>>> ports are  8105, 8205, http ports are 8180, 8280 and https ports  
>>>> are 8543, 8643.
>>>> As soon as I am starting the second instance I receive the  
>>>> following into the logs.
>>>>
>>>> In node1 logs:
>>>>
>>>> bml0042:/usr/local/tomcat/node1/logs janos$ cat catalina. 
>>>> 2008-07-10.log
>>>> Jul 10, 2008 1:03:24 PM  
>>>> org.apache.catalina.core.AprLifecycleListener init
>>>> INFO: The APR based Apache Tomcat Native library which allows  
>>>> optimal performance in production environments was not found on  
>>>> the java.library.path: .:/Library/Java/Extensions:/System/ 
>>>> Library/Java/Extensions:/usr/lib/java
>>>> Jul 10, 2008 1:03:24 PM org.apache.coyote.http11.Http11Protocol  
>>>> init
>>>> INFO: Initializing Coyote HTTP/1.1 on http-8080
>>>> Jul 10, 2008 1:03:24 PM org.apache.catalina.startup.Catalina load
>>>> INFO: Initialization processed in 1187 ms
>>>> Jul 10, 2008 1:03:24 PM org.apache.catalina.core.StandardService  
>>>> start
>>>> INFO: Starting service Catalina
>>>> Jul 10, 2008 1:03:24 PM org.apache.catalina.core.StandardEngine  
>>>> start
>>>> INFO: Starting Servlet Engine: Apache Tomcat/6.0.16
>>>> Jul 10, 2008 1:03:24 PM  
>>>> org.apache.catalina.ha.tcp.SimpleTcpCluster start
>>>> INFO: Cluster is about to start
>>>> Jul 10, 2008 1:03:24 PM  
>>>> org.apache.catalina.tribes.transport.ReceiverBase bind
>>>> INFO: Receiver Server Socket bound to:/10.84.2.42:4000
>>>> Jul 10, 2008 1:03:24 PM  
>>>> org.apache.catalina.tribes.membership.McastServiceImpl setupSocket
>>>> INFO: Setting cluster mcast soTimeout to 500
>>>> Jul 10, 2008 1:03:24 PM  
>>>> org.apache.catalina.tribes.membership.McastServiceImpl  
>>>> waitForMembers
>>>> INFO: Sleeping for 1000 milliseconds to establish cluster  
>>>> membership, start level:4
>>>> Jul 10, 2008 1:03:25 PM  
>>>> org.apache.catalina.tribes.membership.McastServiceImpl  
>>>> waitForMembers
>>>> INFO: Done sleeping, membership established, start level:4
>>>> Jul 10, 2008 1:03:25 PM  
>>>> org.apache.catalina.tribes.membership.McastServiceImpl  
>>>> waitForMembers
>>>> INFO: Sleeping for 1000 milliseconds to establish cluster  
>>>> membership, start level:8
>>>> Jul 10, 2008 1:03:26 PM  
>>>> org.apache.catalina.tribes.membership.McastServiceImpl  
>>>> waitForMembers
>>>> INFO: Done sleeping, membership established, start level:8
>>>> Jul 10, 2008 1:03:26 PM  
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
>>>> INFO: Initializing AbstractReplicatedMap with context name:
>>>> Jul 10, 2008 1:03:27 PM org.apache.catalina.core.StandardContext  
>>>> addApplicationListener
>>>> INFO: The listener "listeners.ContextListener" is already  
>>>> configured for this context. The duplicate definition has been  
>>>> ignored.
>>>> Jul 10, 2008 1:03:27 PM org.apache.catalina.core.StandardContext  
>>>> addApplicationListener
>>>> INFO: The listener "listeners.SessionListener" is already  
>>>> configured for this context. The duplicate definition has been  
>>>> ignored.
>>>> Jul 10, 2008 1:03:27 PM  
>>>> org.apache.catalina.ha.session.DeltaManager start
>>>> INFO: Register manager /examples to cluster element Engine with  
>>>> name Catalina
>>>> Jul 10, 2008 1:03:27 PM  
>>>> org.apache.catalina.ha.session.DeltaManager start
>>>> INFO: Starting clustering manager at /examples
>>>> Jul 10, 2008 1:03:27 PM  
>>>> org.apache.catalina.ha.session.DeltaManager getAllClusterSessions
>>>> INFO: Manager [localhost#/examples]: skipping state transfer. No  
>>>> members active in cluster group.
>>>> Jul 10, 2008 1:03:28 PM  
>>>> org.apache.catalina.ha.session.JvmRouteBinderValve start
>>>> INFO: JvmRouteBinderValve started
>>>> Jul 10, 2008 1:03:28 PM org.apache.coyote.http11.Http11Protocol  
>>>> start
>>>> INFO: Starting Coyote HTTP/1.1 on http-8080
>>>> Jul 10, 2008 1:03:28 PM org.apache.jk.common.ChannelSocket init
>>>> INFO: JK: ajp13 listening on /0.0.0.0:8009
>>>> Jul 10, 2008 1:03:28 PM org.apache.jk.server.JkMain start
>>>> INFO: Jk running ID=0 time=0/45  config=null
>>>> Jul 10, 2008 1:03:28 PM org.apache.catalina.startup.Catalina start
>>>> INFO: Server startup in 3616 ms
>>>> Jul 10, 2008 1:06:41 PM  
>>>> org.apache.catalina.tribes.transport.nio.NioReceiver listen
>>>> SEVERE: Unable to process request in NioReceiver
>>>> java.net.SocketException: Invalid argument
>>>>         at sun.nio.ch.Net.setIntOption0(Native Method)
>>>>         at sun.nio.ch.Net.setIntOption(Net.java:152)
>>>>         at sun.nio.ch.SocketChannelImpl$1.setInt 
>>>> (SocketChannelImpl.java:372)
>>>>         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>>>         at sun.nio.ch.SocketOptsImpl$IP.typeOfService 
>>>> (SocketOptsImpl.java:249)
>>>>         at sun.nio.ch.OptionAdaptor.setTrafficClass 
>>>> (OptionAdaptor.java:158)
>>>>         at sun.nio.ch.SocketAdaptor.setTrafficClass 
>>>> (SocketAdaptor.java:330)
>>>>         at  
>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen 
>>>> (NioReceiver.java:273)
>>>>         at  
>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.run 
>>>> (NioReceiver.java:353)
>>>>         at java.lang.Thread.run(Thread.java:613)
>>>> Jul 10, 2008 1:06:41 PM  
>>>> org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
>>>> INFO: Replication member  
>>>> added:org.apache.catalina.tribes.membership.MemberImpl[tcp:// 
>>>> {10, 84, 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24  
>>>> -65 -46 127 -68 65 119 -107 -12 -61 -24 -12 -122 10 56 },  
>>>> payload={}, command={}, domain={}, ]
>>>> Jul 10, 2008 1:06:41 PM org.apache.catalina.tribes.io.BufferPool  
>>>> getBufferPool
>>>> INFO: Created a buffer pool with max size:104857600 bytes of  
>>>> type:org.apache.catalina.tribes.io.BufferPool15Impl
>>>> Jul 10, 2008 1:06:41 PM  
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>>>> WARNING: Member send is failing for:tcp://{10, 84, 2, 42}:4001 ;  
>>>> Setting to suspect and retrying.
>>>> Jul 10, 2008 1:06:41 PM  
>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector  
>>>> memberDisappeared
>>>> INFO: Received memberDisappeared 
>>>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,  
>>>> 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46  
>>>> 127 -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={},  
>>>> command={}, domain={}, ]] message. Will verify.
>>>> Jul 10, 2008 1:06:41 PM  
>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector  
>>>> memberDisappeared
>>>> INFO: Verification complete. Member still alive 
>>>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,  
>>>> 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46  
>>>> 127 -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={},  
>>>> command={}, domain={}, ]]
>>>> Jul 10, 2008 1:06:41 PM  
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
>>>> SEVERE: Unable to send AbstractReplicatedMap.ping message
>>>> org.apache.catalina.tribes.ChannelException: Send failed,  
>>>> attempt:2 max:1; Faulty members:tcp://{10, 84, 2, 42}:4001;
>>>>         at  
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop 
>>>> (ParallelNioSender.java:172)
>>>>         at  
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMess 
>>>> age(ParallelNioSender.java:78)
>>>>         at  
>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendM 
>>>> essage(PooledParallelSender.java:53)
>>>>         at  
>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMes 
>>>> sage(ReplicationTransmitter.java:80)
>>>>         at  
>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage 
>>>> (ChannelCoordinator.java:78)
>>>>         at  
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
>>>> (ChannelInterceptorBase.java:75)
>>>>         at  
>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.se 
>>>> ndMessage(TcpFailureDetector.java:87)
>>>>         at  
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
>>>> (ChannelInterceptorBase.java:75)
>>>>         at  
>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterc 
>>>> eptor.sendMessage(MessageDispatchInterceptor.java:73)
>>>>         at  
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
>>>> (ChannelInterceptorBase.java:75)
>>>>         at org.apache.catalina.tribes.group.GroupChannel.send 
>>>> (GroupChannel.java:216)
>>>>         at org.apache.catalina.tribes.group.GroupChannel.send 
>>>> (GroupChannel.java:175)
>>>>         at org.apache.catalina.tribes.group.RpcChannel.send 
>>>> (RpcChannel.java:89)
>>>>         at  
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping 
>>>> (AbstractReplicatedMap.java:253)
>>>>         at  
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat 
>>>> (AbstractReplicatedMap.java:793)
>>>>         at  
>>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat 
>>>> (GroupChannel.java:153)
>>>>         at org.apache.catalina.tribes.group.GroupChannel 
>>>> $HeartbeatThread.run(GroupChannel.java:661)
>>>> Caused by: java.net.SocketException: Invalid argument
>>>>         at sun.nio.ch.Net.setIntOption0(Native Method)
>>>>         at sun.nio.ch.Net.setIntOption(Net.java:152)
>>>>         at sun.nio.ch.SocketChannelImpl$1.setInt 
>>>> (SocketChannelImpl.java:372)
>>>>         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>>>         at sun.nio.ch.SocketOptsImpl$IP.typeOfService 
>>>> (SocketOptsImpl.java:249)
>>>>         at sun.nio.ch.OptionAdaptor.setTrafficClass 
>>>> (OptionAdaptor.java:158)
>>>>         at sun.nio.ch.SocketAdaptor.setTrafficClass 
>>>> (SocketAdaptor.java:330)
>>>>         at  
>>>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect( 
>>>> NioSender.java:147)
>>>>         at  
>>>> org.apache.catalina.tribes.transport.nio.NioSender.process 
>>>> (NioSender.java:89)
>>>>         at  
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop 
>>>> (ParallelNioSender.java:130)
>>>>         ... 16 more
>>>> Jul 10, 2008 1:06:41 PM  
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>>>> WARNING: Not retrying send for:tcp://{10, 84, 2, 42}:4001;  
>>>> Sender is disconnected.
>>>> Jul 10, 2008 1:06:41 PM  
>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector  
>>>> memberDisappeared
>>>> INFO: Received memberDisappeared 
>>>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,  
>>>> 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46  
>>>> 127 -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={},  
>>>> command={}, domain={}, ]] message. Will verify.
>>>> Jul 10, 2008 1:06:42 PM  
>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector  
>>>> memberDisappeared
>>>> INFO: Verification complete. Member still alive 
>>>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,  
>>>> 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46  
>>>> 127 -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={},  
>>>> command={}, domain={}, ]]
>>>> Jul 10, 2008 1:06:42 PM  
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
>>>> SEVERE: Unable to send AbstractReplicatedMap.ping message
>>>> org.apache.catalina.tribes.ChannelException: Send failed, and  
>>>> sender is disconnected. Not retrying.; Faulty members:tcp://{10,  
>>>> 84, 2, 42}:4001;
>>>>         at  
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop 
>>>> (ParallelNioSender.java:156)
>>>>         at  
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMess 
>>>> age(ParallelNioSender.java:78)
>>>>         at  
>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendM 
>>>> essage(PooledParallelSender.java:53)
>>>>         at  
>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMes 
>>>> sage(ReplicationTransmitter.java:80)
>>>>         at  
>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage 
>>>> (ChannelCoordinator.java:78)
>>>>         at  
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
>>>> (ChannelInterceptorBase.java:75)
>>>>         at  
>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.se 
>>>> ndMessage(TcpFailureDetector.java:87)
>>>>         at  
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
>>>> (ChannelInterceptorBase.java:75)
>>>>         at  
>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterc 
>>>> eptor.sendMessage(MessageDispatchInterceptor.java:73)
>>>>         at  
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
>>>> (ChannelInterceptorBase.java:75)
>>>>         at org.apache.catalina.tribes.group.GroupChannel.send 
>>>> (GroupChannel.java:216)
>>>>         at org.apache.catalina.tribes.group.GroupChannel.send 
>>>> (GroupChannel.java:175)
>>>>         at org.apache.catalina.tribes.group.RpcChannel.send 
>>>> (RpcChannel.java:89)
>>>>         at  
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping 
>>>> (AbstractReplicatedMap.java:253)
>>>>         at  
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat 
>>>> (AbstractReplicatedMap.java:793)
>>>>         at  
>>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat 
>>>> (GroupChannel.java:158)
>>>>         at org.apache.catalina.tribes.group.GroupChannel 
>>>> $HeartbeatThread.run(GroupChannel.java:661)
>>>> Caused by: java.net.SocketException: Invalid argument
>>>>         at sun.nio.ch.Net.setIntOption0(Native Method)
>>>>         at sun.nio.ch.Net.setIntOption(Net.java:152)
>>>>         at sun.nio.ch.SocketChannelImpl$1.setInt 
>>>> (SocketChannelImpl.java:372)
>>>>         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>>>         at sun.nio.ch.SocketOptsImpl$IP.typeOfService 
>>>> (SocketOptsImpl.java:249)
>>>>         at sun.nio.ch.OptionAdaptor.setTrafficClass 
>>>> (OptionAdaptor.java:158)
>>>>         at sun.nio.ch.SocketAdaptor.setTrafficClass 
>>>> (SocketAdaptor.java:330)
>>>>         at  
>>>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect( 
>>>> NioSender.java:147)
>>>>         at  
>>>> org.apache.catalina.tribes.transport.nio.NioSender.process 
>>>> (NioSender.java:89)
>>>>         at  
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop 
>>>> (ParallelNioSender.java:130)
>>>>         ... 16 more
>>>>
>>>>
>>>> and so on forever.
>>>>
>>>> In node2 logs I see the following:
>>>>
>>>> bml0042:/usr/local/tomcat/node2/logs janos$ cat catalina. 
>>>> 2008-07-10.log
>>>> Jul 10, 2008 1:06:40 PM  
>>>> org.apache.catalina.core.AprLifecycleListener init
>>>> INFO: The APR based Apache Tomcat Native library which allows  
>>>> optimal performance in production environments was not found on  
>>>> the java.library.path: .:/Library/Java/Extensions:/System/ 
>>>> Library/Java/Extensions:/usr/lib/java
>>>> Jul 10, 2008 1:06:40 PM org.apache.coyote.http11.Http11Protocol  
>>>> init
>>>> INFO: Initializing Coyote HTTP/1.1 on http-8180
>>>> Jul 10, 2008 1:06:40 PM org.apache.catalina.startup.Catalina load
>>>> INFO: Initialization processed in 1148 ms
>>>> Jul 10, 2008 1:06:40 PM org.apache.catalina.core.StandardService  
>>>> start
>>>> INFO: Starting service Catalina
>>>> Jul 10, 2008 1:06:40 PM org.apache.catalina.core.StandardEngine  
>>>> start
>>>> INFO: Starting Servlet Engine: Apache Tomcat/6.0.16
>>>> Jul 10, 2008 1:06:40 PM  
>>>> org.apache.catalina.ha.tcp.SimpleTcpCluster start
>>>> INFO: Cluster is about to start
>>>> Jul 10, 2008 1:06:40 PM  
>>>> org.apache.catalina.tribes.transport.ReceiverBase bind
>>>> INFO: Receiver Server Socket bound to:/10.84.2.42:4001
>>>> Jul 10, 2008 1:06:40 PM  
>>>> org.apache.catalina.tribes.membership.McastServiceImpl setupSocket
>>>> INFO: Setting cluster mcast soTimeout to 500
>>>> Jul 10, 2008 1:06:40 PM  
>>>> org.apache.catalina.tribes.membership.McastServiceImpl  
>>>> waitForMembers
>>>> INFO: Sleeping for 1000 milliseconds to establish cluster  
>>>> membership, start level:4
>>>> Jul 10, 2008 1:06:41 PM  
>>>> org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
>>>> INFO: Replication member  
>>>> added:org.apache.catalina.tribes.membership.MemberImpl[tcp:// 
>>>> {10, 84, 2, 42}:4000,{10, 84, 2, 42},4000, alive=196161,id={-25  
>>>> 34 -32 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 },  
>>>> payload={}, command={}, domain={}, ]
>>>> Jul 10, 2008 1:06:41 PM  
>>>> org.apache.catalina.tribes.membership.McastServiceImpl  
>>>> waitForMembers
>>>> INFO: Done sleeping, membership established, start level:4
>>>> Jul 10, 2008 1:06:41 PM  
>>>> org.apache.catalina.tribes.membership.McastServiceImpl  
>>>> waitForMembers
>>>> INFO: Sleeping for 1000 milliseconds to establish cluster  
>>>> membership, start level:8
>>>> Jul 10, 2008 1:06:41 PM  
>>>> org.apache.catalina.tribes.transport.nio.NioReceiver listen
>>>> SEVERE: Unable to process request in NioReceiver
>>>> java.net.SocketException: Invalid argument
>>>>         at sun.nio.ch.Net.setIntOption0(Native Method)
>>>>         at sun.nio.ch.Net.setIntOption(Net.java:152)
>>>>         at sun.nio.ch.SocketChannelImpl$1.setInt 
>>>> (SocketChannelImpl.java:372)
>>>>         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>>>         at sun.nio.ch.SocketOptsImpl$IP.typeOfService 
>>>> (SocketOptsImpl.java:249)
>>>>         at sun.nio.ch.OptionAdaptor.setTrafficClass 
>>>> (OptionAdaptor.java:158)
>>>>         at sun.nio.ch.SocketAdaptor.setTrafficClass 
>>>> (SocketAdaptor.java:330)
>>>>         at  
>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen 
>>>> (NioReceiver.java:273)
>>>>         at  
>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.run 
>>>> (NioReceiver.java:353)
>>>>         at java.lang.Thread.run(Thread.java:613)
>>>> Jul 10, 2008 1:06:42 PM  
>>>> org.apache.catalina.tribes.membership.McastServiceImpl  
>>>> waitForMembers
>>>> INFO: Done sleeping, membership established, start level:8
>>>> Jul 10, 2008 1:06:42 PM  
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
>>>> INFO: Initializing AbstractReplicatedMap with context name:
>>>> Jul 10, 2008 1:06:42 PM org.apache.catalina.tribes.io.BufferPool  
>>>> getBufferPool
>>>> INFO: Created a buffer pool with max size:104857600 bytes of  
>>>> type:org.apache.catalina.tribes.io.BufferPool15Impl
>>>> Jul 10, 2008 1:06:42 PM  
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>>>> WARNING: Member send is failing for:tcp://{10, 84, 2, 42}:4000 ;  
>>>> Setting to suspect and retrying.
>>>> Jul 10, 2008 1:06:42 PM  
>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector  
>>>> memberDisappeared
>>>> INFO: Received memberDisappeared 
>>>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,  
>>>> 2, 42}:4000,{10, 84, 2, 42},4000, alive=197662,id={-25 34 -32  
>>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={},  
>>>> command={}, domain={}, ]] message. Will verify.
>>>> Jul 10, 2008 1:06:42 PM  
>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector  
>>>> memberDisappeared
>>>> INFO: Verification complete. Member still alive 
>>>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,  
>>>> 2, 42}:4000,{10, 84, 2, 42},4000, alive=197662,id={-25 34 -32  
>>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={},  
>>>> command={}, domain={}, ]]
>>>> Jul 10, 2008 1:06:42 PM  
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
>>>> WARNING: Unable to send map start message.
>>>> Jul 10, 2008 1:06:42 PM  
>>>> org.apache.catalina.ha.context.ReplicatedContext start
>>>> SEVERE: Unable to start ReplicatedContext
>>>> java.lang.RuntimeException: Unable to start replicated map.
>>>>         at  
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.init 
>>>> (AbstractReplicatedMap.java:230)
>>>>         at  
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.<init> 
>>>> (AbstractReplicatedMap.java:171)
>>>>         at org.apache.catalina.tribes.tipis.ReplicatedMap.<init> 
>>>> (ReplicatedMap.java:90)
>>>>         at org.apache.catalina.ha.context.ReplicatedContext.start 
>>>> (ReplicatedContext.java:64)
>>>>         at org.apache.catalina.core.ContainerBase.start 
>>>> (ContainerBase.java:1045)
>>>>         at org.apache.catalina.core.StandardHost.start 
>>>> (StandardHost.java:719)
>>>>         at org.apache.catalina.core.ContainerBase.start 
>>>> (ContainerBase.java:1045)
>>>>         at org.apache.catalina.core.StandardEngine.start 
>>>> (StandardEngine.java:443)
>>>>         at org.apache.catalina.core.StandardService.start 
>>>> (StandardService.java:516)
>>>>         at org.apache.catalina.core.StandardServer.start 
>>>> (StandardServer.java:710)
>>>>         at org.apache.catalina.startup.Catalina.start 
>>>> (Catalina.java:578)
>>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native  
>>>> Method)
>>>>         at sun.reflect.NativeMethodAccessorImpl.invoke 
>>>> (NativeMethodAccessorImpl.java:39)
>>>>         at sun.reflect.DelegatingMethodAccessorImpl.invoke 
>>>> (DelegatingMethodAccessorImpl.java:25)
>>>>         at java.lang.reflect.Method.invoke(Method.java:585)
>>>>         at org.apache.catalina.startup.Bootstrap.start 
>>>> (Bootstrap.java:288)
>>>>         at org.apache.catalina.startup.Bootstrap.main 
>>>> (Bootstrap.java:413)
>>>> Caused by: org.apache.catalina.tribes.ChannelException: Send  
>>>> failed, attempt:2 max:1; Faulty members:tcp://{10, 84, 2, 42}:4000;
>>>>         at  
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop 
>>>> (ParallelNioSender.java:172)
>>>>         at  
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMess 
>>>> age(ParallelNioSender.java:78)
>>>>         at  
>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendM 
>>>> essage(PooledParallelSender.java:53)
>>>>         at  
>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMes 
>>>> sage(ReplicationTransmitter.java:80)
>>>>         at  
>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage 
>>>> (ChannelCoordinator.java:78)
>>>>         at  
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
>>>> (ChannelInterceptorBase.java:75)
>>>>         at  
>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.se 
>>>> ndMessage(TcpFailureDetector.java:87)
>>>>         at  
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
>>>> (ChannelInterceptorBase.java:75)
>>>>         at  
>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterc 
>>>> eptor.sendMessage(MessageDispatchInterceptor.java:73)
>>>>         at  
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
>>>> (ChannelInterceptorBase.java:75)
>>>>         at org.apache.catalina.tribes.group.GroupChannel.send 
>>>> (GroupChannel.java:216)
>>>>         at org.apache.catalina.tribes.group.GroupChannel.send 
>>>> (GroupChannel.java:175)
>>>>         at org.apache.catalina.tribes.group.RpcChannel.send 
>>>> (RpcChannel.java:89)
>>>>         at  
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.broadcast 
>>>> (AbstractReplicatedMap.java:300)
>>>>         at  
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.init 
>>>> (AbstractReplicatedMap.java:223)
>>>>         ... 16 more
>>>> Caused by: java.net.SocketException: Invalid argument
>>>>         at sun.nio.ch.Net.setIntOption0(Native Method)
>>>>         at sun.nio.ch.Net.setIntOption(Net.java:152)
>>>>         at sun.nio.ch.SocketChannelImpl$1.setInt 
>>>> (SocketChannelImpl.java:372)
>>>>         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>>>         at sun.nio.ch.SocketOptsImpl$IP.typeOfService 
>>>> (SocketOptsImpl.java:249)
>>>>         at sun.nio.ch.OptionAdaptor.setTrafficClass 
>>>> (OptionAdaptor.java:158)
>>>>         at sun.nio.ch.SocketAdaptor.setTrafficClass 
>>>> (SocketAdaptor.java:330)
>>>>         at  
>>>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect( 
>>>> NioSender.java:147)
>>>>         at  
>>>> org.apache.catalina.tribes.transport.nio.NioSender.process 
>>>> (NioSender.java:89)
>>>>         at  
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop 
>>>> (ParallelNioSender.java:130)
>>>>         ... 30 more
>>>> Jul 10, 2008 1:06:42 PM org.apache.catalina.startup.Catalina start
>>>> SEVERE: Catalina.start:
>>>> LifecycleException:  Failed to start ReplicatedContext:   
>>>> java.lang.RuntimeException: Unable to start replicated map.
>>>>         at org.apache.catalina.ha.context.ReplicatedContext.start 
>>>> (ReplicatedContext.java:73)
>>>>         at org.apache.catalina.core.ContainerBase.start 
>>>> (ContainerBase.java:1045)
>>>>         at org.apache.catalina.core.StandardHost.start 
>>>> (StandardHost.java:719)
>>>>         at org.apache.catalina.core.ContainerBase.start 
>>>> (ContainerBase.java:1045)
>>>>         at org.apache.catalina.core.StandardEngine.start 
>>>> (StandardEngine.java:443)
>>>>         at org.apache.catalina.core.StandardService.start 
>>>> (StandardService.java:516)
>>>>         at org.apache.catalina.core.StandardServer.start 
>>>> (StandardServer.java:710)
>>>>         at org.apache.catalina.startup.Catalina.start 
>>>> (Catalina.java:578)
>>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native  
>>>> Method)
>>>>         at sun.reflect.NativeMethodAccessorImpl.invoke 
>>>> (NativeMethodAccessorImpl.java:39)
>>>>         at sun.reflect.DelegatingMethodAccessorImpl.invoke 
>>>> (DelegatingMethodAccessorImpl.java:25)
>>>>         at java.lang.reflect.Method.invoke(Method.java:585)
>>>>         at org.apache.catalina.startup.Bootstrap.start 
>>>> (Bootstrap.java:288)
>>>>         at org.apache.catalina.startup.Bootstrap.main 
>>>> (Bootstrap.java:413)
>>>> Jul 10, 2008 1:06:42 PM org.apache.catalina.startup.Catalina start
>>>> INFO: Server startup in 2329 ms
>>>> Jul 10, 2008 1:06:47 PM  
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>>>> WARNING: Not retrying send for:tcp://{10, 84, 2, 42}:4000;  
>>>> Sender is disconnected.
>>>> Jul 10, 2008 1:06:47 PM  
>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector  
>>>> memberDisappeared
>>>> INFO: Received memberDisappeared 
>>>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,  
>>>> 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32  
>>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={},  
>>>> command={}, domain={}, ]] message. Will verify.
>>>> Jul 10, 2008 1:06:47 PM  
>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector  
>>>> memberDisappeared
>>>> INFO: Verification complete. Member still alive 
>>>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,  
>>>> 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32  
>>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={},  
>>>> command={}, domain={}, ]]
>>>> Jul 10, 2008 1:06:47 PM  
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
>>>> SEVERE: Unable to send AbstractReplicatedMap.ping message
>>>> org.apache.catalina.tribes.ChannelException: Send failed, and  
>>>> sender is disconnected. Not retrying.; Faulty members:tcp://{10,  
>>>> 84, 2, 42}:4000;
>>>>         at  
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop 
>>>> (ParallelNioSender.java:156)
>>>>         at  
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMess 
>>>> age(ParallelNioSender.java:78)
>>>>         at  
>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendM 
>>>> essage(PooledParallelSender.java:53)
>>>>         at  
>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMes 
>>>> sage(ReplicationTransmitter.java:80)
>>>>         at  
>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage 
>>>> (ChannelCoordinator.java:78)
>>>>         at  
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
>>>> (ChannelInterceptorBase.java:75)
>>>>         at  
>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.se 
>>>> ndMessage(TcpFailureDetector.java:87)
>>>>         at  
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
>>>> (ChannelInterceptorBase.java:75)
>>>>         at  
>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterc 
>>>> eptor.sendMessage(MessageDispatchInterceptor.java:73)
>>>>         at  
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
>>>> (ChannelInterceptorBase.java:75)
>>>>         at org.apache.catalina.tribes.group.GroupChannel.send 
>>>> (GroupChannel.java:216)
>>>>         at org.apache.catalina.tribes.group.GroupChannel.send 
>>>> (GroupChannel.java:175)
>>>>         at org.apache.catalina.tribes.group.RpcChannel.send 
>>>> (RpcChannel.java:89)
>>>>         at  
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping 
>>>> (AbstractReplicatedMap.java:253)
>>>>         at  
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat 
>>>> (AbstractReplicatedMap.java:793)
>>>>         at  
>>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat 
>>>> (GroupChannel.java:153)
>>>>         at org.apache.catalina.tribes.group.GroupChannel 
>>>> $HeartbeatThread.run(GroupChannel.java:661)
>>>> Caused by: java.net.SocketException: Invalid argument
>>>>         at sun.nio.ch.Net.setIntOption0(Native Method)
>>>>         at sun.nio.ch.Net.setIntOption(Net.java:152)
>>>>         at sun.nio.ch.SocketChannelImpl$1.setInt 
>>>> (SocketChannelImpl.java:372)
>>>>         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>>>         at sun.nio.ch.SocketOptsImpl$IP.typeOfService 
>>>> (SocketOptsImpl.java:249)
>>>>         at sun.nio.ch.OptionAdaptor.setTrafficClass 
>>>> (OptionAdaptor.java:158)
>>>>         at sun.nio.ch.SocketAdaptor.setTrafficClass 
>>>> (SocketAdaptor.java:330)
>>>>         at  
>>>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect( 
>>>> NioSender.java:147)
>>>>         at  
>>>> org.apache.catalina.tribes.transport.nio.NioSender.process 
>>>> (NioSender.java:89)
>>>>         at  
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop 
>>>> (ParallelNioSender.java:130)
>>>>         ... 16 more
>>>> Jul 10, 2008 1:06:47 PM  
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>>>> WARNING: Not retrying send for:tcp://{10, 84, 2, 42}:4000;  
>>>> Sender is disconnected.
>>>> Jul 10, 2008 1:06:47 PM  
>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector  
>>>> memberDisappeared
>>>> INFO: Received memberDisappeared 
>>>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,  
>>>> 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32  
>>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={},  
>>>> command={}, domain={}, ]] message. Will verify.
>>>> Jul 10, 2008 1:06:47 PM  
>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector  
>>>> memberDisappeared
>>>> INFO: Verification complete. Member still alive 
>>>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,  
>>>> 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32  
>>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={},  
>>>> command={}, domain={}, ]]
>>>> Jul 10, 2008 1:06:47 PM  
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
>>>> SEVERE: Unable to send AbstractReplicatedMap.ping message
>>>> org.apache.catalina.tribes.ChannelException: Send failed, and  
>>>> sender is disconnected. Not retrying.; Faulty members:tcp://{10,  
>>>> 84, 2, 42}:4000;
>>>>         at  
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop 
>>>> (ParallelNioSender.java:156)
>>>>         at  
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMess 
>>>> age(ParallelNioSender.java:78)
>>>>         at  
>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendM 
>>>> essage(PooledParallelSender.java:53)
>>>>         at  
>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMes 
>>>> sage(ReplicationTransmitter.java:80)
>>>>         at  
>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage 
>>>> (ChannelCoordinator.java:78)
>>>>         at  
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
>>>> (ChannelInterceptorBase.java:75)
>>>>         at  
>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.se 
>>>> ndMessage(TcpFailureDetector.java:87)
>>>>         at  
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
>>>> (ChannelInterceptorBase.java:75)
>>>>         at  
>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterc 
>>>> eptor.sendMessage(MessageDispatchInterceptor.java:73)
>>>>         at  
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
>>>> (ChannelInterceptorBase.java:75)
>>>>         at org.apache.catalina.tribes.group.GroupChannel.send 
>>>> (GroupChannel.java:216)
>>>>         at org.apache.catalina.tribes.group.GroupChannel.send 
>>>> (GroupChannel.java:175)
>>>>         at org.apache.catalina.tribes.group.RpcChannel.send 
>>>> (RpcChannel.java:89)
>>>>         at  
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping 
>>>> (AbstractReplicatedMap.java:253)
>>>>         at  
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat 
>>>> (AbstractReplicatedMap.java:793)
>>>>         at  
>>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat 
>>>> (GroupChannel.java:158)
>>>>         at org.apache.catalina.tribes.group.GroupChannel 
>>>> $HeartbeatThread.run(GroupChannel.java:661)
>>>> Caused by: java.net.SocketException: Invalid argument
>>>>         at sun.nio.ch.Net.setIntOption0(Native Method)
>>>>         at sun.nio.ch.Net.setIntOption(Net.java:152)
>>>>         at sun.nio.ch.SocketChannelImpl$1.setInt 
>>>> (SocketChannelImpl.java:372)
>>>>         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>>>         at sun.nio.ch.SocketOptsImpl$IP.typeOfService 
>>>> (SocketOptsImpl.java:249)
>>>>         at sun.nio.ch.OptionAdaptor.setTrafficClass 
>>>> (OptionAdaptor.java:158)
>>>>         at sun.nio.ch.SocketAdaptor.setTrafficClass 
>>>> (SocketAdaptor.java:330)
>>>>         at  
>>>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect( 
>>>> NioSender.java:147)
>>>>         at  
>>>> org.apache.catalina.tribes.transport.nio.NioSender.process 
>>>> (NioSender.java:89)
>>>>         at  
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop 
>>>> (ParallelNioSender.java:130)
>>>>         ... 16 more
>>>>
>>>>
>>>> and so on.  Top shows that both Java process is Up to 86%, the  
>>>> machine is never idle and all memory is consumed.  The machine  
>>>> is a PowerPc G6 from Apple with OSX 10.4.11.
>>>> Tomcat version is: Apache Tomcat/6.0.16
>>>> JVM version is: 1.5.0_13-b05-241
>>>> I suspect the heartbeat pings have a problem, I just do not know  
>>>> what to do about them.
>>>>
>>>> Thanks ahead,
>>>>
>>>> János
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------- 
>>>> --
>>>> To start a new topic, e-mail: users@tomcat.apache.org
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>>
>>>>
>>>
>>>
>>> -------------------------------------------------------------------- 
>>> -
>>> To start a new topic, e-mail: users@tomcat.apache.org
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To start a new topic, e-mail: users@tomcat.apache.org
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>>
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org


---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: Version 6 Clustering problem

Posted by Filip Hanik - Dev Lists <de...@hanik.com>.
write your own webapp, to make sure you actually replicate serializable data

Filip

János wrote:
> Hi Filip,
>
> I added it in the catalina.sh file on all three nodes and I am not 
> seeing the error.  I also disabled IP6 in Network Preferences.
>
> I added the distrubuted="true" to the context.xml file on all three 
> nodes and added the <distributable/> tag to the 
> webapps/examples/WEB-INF/web.xml on all 3 nodes and tried the 
> SessionExample servlet
>
> http://bml0042.yalepath.org:8080/examples/servlets/servlet/SessionExample
>
> to see if I have a working cluster.  I am following the book at
>
> http://proquest.safaribooksonline.com:80/9780596101060/tomcat_6_clustering_implementation 
>
>
> and it says that I should see messages similar to this one in the 
> nodes' logs"
>
> Sep 27, 2008 5:39:38 PM org.apache.catalina.core.ApplicationContext log
> INFO: SessionListener: contextInitialized(  )
> Sep 27, 2008 5:40:07 PM org.apache.catalina.core.ApplicationContext log
> INFO: SessionListener: 
> sessionCreated('F4B2D7191C1F335FFAAC93DA461CA95F.tc1')
> Sep 27, 2008 5:40:19 PM org.apache.catalina.core.ApplicationContext log
> INFO: SessionListener: 
> attributeAdded('F4B2D7191C1F335FFAAC93DA461CA95F.tc1', 'test',
>
>
> But I am not seeing them.  This is the resent log from node1:
>
> Jul 11, 2008 3:28:01 PM org.apache.catalina.core.AprLifecycleListener 
> init
> INFO: The APR based Apache Tomcat Native library which allows optimal 
> performance in production environments was not found on the 
> java.library.path: 
> .:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java
> Jul 11, 2008 3:28:01 PM org.apache.coyote.http11.Http11Protocol init
> INFO: Initializing Coyote HTTP/1.1 on http-8080
> Jul 11, 2008 3:28:01 PM org.apache.catalina.startup.Catalina load
> INFO: Initialization processed in 1098 ms
> Jul 11, 2008 3:28:01 PM org.apache.catalina.core.StandardService start
> INFO: Starting service Catalina
> Jul 11, 2008 3:28:01 PM org.apache.catalina.core.StandardEngine start
> INFO: Starting Servlet Engine: Apache Tomcat/6.0.16
> Jul 11, 2008 3:28:01 PM org.apache.catalina.ha.tcp.SimpleTcpCluster start
> INFO: Cluster is about to start
> Jul 11, 2008 3:28:01 PM 
> org.apache.catalina.tribes.transport.ReceiverBase bind
> INFO: Receiver Server Socket bound to:/10.84.2.42:4000
> Jul 11, 2008 3:28:01 PM 
> org.apache.catalina.tribes.membership.McastServiceImpl setupSocket
> INFO: Setting cluster mcast soTimeout to 500
> Jul 11, 2008 3:28:01 PM 
> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
> INFO: Sleeping for 1000 milliseconds to establish cluster membership, 
> start level:4
> Jul 11, 2008 3:28:02 PM 
> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
> INFO: Done sleeping, membership established, start level:4
> Jul 11, 2008 3:28:02 PM 
> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
> INFO: Sleeping for 1000 milliseconds to establish cluster membership, 
> start level:8
> Jul 11, 2008 3:28:03 PM 
> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
> INFO: Done sleeping, membership established, start level:8
> Jul 11, 2008 3:28:03 PM 
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
> INFO: Initializing AbstractReplicatedMap with context name:
> Jul 11, 2008 3:28:04 PM org.apache.catalina.core.StandardContext 
> addApplicationListener
> INFO: The listener "listeners.ContextListener" is already configured 
> for this context. The duplicate definition has been ignored.
> Jul 11, 2008 3:28:04 PM org.apache.catalina.core.StandardContext 
> addApplicationListener
> INFO: The listener "listeners.SessionListener" is already configured 
> for this context. The duplicate definition has been ignored.
> Jul 11, 2008 3:28:04 PM org.apache.catalina.ha.session.DeltaManager start
> INFO: Register manager /examples to cluster element Engine with name 
> Catalina
> Jul 11, 2008 3:28:04 PM org.apache.catalina.ha.session.DeltaManager start
> INFO: Starting clustering manager at /examples
> Jul 11, 2008 3:28:04 PM org.apache.catalina.ha.session.DeltaManager 
> getAllClusterSessions
> INFO: Manager [localhost#/examples]: skipping state transfer. No 
> members active in cluster group.
> Jul 11, 2008 3:28:04 PM 
> org.apache.catalina.ha.session.JvmRouteBinderValve start
> INFO: JvmRouteBinderValve started
> Jul 11, 2008 3:28:04 PM org.apache.coyote.http11.Http11Protocol start
> INFO: Starting Coyote HTTP/1.1 on http-8080
> Jul 11, 2008 3:28:05 PM org.apache.jk.common.ChannelSocket init
> INFO: JK: ajp13 listening on /0.0.0.0:8009
> Jul 11, 2008 3:28:05 PM org.apache.jk.server.JkMain start
> INFO: Jk running ID=0 time=0/45  config=null
> Jul 11, 2008 3:28:05 PM org.apache.catalina.startup.Catalina start
> INFO: Server startup in 3601 ms
> Jul 11, 2008 3:28:37 PM org.apache.catalina.tribes.io.BufferPool 
> getBufferPool
> INFO: Created a buffer pool with max size:104857600 bytes of 
> type:org.apache.catalina.tribes.io.BufferPool15Impl
> Jul 11, 2008 3:28:37 PM org.apache.catalina.ha.tcp.SimpleTcpCluster 
> memberAdded
> INFO: Replication member 
> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84, 
> 2, 42}:4001,{10, 84, 2, 42},4001, alive=1020,id={60 103 -111 -34 98 62 
> 71 -116 -104 76 -100 115 -77 15 -52 -92 }, payload={}, command={}, 
> domain={}, ]
> Jul 11, 2008 3:29:08 PM org.apache.catalina.ha.tcp.SimpleTcpCluster 
> memberAdded
> INFO: Replication member 
> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84, 
> 2, 42}:4002,{10, 84, 2, 42},4002, alive=1019,id={27 24 -73 72 48 -6 70 
> 32 -90 57 51 106 -94 -54 126 -52 }, payload={}, command={}, domain={}, ]
>
>
>
> Similarly on the other two nodes.  What am I missing ??  Should I add 
> the <Context distributed=True" /> to the server.xml files ?
>
> Thanks ahead,
>
> János
> On Jul 10, 2008, at 6:03 PM, Filip Hanik - Dev Lists wrote:
>
>> try setting the
>> |-Djava.net.preferIPv4Stack=true
>>
>> option for your tomcat launch
>>
>> Filip
>> |
>> János wrote:
>>> Hi,
>>>
>>> I am trying to establish a 3 member cluster on the same machine.  
>>> Here is my server.xml copied from a sample I found on this list.
>>>
>>> <?xml version='1.0' encoding='utf-8'?>
>>> <Server port="8005" shutdown="SHUTDOWN">
>>>   <Listener 
>>> className="org.apache.catalina.core.AprLifecycleListener" 
>>> SSLEngine="on" />
>>>   <Listener className="org.apache.catalina.core.JasperListener" />
>>>   <Listener 
>>> className="org.apache.catalina.mbeans.ServerLifecycleListener" />
>>>   <Listener 
>>> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" 
>>> />
>>>
>>>   <GlobalNamingResources>
>>>     <Resource name="UserDatabase" auth="Container"
>>>               type="org.apache.catalina.UserDatabase"
>>>               description="User database that can be updated and saved"
>>>               
>>> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
>>>               pathname="conf/tomcat-users.xml" />
>>>   </GlobalNamingResources>
>>>
>>>   <Service name="Catalina">
>>>
>>>     <Connector port="8080" protocol="HTTP/1.1"
>>>                connectionTimeout="20000"
>>>                redirectPort="8443" />
>>>     <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
>>>
>>>     <Engine name="Catalina" defaultHost="localhost" jvmRoute="node1">
>>>
>>>       <Cluster 
>>> className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
>>>       <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
>>>              resourceName="UserDatabase"/>
>>>
>>>       <Host name="localhost"  appBase="webapps"
>>>             unpackWARs="true" autoDeploy="true"
>>>             xmlValidation="false" xmlNamespaceAware="false">
>>>         <Context path="" docBase="ROOT" 
>>> className="org.apache.catalina.ha.context.ReplicatedContext"/>
>>>       </Host>
>>>     </Engine>
>>>   </Service>
>>> </Server>
>>>
>>> The other two server.xml-s are the same, except the shutdown ports 
>>> are  8105, 8205, http ports are 8180, 8280 and https ports are 8543, 
>>> 8643.
>>> As soon as I am starting the second instance I receive the following 
>>> into the logs.
>>>
>>> In node1 logs:
>>>
>>> bml0042:/usr/local/tomcat/node1/logs janos$ cat catalina.2008-07-10.log
>>> Jul 10, 2008 1:03:24 PM 
>>> org.apache.catalina.core.AprLifecycleListener init
>>> INFO: The APR based Apache Tomcat Native library which allows 
>>> optimal performance in production environments was not found on the 
>>> java.library.path: 
>>> .:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java 
>>>
>>> Jul 10, 2008 1:03:24 PM org.apache.coyote.http11.Http11Protocol init
>>> INFO: Initializing Coyote HTTP/1.1 on http-8080
>>> Jul 10, 2008 1:03:24 PM org.apache.catalina.startup.Catalina load
>>> INFO: Initialization processed in 1187 ms
>>> Jul 10, 2008 1:03:24 PM org.apache.catalina.core.StandardService start
>>> INFO: Starting service Catalina
>>> Jul 10, 2008 1:03:24 PM org.apache.catalina.core.StandardEngine start
>>> INFO: Starting Servlet Engine: Apache Tomcat/6.0.16
>>> Jul 10, 2008 1:03:24 PM org.apache.catalina.ha.tcp.SimpleTcpCluster 
>>> start
>>> INFO: Cluster is about to start
>>> Jul 10, 2008 1:03:24 PM 
>>> org.apache.catalina.tribes.transport.ReceiverBase bind
>>> INFO: Receiver Server Socket bound to:/10.84.2.42:4000
>>> Jul 10, 2008 1:03:24 PM 
>>> org.apache.catalina.tribes.membership.McastServiceImpl setupSocket
>>> INFO: Setting cluster mcast soTimeout to 500
>>> Jul 10, 2008 1:03:24 PM 
>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>> INFO: Sleeping for 1000 milliseconds to establish cluster 
>>> membership, start level:4
>>> Jul 10, 2008 1:03:25 PM 
>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>> INFO: Done sleeping, membership established, start level:4
>>> Jul 10, 2008 1:03:25 PM 
>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>> INFO: Sleeping for 1000 milliseconds to establish cluster 
>>> membership, start level:8
>>> Jul 10, 2008 1:03:26 PM 
>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>> INFO: Done sleeping, membership established, start level:8
>>> Jul 10, 2008 1:03:26 PM 
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
>>> INFO: Initializing AbstractReplicatedMap with context name:
>>> Jul 10, 2008 1:03:27 PM org.apache.catalina.core.StandardContext 
>>> addApplicationListener
>>> INFO: The listener "listeners.ContextListener" is already configured 
>>> for this context. The duplicate definition has been ignored.
>>> Jul 10, 2008 1:03:27 PM org.apache.catalina.core.StandardContext 
>>> addApplicationListener
>>> INFO: The listener "listeners.SessionListener" is already configured 
>>> for this context. The duplicate definition has been ignored.
>>> Jul 10, 2008 1:03:27 PM org.apache.catalina.ha.session.DeltaManager 
>>> start
>>> INFO: Register manager /examples to cluster element Engine with name 
>>> Catalina
>>> Jul 10, 2008 1:03:27 PM org.apache.catalina.ha.session.DeltaManager 
>>> start
>>> INFO: Starting clustering manager at /examples
>>> Jul 10, 2008 1:03:27 PM org.apache.catalina.ha.session.DeltaManager 
>>> getAllClusterSessions
>>> INFO: Manager [localhost#/examples]: skipping state transfer. No 
>>> members active in cluster group.
>>> Jul 10, 2008 1:03:28 PM 
>>> org.apache.catalina.ha.session.JvmRouteBinderValve start
>>> INFO: JvmRouteBinderValve started
>>> Jul 10, 2008 1:03:28 PM org.apache.coyote.http11.Http11Protocol start
>>> INFO: Starting Coyote HTTP/1.1 on http-8080
>>> Jul 10, 2008 1:03:28 PM org.apache.jk.common.ChannelSocket init
>>> INFO: JK: ajp13 listening on /0.0.0.0:8009
>>> Jul 10, 2008 1:03:28 PM org.apache.jk.server.JkMain start
>>> INFO: Jk running ID=0 time=0/45  config=null
>>> Jul 10, 2008 1:03:28 PM org.apache.catalina.startup.Catalina start
>>> INFO: Server startup in 3616 ms
>>> Jul 10, 2008 1:06:41 PM 
>>> org.apache.catalina.tribes.transport.nio.NioReceiver listen
>>> SEVERE: Unable to process request in NioReceiver
>>> java.net.SocketException: Invalid argument
>>>         at sun.nio.ch.Net.setIntOption0(Native Method)
>>>         at sun.nio.ch.Net.setIntOption(Net.java:152)
>>>         at 
>>> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
>>>         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>>         at 
>>> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
>>>         at 
>>> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
>>>         at 
>>> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
>>>         at 
>>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:273) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioReceiver.java:353) 
>>>
>>>         at java.lang.Thread.run(Thread.java:613)
>>> Jul 10, 2008 1:06:41 PM org.apache.catalina.ha.tcp.SimpleTcpCluster 
>>> memberAdded
>>> INFO: Replication member 
>>> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 
>>> 84, 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46 
>>> 127 -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={}, 
>>> command={}, domain={}, ]
>>> Jul 10, 2008 1:06:41 PM org.apache.catalina.tribes.io.BufferPool 
>>> getBufferPool
>>> INFO: Created a buffer pool with max size:104857600 bytes of 
>>> type:org.apache.catalina.tribes.io.BufferPool15Impl
>>> Jul 10, 2008 1:06:41 PM 
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>>> WARNING: Member send is failing for:tcp://{10, 84, 2, 42}:4001 ; 
>>> Setting to suspect and retrying.
>>> Jul 10, 2008 1:06:41 PM 
>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector 
>>> memberDisappeared
>>> INFO: Received 
>>> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 
>>> 84, 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46 
>>> 127 -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={}, 
>>> command={}, domain={}, ]] message. Will verify.
>>> Jul 10, 2008 1:06:41 PM 
>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector 
>>> memberDisappeared
>>> INFO: Verification complete. Member still 
>>> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 
>>> 84, 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46 
>>> 127 -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={}, 
>>> command={}, domain={}, ]]
>>> Jul 10, 2008 1:06:41 PM 
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
>>> SEVERE: Unable to send AbstractReplicatedMap.ping message
>>> org.apache.catalina.tribes.ChannelException: Send failed, attempt:2 
>>> max:1; Faulty members:tcp://{10, 84, 2, 42}:4001;
>>>         at 
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:172) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:78) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:73) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:89)
>>>         at 
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:253) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:793) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:153) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:661) 
>>>
>>> Caused by: java.net.SocketException: Invalid argument
>>>         at sun.nio.ch.Net.setIntOption0(Native Method)
>>>         at sun.nio.ch.Net.setIntOption(Net.java:152)
>>>         at 
>>> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
>>>         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>>         at 
>>> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
>>>         at 
>>> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
>>>         at 
>>> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
>>>         at 
>>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect(NioSender.java:147) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.transport.nio.NioSender.process(NioSender.java:89) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:130) 
>>>
>>>         ... 16 more
>>> Jul 10, 2008 1:06:41 PM 
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>>> WARNING: Not retrying send for:tcp://{10, 84, 2, 42}:4001; Sender is 
>>> disconnected.
>>> Jul 10, 2008 1:06:41 PM 
>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector 
>>> memberDisappeared
>>> INFO: Received 
>>> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 
>>> 84, 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46 
>>> 127 -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={}, 
>>> command={}, domain={}, ]] message. Will verify.
>>> Jul 10, 2008 1:06:42 PM 
>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector 
>>> memberDisappeared
>>> INFO: Verification complete. Member still 
>>> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 
>>> 84, 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46 
>>> 127 -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={}, 
>>> command={}, domain={}, ]]
>>> Jul 10, 2008 1:06:42 PM 
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
>>> SEVERE: Unable to send AbstractReplicatedMap.ping message
>>> org.apache.catalina.tribes.ChannelException: Send failed, and sender 
>>> is disconnected. Not retrying.; Faulty members:tcp://{10, 84, 2, 
>>> 42}:4001;
>>>         at 
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:156) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:78) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:73) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:89)
>>>         at 
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:253) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:793) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:158) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:661) 
>>>
>>> Caused by: java.net.SocketException: Invalid argument
>>>         at sun.nio.ch.Net.setIntOption0(Native Method)
>>>         at sun.nio.ch.Net.setIntOption(Net.java:152)
>>>         at 
>>> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
>>>         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>>         at 
>>> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
>>>         at 
>>> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
>>>         at 
>>> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
>>>         at 
>>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect(NioSender.java:147) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.transport.nio.NioSender.process(NioSender.java:89) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:130) 
>>>
>>>         ... 16 more
>>>
>>>
>>> and so on forever.
>>>
>>> In node2 logs I see the following:
>>>
>>> bml0042:/usr/local/tomcat/node2/logs janos$ cat catalina.2008-07-10.log
>>> Jul 10, 2008 1:06:40 PM 
>>> org.apache.catalina.core.AprLifecycleListener init
>>> INFO: The APR based Apache Tomcat Native library which allows 
>>> optimal performance in production environments was not found on the 
>>> java.library.path: 
>>> .:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java 
>>>
>>> Jul 10, 2008 1:06:40 PM org.apache.coyote.http11.Http11Protocol init
>>> INFO: Initializing Coyote HTTP/1.1 on http-8180
>>> Jul 10, 2008 1:06:40 PM org.apache.catalina.startup.Catalina load
>>> INFO: Initialization processed in 1148 ms
>>> Jul 10, 2008 1:06:40 PM org.apache.catalina.core.StandardService start
>>> INFO: Starting service Catalina
>>> Jul 10, 2008 1:06:40 PM org.apache.catalina.core.StandardEngine start
>>> INFO: Starting Servlet Engine: Apache Tomcat/6.0.16
>>> Jul 10, 2008 1:06:40 PM org.apache.catalina.ha.tcp.SimpleTcpCluster 
>>> start
>>> INFO: Cluster is about to start
>>> Jul 10, 2008 1:06:40 PM 
>>> org.apache.catalina.tribes.transport.ReceiverBase bind
>>> INFO: Receiver Server Socket bound to:/10.84.2.42:4001
>>> Jul 10, 2008 1:06:40 PM 
>>> org.apache.catalina.tribes.membership.McastServiceImpl setupSocket
>>> INFO: Setting cluster mcast soTimeout to 500
>>> Jul 10, 2008 1:06:40 PM 
>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>> INFO: Sleeping for 1000 milliseconds to establish cluster 
>>> membership, start level:4
>>> Jul 10, 2008 1:06:41 PM org.apache.catalina.ha.tcp.SimpleTcpCluster 
>>> memberAdded
>>> INFO: Replication member 
>>> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 
>>> 84, 2, 42}:4000,{10, 84, 2, 42},4000, alive=196161,id={-25 34 -32 
>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, 
>>> command={}, domain={}, ]
>>> Jul 10, 2008 1:06:41 PM 
>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>> INFO: Done sleeping, membership established, start level:4
>>> Jul 10, 2008 1:06:41 PM 
>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>> INFO: Sleeping for 1000 milliseconds to establish cluster 
>>> membership, start level:8
>>> Jul 10, 2008 1:06:41 PM 
>>> org.apache.catalina.tribes.transport.nio.NioReceiver listen
>>> SEVERE: Unable to process request in NioReceiver
>>> java.net.SocketException: Invalid argument
>>>         at sun.nio.ch.Net.setIntOption0(Native Method)
>>>         at sun.nio.ch.Net.setIntOption(Net.java:152)
>>>         at 
>>> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
>>>         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>>         at 
>>> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
>>>         at 
>>> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
>>>         at 
>>> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
>>>         at 
>>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:273) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioReceiver.java:353) 
>>>
>>>         at java.lang.Thread.run(Thread.java:613)
>>> Jul 10, 2008 1:06:42 PM 
>>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>>> INFO: Done sleeping, membership established, start level:8
>>> Jul 10, 2008 1:06:42 PM 
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
>>> INFO: Initializing AbstractReplicatedMap with context name:
>>> Jul 10, 2008 1:06:42 PM org.apache.catalina.tribes.io.BufferPool 
>>> getBufferPool
>>> INFO: Created a buffer pool with max size:104857600 bytes of 
>>> type:org.apache.catalina.tribes.io.BufferPool15Impl
>>> Jul 10, 2008 1:06:42 PM 
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>>> WARNING: Member send is failing for:tcp://{10, 84, 2, 42}:4000 ; 
>>> Setting to suspect and retrying.
>>> Jul 10, 2008 1:06:42 PM 
>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector 
>>> memberDisappeared
>>> INFO: Received 
>>> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 
>>> 84, 2, 42}:4000,{10, 84, 2, 42},4000, alive=197662,id={-25 34 -32 
>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, 
>>> command={}, domain={}, ]] message. Will verify.
>>> Jul 10, 2008 1:06:42 PM 
>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector 
>>> memberDisappeared
>>> INFO: Verification complete. Member still 
>>> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 
>>> 84, 2, 42}:4000,{10, 84, 2, 42},4000, alive=197662,id={-25 34 -32 
>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, 
>>> command={}, domain={}, ]]
>>> Jul 10, 2008 1:06:42 PM 
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
>>> WARNING: Unable to send map start message.
>>> Jul 10, 2008 1:06:42 PM 
>>> org.apache.catalina.ha.context.ReplicatedContext start
>>> SEVERE: Unable to start ReplicatedContext
>>> java.lang.RuntimeException: Unable to start replicated map.
>>>         at 
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.init(AbstractReplicatedMap.java:230) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.<init>(AbstractReplicatedMap.java:171) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.tipis.ReplicatedMap.<init>(ReplicatedMap.java:90) 
>>>
>>>         at 
>>> org.apache.catalina.ha.context.ReplicatedContext.start(ReplicatedContext.java:64) 
>>>
>>>         at 
>>> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
>>>         at 
>>> org.apache.catalina.core.StandardHost.start(StandardHost.java:719)
>>>         at 
>>> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
>>>         at 
>>> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443)
>>>         at 
>>> org.apache.catalina.core.StandardService.start(StandardService.java:516) 
>>>
>>>         at 
>>> org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
>>>         at 
>>> org.apache.catalina.startup.Catalina.start(Catalina.java:578)
>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>         at 
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
>>>
>>>         at 
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
>>>
>>>         at java.lang.reflect.Method.invoke(Method.java:585)
>>>         at 
>>> org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
>>>         at 
>>> org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
>>> Caused by: org.apache.catalina.tribes.ChannelException: Send failed, 
>>> attempt:2 max:1; Faulty members:tcp://{10, 84, 2, 42}:4000;
>>>         at 
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:172) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:78) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:73) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:89)
>>>         at 
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.broadcast(AbstractReplicatedMap.java:300) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.init(AbstractReplicatedMap.java:223) 
>>>
>>>         ... 16 more
>>> Caused by: java.net.SocketException: Invalid argument
>>>         at sun.nio.ch.Net.setIntOption0(Native Method)
>>>         at sun.nio.ch.Net.setIntOption(Net.java:152)
>>>         at 
>>> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
>>>         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>>         at 
>>> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
>>>         at 
>>> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
>>>         at 
>>> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
>>>         at 
>>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect(NioSender.java:147) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.transport.nio.NioSender.process(NioSender.java:89) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:130) 
>>>
>>>         ... 30 more
>>> Jul 10, 2008 1:06:42 PM org.apache.catalina.startup.Catalina start
>>> SEVERE: Catalina.start:
>>> LifecycleException:  Failed to start ReplicatedContext:  
>>> java.lang.RuntimeException: Unable to start replicated map.
>>>         at 
>>> org.apache.catalina.ha.context.ReplicatedContext.start(ReplicatedContext.java:73) 
>>>
>>>         at 
>>> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
>>>         at 
>>> org.apache.catalina.core.StandardHost.start(StandardHost.java:719)
>>>         at 
>>> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
>>>         at 
>>> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443)
>>>         at 
>>> org.apache.catalina.core.StandardService.start(StandardService.java:516) 
>>>
>>>         at 
>>> org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
>>>         at 
>>> org.apache.catalina.startup.Catalina.start(Catalina.java:578)
>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>         at 
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
>>>
>>>         at 
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
>>>
>>>         at java.lang.reflect.Method.invoke(Method.java:585)
>>>         at 
>>> org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
>>>         at 
>>> org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
>>> Jul 10, 2008 1:06:42 PM org.apache.catalina.startup.Catalina start
>>> INFO: Server startup in 2329 ms
>>> Jul 10, 2008 1:06:47 PM 
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>>> WARNING: Not retrying send for:tcp://{10, 84, 2, 42}:4000; Sender is 
>>> disconnected.
>>> Jul 10, 2008 1:06:47 PM 
>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector 
>>> memberDisappeared
>>> INFO: Received 
>>> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 
>>> 84, 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32 
>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, 
>>> command={}, domain={}, ]] message. Will verify.
>>> Jul 10, 2008 1:06:47 PM 
>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector 
>>> memberDisappeared
>>> INFO: Verification complete. Member still 
>>> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 
>>> 84, 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32 
>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, 
>>> command={}, domain={}, ]]
>>> Jul 10, 2008 1:06:47 PM 
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
>>> SEVERE: Unable to send AbstractReplicatedMap.ping message
>>> org.apache.catalina.tribes.ChannelException: Send failed, and sender 
>>> is disconnected. Not retrying.; Faulty members:tcp://{10, 84, 2, 
>>> 42}:4000;
>>>         at 
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:156) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:78) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:73) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:89)
>>>         at 
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:253) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:793) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:153) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:661) 
>>>
>>> Caused by: java.net.SocketException: Invalid argument
>>>         at sun.nio.ch.Net.setIntOption0(Native Method)
>>>         at sun.nio.ch.Net.setIntOption(Net.java:152)
>>>         at 
>>> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
>>>         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>>         at 
>>> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
>>>         at 
>>> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
>>>         at 
>>> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
>>>         at 
>>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect(NioSender.java:147) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.transport.nio.NioSender.process(NioSender.java:89) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:130) 
>>>
>>>         ... 16 more
>>> Jul 10, 2008 1:06:47 PM 
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>>> WARNING: Not retrying send for:tcp://{10, 84, 2, 42}:4000; Sender is 
>>> disconnected.
>>> Jul 10, 2008 1:06:47 PM 
>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector 
>>> memberDisappeared
>>> INFO: Received 
>>> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 
>>> 84, 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32 
>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, 
>>> command={}, domain={}, ]] message. Will verify.
>>> Jul 10, 2008 1:06:47 PM 
>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector 
>>> memberDisappeared
>>> INFO: Verification complete. Member still 
>>> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 
>>> 84, 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32 
>>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, 
>>> command={}, domain={}, ]]
>>> Jul 10, 2008 1:06:47 PM 
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
>>> SEVERE: Unable to send AbstractReplicatedMap.ping message
>>> org.apache.catalina.tribes.ChannelException: Send failed, and sender 
>>> is disconnected. Not retrying.; Faulty members:tcp://{10, 84, 2, 
>>> 42}:4000;
>>>         at 
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:156) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:78) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:73) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:89)
>>>         at 
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:253) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:793) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:158) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:661) 
>>>
>>> Caused by: java.net.SocketException: Invalid argument
>>>         at sun.nio.ch.Net.setIntOption0(Native Method)
>>>         at sun.nio.ch.Net.setIntOption(Net.java:152)
>>>         at 
>>> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
>>>         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>>         at 
>>> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
>>>         at 
>>> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
>>>         at 
>>> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
>>>         at 
>>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect(NioSender.java:147) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.transport.nio.NioSender.process(NioSender.java:89) 
>>>
>>>         at 
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:130) 
>>>
>>>         ... 16 more
>>>
>>>
>>> and so on.  Top shows that both Java process is Up to 86%, the 
>>> machine is never idle and all memory is consumed.  The machine is a 
>>> PowerPc G6 from Apple with OSX 10.4.11.
>>> Tomcat version is: Apache Tomcat/6.0.16
>>> JVM version is: 1.5.0_13-b05-241
>>> I suspect the heartbeat pings have a problem, I just do not know 
>>> what to do about them.
>>>
>>> Thanks ahead,
>>>
>>> János
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To start a new topic, e-mail: users@tomcat.apache.org
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To start a new topic, e-mail: users@tomcat.apache.org
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>


---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: Version 6 Clustering problem

Posted by János <ja...@yale.edu>.
Hi Filip,

I added it in the catalina.sh file on all three nodes and I am not  
seeing the error.  I also disabled IP6 in Network Preferences.

I added the distrubuted="true" to the context.xml file on all three  
nodes and added the <distributable/> tag to the webapps/examples/WEB- 
INF/web.xml on all 3 nodes and tried the SessionExample servlet

http://bml0042.yalepath.org:8080/examples/servlets/servlet/ 
SessionExample

to see if I have a working cluster.  I am following the book at

http://proquest.safaribooksonline.com:80/9780596101060/ 
tomcat_6_clustering_implementation

and it says that I should see messages similar to this one in the  
nodes' logs"

Sep 27, 2008 5:39:38 PM org.apache.catalina.core.ApplicationContext log
INFO: SessionListener: contextInitialized(  )
Sep 27, 2008 5:40:07 PM org.apache.catalina.core.ApplicationContext log
INFO: SessionListener: sessionCreated 
('F4B2D7191C1F335FFAAC93DA461CA95F.tc1')
Sep 27, 2008 5:40:19 PM org.apache.catalina.core.ApplicationContext log
INFO: SessionListener: attributeAdded 
('F4B2D7191C1F335FFAAC93DA461CA95F.tc1', 'test',


But I am not seeing them.  This is the resent log from node1:

Jul 11, 2008 3:28:01 PM org.apache.catalina.core.AprLifecycleListener  
init
INFO: The APR based Apache Tomcat Native library which allows optimal  
performance in production environments was not found on the  
java.library.path: .:/Library/Java/Extensions:/System/Library/Java/ 
Extensions:/usr/lib/java
Jul 11, 2008 3:28:01 PM org.apache.coyote.http11.Http11Protocol init
INFO: Initializing Coyote HTTP/1.1 on http-8080
Jul 11, 2008 3:28:01 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 1098 ms
Jul 11, 2008 3:28:01 PM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
Jul 11, 2008 3:28:01 PM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.16
Jul 11, 2008 3:28:01 PM org.apache.catalina.ha.tcp.SimpleTcpCluster  
start
INFO: Cluster is about to start
Jul 11, 2008 3:28:01 PM  
org.apache.catalina.tribes.transport.ReceiverBase bind
INFO: Receiver Server Socket bound to:/10.84.2.42:4000
Jul 11, 2008 3:28:01 PM  
org.apache.catalina.tribes.membership.McastServiceImpl setupSocket
INFO: Setting cluster mcast soTimeout to 500
Jul 11, 2008 3:28:01 PM  
org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
INFO: Sleeping for 1000 milliseconds to establish cluster membership,  
start level:4
Jul 11, 2008 3:28:02 PM  
org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
INFO: Done sleeping, membership established, start level:4
Jul 11, 2008 3:28:02 PM  
org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
INFO: Sleeping for 1000 milliseconds to establish cluster membership,  
start level:8
Jul 11, 2008 3:28:03 PM  
org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
INFO: Done sleeping, membership established, start level:8
Jul 11, 2008 3:28:03 PM  
org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
INFO: Initializing AbstractReplicatedMap with context name:
Jul 11, 2008 3:28:04 PM org.apache.catalina.core.StandardContext  
addApplicationListener
INFO: The listener "listeners.ContextListener" is already configured  
for this context. The duplicate definition has been ignored.
Jul 11, 2008 3:28:04 PM org.apache.catalina.core.StandardContext  
addApplicationListener
INFO: The listener "listeners.SessionListener" is already configured  
for this context. The duplicate definition has been ignored.
Jul 11, 2008 3:28:04 PM org.apache.catalina.ha.session.DeltaManager  
start
INFO: Register manager /examples to cluster element Engine with name  
Catalina
Jul 11, 2008 3:28:04 PM org.apache.catalina.ha.session.DeltaManager  
start
INFO: Starting clustering manager at /examples
Jul 11, 2008 3:28:04 PM org.apache.catalina.ha.session.DeltaManager  
getAllClusterSessions
INFO: Manager [localhost#/examples]: skipping state transfer. No  
members active in cluster group.
Jul 11, 2008 3:28:04 PM  
org.apache.catalina.ha.session.JvmRouteBinderValve start
INFO: JvmRouteBinderValve started
Jul 11, 2008 3:28:04 PM org.apache.coyote.http11.Http11Protocol start
INFO: Starting Coyote HTTP/1.1 on http-8080
Jul 11, 2008 3:28:05 PM org.apache.jk.common.ChannelSocket init
INFO: JK: ajp13 listening on /0.0.0.0:8009
Jul 11, 2008 3:28:05 PM org.apache.jk.server.JkMain start
INFO: Jk running ID=0 time=0/45  config=null
Jul 11, 2008 3:28:05 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 3601 ms
Jul 11, 2008 3:28:37 PM org.apache.catalina.tribes.io.BufferPool  
getBufferPool
INFO: Created a buffer pool with max size:104857600 bytes of  
type:org.apache.catalina.tribes.io.BufferPool15Impl
Jul 11, 2008 3:28:37 PM org.apache.catalina.ha.tcp.SimpleTcpCluster  
memberAdded
INFO: Replication member  
added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,  
2, 42}:4001,{10, 84, 2, 42},4001, alive=1020,id={60 103 -111 -34 98  
62 71 -116 -104 76 -100 115 -77 15 -52 -92 }, payload={}, command={},  
domain={}, ]
Jul 11, 2008 3:29:08 PM org.apache.catalina.ha.tcp.SimpleTcpCluster  
memberAdded
INFO: Replication member  
added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,  
2, 42}:4002,{10, 84, 2, 42},4002, alive=1019,id={27 24 -73 72 48 -6  
70 32 -90 57 51 106 -94 -54 126 -52 }, payload={}, command={}, domain= 
{}, ]



Similarly on the other two nodes.  What am I missing ??  Should I add  
the <Context distributed=True" /> to the server.xml files ?

Thanks ahead,

János
On Jul 10, 2008, at 6:03 PM, Filip Hanik - Dev Lists wrote:

> try setting the
> |-Djava.net.preferIPv4Stack=true
>
> option for your tomcat launch
>
> Filip
> |
> János wrote:
>> Hi,
>>
>> I am trying to establish a 3 member cluster on the same machine.   
>> Here is my server.xml copied from a sample I found on this list.
>>
>> <?xml version='1.0' encoding='utf-8'?>
>> <Server port="8005" shutdown="SHUTDOWN">
>>   <Listener  
>> className="org.apache.catalina.core.AprLifecycleListener"  
>> SSLEngine="on" />
>>   <Listener className="org.apache.catalina.core.JasperListener" />
>>   <Listener  
>> className="org.apache.catalina.mbeans.ServerLifecycleListener" />
>>   <Listener  
>> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListene 
>> r" />
>>
>>   <GlobalNamingResources>
>>     <Resource name="UserDatabase" auth="Container"
>>               type="org.apache.catalina.UserDatabase"
>>               description="User database that can be updated and  
>> saved"
>>                
>> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
>>               pathname="conf/tomcat-users.xml" />
>>   </GlobalNamingResources>
>>
>>   <Service name="Catalina">
>>
>>     <Connector port="8080" protocol="HTTP/1.1"
>>                connectionTimeout="20000"
>>                redirectPort="8443" />
>>     <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
>>
>>     <Engine name="Catalina" defaultHost="localhost" jvmRoute="node1">
>>
>>       <Cluster  
>> className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
>>       <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
>>              resourceName="UserDatabase"/>
>>
>>       <Host name="localhost"  appBase="webapps"
>>             unpackWARs="true" autoDeploy="true"
>>             xmlValidation="false" xmlNamespaceAware="false">
>>         <Context path="" docBase="ROOT"  
>> className="org.apache.catalina.ha.context.ReplicatedContext"/>
>>       </Host>
>>     </Engine>
>>   </Service>
>> </Server>
>>
>> The other two server.xml-s are the same, except the shutdown ports  
>> are  8105, 8205, http ports are 8180, 8280 and https ports are  
>> 8543, 8643.
>> As soon as I am starting the second instance I receive the  
>> following into the logs.
>>
>> In node1 logs:
>>
>> bml0042:/usr/local/tomcat/node1/logs janos$ cat catalina. 
>> 2008-07-10.log
>> Jul 10, 2008 1:03:24 PM  
>> org.apache.catalina.core.AprLifecycleListener init
>> INFO: The APR based Apache Tomcat Native library which allows  
>> optimal performance in production environments was not found on  
>> the java.library.path: .:/Library/Java/Extensions:/System/Library/ 
>> Java/Extensions:/usr/lib/java
>> Jul 10, 2008 1:03:24 PM org.apache.coyote.http11.Http11Protocol init
>> INFO: Initializing Coyote HTTP/1.1 on http-8080
>> Jul 10, 2008 1:03:24 PM org.apache.catalina.startup.Catalina load
>> INFO: Initialization processed in 1187 ms
>> Jul 10, 2008 1:03:24 PM org.apache.catalina.core.StandardService  
>> start
>> INFO: Starting service Catalina
>> Jul 10, 2008 1:03:24 PM org.apache.catalina.core.StandardEngine start
>> INFO: Starting Servlet Engine: Apache Tomcat/6.0.16
>> Jul 10, 2008 1:03:24 PM  
>> org.apache.catalina.ha.tcp.SimpleTcpCluster start
>> INFO: Cluster is about to start
>> Jul 10, 2008 1:03:24 PM  
>> org.apache.catalina.tribes.transport.ReceiverBase bind
>> INFO: Receiver Server Socket bound to:/10.84.2.42:4000
>> Jul 10, 2008 1:03:24 PM  
>> org.apache.catalina.tribes.membership.McastServiceImpl setupSocket
>> INFO: Setting cluster mcast soTimeout to 500
>> Jul 10, 2008 1:03:24 PM  
>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>> INFO: Sleeping for 1000 milliseconds to establish cluster  
>> membership, start level:4
>> Jul 10, 2008 1:03:25 PM  
>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>> INFO: Done sleeping, membership established, start level:4
>> Jul 10, 2008 1:03:25 PM  
>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>> INFO: Sleeping for 1000 milliseconds to establish cluster  
>> membership, start level:8
>> Jul 10, 2008 1:03:26 PM  
>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>> INFO: Done sleeping, membership established, start level:8
>> Jul 10, 2008 1:03:26 PM  
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
>> INFO: Initializing AbstractReplicatedMap with context name:
>> Jul 10, 2008 1:03:27 PM org.apache.catalina.core.StandardContext  
>> addApplicationListener
>> INFO: The listener "listeners.ContextListener" is already  
>> configured for this context. The duplicate definition has been  
>> ignored.
>> Jul 10, 2008 1:03:27 PM org.apache.catalina.core.StandardContext  
>> addApplicationListener
>> INFO: The listener "listeners.SessionListener" is already  
>> configured for this context. The duplicate definition has been  
>> ignored.
>> Jul 10, 2008 1:03:27 PM  
>> org.apache.catalina.ha.session.DeltaManager start
>> INFO: Register manager /examples to cluster element Engine with  
>> name Catalina
>> Jul 10, 2008 1:03:27 PM  
>> org.apache.catalina.ha.session.DeltaManager start
>> INFO: Starting clustering manager at /examples
>> Jul 10, 2008 1:03:27 PM  
>> org.apache.catalina.ha.session.DeltaManager getAllClusterSessions
>> INFO: Manager [localhost#/examples]: skipping state transfer. No  
>> members active in cluster group.
>> Jul 10, 2008 1:03:28 PM  
>> org.apache.catalina.ha.session.JvmRouteBinderValve start
>> INFO: JvmRouteBinderValve started
>> Jul 10, 2008 1:03:28 PM org.apache.coyote.http11.Http11Protocol start
>> INFO: Starting Coyote HTTP/1.1 on http-8080
>> Jul 10, 2008 1:03:28 PM org.apache.jk.common.ChannelSocket init
>> INFO: JK: ajp13 listening on /0.0.0.0:8009
>> Jul 10, 2008 1:03:28 PM org.apache.jk.server.JkMain start
>> INFO: Jk running ID=0 time=0/45  config=null
>> Jul 10, 2008 1:03:28 PM org.apache.catalina.startup.Catalina start
>> INFO: Server startup in 3616 ms
>> Jul 10, 2008 1:06:41 PM  
>> org.apache.catalina.tribes.transport.nio.NioReceiver listen
>> SEVERE: Unable to process request in NioReceiver
>> java.net.SocketException: Invalid argument
>>         at sun.nio.ch.Net.setIntOption0(Native Method)
>>         at sun.nio.ch.Net.setIntOption(Net.java:152)
>>         at sun.nio.ch.SocketChannelImpl$1.setInt 
>> (SocketChannelImpl.java:372)
>>         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>         at sun.nio.ch.SocketOptsImpl$IP.typeOfService 
>> (SocketOptsImpl.java:249)
>>         at sun.nio.ch.OptionAdaptor.setTrafficClass 
>> (OptionAdaptor.java:158)
>>         at sun.nio.ch.SocketAdaptor.setTrafficClass 
>> (SocketAdaptor.java:330)
>>         at  
>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen 
>> (NioReceiver.java:273)
>>         at org.apache.catalina.tribes.transport.nio.NioReceiver.run 
>> (NioReceiver.java:353)
>>         at java.lang.Thread.run(Thread.java:613)
>> Jul 10, 2008 1:06:41 PM  
>> org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
>> INFO: Replication member  
>> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,  
>> 84, 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46  
>> 127 -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={},  
>> command={}, domain={}, ]
>> Jul 10, 2008 1:06:41 PM org.apache.catalina.tribes.io.BufferPool  
>> getBufferPool
>> INFO: Created a buffer pool with max size:104857600 bytes of  
>> type:org.apache.catalina.tribes.io.BufferPool15Impl
>> Jul 10, 2008 1:06:41 PM  
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>> WARNING: Member send is failing for:tcp://{10, 84, 2, 42}:4001 ;  
>> Setting to suspect and retrying.
>> Jul 10, 2008 1:06:41 PM  
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector  
>> memberDisappeared
>> INFO: Received memberDisappeared 
>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,  
>> 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46 127  
>> -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={}, command= 
>> {}, domain={}, ]] message. Will verify.
>> Jul 10, 2008 1:06:41 PM  
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector  
>> memberDisappeared
>> INFO: Verification complete. Member still alive 
>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,  
>> 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46 127  
>> -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={}, command= 
>> {}, domain={}, ]]
>> Jul 10, 2008 1:06:41 PM  
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
>> SEVERE: Unable to send AbstractReplicatedMap.ping message
>> org.apache.catalina.tribes.ChannelException: Send failed, attempt: 
>> 2 max:1; Faulty members:tcp://{10, 84, 2, 42}:4001;
>>         at  
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop 
>> (ParallelNioSender.java:172)
>>         at  
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessag 
>> e(ParallelNioSender.java:78)
>>         at  
>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMes 
>> sage(PooledParallelSender.java:53)
>>         at  
>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessa 
>> ge(ReplicationTransmitter.java:80)
>>         at  
>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage 
>> (ChannelCoordinator.java:78)
>>         at  
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
>> (ChannelInterceptorBase.java:75)
>>         at  
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.send 
>> Message(TcpFailureDetector.java:87)
>>         at  
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
>> (ChannelInterceptorBase.java:75)
>>         at  
>> org.apache.catalina.tribes.group.interceptors.MessageDispatchIntercep 
>> tor.sendMessage(MessageDispatchInterceptor.java:73)
>>         at  
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
>> (ChannelInterceptorBase.java:75)
>>         at org.apache.catalina.tribes.group.GroupChannel.send 
>> (GroupChannel.java:216)
>>         at org.apache.catalina.tribes.group.GroupChannel.send 
>> (GroupChannel.java:175)
>>         at org.apache.catalina.tribes.group.RpcChannel.send 
>> (RpcChannel.java:89)
>>         at  
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping 
>> (AbstractReplicatedMap.java:253)
>>         at  
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat 
>> (AbstractReplicatedMap.java:793)
>>         at org.apache.catalina.tribes.group.GroupChannel.heartbeat 
>> (GroupChannel.java:153)
>>         at org.apache.catalina.tribes.group.GroupChannel 
>> $HeartbeatThread.run(GroupChannel.java:661)
>> Caused by: java.net.SocketException: Invalid argument
>>         at sun.nio.ch.Net.setIntOption0(Native Method)
>>         at sun.nio.ch.Net.setIntOption(Net.java:152)
>>         at sun.nio.ch.SocketChannelImpl$1.setInt 
>> (SocketChannelImpl.java:372)
>>         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>         at sun.nio.ch.SocketOptsImpl$IP.typeOfService 
>> (SocketOptsImpl.java:249)
>>         at sun.nio.ch.OptionAdaptor.setTrafficClass 
>> (OptionAdaptor.java:158)
>>         at sun.nio.ch.SocketAdaptor.setTrafficClass 
>> (SocketAdaptor.java:330)
>>         at  
>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect 
>> (NioSender.java:147)
>>         at  
>> org.apache.catalina.tribes.transport.nio.NioSender.process 
>> (NioSender.java:89)
>>         at  
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop 
>> (ParallelNioSender.java:130)
>>         ... 16 more
>> Jul 10, 2008 1:06:41 PM  
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>> WARNING: Not retrying send for:tcp://{10, 84, 2, 42}:4001; Sender  
>> is disconnected.
>> Jul 10, 2008 1:06:41 PM  
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector  
>> memberDisappeared
>> INFO: Received memberDisappeared 
>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,  
>> 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46 127  
>> -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={}, command= 
>> {}, domain={}, ]] message. Will verify.
>> Jul 10, 2008 1:06:42 PM  
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector  
>> memberDisappeared
>> INFO: Verification complete. Member still alive 
>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,  
>> 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46 127  
>> -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={}, command= 
>> {}, domain={}, ]]
>> Jul 10, 2008 1:06:42 PM  
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
>> SEVERE: Unable to send AbstractReplicatedMap.ping message
>> org.apache.catalina.tribes.ChannelException: Send failed, and  
>> sender is disconnected. Not retrying.; Faulty members:tcp://{10,  
>> 84, 2, 42}:4001;
>>         at  
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop 
>> (ParallelNioSender.java:156)
>>         at  
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessag 
>> e(ParallelNioSender.java:78)
>>         at  
>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMes 
>> sage(PooledParallelSender.java:53)
>>         at  
>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessa 
>> ge(ReplicationTransmitter.java:80)
>>         at  
>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage 
>> (ChannelCoordinator.java:78)
>>         at  
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
>> (ChannelInterceptorBase.java:75)
>>         at  
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.send 
>> Message(TcpFailureDetector.java:87)
>>         at  
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
>> (ChannelInterceptorBase.java:75)
>>         at  
>> org.apache.catalina.tribes.group.interceptors.MessageDispatchIntercep 
>> tor.sendMessage(MessageDispatchInterceptor.java:73)
>>         at  
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
>> (ChannelInterceptorBase.java:75)
>>         at org.apache.catalina.tribes.group.GroupChannel.send 
>> (GroupChannel.java:216)
>>         at org.apache.catalina.tribes.group.GroupChannel.send 
>> (GroupChannel.java:175)
>>         at org.apache.catalina.tribes.group.RpcChannel.send 
>> (RpcChannel.java:89)
>>         at  
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping 
>> (AbstractReplicatedMap.java:253)
>>         at  
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat 
>> (AbstractReplicatedMap.java:793)
>>         at org.apache.catalina.tribes.group.GroupChannel.heartbeat 
>> (GroupChannel.java:158)
>>         at org.apache.catalina.tribes.group.GroupChannel 
>> $HeartbeatThread.run(GroupChannel.java:661)
>> Caused by: java.net.SocketException: Invalid argument
>>         at sun.nio.ch.Net.setIntOption0(Native Method)
>>         at sun.nio.ch.Net.setIntOption(Net.java:152)
>>         at sun.nio.ch.SocketChannelImpl$1.setInt 
>> (SocketChannelImpl.java:372)
>>         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>         at sun.nio.ch.SocketOptsImpl$IP.typeOfService 
>> (SocketOptsImpl.java:249)
>>         at sun.nio.ch.OptionAdaptor.setTrafficClass 
>> (OptionAdaptor.java:158)
>>         at sun.nio.ch.SocketAdaptor.setTrafficClass 
>> (SocketAdaptor.java:330)
>>         at  
>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect 
>> (NioSender.java:147)
>>         at  
>> org.apache.catalina.tribes.transport.nio.NioSender.process 
>> (NioSender.java:89)
>>         at  
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop 
>> (ParallelNioSender.java:130)
>>         ... 16 more
>>
>>
>> and so on forever.
>>
>> In node2 logs I see the following:
>>
>> bml0042:/usr/local/tomcat/node2/logs janos$ cat catalina. 
>> 2008-07-10.log
>> Jul 10, 2008 1:06:40 PM  
>> org.apache.catalina.core.AprLifecycleListener init
>> INFO: The APR based Apache Tomcat Native library which allows  
>> optimal performance in production environments was not found on  
>> the java.library.path: .:/Library/Java/Extensions:/System/Library/ 
>> Java/Extensions:/usr/lib/java
>> Jul 10, 2008 1:06:40 PM org.apache.coyote.http11.Http11Protocol init
>> INFO: Initializing Coyote HTTP/1.1 on http-8180
>> Jul 10, 2008 1:06:40 PM org.apache.catalina.startup.Catalina load
>> INFO: Initialization processed in 1148 ms
>> Jul 10, 2008 1:06:40 PM org.apache.catalina.core.StandardService  
>> start
>> INFO: Starting service Catalina
>> Jul 10, 2008 1:06:40 PM org.apache.catalina.core.StandardEngine start
>> INFO: Starting Servlet Engine: Apache Tomcat/6.0.16
>> Jul 10, 2008 1:06:40 PM  
>> org.apache.catalina.ha.tcp.SimpleTcpCluster start
>> INFO: Cluster is about to start
>> Jul 10, 2008 1:06:40 PM  
>> org.apache.catalina.tribes.transport.ReceiverBase bind
>> INFO: Receiver Server Socket bound to:/10.84.2.42:4001
>> Jul 10, 2008 1:06:40 PM  
>> org.apache.catalina.tribes.membership.McastServiceImpl setupSocket
>> INFO: Setting cluster mcast soTimeout to 500
>> Jul 10, 2008 1:06:40 PM  
>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>> INFO: Sleeping for 1000 milliseconds to establish cluster  
>> membership, start level:4
>> Jul 10, 2008 1:06:41 PM  
>> org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
>> INFO: Replication member  
>> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,  
>> 84, 2, 42}:4000,{10, 84, 2, 42},4000, alive=196161,id={-25 34 -32  
>> 116 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={},  
>> command={}, domain={}, ]
>> Jul 10, 2008 1:06:41 PM  
>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>> INFO: Done sleeping, membership established, start level:4
>> Jul 10, 2008 1:06:41 PM  
>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>> INFO: Sleeping for 1000 milliseconds to establish cluster  
>> membership, start level:8
>> Jul 10, 2008 1:06:41 PM  
>> org.apache.catalina.tribes.transport.nio.NioReceiver listen
>> SEVERE: Unable to process request in NioReceiver
>> java.net.SocketException: Invalid argument
>>         at sun.nio.ch.Net.setIntOption0(Native Method)
>>         at sun.nio.ch.Net.setIntOption(Net.java:152)
>>         at sun.nio.ch.SocketChannelImpl$1.setInt 
>> (SocketChannelImpl.java:372)
>>         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>         at sun.nio.ch.SocketOptsImpl$IP.typeOfService 
>> (SocketOptsImpl.java:249)
>>         at sun.nio.ch.OptionAdaptor.setTrafficClass 
>> (OptionAdaptor.java:158)
>>         at sun.nio.ch.SocketAdaptor.setTrafficClass 
>> (SocketAdaptor.java:330)
>>         at  
>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen 
>> (NioReceiver.java:273)
>>         at org.apache.catalina.tribes.transport.nio.NioReceiver.run 
>> (NioReceiver.java:353)
>>         at java.lang.Thread.run(Thread.java:613)
>> Jul 10, 2008 1:06:42 PM  
>> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
>> INFO: Done sleeping, membership established, start level:8
>> Jul 10, 2008 1:06:42 PM  
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
>> INFO: Initializing AbstractReplicatedMap with context name:
>> Jul 10, 2008 1:06:42 PM org.apache.catalina.tribes.io.BufferPool  
>> getBufferPool
>> INFO: Created a buffer pool with max size:104857600 bytes of  
>> type:org.apache.catalina.tribes.io.BufferPool15Impl
>> Jul 10, 2008 1:06:42 PM  
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>> WARNING: Member send is failing for:tcp://{10, 84, 2, 42}:4000 ;  
>> Setting to suspect and retrying.
>> Jul 10, 2008 1:06:42 PM  
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector  
>> memberDisappeared
>> INFO: Received memberDisappeared 
>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,  
>> 2, 42}:4000,{10, 84, 2, 42},4000, alive=197662,id={-25 34 -32 116  
>> 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command= 
>> {}, domain={}, ]] message. Will verify.
>> Jul 10, 2008 1:06:42 PM  
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector  
>> memberDisappeared
>> INFO: Verification complete. Member still alive 
>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,  
>> 2, 42}:4000,{10, 84, 2, 42},4000, alive=197662,id={-25 34 -32 116  
>> 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command= 
>> {}, domain={}, ]]
>> Jul 10, 2008 1:06:42 PM  
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
>> WARNING: Unable to send map start message.
>> Jul 10, 2008 1:06:42 PM  
>> org.apache.catalina.ha.context.ReplicatedContext start
>> SEVERE: Unable to start ReplicatedContext
>> java.lang.RuntimeException: Unable to start replicated map.
>>         at  
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.init 
>> (AbstractReplicatedMap.java:230)
>>         at  
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.<init> 
>> (AbstractReplicatedMap.java:171)
>>         at org.apache.catalina.tribes.tipis.ReplicatedMap.<init> 
>> (ReplicatedMap.java:90)
>>         at org.apache.catalina.ha.context.ReplicatedContext.start 
>> (ReplicatedContext.java:64)
>>         at org.apache.catalina.core.ContainerBase.start 
>> (ContainerBase.java:1045)
>>         at org.apache.catalina.core.StandardHost.start 
>> (StandardHost.java:719)
>>         at org.apache.catalina.core.ContainerBase.start 
>> (ContainerBase.java:1045)
>>         at org.apache.catalina.core.StandardEngine.start 
>> (StandardEngine.java:443)
>>         at org.apache.catalina.core.StandardService.start 
>> (StandardService.java:516)
>>         at org.apache.catalina.core.StandardServer.start 
>> (StandardServer.java:710)
>>         at org.apache.catalina.startup.Catalina.start 
>> (Catalina.java:578)
>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native  
>> Method)
>>         at sun.reflect.NativeMethodAccessorImpl.invoke 
>> (NativeMethodAccessorImpl.java:39)
>>         at sun.reflect.DelegatingMethodAccessorImpl.invoke 
>> (DelegatingMethodAccessorImpl.java:25)
>>         at java.lang.reflect.Method.invoke(Method.java:585)
>>         at org.apache.catalina.startup.Bootstrap.start 
>> (Bootstrap.java:288)
>>         at org.apache.catalina.startup.Bootstrap.main 
>> (Bootstrap.java:413)
>> Caused by: org.apache.catalina.tribes.ChannelException: Send  
>> failed, attempt:2 max:1; Faulty members:tcp://{10, 84, 2, 42}:4000;
>>         at  
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop 
>> (ParallelNioSender.java:172)
>>         at  
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessag 
>> e(ParallelNioSender.java:78)
>>         at  
>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMes 
>> sage(PooledParallelSender.java:53)
>>         at  
>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessa 
>> ge(ReplicationTransmitter.java:80)
>>         at  
>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage 
>> (ChannelCoordinator.java:78)
>>         at  
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
>> (ChannelInterceptorBase.java:75)
>>         at  
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.send 
>> Message(TcpFailureDetector.java:87)
>>         at  
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
>> (ChannelInterceptorBase.java:75)
>>         at  
>> org.apache.catalina.tribes.group.interceptors.MessageDispatchIntercep 
>> tor.sendMessage(MessageDispatchInterceptor.java:73)
>>         at  
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
>> (ChannelInterceptorBase.java:75)
>>         at org.apache.catalina.tribes.group.GroupChannel.send 
>> (GroupChannel.java:216)
>>         at org.apache.catalina.tribes.group.GroupChannel.send 
>> (GroupChannel.java:175)
>>         at org.apache.catalina.tribes.group.RpcChannel.send 
>> (RpcChannel.java:89)
>>         at  
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.broadcast 
>> (AbstractReplicatedMap.java:300)
>>         at  
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.init 
>> (AbstractReplicatedMap.java:223)
>>         ... 16 more
>> Caused by: java.net.SocketException: Invalid argument
>>         at sun.nio.ch.Net.setIntOption0(Native Method)
>>         at sun.nio.ch.Net.setIntOption(Net.java:152)
>>         at sun.nio.ch.SocketChannelImpl$1.setInt 
>> (SocketChannelImpl.java:372)
>>         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>         at sun.nio.ch.SocketOptsImpl$IP.typeOfService 
>> (SocketOptsImpl.java:249)
>>         at sun.nio.ch.OptionAdaptor.setTrafficClass 
>> (OptionAdaptor.java:158)
>>         at sun.nio.ch.SocketAdaptor.setTrafficClass 
>> (SocketAdaptor.java:330)
>>         at  
>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect 
>> (NioSender.java:147)
>>         at  
>> org.apache.catalina.tribes.transport.nio.NioSender.process 
>> (NioSender.java:89)
>>         at  
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop 
>> (ParallelNioSender.java:130)
>>         ... 30 more
>> Jul 10, 2008 1:06:42 PM org.apache.catalina.startup.Catalina start
>> SEVERE: Catalina.start:
>> LifecycleException:  Failed to start ReplicatedContext:   
>> java.lang.RuntimeException: Unable to start replicated map.
>>         at org.apache.catalina.ha.context.ReplicatedContext.start 
>> (ReplicatedContext.java:73)
>>         at org.apache.catalina.core.ContainerBase.start 
>> (ContainerBase.java:1045)
>>         at org.apache.catalina.core.StandardHost.start 
>> (StandardHost.java:719)
>>         at org.apache.catalina.core.ContainerBase.start 
>> (ContainerBase.java:1045)
>>         at org.apache.catalina.core.StandardEngine.start 
>> (StandardEngine.java:443)
>>         at org.apache.catalina.core.StandardService.start 
>> (StandardService.java:516)
>>         at org.apache.catalina.core.StandardServer.start 
>> (StandardServer.java:710)
>>         at org.apache.catalina.startup.Catalina.start 
>> (Catalina.java:578)
>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native  
>> Method)
>>         at sun.reflect.NativeMethodAccessorImpl.invoke 
>> (NativeMethodAccessorImpl.java:39)
>>         at sun.reflect.DelegatingMethodAccessorImpl.invoke 
>> (DelegatingMethodAccessorImpl.java:25)
>>         at java.lang.reflect.Method.invoke(Method.java:585)
>>         at org.apache.catalina.startup.Bootstrap.start 
>> (Bootstrap.java:288)
>>         at org.apache.catalina.startup.Bootstrap.main 
>> (Bootstrap.java:413)
>> Jul 10, 2008 1:06:42 PM org.apache.catalina.startup.Catalina start
>> INFO: Server startup in 2329 ms
>> Jul 10, 2008 1:06:47 PM  
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>> WARNING: Not retrying send for:tcp://{10, 84, 2, 42}:4000; Sender  
>> is disconnected.
>> Jul 10, 2008 1:06:47 PM  
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector  
>> memberDisappeared
>> INFO: Received memberDisappeared 
>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,  
>> 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32 116  
>> 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command= 
>> {}, domain={}, ]] message. Will verify.
>> Jul 10, 2008 1:06:47 PM  
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector  
>> memberDisappeared
>> INFO: Verification complete. Member still alive 
>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,  
>> 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32 116  
>> 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command= 
>> {}, domain={}, ]]
>> Jul 10, 2008 1:06:47 PM  
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
>> SEVERE: Unable to send AbstractReplicatedMap.ping message
>> org.apache.catalina.tribes.ChannelException: Send failed, and  
>> sender is disconnected. Not retrying.; Faulty members:tcp://{10,  
>> 84, 2, 42}:4000;
>>         at  
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop 
>> (ParallelNioSender.java:156)
>>         at  
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessag 
>> e(ParallelNioSender.java:78)
>>         at  
>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMes 
>> sage(PooledParallelSender.java:53)
>>         at  
>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessa 
>> ge(ReplicationTransmitter.java:80)
>>         at  
>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage 
>> (ChannelCoordinator.java:78)
>>         at  
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
>> (ChannelInterceptorBase.java:75)
>>         at  
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.send 
>> Message(TcpFailureDetector.java:87)
>>         at  
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
>> (ChannelInterceptorBase.java:75)
>>         at  
>> org.apache.catalina.tribes.group.interceptors.MessageDispatchIntercep 
>> tor.sendMessage(MessageDispatchInterceptor.java:73)
>>         at  
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
>> (ChannelInterceptorBase.java:75)
>>         at org.apache.catalina.tribes.group.GroupChannel.send 
>> (GroupChannel.java:216)
>>         at org.apache.catalina.tribes.group.GroupChannel.send 
>> (GroupChannel.java:175)
>>         at org.apache.catalina.tribes.group.RpcChannel.send 
>> (RpcChannel.java:89)
>>         at  
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping 
>> (AbstractReplicatedMap.java:253)
>>         at  
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat 
>> (AbstractReplicatedMap.java:793)
>>         at org.apache.catalina.tribes.group.GroupChannel.heartbeat 
>> (GroupChannel.java:153)
>>         at org.apache.catalina.tribes.group.GroupChannel 
>> $HeartbeatThread.run(GroupChannel.java:661)
>> Caused by: java.net.SocketException: Invalid argument
>>         at sun.nio.ch.Net.setIntOption0(Native Method)
>>         at sun.nio.ch.Net.setIntOption(Net.java:152)
>>         at sun.nio.ch.SocketChannelImpl$1.setInt 
>> (SocketChannelImpl.java:372)
>>         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>         at sun.nio.ch.SocketOptsImpl$IP.typeOfService 
>> (SocketOptsImpl.java:249)
>>         at sun.nio.ch.OptionAdaptor.setTrafficClass 
>> (OptionAdaptor.java:158)
>>         at sun.nio.ch.SocketAdaptor.setTrafficClass 
>> (SocketAdaptor.java:330)
>>         at  
>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect 
>> (NioSender.java:147)
>>         at  
>> org.apache.catalina.tribes.transport.nio.NioSender.process 
>> (NioSender.java:89)
>>         at  
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop 
>> (ParallelNioSender.java:130)
>>         ... 16 more
>> Jul 10, 2008 1:06:47 PM  
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
>> WARNING: Not retrying send for:tcp://{10, 84, 2, 42}:4000; Sender  
>> is disconnected.
>> Jul 10, 2008 1:06:47 PM  
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector  
>> memberDisappeared
>> INFO: Received memberDisappeared 
>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,  
>> 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32 116  
>> 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command= 
>> {}, domain={}, ]] message. Will verify.
>> Jul 10, 2008 1:06:47 PM  
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector  
>> memberDisappeared
>> INFO: Verification complete. Member still alive 
>> [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,  
>> 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32 116  
>> 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command= 
>> {}, domain={}, ]]
>> Jul 10, 2008 1:06:47 PM  
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
>> SEVERE: Unable to send AbstractReplicatedMap.ping message
>> org.apache.catalina.tribes.ChannelException: Send failed, and  
>> sender is disconnected. Not retrying.; Faulty members:tcp://{10,  
>> 84, 2, 42}:4000;
>>         at  
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop 
>> (ParallelNioSender.java:156)
>>         at  
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessag 
>> e(ParallelNioSender.java:78)
>>         at  
>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMes 
>> sage(PooledParallelSender.java:53)
>>         at  
>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessa 
>> ge(ReplicationTransmitter.java:80)
>>         at  
>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage 
>> (ChannelCoordinator.java:78)
>>         at  
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
>> (ChannelInterceptorBase.java:75)
>>         at  
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.send 
>> Message(TcpFailureDetector.java:87)
>>         at  
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
>> (ChannelInterceptorBase.java:75)
>>         at  
>> org.apache.catalina.tribes.group.interceptors.MessageDispatchIntercep 
>> tor.sendMessage(MessageDispatchInterceptor.java:73)
>>         at  
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
>> (ChannelInterceptorBase.java:75)
>>         at org.apache.catalina.tribes.group.GroupChannel.send 
>> (GroupChannel.java:216)
>>         at org.apache.catalina.tribes.group.GroupChannel.send 
>> (GroupChannel.java:175)
>>         at org.apache.catalina.tribes.group.RpcChannel.send 
>> (RpcChannel.java:89)
>>         at  
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping 
>> (AbstractReplicatedMap.java:253)
>>         at  
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat 
>> (AbstractReplicatedMap.java:793)
>>         at org.apache.catalina.tribes.group.GroupChannel.heartbeat 
>> (GroupChannel.java:158)
>>         at org.apache.catalina.tribes.group.GroupChannel 
>> $HeartbeatThread.run(GroupChannel.java:661)
>> Caused by: java.net.SocketException: Invalid argument
>>         at sun.nio.ch.Net.setIntOption0(Native Method)
>>         at sun.nio.ch.Net.setIntOption(Net.java:152)
>>         at sun.nio.ch.SocketChannelImpl$1.setInt 
>> (SocketChannelImpl.java:372)
>>         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>>         at sun.nio.ch.SocketOptsImpl$IP.typeOfService 
>> (SocketOptsImpl.java:249)
>>         at sun.nio.ch.OptionAdaptor.setTrafficClass 
>> (OptionAdaptor.java:158)
>>         at sun.nio.ch.SocketAdaptor.setTrafficClass 
>> (SocketAdaptor.java:330)
>>         at  
>> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect 
>> (NioSender.java:147)
>>         at  
>> org.apache.catalina.tribes.transport.nio.NioSender.process 
>> (NioSender.java:89)
>>         at  
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop 
>> (ParallelNioSender.java:130)
>>         ... 16 more
>>
>>
>> and so on.  Top shows that both Java process is Up to 86%, the  
>> machine is never idle and all memory is consumed.  The machine is  
>> a PowerPc G6 from Apple with OSX 10.4.11.
>> Tomcat version is: Apache Tomcat/6.0.16
>> JVM version is: 1.5.0_13-b05-241
>> I suspect the heartbeat pings have a problem, I just do not know  
>> what to do about them.
>>
>> Thanks ahead,
>>
>> János
>>
>>
>>
>> ---------------------------------------------------------------------
>> To start a new topic, e-mail: users@tomcat.apache.org
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>>
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org


---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: Version 6 Clustering problem

Posted by Filip Hanik - Dev Lists <de...@hanik.com>.
try setting the
|-Djava.net.preferIPv4Stack=true

option for your tomcat launch

Filip
|
János wrote:
> Hi,
>
> I am trying to establish a 3 member cluster on the same machine.  Here 
> is my server.xml copied from a sample I found on this list.
>
> <?xml version='1.0' encoding='utf-8'?>
> <Server port="8005" shutdown="SHUTDOWN">
>   <Listener className="org.apache.catalina.core.AprLifecycleListener" 
> SSLEngine="on" />
>   <Listener className="org.apache.catalina.core.JasperListener" />
>   <Listener 
> className="org.apache.catalina.mbeans.ServerLifecycleListener" />
>   <Listener 
> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" 
> />
>
>   <GlobalNamingResources>
>     <Resource name="UserDatabase" auth="Container"
>               type="org.apache.catalina.UserDatabase"
>               description="User database that can be updated and saved"
>               
> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
>               pathname="conf/tomcat-users.xml" />
>   </GlobalNamingResources>
>
>   <Service name="Catalina">
>
>     <Connector port="8080" protocol="HTTP/1.1"
>                connectionTimeout="20000"
>                redirectPort="8443" />
>     <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
>
>     <Engine name="Catalina" defaultHost="localhost" jvmRoute="node1">
>
>       <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
>       <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
>              resourceName="UserDatabase"/>
>
>       <Host name="localhost"  appBase="webapps"
>             unpackWARs="true" autoDeploy="true"
>             xmlValidation="false" xmlNamespaceAware="false">
>         <Context path="" docBase="ROOT" 
> className="org.apache.catalina.ha.context.ReplicatedContext"/>
>       </Host>
>     </Engine>
>   </Service>
> </Server>
>
> The other two server.xml-s are the same, except the shutdown ports 
> are  8105, 8205, http ports are 8180, 8280 and https ports are 8543, 
> 8643.
> As soon as I am starting the second instance I receive the following 
> into the logs.
>
> In node1 logs:
>
> bml0042:/usr/local/tomcat/node1/logs janos$ cat catalina.2008-07-10.log
> Jul 10, 2008 1:03:24 PM org.apache.catalina.core.AprLifecycleListener 
> init
> INFO: The APR based Apache Tomcat Native library which allows optimal 
> performance in production environments was not found on the 
> java.library.path: 
> .:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java
> Jul 10, 2008 1:03:24 PM org.apache.coyote.http11.Http11Protocol init
> INFO: Initializing Coyote HTTP/1.1 on http-8080
> Jul 10, 2008 1:03:24 PM org.apache.catalina.startup.Catalina load
> INFO: Initialization processed in 1187 ms
> Jul 10, 2008 1:03:24 PM org.apache.catalina.core.StandardService start
> INFO: Starting service Catalina
> Jul 10, 2008 1:03:24 PM org.apache.catalina.core.StandardEngine start
> INFO: Starting Servlet Engine: Apache Tomcat/6.0.16
> Jul 10, 2008 1:03:24 PM org.apache.catalina.ha.tcp.SimpleTcpCluster start
> INFO: Cluster is about to start
> Jul 10, 2008 1:03:24 PM 
> org.apache.catalina.tribes.transport.ReceiverBase bind
> INFO: Receiver Server Socket bound to:/10.84.2.42:4000
> Jul 10, 2008 1:03:24 PM 
> org.apache.catalina.tribes.membership.McastServiceImpl setupSocket
> INFO: Setting cluster mcast soTimeout to 500
> Jul 10, 2008 1:03:24 PM 
> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
> INFO: Sleeping for 1000 milliseconds to establish cluster membership, 
> start level:4
> Jul 10, 2008 1:03:25 PM 
> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
> INFO: Done sleeping, membership established, start level:4
> Jul 10, 2008 1:03:25 PM 
> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
> INFO: Sleeping for 1000 milliseconds to establish cluster membership, 
> start level:8
> Jul 10, 2008 1:03:26 PM 
> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
> INFO: Done sleeping, membership established, start level:8
> Jul 10, 2008 1:03:26 PM 
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
> INFO: Initializing AbstractReplicatedMap with context name:
> Jul 10, 2008 1:03:27 PM org.apache.catalina.core.StandardContext 
> addApplicationListener
> INFO: The listener "listeners.ContextListener" is already configured 
> for this context. The duplicate definition has been ignored.
> Jul 10, 2008 1:03:27 PM org.apache.catalina.core.StandardContext 
> addApplicationListener
> INFO: The listener "listeners.SessionListener" is already configured 
> for this context. The duplicate definition has been ignored.
> Jul 10, 2008 1:03:27 PM org.apache.catalina.ha.session.DeltaManager start
> INFO: Register manager /examples to cluster element Engine with name 
> Catalina
> Jul 10, 2008 1:03:27 PM org.apache.catalina.ha.session.DeltaManager start
> INFO: Starting clustering manager at /examples
> Jul 10, 2008 1:03:27 PM org.apache.catalina.ha.session.DeltaManager 
> getAllClusterSessions
> INFO: Manager [localhost#/examples]: skipping state transfer. No 
> members active in cluster group.
> Jul 10, 2008 1:03:28 PM 
> org.apache.catalina.ha.session.JvmRouteBinderValve start
> INFO: JvmRouteBinderValve started
> Jul 10, 2008 1:03:28 PM org.apache.coyote.http11.Http11Protocol start
> INFO: Starting Coyote HTTP/1.1 on http-8080
> Jul 10, 2008 1:03:28 PM org.apache.jk.common.ChannelSocket init
> INFO: JK: ajp13 listening on /0.0.0.0:8009
> Jul 10, 2008 1:03:28 PM org.apache.jk.server.JkMain start
> INFO: Jk running ID=0 time=0/45  config=null
> Jul 10, 2008 1:03:28 PM org.apache.catalina.startup.Catalina start
> INFO: Server startup in 3616 ms
> Jul 10, 2008 1:06:41 PM 
> org.apache.catalina.tribes.transport.nio.NioReceiver listen
> SEVERE: Unable to process request in NioReceiver
> java.net.SocketException: Invalid argument
>         at sun.nio.ch.Net.setIntOption0(Native Method)
>         at sun.nio.ch.Net.setIntOption(Net.java:152)
>         at 
> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
>         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>         at 
> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
>         at 
> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
>         at 
> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
>         at 
> org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:273) 
>
>         at 
> org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioReceiver.java:353) 
>
>         at java.lang.Thread.run(Thread.java:613)
> Jul 10, 2008 1:06:41 PM org.apache.catalina.ha.tcp.SimpleTcpCluster 
> memberAdded
> INFO: Replication member 
> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84, 
> 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46 127 -68 
> 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={}, command={}, 
> domain={}, ]
> Jul 10, 2008 1:06:41 PM org.apache.catalina.tribes.io.BufferPool 
> getBufferPool
> INFO: Created a buffer pool with max size:104857600 bytes of 
> type:org.apache.catalina.tribes.io.BufferPool15Impl
> Jul 10, 2008 1:06:41 PM 
> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
> WARNING: Member send is failing for:tcp://{10, 84, 2, 42}:4001 ; 
> Setting to suspect and retrying.
> Jul 10, 2008 1:06:41 PM 
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector 
> memberDisappeared
> INFO: Received 
> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 
> 84, 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46 127 
> -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={}, command={}, 
> domain={}, ]] message. Will verify.
> Jul 10, 2008 1:06:41 PM 
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector 
> memberDisappeared
> INFO: Verification complete. Member still 
> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84, 
> 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46 127 -68 
> 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={}, command={}, 
> domain={}, ]]
> Jul 10, 2008 1:06:41 PM 
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
> SEVERE: Unable to send AbstractReplicatedMap.ping message
> org.apache.catalina.tribes.ChannelException: Send failed, attempt:2 
> max:1; Faulty members:tcp://{10, 84, 2, 42}:4001;
>         at 
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:172) 
>
>         at 
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:78) 
>
>         at 
> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53) 
>
>         at 
> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80) 
>
>         at 
> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78) 
>
>         at 
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>
>         at 
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87) 
>
>         at 
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>
>         at 
> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:73) 
>
>         at 
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>
>         at 
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
>         at 
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
>         at 
> org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:89)
>         at 
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:253) 
>
>         at 
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:793) 
>
>         at 
> org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:153) 
>
>         at 
> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:661) 
>
> Caused by: java.net.SocketException: Invalid argument
>         at sun.nio.ch.Net.setIntOption0(Native Method)
>         at sun.nio.ch.Net.setIntOption(Net.java:152)
>         at 
> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
>         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>         at 
> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
>         at 
> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
>         at 
> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
>         at 
> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect(NioSender.java:147) 
>
>         at 
> org.apache.catalina.tribes.transport.nio.NioSender.process(NioSender.java:89) 
>
>         at 
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:130) 
>
>         ... 16 more
> Jul 10, 2008 1:06:41 PM 
> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
> WARNING: Not retrying send for:tcp://{10, 84, 2, 42}:4001; Sender is 
> disconnected.
> Jul 10, 2008 1:06:41 PM 
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector 
> memberDisappeared
> INFO: Received 
> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 
> 84, 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46 127 
> -68 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={}, command={}, 
> domain={}, ]] message. Will verify.
> Jul 10, 2008 1:06:42 PM 
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector 
> memberDisappeared
> INFO: Verification complete. Member still 
> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84, 
> 2, 42}:4001,{10, 84, 2, 42},4001, alive=1022,id={96 24 -65 -46 127 -68 
> 65 119 -107 -12 -61 -24 -12 -122 10 56 }, payload={}, command={}, 
> domain={}, ]]
> Jul 10, 2008 1:06:42 PM 
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
> SEVERE: Unable to send AbstractReplicatedMap.ping message
> org.apache.catalina.tribes.ChannelException: Send failed, and sender 
> is disconnected. Not retrying.; Faulty members:tcp://{10, 84, 2, 
> 42}:4001;
>         at 
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:156) 
>
>         at 
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:78) 
>
>         at 
> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53) 
>
>         at 
> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80) 
>
>         at 
> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78) 
>
>         at 
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>
>         at 
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87) 
>
>         at 
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>
>         at 
> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:73) 
>
>         at 
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>
>         at 
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
>         at 
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
>         at 
> org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:89)
>         at 
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:253) 
>
>         at 
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:793) 
>
>         at 
> org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:158) 
>
>         at 
> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:661) 
>
> Caused by: java.net.SocketException: Invalid argument
>         at sun.nio.ch.Net.setIntOption0(Native Method)
>         at sun.nio.ch.Net.setIntOption(Net.java:152)
>         at 
> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
>         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>         at 
> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
>         at 
> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
>         at 
> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
>         at 
> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect(NioSender.java:147) 
>
>         at 
> org.apache.catalina.tribes.transport.nio.NioSender.process(NioSender.java:89) 
>
>         at 
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:130) 
>
>         ... 16 more
>
>
> and so on forever.
>
> In node2 logs I see the following:
>
> bml0042:/usr/local/tomcat/node2/logs janos$ cat catalina.2008-07-10.log
> Jul 10, 2008 1:06:40 PM org.apache.catalina.core.AprLifecycleListener 
> init
> INFO: The APR based Apache Tomcat Native library which allows optimal 
> performance in production environments was not found on the 
> java.library.path: 
> .:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java
> Jul 10, 2008 1:06:40 PM org.apache.coyote.http11.Http11Protocol init
> INFO: Initializing Coyote HTTP/1.1 on http-8180
> Jul 10, 2008 1:06:40 PM org.apache.catalina.startup.Catalina load
> INFO: Initialization processed in 1148 ms
> Jul 10, 2008 1:06:40 PM org.apache.catalina.core.StandardService start
> INFO: Starting service Catalina
> Jul 10, 2008 1:06:40 PM org.apache.catalina.core.StandardEngine start
> INFO: Starting Servlet Engine: Apache Tomcat/6.0.16
> Jul 10, 2008 1:06:40 PM org.apache.catalina.ha.tcp.SimpleTcpCluster start
> INFO: Cluster is about to start
> Jul 10, 2008 1:06:40 PM 
> org.apache.catalina.tribes.transport.ReceiverBase bind
> INFO: Receiver Server Socket bound to:/10.84.2.42:4001
> Jul 10, 2008 1:06:40 PM 
> org.apache.catalina.tribes.membership.McastServiceImpl setupSocket
> INFO: Setting cluster mcast soTimeout to 500
> Jul 10, 2008 1:06:40 PM 
> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
> INFO: Sleeping for 1000 milliseconds to establish cluster membership, 
> start level:4
> Jul 10, 2008 1:06:41 PM org.apache.catalina.ha.tcp.SimpleTcpCluster 
> memberAdded
> INFO: Replication member 
> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84, 
> 2, 42}:4000,{10, 84, 2, 42},4000, alive=196161,id={-25 34 -32 116 42 
> -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command={}, 
> domain={}, ]
> Jul 10, 2008 1:06:41 PM 
> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
> INFO: Done sleeping, membership established, start level:4
> Jul 10, 2008 1:06:41 PM 
> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
> INFO: Sleeping for 1000 milliseconds to establish cluster membership, 
> start level:8
> Jul 10, 2008 1:06:41 PM 
> org.apache.catalina.tribes.transport.nio.NioReceiver listen
> SEVERE: Unable to process request in NioReceiver
> java.net.SocketException: Invalid argument
>         at sun.nio.ch.Net.setIntOption0(Native Method)
>         at sun.nio.ch.Net.setIntOption(Net.java:152)
>         at 
> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
>         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>         at 
> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
>         at 
> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
>         at 
> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
>         at 
> org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:273) 
>
>         at 
> org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioReceiver.java:353) 
>
>         at java.lang.Thread.run(Thread.java:613)
> Jul 10, 2008 1:06:42 PM 
> org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
> INFO: Done sleeping, membership established, start level:8
> Jul 10, 2008 1:06:42 PM 
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
> INFO: Initializing AbstractReplicatedMap with context name:
> Jul 10, 2008 1:06:42 PM org.apache.catalina.tribes.io.BufferPool 
> getBufferPool
> INFO: Created a buffer pool with max size:104857600 bytes of 
> type:org.apache.catalina.tribes.io.BufferPool15Impl
> Jul 10, 2008 1:06:42 PM 
> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
> WARNING: Member send is failing for:tcp://{10, 84, 2, 42}:4000 ; 
> Setting to suspect and retrying.
> Jul 10, 2008 1:06:42 PM 
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector 
> memberDisappeared
> INFO: Received 
> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 
> 84, 2, 42}:4000,{10, 84, 2, 42},4000, alive=197662,id={-25 34 -32 116 
> 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command={}, 
> domain={}, ]] message. Will verify.
> Jul 10, 2008 1:06:42 PM 
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector 
> memberDisappeared
> INFO: Verification complete. Member still 
> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84, 
> 2, 42}:4000,{10, 84, 2, 42},4000, alive=197662,id={-25 34 -32 116 42 
> -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command={}, 
> domain={}, ]]
> Jul 10, 2008 1:06:42 PM 
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap init
> WARNING: Unable to send map start message.
> Jul 10, 2008 1:06:42 PM 
> org.apache.catalina.ha.context.ReplicatedContext start
> SEVERE: Unable to start ReplicatedContext
> java.lang.RuntimeException: Unable to start replicated map.
>         at 
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.init(AbstractReplicatedMap.java:230) 
>
>         at 
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.<init>(AbstractReplicatedMap.java:171) 
>
>         at 
> org.apache.catalina.tribes.tipis.ReplicatedMap.<init>(ReplicatedMap.java:90) 
>
>         at 
> org.apache.catalina.ha.context.ReplicatedContext.start(ReplicatedContext.java:64) 
>
>         at 
> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
>         at 
> org.apache.catalina.core.StandardHost.start(StandardHost.java:719)
>         at 
> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
>         at 
> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443)
>         at 
> org.apache.catalina.core.StandardService.start(StandardService.java:516)
>         at 
> org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
>         at org.apache.catalina.startup.Catalina.start(Catalina.java:578)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
>
>         at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
>
>         at java.lang.reflect.Method.invoke(Method.java:585)
>         at 
> org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
>         at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
> Caused by: org.apache.catalina.tribes.ChannelException: Send failed, 
> attempt:2 max:1; Faulty members:tcp://{10, 84, 2, 42}:4000;
>         at 
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:172) 
>
>         at 
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:78) 
>
>         at 
> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53) 
>
>         at 
> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80) 
>
>         at 
> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78) 
>
>         at 
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>
>         at 
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87) 
>
>         at 
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>
>         at 
> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:73) 
>
>         at 
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>
>         at 
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
>         at 
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
>         at 
> org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:89)
>         at 
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.broadcast(AbstractReplicatedMap.java:300) 
>
>         at 
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.init(AbstractReplicatedMap.java:223) 
>
>         ... 16 more
> Caused by: java.net.SocketException: Invalid argument
>         at sun.nio.ch.Net.setIntOption0(Native Method)
>         at sun.nio.ch.Net.setIntOption(Net.java:152)
>         at 
> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
>         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>         at 
> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
>         at 
> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
>         at 
> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
>         at 
> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect(NioSender.java:147) 
>
>         at 
> org.apache.catalina.tribes.transport.nio.NioSender.process(NioSender.java:89) 
>
>         at 
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:130) 
>
>         ... 30 more
> Jul 10, 2008 1:06:42 PM org.apache.catalina.startup.Catalina start
> SEVERE: Catalina.start:
> LifecycleException:  Failed to start ReplicatedContext:  
> java.lang.RuntimeException: Unable to start replicated map.
>         at 
> org.apache.catalina.ha.context.ReplicatedContext.start(ReplicatedContext.java:73) 
>
>         at 
> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
>         at 
> org.apache.catalina.core.StandardHost.start(StandardHost.java:719)
>         at 
> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
>         at 
> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443)
>         at 
> org.apache.catalina.core.StandardService.start(StandardService.java:516)
>         at 
> org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
>         at org.apache.catalina.startup.Catalina.start(Catalina.java:578)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
>
>         at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
>
>         at java.lang.reflect.Method.invoke(Method.java:585)
>         at 
> org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
>         at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
> Jul 10, 2008 1:06:42 PM org.apache.catalina.startup.Catalina start
> INFO: Server startup in 2329 ms
> Jul 10, 2008 1:06:47 PM 
> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
> WARNING: Not retrying send for:tcp://{10, 84, 2, 42}:4000; Sender is 
> disconnected.
> Jul 10, 2008 1:06:47 PM 
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector 
> memberDisappeared
> INFO: Received 
> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 
> 84, 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32 116 
> 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command={}, 
> domain={}, ]] message. Will verify.
> Jul 10, 2008 1:06:47 PM 
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector 
> memberDisappeared
> INFO: Verification complete. Member still 
> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84, 
> 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32 116 42 
> -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command={}, 
> domain={}, ]]
> Jul 10, 2008 1:06:47 PM 
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
> SEVERE: Unable to send AbstractReplicatedMap.ping message
> org.apache.catalina.tribes.ChannelException: Send failed, and sender 
> is disconnected. Not retrying.; Faulty members:tcp://{10, 84, 2, 
> 42}:4000;
>         at 
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:156) 
>
>         at 
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:78) 
>
>         at 
> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53) 
>
>         at 
> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80) 
>
>         at 
> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78) 
>
>         at 
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>
>         at 
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87) 
>
>         at 
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>
>         at 
> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:73) 
>
>         at 
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>
>         at 
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
>         at 
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
>         at 
> org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:89)
>         at 
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:253) 
>
>         at 
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:793) 
>
>         at 
> org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:153) 
>
>         at 
> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:661) 
>
> Caused by: java.net.SocketException: Invalid argument
>         at sun.nio.ch.Net.setIntOption0(Native Method)
>         at sun.nio.ch.Net.setIntOption(Net.java:152)
>         at 
> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
>         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>         at 
> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
>         at 
> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
>         at 
> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
>         at 
> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect(NioSender.java:147) 
>
>         at 
> org.apache.catalina.tribes.transport.nio.NioSender.process(NioSender.java:89) 
>
>         at 
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:130) 
>
>         ... 16 more
> Jul 10, 2008 1:06:47 PM 
> org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
> WARNING: Not retrying send for:tcp://{10, 84, 2, 42}:4000; Sender is 
> disconnected.
> Jul 10, 2008 1:06:47 PM 
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector 
> memberDisappeared
> INFO: Received 
> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 
> 84, 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32 116 
> 42 -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command={}, 
> domain={}, ]] message. Will verify.
> Jul 10, 2008 1:06:47 PM 
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector 
> memberDisappeared
> INFO: Verification complete. Member still 
> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84, 
> 2, 42}:4000,{10, 84, 2, 42},4000, alive=202684,id={-25 34 -32 116 42 
> -104 70 78 -115 -32 93 74 -41 -122 -66 0 }, payload={}, command={}, 
> domain={}, ]]
> Jul 10, 2008 1:06:47 PM 
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap heartbeat
> SEVERE: Unable to send AbstractReplicatedMap.ping message
> org.apache.catalina.tribes.ChannelException: Send failed, and sender 
> is disconnected. Not retrying.; Faulty members:tcp://{10, 84, 2, 
> 42}:4000;
>         at 
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:156) 
>
>         at 
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:78) 
>
>         at 
> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53) 
>
>         at 
> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80) 
>
>         at 
> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78) 
>
>         at 
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>
>         at 
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87) 
>
>         at 
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>
>         at 
> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:73) 
>
>         at 
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
>
>         at 
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
>         at 
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
>         at 
> org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:89)
>         at 
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:253) 
>
>         at 
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:793) 
>
>         at 
> org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:158) 
>
>         at 
> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:661) 
>
> Caused by: java.net.SocketException: Invalid argument
>         at sun.nio.ch.Net.setIntOption0(Native Method)
>         at sun.nio.ch.Net.setIntOption(Net.java:152)
>         at 
> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372)
>         at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
>         at 
> sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249)
>         at 
> sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158)
>         at 
> sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330)
>         at 
> org.apache.catalina.tribes.transport.nio.NioSender.completeConnect(NioSender.java:147) 
>
>         at 
> org.apache.catalina.tribes.transport.nio.NioSender.process(NioSender.java:89) 
>
>         at 
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:130) 
>
>         ... 16 more
>
>
> and so on.  Top shows that both Java process is Up to 86%, the machine 
> is never idle and all memory is consumed.  The machine is a PowerPc G6 
> from Apple with OSX 10.4.11.
> Tomcat version is: Apache Tomcat/6.0.16
> JVM version is: 1.5.0_13-b05-241
> I suspect the heartbeat pings have a problem, I just do not know what 
> to do about them.
>
> Thanks ahead,
>
> János
>
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>


---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org