You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Raúl García <rg...@emovilia.com> on 2008/01/17 16:12:21 UTC
Tomcat 6 - Cluster error.
Hi,
I repost the problem with a better subject:
We had a tomcat 5.0.28 server configured with 2 clustered instances(working
perfectly), and we decided to migrate to java6 and the new tomcat 6.0.14.
We modified the configuration files to match the new tomcat 6.0.14
structure.
But now we have really annoying problems with the cluster.
This server receives aprox. 1 petition per second. We have pen as the load
balancer between 2 instances.
Once started, both nodes report member.added and they are clustered.
Works fine. I can restart any node and no session is lost.
But around 12 days after being started, instance number 2, report a
cluster.member disappear, then the session replication doesn't work,
instance 2 is unstable and gives a timeout error (But instance 1 is still
alive!!).
So we have to reload both nodes every 12 days.
Intance2 Log:
=============
Jan 14, 2008 7:05:17 PM org.apache.catalina.tribes.transport.nio.NioReceiver
socketTimeouts
WARNING: Channel key is registered, but has had no interest ops for the last
3000 ms. (cancelled:false):sun.nio.ch.SelectionKeyImpl@29abfd last
access:2008-01-14 19:05:12.847
Jan 14, 2008 7:05:22 PM org.apache.catalina.tribes.transport.nio.NioReceiver
socketTimeouts
WARNING: Channel key is registered, but has had no interest ops for the last
3000 ms. (cancelled:false):sun.nio.ch.SelectionKeyImpl@29abfd last
access:2008-01-14 19:05:17.848
Jan 14, 2008 7:05:27 PM org.apache.catalina.tribes.transport.nio.NioReceiver
socketTimeouts
WARNING: Channel key is registered, but has had no interest ops for the last
3000 ms. (cancelled:false):sun.nio.ch.SelectionKeyImpl@29abfd last
access:2008-01-14 19:05:22.85
Jan 14, 2008 7:05:35 PM org.apache.catalina.tribes.transport.nio.NioReceiver
socketTimeouts
WARNING: Channel key is registered, but has had no interest ops for the last
3000 ms. (cancelled:false):sun.nio.ch.SelectionKeyImpl@19dc5ee last
access:2008-01-14 19:05:30.111
Jan 14, 2008 7:05:35 PM org.apache.catalina.tribes.transport.nio.NioReceiver
socketTimeouts
WARNING: Channel key is registered, but has had no interest ops for the last
3000 ms. (cancelled:false):sun.nio.ch.SelectionKeyImpl@29abfd last
access:2008-01-14 19:05:27.852
Jan 15, 2008 1:56:37 AM
org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor report
INFO: ThroughputInterceptor Report[
Tx Msg:20000 messages
Sent:11.00 MB (total)
Sent:11.00 MB (application)
Time:28.14 seconds
Tx Speed:0.39 MB/sec (total)
TxSpeed:0.39 MB/sec (application)
Error Msg:0
Rx Msg:12195 messages
Rx Speed:0.00 MB/sec (since 1st msg)
Received:6.74 MB]
Jan 15, 2008 8:48:36 AM org.apache.catalina.ha.tcp.SimpleTcpCluster
memberDisappeared
INFO: Received member
disappeared:org.apache.catalina.tribes.membership.MemberImpl[tcp://localhost
:4001,localhost,4001, alive=72702012,id={-44 -75 50 38 26 -53 72 -63 -76 -94
12 82 127 106 126 -61 }, payload={}, command={}, domain={}, ] Jan 15, 2008
8:48:36 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
performBasicCheck
INFO: Suspect member, confirmed
dead.[org.apache.catalina.tribes.membership.MemberImpl[tcp://localhost:4001,
localhost,4001, alive=72702012,id={-44 -75 50 38 26 -53 72 -63 -76 -94 12 82
127 106 126 -61 }, payload={}, command={}, domain={}, ]] Jan 15, 2008
8:48:39 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Received
memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://loc
alhost:4001,localhost,4001, alive=72705018,id={-44 -75 50 38 26 -53 72 -63
-76 -94 12 82 127 106 126 -61 }, payload={}, command={}, domain={}, ]]
message. Will verify.
Jan 15, 2008 8:48:39 AM
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Verification complete. Member still
alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://localhost:4001,
localhost,4001, alive=72705018,id={-44 -75 50 38 26 -53 72 -63 -76 -94 12 82
127 106 126 -61 }, payload={}, command={}, domain={}, ]] Jan 15, 2008
8:48:39 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out(60000
ms.).; Faulty members:tcp://localhost:4001;
at
org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(Paral
lelNioSender.java:97)
at
org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(Po
oledParallelSender.java:53)
at
org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(Repl
icationTransmitter.java:80)
at
org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoord
inator.java:78)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
nterceptorBase.java:75)
at
org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor.sendMess
age(ThroughputInterceptor.java:61)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
nterceptorBase.java:75)
at
org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sen
dMessage(MessageDispatchInterceptor.java:73)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
nterceptorBase.java:75)
at
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage
(TcpFailureDetector.java:87)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
nterceptorBase.java:75)
at
org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
at
org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
at
org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:835)
at
org.apache.catalina.ha.tcp.SimpleTcpCluster.sendClusterDomain(SimpleTcpClust
er.java:814)
at
org.apache.catalina.ha.session.DeltaManager.send(DeltaManager.java:586)
at
org.apache.catalina.ha.session.DeltaManager.sendCreateSession(DeltaManager.j
ava:575)
at
org.apache.catalina.ha.session.DeltaManager.createSession(DeltaManager.java:
551)
at
org.apache.catalina.ha.session.DeltaManager.createSession(DeltaManager.java:
534)
at
org.apache.catalina.connector.Request.doGetSession(Request.java:2312)
at
org.apache.catalina.connector.Request.getSession(Request.java:2075)
at
org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:83
3)
at pad.kernel.Resolver.service(Resolver.java:266)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
FilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
ain.java:206)
at pad.kernel.EntryPointFilter.doFilter(EntryPointFilter.java:365)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
FilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
ain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
va:219)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
va:175)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128
)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102
)
at
org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.jav
a:269)
at
org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:81)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
:109)
at
org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:347
)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:263)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http
11Protocol.java:584)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:619)
[...]
SEVERE: Unable to send message through cluster sender.
[...] <---- Repeats every 60000ms
Resolver.java at line 266
==========================
//I just get a new session (request is a
javax.servlet.http.HttpServletRequest)
javax.servlet.http.HttpSession sesion = request.getSession(true);
==================
We modified server.xml to be exactly what is recommended by the
documentation. Helped by some guys of dev-apache mailing list (wrong list,
so I post this problem again here)
The config files of each node are:
INSTANCE-1 --- Server.xml
==========================
NOTE:: 111.111.111.111 is the server ip address.
==========================
<Server port="8006" shutdown="SHUTDOWN" debug="0">
<Listener className="org.apache.catalina.core.JasperListener" debug="0"/>
<Listener className="org.apache.catalina.mbeans.ServerLifecycleListener"
debug="0"/>
<Listener
className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
debug="0"/>
<GlobalNamingResources>
<Environment name="InstanceName" type="java.lang.String" value="pro1"/>
<Resource name="UserDatabase" auth="Container"
type="org.apache.catalina.UserDatabase"
description="User database that can be updated and saved"
factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
pathname="conf/tomcat-users.xml" />
</GlobalNamingResources>
<Service name="Catalina">
<Connector port="8081" protocol="HTTP/1.1" maxHttpHeaderSize="8192"
emptySessionPath="true"
maxThreads="150" minSpareThreads="100" maxSpareThreads="300"
enableLookups="false" redirectPort="81443" acceptCount="1000"
debug="0" connectionTimeout="20000"
disableUploadTimeout="true"
compression="on"
compressionMinSize="2048"
noCompressionUserAgents="gozilla, traviata"
compressableMimeType="text/html,text/xml" />
<Engine name="Catalina" defaultHost="localhost" debug="0"
jvmRoute="PR1">
<Cluster
className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
channelSendOptions="6">
<Manager className="org.apache.catalina.ha.session.DeltaManager"
expireSessionsOnShutdown="false"
notifyListenersOnReplication="true"/>
<Channel
className="org.apache.catalina.tribes.group.GroupChannel">
<Membership
className="org.apache.catalina.tribes.membership.McastService"
address="228.0.0.4"
port="45564"
frequency="1000"
dropTime="30000"/>
<Receiver
className="org.apache.catalina.tribes.transport.nio.NioReceiver"
address="127.0.0.1"
port="4001"
autoBind="100"
selectorTimeout="5000"
maxThreads="12"/>
<Sender
className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
<Transport
className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
timeout="60000"/>
</Sender>
<Interceptor
className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"
/>
<Interceptor
className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15In
terceptor"/>
<Interceptor
className="org.apache.catalina.tribes.group.interceptors.ThroughputIntercept
or"/>
</Channel>
<Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.css;.*\.txt;"/>
<Deployer
className="org.apache.catalina.ha.deploy.FarmWarDeployer"
tempDir="/tmp/war-temp/"
deployDir="/tmp/war-deploy/"
watchDir="/tmp/war-listen/"
watchEnabled="false"/>
<ClusterListener
className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
<ClusterListener
className="org.apache.catalina.ha.session.ClusterSessionListener"/>
</Cluster>
<Realm className="org.apache.catalina.realm.UserDatabaseRealm"
debug="0" resourceName="UserDatabase"/>
<Host name="localhost" debug="0" appBase="webapps"
unpackWARs="true" autoDeploy="true"
xmlValidation="false" xmlNamespaceAware="false">
<Valve className="org.apache.catalina.valves.RemoteAddrValve"
allow="10.0.0.*,127.0.0.1,228.0.0.4,111.111.111.111"/>
</Host>
</Engine>
</Service>
</Server>
==============================================
INSTANCE-2 server.xml
=====================
<Server port="8007" shutdown="SHUTDOWN" debug="0">
<Listener className="org.apache.catalina.core.JasperListener" debug="0"/>
<Listener className="org.apache.catalina.mbeans.ServerLifecycleListener"
debug="0"/>
<Listener
className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
debug="0"/>
<GlobalNamingResources>
<Environment name="InstanceName" type="java.lang.String" value="pro2"/>
<Resource name="UserDatabase" auth="Container"
type="org.apache.catalina.UserDatabase"
description="User database that can be updated and saved"
factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
pathname="conf/tomcat-users.xml"/>
</GlobalNamingResources>
<Service name="Catalina">
<Connector port="8082" protocol="HTTP/1.1" maxHttpHeaderSize="8192"
emptySessionPath="true"
maxThreads="150" minSpareThreads="100" maxSpareThreads="300"
enableLookups="false" redirectPort="82443" acceptCount="1000"
debug="0" connectionTimeout="20000"
disableUploadTimeout="true"
compression="on"
compressionMinSize="2048"
noCompressionUserAgents="gozilla, traviata"
compressableMimeType="text/html,text/xml" />
<Engine name="Catalina" defaultHost="localhost" debug="0"
jvmRoute="PR2">
<Cluster
className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
channelSendOptions="6">
<Manager className="org.apache.catalina.ha.session.DeltaManager"
expireSessionsOnShutdown="false"
notifyListenersOnReplication="true"/>
<Channel
className="org.apache.catalina.tribes.group.GroupChannel">
<Membership
className="org.apache.catalina.tribes.membership.McastService"
address="228.0.0.4"
port="45564"
frequency="1000"
dropTime="30000"/>
<Receiver
className="org.apache.catalina.tribes.transport.nio.NioReceiver"
address="127.0.0.1"
port="4002"
autoBind="100"
selectorTimeout="5000"
maxThreads="12"/>
<Sender
className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
<Transport
className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
timeout="60000"/>
</Sender>
<Interceptor
className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"
/>
<Interceptor
className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15In
terceptor"/>
<Interceptor
className="org.apache.catalina.tribes.group.interceptors.ThroughputIntercept
or"/>
</Channel>
<Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.css;.*\.txt;"/>
<!-- <Valve
className="org.apache.catalina.ha.session.JvmRouteBinderValve"/> -->
<Deployer
className="org.apache.catalina.ha.deploy.FarmWarDeployer"
tempDir="/tmp/war-temp/"
deployDir="/tmp/war-deploy/"
watchDir="/tmp/war-listen/"
watchEnabled="false"/>
<ClusterListener
className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
<ClusterListener
className="org.apache.catalina.ha.session.ClusterSessionListener"/>
</Cluster>
<Realm className="org.apache.catalina.realm.UserDatabaseRealm"
resourceName="UserDatabase" debug="0"/>
<Host name="localhost" debug="0" appBase="webapps"
unpackWARs="true" autoDeploy="true"
xmlValidation="false" xmlNamespaceAware="false">
<Valve className="org.apache.catalina.valves.RemoteAddrValve"
allow="10.0.0.*,127.0.0.1,228.0.0.4,111.111.111.111"/>
</Host>
</Engine>
</Service>
</Server>
===============================
Can you see something wrong that can cause that timeouts?
I can paste more config files if you need.
Thank you very much
Raúl.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat 6 - Cluster error.
Posted by Raúl García <rg...@emovilia.com>.
Hi again,
After some weeks, Mr. Hanik help us solving the problem.
And he found it!
The problem was the sticky sessions, and our balancer: pen.
Pen, (http://siag.nu/pen/) it's a simple load balancer, and it doesn't care
about sessions between nodes.
That provokes dead locks at tomcat.
The problem had been solved at a tomcat revision:
http://svn.apache.org/viewvc?view=rev&revision=618823
Right now, our 2 nodes are up and running without errors since 1th of march
(2008).
Clean logs.
Once again,
Thanks Filip, and thank you all for this great dev-list.
Best Regards
Raúl.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 6 - Cluster error.
Posted by Filip Hanik - Dev Lists <de...@hanik.com>.
the error is because a member disappeared, the cluster code should not
be doing
channel.send(channel.getMembers());
but it should be doing
channel.getMembers() can return an empty array, correct your code to do this
Member[] dest = channel.getMembers();
if(dest.length>0)channel.send(dest)
Filip
Raúl García wrote:
> Hi again,
> I'm getting an error when one of the nodes are shutdown and I make a hit to
> the server.
>
> Seems like the node 2, is trying to send the session to the other node (But
> is offline)
>
> LOG_INSTANCE-2
> ===============
> === I lauch the shutdown command of node1: ====
> 21-ene-2008 17:15:10
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
> memberDisappeared
> INFO: Verification complete. Member
> disappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://localhost
> .localdomain:4001,localhost.localdomain,4001, alive=161284,id={-87 -124 123
> -76 30 -1 76 49 -120 -67 -116 87 -2 59 77 29 }, payload={}, command={66 65
> 66 89 45 65 76 69 88 ...(9)}, domain={}, ]]
> 21-ene-2008 17:15:10 org.apache.catalina.ha.tcp.SimpleTcpCluster
> memberDisappeared
> INFO: Received member
> disappeared:org.apache.catalina.tribes.membership.MemberImpl[tcp://localhost
> .localdomain:4001,localhost.localdomain,4001, alive=161284,id={-87 -124 123
> -76 30 -1 76 49 -120 -67 -116 87 -2 59 77 29 }, payload={}, command={66 65
> 66 89 45 65 76 69 88 ...(9)}, domain={}, ]
> 21-ene-2008 17:15:33 org.apache.catalina.ha.tcp.SimpleTcpCluster send
>
> == Now I browse to http://localhost/MyServlet =====
>
> SEVERE: Unable to send message through cluster sender.
> org.apache.catalina.tribes.ChannelException: No destination given; No faulty
> members identified.
> at
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:194)
> at
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
> at
> org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:835)
> at
> org.apache.catalina.ha.tcp.SimpleTcpCluster.sendClusterDomain(SimpleTcpClust
> er.java:814)
> at
> org.apache.catalina.ha.session.DeltaManager.send(DeltaManager.java:586)
> at
> org.apache.catalina.ha.session.DeltaManager.sessionExpired(DeltaManager.java
> :1211)
> at
> org.apache.catalina.ha.session.DeltaSession.expire(DeltaSession.java:366)
> at
> org.apache.catalina.ha.session.DeltaSession.expire(DeltaSession.java:352)
> at
> org.apache.catalina.session.StandardSession.expire(StandardSession.java:660)
> at
> org.apache.catalina.session.StandardSession.invalidate(StandardSession.java:
> 1111)
> at
> org.apache.catalina.session.StandardSessionFacade.invalidate(StandardSession
> Facade.java:150)
> at pad.kernel.Resolver.service(Resolver.java:623)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
> FilterChain.java:290)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
> ain.java:206)
> at pad.kernel.EntryPointFilter.doFilter(EntryPointFilter.java:365)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
> FilterChain.java:235)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
> ain.java:206)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
> va:219)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
> va:175)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128
> )
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102
> )
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
> :109)
> at
> org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:347
> )
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:263)
> at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
> at
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http
> 11Protocol.java:584)
> at
> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
> at java.lang.Thread.run(Thread.java:619)
>
> --> Every new hit at the sever provokes a new:
> "SEVERE: Unable to send message through cluster sender.[...]"
>
>
> This is a normal issue? Or it can be desactivated?
>
> Thanks
> Best regards
> Raúl.
>
>
> -----Mensaje original-----
> De: Raúl García [mailto:rgarcia@emovilia.com]
> Enviado el: viernes, 18 de enero de 2008 11:23
> Para: 'Tomcat Users List'
> Asunto: RE: Tomcat 6 - Cluster error.
>
> Hi Filip,
>
> Thank you for that suggestion. We are going to try it now.
>
> On the other side, if You want to reproduce the error:
>
> We have used a server here configured with the same config.xml files.
> To force it to the "crash" we send a high amount of "login" request (Those
> that creates a new session), for a long time. Not always, but this can
> reproduce the error without having to wait 12 days.
>
> At the moment, we tried once the new keepAliveCount="0", and both nodes
> worked perfectly during the "experiment" (45 min, 11.000 hits).
>
> It will be a pleasure that you get the debug error, let me know anything
> more you need.
>
> Thank you for the response.
>
> We will see the real results in 13 days, I'll update.
>
> Best Regards
> Raúl.
>
> -----Mensaje original-----
> De: Filip Hanik - Dev Lists [mailto:devlists@hanik.com]
> Enviado el: jueves, 17 de enero de 2008 19:01
> Para: Tomcat Users List
> Asunto: Re: Tomcat 6 - Cluster error.
>
> already replied to your old thread
>
> ok, it looks like you might have ended up with a rogue socket,
> and what happens is that any message sent to that socket just gets lost
> in the ether, since it doesn't have any interest ops.
> There is a workaround for this, turn off keep alives all together, or
> implement a keep alive timeout
>
> Option 1 - no keep alives at all
>
> <Transport
> className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
> timeout="60000"
> keepAliveCount="0"/>
>
> Option 2 - implement a keep alive timeout
>
> <Transport
> className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
> timeout="60000"
> keepAliveTime="120000"/>
>
> or make a combination of both values
>
> either option should work for you.
>
> On a side note, I'm interested if the scenario you run into is
> reproducible, it keeps happening over and over again, then if possible,
> I'd like to get some debug logs from you
>
> Filip
>
>
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat 6 - Cluster error.
Posted by Raúl García <rg...@emovilia.com>.
Hi again,
I'm getting an error when one of the nodes are shutdown and I make a hit to
the server.
Seems like the node 2, is trying to send the session to the other node (But
is offline)
LOG_INSTANCE-2
===============
=== I lauch the shutdown command of node1: ====
21-ene-2008 17:15:10
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Verification complete. Member
disappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://localhost
.localdomain:4001,localhost.localdomain,4001, alive=161284,id={-87 -124 123
-76 30 -1 76 49 -120 -67 -116 87 -2 59 77 29 }, payload={}, command={66 65
66 89 45 65 76 69 88 ...(9)}, domain={}, ]]
21-ene-2008 17:15:10 org.apache.catalina.ha.tcp.SimpleTcpCluster
memberDisappeared
INFO: Received member
disappeared:org.apache.catalina.tribes.membership.MemberImpl[tcp://localhost
.localdomain:4001,localhost.localdomain,4001, alive=161284,id={-87 -124 123
-76 30 -1 76 49 -120 -67 -116 87 -2 59 77 29 }, payload={}, command={66 65
66 89 45 65 76 69 88 ...(9)}, domain={}, ]
21-ene-2008 17:15:33 org.apache.catalina.ha.tcp.SimpleTcpCluster send
== Now I browse to http://localhost/MyServlet =====
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: No destination given; No faulty
members identified.
at
org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:194)
at
org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
at
org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:835)
at
org.apache.catalina.ha.tcp.SimpleTcpCluster.sendClusterDomain(SimpleTcpClust
er.java:814)
at
org.apache.catalina.ha.session.DeltaManager.send(DeltaManager.java:586)
at
org.apache.catalina.ha.session.DeltaManager.sessionExpired(DeltaManager.java
:1211)
at
org.apache.catalina.ha.session.DeltaSession.expire(DeltaSession.java:366)
at
org.apache.catalina.ha.session.DeltaSession.expire(DeltaSession.java:352)
at
org.apache.catalina.session.StandardSession.expire(StandardSession.java:660)
at
org.apache.catalina.session.StandardSession.invalidate(StandardSession.java:
1111)
at
org.apache.catalina.session.StandardSessionFacade.invalidate(StandardSession
Facade.java:150)
at pad.kernel.Resolver.service(Resolver.java:623)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
FilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
ain.java:206)
at pad.kernel.EntryPointFilter.doFilter(EntryPointFilter.java:365)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
FilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
ain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
va:219)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
va:175)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128
)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102
)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
:109)
at
org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:347
)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:263)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http
11Protocol.java:584)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:619)
--> Every new hit at the sever provokes a new:
"SEVERE: Unable to send message through cluster sender.[...]"
This is a normal issue? Or it can be desactivated?
Thanks
Best regards
Raúl.
-----Mensaje original-----
De: Raúl García [mailto:rgarcia@emovilia.com]
Enviado el: viernes, 18 de enero de 2008 11:23
Para: 'Tomcat Users List'
Asunto: RE: Tomcat 6 - Cluster error.
Hi Filip,
Thank you for that suggestion. We are going to try it now.
On the other side, if You want to reproduce the error:
We have used a server here configured with the same config.xml files.
To force it to the "crash" we send a high amount of "login" request (Those
that creates a new session), for a long time. Not always, but this can
reproduce the error without having to wait 12 days.
At the moment, we tried once the new keepAliveCount="0", and both nodes
worked perfectly during the "experiment" (45 min, 11.000 hits).
It will be a pleasure that you get the debug error, let me know anything
more you need.
Thank you for the response.
We will see the real results in 13 days, I'll update.
Best Regards
Raúl.
-----Mensaje original-----
De: Filip Hanik - Dev Lists [mailto:devlists@hanik.com]
Enviado el: jueves, 17 de enero de 2008 19:01
Para: Tomcat Users List
Asunto: Re: Tomcat 6 - Cluster error.
already replied to your old thread
ok, it looks like you might have ended up with a rogue socket,
and what happens is that any message sent to that socket just gets lost
in the ether, since it doesn't have any interest ops.
There is a workaround for this, turn off keep alives all together, or
implement a keep alive timeout
Option 1 - no keep alives at all
<Transport
className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
timeout="60000"
keepAliveCount="0"/>
Option 2 - implement a keep alive timeout
<Transport
className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
timeout="60000"
keepAliveTime="120000"/>
or make a combination of both values
either option should work for you.
On a side note, I'm interested if the scenario you run into is
reproducible, it keeps happening over and over again, then if possible,
I'd like to get some debug logs from you
Filip
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat 6 - Cluster error.
Posted by Raúl García <rg...@emovilia.com>.
Hi Filip,
Thank you for that suggestion. We are going to try it now.
On the other side, if You want to reproduce the error:
We have used a server here configured with the same config.xml files.
To force it to the "crash" we send a high amount of "login" request (Those
that creates a new session), for a long time. Not always, but this can
reproduce the error without having to wait 12 days.
At the moment, we tried once the new keepAliveCount="0", and both nodes
worked perfectly during the "experiment" (45 min, 11.000 hits).
It will be a pleasure that you get the debug error, let me know anything
more you need.
Thank you for the response.
We will see the real results in 13 days, I'll update.
Best Regards
Raúl.
-----Mensaje original-----
De: Filip Hanik - Dev Lists [mailto:devlists@hanik.com]
Enviado el: jueves, 17 de enero de 2008 19:01
Para: Tomcat Users List
Asunto: Re: Tomcat 6 - Cluster error.
already replied to your old thread
ok, it looks like you might have ended up with a rogue socket,
and what happens is that any message sent to that socket just gets lost
in the ether, since it doesn't have any interest ops.
There is a workaround for this, turn off keep alives all together, or
implement a keep alive timeout
Option 1 - no keep alives at all
<Transport
className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
timeout="60000"
keepAliveCount="0"/>
Option 2 - implement a keep alive timeout
<Transport
className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
timeout="60000"
keepAliveTime="120000"/>
or make a combination of both values
either option should work for you.
On a side note, I'm interested if the scenario you run into is
reproducible, it keeps happening over and over again, then if possible,
I'd like to get some debug logs from you
Filip
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 6 - Cluster error.
Posted by Filip Hanik - Dev Lists <de...@hanik.com>.
I'll take this offline with you, and if we resolve it, we will post the
solution here
Filip
Raúl García wrote:
> Hi again,
>
> I try the config using keepAliveTime to 10:
>
> <Transport
> className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
> timeout="60000" keepAliveTime="10"
> keepAliveCount="0"/>
>
> One more time, the cluster is not working, the big problem is that I cannot
> reproduce the error at my backup server that works perfectly.
>
> Node 2, drops a log error at 12:58 AM, then, at the same time, node 1 report
> "ClusterError" continuously
> (Continuous errors are on every hit; the server supports 1 hit per second)
>
> Logs:
>
> NODE 2 - LOG
> =============
> Jan 31, 2008 12:58:13 PM
> org.apache.catalina.tribes.transport.nio.NioReceiver socketTimeouts
> WARNING: Channel key is registered, but has had no interest ops for the last
> 3000 ms. (cancelled:false):sun.nio.ch.SelectionKeyImpl@c3271d last
> access:2008-01-31 12:58:10.208
>
>
> NODE 1 - LOG
> =============
> Jan 31, 2008 12:58:04 PM
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
> memberDisappeared
> INFO: Received
> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://loc
> alhost:4002,localhost,4002, alive=101194547,id={123 -66 95 -10 88
> 24 77 -32 -93 16 -13 -112 90 52 -18 78 }, payload={}, command={}, domain={},
> ]] message. Will verify.
> Jan 31, 2008 12:58:04 PM
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
> memberDisappeared
> INFO: Verification complete. Member still
> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://localhost:4002,
> localhost,4002, alive=101194547,id={123
> -66 95 -10 88 24 77 -32 -93 16 -13 -112 90 52 -18 78 }, payload={},
> command={}, domain={}, ]]
> Jan 31, 2008 12:58:04 PM org.apache.catalina.ha.tcp.SimpleTcpCluster send
> SEVERE: Unable to send message through cluster sender.
> org.apache.catalina.tribes.ChannelException: Operation has timed out(60000
> ms.).; Faulty members:tcp://localhost:4002;
> at
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(Paral
> lelNioSender.java:97)
> at
> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(Po
> oledParallelSender.java:53)
> at
> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(Repl
> icationTransmitter.java:80)
> at
> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoord
> inator.java:78)
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
> nterceptorBase.java:75)
> at
> org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor.sendMess
> age(ThroughputInterceptor.java:61)
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
> nterceptorBase.java:75)
> at
> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sen
> dMessage(MessageDispatchInterceptor.java:73)
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
> nterceptorBase.java:75)
> at
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage
> (TcpFailureDetector.java:87)
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
> nterceptorBase.java:75)
> at
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
> at
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
> at
> org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:835)
> at
> org.apache.catalina.ha.tcp.SimpleTcpCluster.sendClusterDomain(SimpleTcpClust
> er.java:814)
> at
> org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:551)
> at
> org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.jav
> a:535)
> at
> org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(Re
> plicationValve.java:517)
> at
> org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(Replicati
> onValve.java:428)
> at
> org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:362
> )
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:263)
> at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
> at
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http
> 11Protocol.java:584)
> at
> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
> at java.lang.Thread.run(Thread.java:619)
> Jan 31, 2008 12:58:07 PM
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
> memberDisappeared
> INFO: Received
> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://loc
> alhost:4002,localhost,4002, alive=101197553,id={123 -66 95 -10 88
> 24 77 -32 -93 16 -13 -112 90 52 -18 78 }, payload={}, command={}, domain={},
> ]] message. Will verify.
> Jan 31, 2008 12:58:07 PM
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
> memberDisappeared
> INFO: Verification complete. Member still
> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://localhost:4002,
> localhost,4002, alive=101197553,id={123
> -66 95 -10 88 24 77 -32 -93 16 -13 -112 90 52 -18 78 }, payload={},
> command={}, domain={}, ]]
> [...] repeats on every hit.
> ========================
>
> I cannot understand the node 2 log, why is the node 2 crashing??
>
> What can I do??
>
> Thanks on advance.
>
> Raúl.
>
>
> -----Mensaje original-----
> De: Filip Hanik - Dev Lists [mailto:devlists@hanik.com]
> Enviado el: lunes, 28 de enero de 2008 1:45
> Para: Tomcat Users List
> Asunto: Re: Tomcat 6 - Cluster error.
>
> I'd set keepAliveTime to 10 as well,
>
> Filip
>
> Raúl García wrote:
>
>> Hi Again, once again thanks for your time, but we still have problems,
>>
>> We applied the "keepAliveCount=0" param. and last Wednesday 23 Jan we
>> restart both nodes.
>>
>> Around 11 hour after the startup, node 1 reports a new error, but both
>>
> nodes
>
>> are working perfectly.
>>
>> I cannot imagine why the member disappear unexpectedly, I repost the
>>
> error,
>
>> and the config files.
>>
>> INSTANCE 1 - LOG
>> ================
>> Jan 24, 2008 10:25:54 PM
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>> memberDisappeared
>> INFO: Received
>>
>>
> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://loc
>
>> alhost:4002,localhost,4002, alive=123412856,id={-31 -91 -122 -60 -58 -5 68
>> 25 -87 13 -20 -12 -100 5 -16 94 }, payload={}, command={}, domain={}, ]]
>> message. Will verify.
>> Jan 24, 2008 10:25:54 PM
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>> memberDisappeared
>> INFO: Verification complete. Member still
>>
>>
> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://localhost:4002,
>
>> localhost,4002, alive=123412856,id={-31 -91 -122 -60 -58 -5 68 25 -87 13
>>
> -20
>
>> -12 -100 5 -16 94 }, payload={}, command={}, domain={}, ]]
>> Jan 24, 2008 10:25:54 PM org.apache.catalina.ha.tcp.SimpleTcpCluster send
>> SEVERE: Unable to send message through cluster sender.
>> org.apache.catalina.tribes.ChannelException: Operation has timed out(60000
>> ms.).; Faulty members:tcp://localhost:4002;
>> at
>>
>>
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(Paral
>
>> lelNioSender.java:97)
>> at
>>
>>
> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(Po
>
>> oledParallelSender.java:53)
>> at
>>
>>
> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(Repl
>
>> icationTransmitter.java:80)
>> at
>>
>>
> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoord
>
>> inator.java:78)
>> at
>>
>>
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
>
>> nterceptorBase.java:75)
>> at
>>
>>
> org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor.sendMess
>
>> age(ThroughputInterceptor.java:61)
>> at
>>
>>
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
>
>> nterceptorBase.java:75)
>> at
>>
>>
> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sen
>
>> dMessage(MessageDispatchInterceptor.java:73)
>> at
>>
>>
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
>
>> nterceptorBase.java:75)
>> at
>>
>>
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage
>
>> (TcpFailureDetector.java:87)
>> at
>>
>>
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
>
>> nterceptorBase.java:75)
>> at
>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
>> at
>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
>> at
>>
>>
> org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:835)
>
>> at
>>
>>
> org.apache.catalina.ha.tcp.SimpleTcpCluster.sendClusterDomain(SimpleTcpClust
>
>> er.java:814)
>> at
>>
>>
> org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:551)
>
>> at
>>
>>
> org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.jav
>
>> a:535)
>> at
>>
>>
> org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(Re
>
>> plicationValve.java:517)
>> at
>>
>>
> org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(Replicati
>
>> onValve.java:428)
>> at
>>
>>
> org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:362
>
>> )
>> at
>>
>>
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:263)
>
>> at
>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
>> at
>>
>>
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http
>
>> 11Protocol.java:584)
>> at
>> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
>> at java.lang.Thread.run(Thread.java:619)
>>
>> Jan 24, 2008 10:26:54 PM
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>> memberDisappeared
>> INFO: Received memberDisappeared [...] repeats only once again.
>>
>> Jan 25, 2008 5:37:52 AM
>> org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor report
>> INFO: ThroughputInterceptor Report[
>> Tx Msg:66167 messages
>> Sent:37.02 MB (total)
>> Sent:37.02 MB (application)
>> Time:118.53 seconds
>> Tx Speed:0.31 MB/sec (total)
>> TxSpeed:0.31 MB/sec (application)
>> Error Msg:2
>> Rx Msg:90000 messages
>> Rx Speed:0.00 MB/sec (since 1st msg)
>> Received:41.06 MB]
>>
>>
>>
>>
>> INSTANCE-1 --- Server.xml
>> ==========================
>> NOTE:: 111.111.111.111 is the server ip address.
>> ==========================
>> <Server port="8006" shutdown="SHUTDOWN" debug="0">
>> <Listener className="org.apache.catalina.core.JasperListener"
>>
> debug="0"/>
>
>> <Listener className="org.apache.catalina.mbeans.ServerLifecycleListener"
>> debug="0"/>
>> <Listener
>> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
>> debug="0"/>
>>
>> <GlobalNamingResources>
>> <Environment name="InstanceName" type="java.lang.String"
>>
> value="pro1"/>
>
>> <Resource name="UserDatabase" auth="Container"
>> type="org.apache.catalina.UserDatabase"
>> description="User database that can be updated and saved"
>>
>>
> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
>
>> pathname="conf/tomcat-users.xml" />
>> </GlobalNamingResources>
>>
>> <Service name="Catalina">
>>
>> <Connector port="8081" protocol="HTTP/1.1" maxHttpHeaderSize="8192"
>> emptySessionPath="true"
>> maxThreads="150" minSpareThreads="100"
>>
> maxSpareThreads="300"
>
>> enableLookups="false" redirectPort="81443"
>>
> acceptCount="1000"
>
>> debug="0" connectionTimeout="20000"
>> disableUploadTimeout="true"
>> compression="on"
>> compressionMinSize="2048"
>> noCompressionUserAgents="gozilla, traviata"
>> compressableMimeType="text/html,text/xml" />
>>
>> <Engine name="Catalina" defaultHost="localhost" debug="0"
>> jvmRoute="PR1">
>> <Cluster
>> className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
>> channelSendOptions="6">
>>
>> <Manager className="org.apache.catalina.ha.session.DeltaManager"
>> expireSessionsOnShutdown="false"
>> notifyListenersOnReplication="true"/>
>>
>> <Channel
>> className="org.apache.catalina.tribes.group.GroupChannel">
>> <Membership
>> className="org.apache.catalina.tribes.membership.McastService"
>> address="228.0.0.4"
>> port="45564"
>> frequency="1000"
>> dropTime="30000"/>
>> <Receiver
>> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
>> address="127.0.0.1"
>> port="4001"
>> autoBind="100"
>> selectorTimeout="5000"
>> maxThreads="12"/>
>>
>> <Sender
>> className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
>> <Transport
>> className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
>> timeout="60000" keepAliveCount="0"/>
>> </Sender>
>> <Interceptor
>>
>>
> className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"
>
>> />
>> <Interceptor
>>
>>
> className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15In
>
>> terceptor"/>
>> <Interceptor
>>
>>
> className="org.apache.catalina.tribes.group.interceptors.ThroughputIntercept
>
>> or"/>
>> </Channel>
>>
>> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
>>
>>
>>
> filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.css;.*\.txt;"/>
>
>> <Deployer
>> className="org.apache.catalina.ha.deploy.FarmWarDeployer"
>> tempDir="/tmp/war-temp/"
>> deployDir="/tmp/war-deploy/"
>> watchDir="/tmp/war-listen/"
>> watchEnabled="false"/>
>>
>> <ClusterListener
>>
>>
> className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
>
>> <ClusterListener
>> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
>> </Cluster>
>> <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
>> debug="0" resourceName="UserDatabase"/>
>> <Host name="localhost" debug="0" appBase="webapps"
>> unpackWARs="true" autoDeploy="true"
>> xmlValidation="false" xmlNamespaceAware="false">
>> <Valve className="org.apache.catalina.valves.RemoteAddrValve"
>> allow="10.0.0.*,127.0.0.1,228.0.0.4,111.111.111.111"/>
>> </Host>
>> </Engine>
>> </Service>
>> </Server>
>> ==============================================
>>
>>
>> INSTANCE-2 server.xml
>> =====================
>> <Server port="8007" shutdown="SHUTDOWN" debug="0">
>>
>> <Listener className="org.apache.catalina.core.JasperListener"
>>
> debug="0"/>
>
>> <Listener className="org.apache.catalina.mbeans.ServerLifecycleListener"
>> debug="0"/>
>> <Listener
>> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
>> debug="0"/>
>>
>> <GlobalNamingResources>
>>
>> <Environment name="InstanceName" type="java.lang.String"
>>
> value="pro2"/>
>
>> <Resource name="UserDatabase" auth="Container"
>> type="org.apache.catalina.UserDatabase"
>> description="User database that can be updated and saved"
>>
>>
> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
>
>> pathname="conf/tomcat-users.xml"/>
>> </GlobalNamingResources>
>>
>> <Service name="Catalina">
>>
>> <Connector port="8082" protocol="HTTP/1.1" maxHttpHeaderSize="8192"
>> emptySessionPath="true"
>> maxThreads="150" minSpareThreads="100"
>>
> maxSpareThreads="300"
>
>> enableLookups="false" redirectPort="82443"
>>
> acceptCount="1000"
>
>> debug="0" connectionTimeout="20000"
>> disableUploadTimeout="true"
>> compression="on"
>> compressionMinSize="2048"
>> noCompressionUserAgents="gozilla, traviata"
>> compressableMimeType="text/html,text/xml" />
>> <Engine name="Catalina" defaultHost="localhost" debug="0"
>> jvmRoute="PR2">
>>
>> <Cluster
>> className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
>> channelSendOptions="6">
>>
>>
>> <Manager className="org.apache.catalina.ha.session.DeltaManager"
>> expireSessionsOnShutdown="false"
>> notifyListenersOnReplication="true"/>
>>
>> <Channel
>> className="org.apache.catalina.tribes.group.GroupChannel">
>> <Membership
>> className="org.apache.catalina.tribes.membership.McastService"
>> address="228.0.0.4"
>> port="45564"
>> frequency="1000"
>> dropTime="30000"/>
>> <Receiver
>> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
>> address="127.0.0.1"
>> port="4002"
>> autoBind="100"
>> selectorTimeout="5000"
>> maxThreads="12"/>
>>
>> <Sender
>> className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
>> <Transport
>> className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
>> timeout="60000" keepAliveCount="0"/>
>> </Sender>
>> <Interceptor
>>
>>
> className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"
>
>> />
>> <Interceptor
>>
>>
> className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15In
>
>> terceptor"/>
>> <Interceptor
>>
>>
> className="org.apache.catalina.tribes.group.interceptors.ThroughputIntercept
>
>> or"/>
>> </Channel>
>>
>> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
>>
>>
>>
> filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.css;.*\.txt;"/>
>
>> <!-- <Valve
>> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/> -->
>>
>> <Deployer
>> className="org.apache.catalina.ha.deploy.FarmWarDeployer"
>> tempDir="/tmp/war-temp/"
>> deployDir="/tmp/war-deploy/"
>> watchDir="/tmp/war-listen/"
>> watchEnabled="false"/>
>>
>> <ClusterListener
>>
>>
> className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
>
>> <ClusterListener
>> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
>> </Cluster>
>>
>> <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
>> resourceName="UserDatabase" debug="0"/>
>>
>> <Host name="localhost" debug="0" appBase="webapps"
>> unpackWARs="true" autoDeploy="true"
>> xmlValidation="false" xmlNamespaceAware="false">
>>
>> <Valve className="org.apache.catalina.valves.RemoteAddrValve"
>> allow="10.0.0.*,127.0.0.1,228.0.0.4,111.111.111.111"/>
>> </Host>
>> </Engine>
>> </Service>
>> </Server>
>> ===============================
>>
>> -----Mensaje original-----
>> De: Filip Hanik - Dev Lists [mailto:devlists@hanik.com]
>> Enviado el: jueves, 17 de enero de 2008 19:01
>> Para: Tomcat Users List
>> Asunto: Re: Tomcat 6 - Cluster error.
>>
>> already replied to your old thread
>>
>> ok, it looks like you might have ended up with a rogue socket,
>> and what happens is that any message sent to that socket just gets lost
>> in the ether, since it doesn't have any interest ops.
>> There is a workaround for this, turn off keep alives all together, or
>> implement a keep alive timeout
>>
>> Option 1 - no keep alives at all
>>
>> <Transport
>> className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
>> timeout="60000"
>> keepAliveCount="0"/>
>>
>> Option 2 - implement a keep alive timeout
>>
>> <Transport
>> className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
>> timeout="60000"
>> keepAliveTime="120000"/>
>>
>> or make a combination of both values
>>
>> either option should work for you.
>>
>> On a side note, I'm interested if the scenario you run into is
>> reproducible, it keeps happening over and over again, then if possible,
>> I'd like to get some debug logs from you
>>
>> Filip
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To start a new topic, e-mail: users@tomcat.apache.org
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To start a new topic, e-mail: users@tomcat.apache.org
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>>
>>
>>
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat 6 - Cluster error.
Posted by Raúl García <rg...@emovilia.com>.
Hi again,
I try the config using keepAliveTime to 10:
<Transport
className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
timeout="60000" keepAliveTime="10"
keepAliveCount="0"/>
One more time, the cluster is not working, the big problem is that I cannot
reproduce the error at my backup server that works perfectly.
Node 2, drops a log error at 12:58 AM, then, at the same time, node 1 report
"ClusterError" continuously
(Continuous errors are on every hit; the server supports 1 hit per second)
Logs:
NODE 2 - LOG
=============
Jan 31, 2008 12:58:13 PM
org.apache.catalina.tribes.transport.nio.NioReceiver socketTimeouts
WARNING: Channel key is registered, but has had no interest ops for the last
3000 ms. (cancelled:false):sun.nio.ch.SelectionKeyImpl@c3271d last
access:2008-01-31 12:58:10.208
NODE 1 - LOG
=============
Jan 31, 2008 12:58:04 PM
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Received
memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://loc
alhost:4002,localhost,4002, alive=101194547,id={123 -66 95 -10 88
24 77 -32 -93 16 -13 -112 90 52 -18 78 }, payload={}, command={}, domain={},
]] message. Will verify.
Jan 31, 2008 12:58:04 PM
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Verification complete. Member still
alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://localhost:4002,
localhost,4002, alive=101194547,id={123
-66 95 -10 88 24 77 -32 -93 16 -13 -112 90 52 -18 78 }, payload={},
command={}, domain={}, ]]
Jan 31, 2008 12:58:04 PM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out(60000
ms.).; Faulty members:tcp://localhost:4002;
at
org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(Paral
lelNioSender.java:97)
at
org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(Po
oledParallelSender.java:53)
at
org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(Repl
icationTransmitter.java:80)
at
org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoord
inator.java:78)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
nterceptorBase.java:75)
at
org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor.sendMess
age(ThroughputInterceptor.java:61)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
nterceptorBase.java:75)
at
org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sen
dMessage(MessageDispatchInterceptor.java:73)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
nterceptorBase.java:75)
at
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage
(TcpFailureDetector.java:87)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
nterceptorBase.java:75)
at
org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
at
org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
at
org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:835)
at
org.apache.catalina.ha.tcp.SimpleTcpCluster.sendClusterDomain(SimpleTcpClust
er.java:814)
at
org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:551)
at
org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.jav
a:535)
at
org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(Re
plicationValve.java:517)
at
org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(Replicati
onValve.java:428)
at
org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:362
)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:263)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http
11Protocol.java:584)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:619)
Jan 31, 2008 12:58:07 PM
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Received
memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://loc
alhost:4002,localhost,4002, alive=101197553,id={123 -66 95 -10 88
24 77 -32 -93 16 -13 -112 90 52 -18 78 }, payload={}, command={}, domain={},
]] message. Will verify.
Jan 31, 2008 12:58:07 PM
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Verification complete. Member still
alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://localhost:4002,
localhost,4002, alive=101197553,id={123
-66 95 -10 88 24 77 -32 -93 16 -13 -112 90 52 -18 78 }, payload={},
command={}, domain={}, ]]
[...] repeats on every hit.
========================
I cannot understand the node 2 log, why is the node 2 crashing??
What can I do??
Thanks on advance.
Raúl.
-----Mensaje original-----
De: Filip Hanik - Dev Lists [mailto:devlists@hanik.com]
Enviado el: lunes, 28 de enero de 2008 1:45
Para: Tomcat Users List
Asunto: Re: Tomcat 6 - Cluster error.
I'd set keepAliveTime to 10 as well,
Filip
Raúl García wrote:
> Hi Again, once again thanks for your time, but we still have problems,
>
> We applied the "keepAliveCount=0" param. and last Wednesday 23 Jan we
> restart both nodes.
>
> Around 11 hour after the startup, node 1 reports a new error, but both
nodes
> are working perfectly.
>
> I cannot imagine why the member disappear unexpectedly, I repost the
error,
> and the config files.
>
> INSTANCE 1 - LOG
> ================
> Jan 24, 2008 10:25:54 PM
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
> memberDisappeared
> INFO: Received
>
memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://loc
> alhost:4002,localhost,4002, alive=123412856,id={-31 -91 -122 -60 -58 -5 68
> 25 -87 13 -20 -12 -100 5 -16 94 }, payload={}, command={}, domain={}, ]]
> message. Will verify.
> Jan 24, 2008 10:25:54 PM
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
> memberDisappeared
> INFO: Verification complete. Member still
>
alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://localhost:4002,
> localhost,4002, alive=123412856,id={-31 -91 -122 -60 -58 -5 68 25 -87 13
-20
> -12 -100 5 -16 94 }, payload={}, command={}, domain={}, ]]
> Jan 24, 2008 10:25:54 PM org.apache.catalina.ha.tcp.SimpleTcpCluster send
> SEVERE: Unable to send message through cluster sender.
> org.apache.catalina.tribes.ChannelException: Operation has timed out(60000
> ms.).; Faulty members:tcp://localhost:4002;
> at
>
org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(Paral
> lelNioSender.java:97)
> at
>
org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(Po
> oledParallelSender.java:53)
> at
>
org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(Repl
> icationTransmitter.java:80)
> at
>
org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoord
> inator.java:78)
> at
>
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
> nterceptorBase.java:75)
> at
>
org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor.sendMess
> age(ThroughputInterceptor.java:61)
> at
>
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
> nterceptorBase.java:75)
> at
>
org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sen
> dMessage(MessageDispatchInterceptor.java:73)
> at
>
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
> nterceptorBase.java:75)
> at
>
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage
> (TcpFailureDetector.java:87)
> at
>
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
> nterceptorBase.java:75)
> at
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
> at
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
> at
>
org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:835)
> at
>
org.apache.catalina.ha.tcp.SimpleTcpCluster.sendClusterDomain(SimpleTcpClust
> er.java:814)
> at
>
org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:551)
> at
>
org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.jav
> a:535)
> at
>
org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(Re
> plicationValve.java:517)
> at
>
org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(Replicati
> onValve.java:428)
> at
>
org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:362
> )
> at
>
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:263)
> at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
> at
>
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http
> 11Protocol.java:584)
> at
> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
> at java.lang.Thread.run(Thread.java:619)
>
> Jan 24, 2008 10:26:54 PM
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
> memberDisappeared
> INFO: Received memberDisappeared [...] repeats only once again.
>
> Jan 25, 2008 5:37:52 AM
> org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor report
> INFO: ThroughputInterceptor Report[
> Tx Msg:66167 messages
> Sent:37.02 MB (total)
> Sent:37.02 MB (application)
> Time:118.53 seconds
> Tx Speed:0.31 MB/sec (total)
> TxSpeed:0.31 MB/sec (application)
> Error Msg:2
> Rx Msg:90000 messages
> Rx Speed:0.00 MB/sec (since 1st msg)
> Received:41.06 MB]
>
>
>
>
> INSTANCE-1 --- Server.xml
> ==========================
> NOTE:: 111.111.111.111 is the server ip address.
> ==========================
> <Server port="8006" shutdown="SHUTDOWN" debug="0">
> <Listener className="org.apache.catalina.core.JasperListener"
debug="0"/>
> <Listener className="org.apache.catalina.mbeans.ServerLifecycleListener"
> debug="0"/>
> <Listener
> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
> debug="0"/>
>
> <GlobalNamingResources>
> <Environment name="InstanceName" type="java.lang.String"
value="pro1"/>
>
> <Resource name="UserDatabase" auth="Container"
> type="org.apache.catalina.UserDatabase"
> description="User database that can be updated and saved"
>
factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
> pathname="conf/tomcat-users.xml" />
> </GlobalNamingResources>
>
> <Service name="Catalina">
>
> <Connector port="8081" protocol="HTTP/1.1" maxHttpHeaderSize="8192"
> emptySessionPath="true"
> maxThreads="150" minSpareThreads="100"
maxSpareThreads="300"
> enableLookups="false" redirectPort="81443"
acceptCount="1000"
> debug="0" connectionTimeout="20000"
> disableUploadTimeout="true"
> compression="on"
> compressionMinSize="2048"
> noCompressionUserAgents="gozilla, traviata"
> compressableMimeType="text/html,text/xml" />
>
> <Engine name="Catalina" defaultHost="localhost" debug="0"
> jvmRoute="PR1">
> <Cluster
> className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
> channelSendOptions="6">
>
> <Manager className="org.apache.catalina.ha.session.DeltaManager"
> expireSessionsOnShutdown="false"
> notifyListenersOnReplication="true"/>
>
> <Channel
> className="org.apache.catalina.tribes.group.GroupChannel">
> <Membership
> className="org.apache.catalina.tribes.membership.McastService"
> address="228.0.0.4"
> port="45564"
> frequency="1000"
> dropTime="30000"/>
> <Receiver
> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
> address="127.0.0.1"
> port="4001"
> autoBind="100"
> selectorTimeout="5000"
> maxThreads="12"/>
>
> <Sender
> className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
> <Transport
> className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
> timeout="60000" keepAliveCount="0"/>
> </Sender>
> <Interceptor
>
className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"
> />
> <Interceptor
>
className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15In
> terceptor"/>
> <Interceptor
>
className="org.apache.catalina.tribes.group.interceptors.ThroughputIntercept
> or"/>
> </Channel>
>
> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
>
>
filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.css;.*\.txt;"/>
>
> <Deployer
> className="org.apache.catalina.ha.deploy.FarmWarDeployer"
> tempDir="/tmp/war-temp/"
> deployDir="/tmp/war-deploy/"
> watchDir="/tmp/war-listen/"
> watchEnabled="false"/>
>
> <ClusterListener
>
className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
> <ClusterListener
> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
> </Cluster>
> <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
> debug="0" resourceName="UserDatabase"/>
> <Host name="localhost" debug="0" appBase="webapps"
> unpackWARs="true" autoDeploy="true"
> xmlValidation="false" xmlNamespaceAware="false">
> <Valve className="org.apache.catalina.valves.RemoteAddrValve"
> allow="10.0.0.*,127.0.0.1,228.0.0.4,111.111.111.111"/>
> </Host>
> </Engine>
> </Service>
> </Server>
> ==============================================
>
>
> INSTANCE-2 server.xml
> =====================
> <Server port="8007" shutdown="SHUTDOWN" debug="0">
>
> <Listener className="org.apache.catalina.core.JasperListener"
debug="0"/>
> <Listener className="org.apache.catalina.mbeans.ServerLifecycleListener"
> debug="0"/>
> <Listener
> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
> debug="0"/>
>
> <GlobalNamingResources>
>
> <Environment name="InstanceName" type="java.lang.String"
value="pro2"/>
>
> <Resource name="UserDatabase" auth="Container"
> type="org.apache.catalina.UserDatabase"
> description="User database that can be updated and saved"
>
factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
> pathname="conf/tomcat-users.xml"/>
> </GlobalNamingResources>
>
> <Service name="Catalina">
>
> <Connector port="8082" protocol="HTTP/1.1" maxHttpHeaderSize="8192"
> emptySessionPath="true"
> maxThreads="150" minSpareThreads="100"
maxSpareThreads="300"
> enableLookups="false" redirectPort="82443"
acceptCount="1000"
> debug="0" connectionTimeout="20000"
> disableUploadTimeout="true"
> compression="on"
> compressionMinSize="2048"
> noCompressionUserAgents="gozilla, traviata"
> compressableMimeType="text/html,text/xml" />
> <Engine name="Catalina" defaultHost="localhost" debug="0"
> jvmRoute="PR2">
>
> <Cluster
> className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
> channelSendOptions="6">
>
>
> <Manager className="org.apache.catalina.ha.session.DeltaManager"
> expireSessionsOnShutdown="false"
> notifyListenersOnReplication="true"/>
>
> <Channel
> className="org.apache.catalina.tribes.group.GroupChannel">
> <Membership
> className="org.apache.catalina.tribes.membership.McastService"
> address="228.0.0.4"
> port="45564"
> frequency="1000"
> dropTime="30000"/>
> <Receiver
> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
> address="127.0.0.1"
> port="4002"
> autoBind="100"
> selectorTimeout="5000"
> maxThreads="12"/>
>
> <Sender
> className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
> <Transport
> className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
> timeout="60000" keepAliveCount="0"/>
> </Sender>
> <Interceptor
>
className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"
> />
> <Interceptor
>
className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15In
> terceptor"/>
> <Interceptor
>
className="org.apache.catalina.tribes.group.interceptors.ThroughputIntercept
> or"/>
> </Channel>
>
> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
>
>
filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.css;.*\.txt;"/>
> <!-- <Valve
> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/> -->
>
> <Deployer
> className="org.apache.catalina.ha.deploy.FarmWarDeployer"
> tempDir="/tmp/war-temp/"
> deployDir="/tmp/war-deploy/"
> watchDir="/tmp/war-listen/"
> watchEnabled="false"/>
>
> <ClusterListener
>
className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
> <ClusterListener
> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
> </Cluster>
>
> <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
> resourceName="UserDatabase" debug="0"/>
>
> <Host name="localhost" debug="0" appBase="webapps"
> unpackWARs="true" autoDeploy="true"
> xmlValidation="false" xmlNamespaceAware="false">
>
> <Valve className="org.apache.catalina.valves.RemoteAddrValve"
> allow="10.0.0.*,127.0.0.1,228.0.0.4,111.111.111.111"/>
> </Host>
> </Engine>
> </Service>
> </Server>
> ===============================
>
> -----Mensaje original-----
> De: Filip Hanik - Dev Lists [mailto:devlists@hanik.com]
> Enviado el: jueves, 17 de enero de 2008 19:01
> Para: Tomcat Users List
> Asunto: Re: Tomcat 6 - Cluster error.
>
> already replied to your old thread
>
> ok, it looks like you might have ended up with a rogue socket,
> and what happens is that any message sent to that socket just gets lost
> in the ether, since it doesn't have any interest ops.
> There is a workaround for this, turn off keep alives all together, or
> implement a keep alive timeout
>
> Option 1 - no keep alives at all
>
> <Transport
> className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
> timeout="60000"
> keepAliveCount="0"/>
>
> Option 2 - implement a keep alive timeout
>
> <Transport
> className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
> timeout="60000"
> keepAliveTime="120000"/>
>
> or make a combination of both values
>
> either option should work for you.
>
> On a side note, I'm interested if the scenario you run into is
> reproducible, it keeps happening over and over again, then if possible,
> I'd like to get some debug logs from you
>
> Filip
>
>
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 6 - Cluster error.
Posted by Filip Hanik - Dev Lists <de...@hanik.com>.
I'd set keepAliveTime to 10 as well,
Filip
Raúl García wrote:
> Hi Again, once again thanks for your time, but we still have problems,
>
> We applied the "keepAliveCount=0" param. and last Wednesday 23 Jan we
> restart both nodes.
>
> Around 11 hour after the startup, node 1 reports a new error, but both nodes
> are working perfectly.
>
> I cannot imagine why the member disappear unexpectedly, I repost the error,
> and the config files.
>
> INSTANCE 1 - LOG
> ================
> Jan 24, 2008 10:25:54 PM
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
> memberDisappeared
> INFO: Received
> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://loc
> alhost:4002,localhost,4002, alive=123412856,id={-31 -91 -122 -60 -58 -5 68
> 25 -87 13 -20 -12 -100 5 -16 94 }, payload={}, command={}, domain={}, ]]
> message. Will verify.
> Jan 24, 2008 10:25:54 PM
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
> memberDisappeared
> INFO: Verification complete. Member still
> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://localhost:4002,
> localhost,4002, alive=123412856,id={-31 -91 -122 -60 -58 -5 68 25 -87 13 -20
> -12 -100 5 -16 94 }, payload={}, command={}, domain={}, ]]
> Jan 24, 2008 10:25:54 PM org.apache.catalina.ha.tcp.SimpleTcpCluster send
> SEVERE: Unable to send message through cluster sender.
> org.apache.catalina.tribes.ChannelException: Operation has timed out(60000
> ms.).; Faulty members:tcp://localhost:4002;
> at
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(Paral
> lelNioSender.java:97)
> at
> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(Po
> oledParallelSender.java:53)
> at
> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(Repl
> icationTransmitter.java:80)
> at
> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoord
> inator.java:78)
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
> nterceptorBase.java:75)
> at
> org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor.sendMess
> age(ThroughputInterceptor.java:61)
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
> nterceptorBase.java:75)
> at
> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sen
> dMessage(MessageDispatchInterceptor.java:73)
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
> nterceptorBase.java:75)
> at
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage
> (TcpFailureDetector.java:87)
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
> nterceptorBase.java:75)
> at
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
> at
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
> at
> org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:835)
> at
> org.apache.catalina.ha.tcp.SimpleTcpCluster.sendClusterDomain(SimpleTcpClust
> er.java:814)
> at
> org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:551)
> at
> org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.jav
> a:535)
> at
> org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(Re
> plicationValve.java:517)
> at
> org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(Replicati
> onValve.java:428)
> at
> org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:362
> )
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:263)
> at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
> at
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http
> 11Protocol.java:584)
> at
> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
> at java.lang.Thread.run(Thread.java:619)
>
> Jan 24, 2008 10:26:54 PM
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
> memberDisappeared
> INFO: Received memberDisappeared [...] repeats only once again.
>
> Jan 25, 2008 5:37:52 AM
> org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor report
> INFO: ThroughputInterceptor Report[
> Tx Msg:66167 messages
> Sent:37.02 MB (total)
> Sent:37.02 MB (application)
> Time:118.53 seconds
> Tx Speed:0.31 MB/sec (total)
> TxSpeed:0.31 MB/sec (application)
> Error Msg:2
> Rx Msg:90000 messages
> Rx Speed:0.00 MB/sec (since 1st msg)
> Received:41.06 MB]
>
>
>
>
> INSTANCE-1 --- Server.xml
> ==========================
> NOTE:: 111.111.111.111 is the server ip address.
> ==========================
> <Server port="8006" shutdown="SHUTDOWN" debug="0">
> <Listener className="org.apache.catalina.core.JasperListener" debug="0"/>
> <Listener className="org.apache.catalina.mbeans.ServerLifecycleListener"
> debug="0"/>
> <Listener
> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
> debug="0"/>
>
> <GlobalNamingResources>
> <Environment name="InstanceName" type="java.lang.String" value="pro1"/>
>
> <Resource name="UserDatabase" auth="Container"
> type="org.apache.catalina.UserDatabase"
> description="User database that can be updated and saved"
> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
> pathname="conf/tomcat-users.xml" />
> </GlobalNamingResources>
>
> <Service name="Catalina">
>
> <Connector port="8081" protocol="HTTP/1.1" maxHttpHeaderSize="8192"
> emptySessionPath="true"
> maxThreads="150" minSpareThreads="100" maxSpareThreads="300"
> enableLookups="false" redirectPort="81443" acceptCount="1000"
> debug="0" connectionTimeout="20000"
> disableUploadTimeout="true"
> compression="on"
> compressionMinSize="2048"
> noCompressionUserAgents="gozilla, traviata"
> compressableMimeType="text/html,text/xml" />
>
> <Engine name="Catalina" defaultHost="localhost" debug="0"
> jvmRoute="PR1">
> <Cluster
> className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
> channelSendOptions="6">
>
> <Manager className="org.apache.catalina.ha.session.DeltaManager"
> expireSessionsOnShutdown="false"
> notifyListenersOnReplication="true"/>
>
> <Channel
> className="org.apache.catalina.tribes.group.GroupChannel">
> <Membership
> className="org.apache.catalina.tribes.membership.McastService"
> address="228.0.0.4"
> port="45564"
> frequency="1000"
> dropTime="30000"/>
> <Receiver
> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
> address="127.0.0.1"
> port="4001"
> autoBind="100"
> selectorTimeout="5000"
> maxThreads="12"/>
>
> <Sender
> className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
> <Transport
> className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
> timeout="60000" keepAliveCount="0"/>
> </Sender>
> <Interceptor
> className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"
> />
> <Interceptor
> className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15In
> terceptor"/>
> <Interceptor
> className="org.apache.catalina.tribes.group.interceptors.ThroughputIntercept
> or"/>
> </Channel>
>
> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
>
> filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.css;.*\.txt;"/>
>
> <Deployer
> className="org.apache.catalina.ha.deploy.FarmWarDeployer"
> tempDir="/tmp/war-temp/"
> deployDir="/tmp/war-deploy/"
> watchDir="/tmp/war-listen/"
> watchEnabled="false"/>
>
> <ClusterListener
> className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
> <ClusterListener
> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
> </Cluster>
> <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
> debug="0" resourceName="UserDatabase"/>
> <Host name="localhost" debug="0" appBase="webapps"
> unpackWARs="true" autoDeploy="true"
> xmlValidation="false" xmlNamespaceAware="false">
> <Valve className="org.apache.catalina.valves.RemoteAddrValve"
> allow="10.0.0.*,127.0.0.1,228.0.0.4,111.111.111.111"/>
> </Host>
> </Engine>
> </Service>
> </Server>
> ==============================================
>
>
> INSTANCE-2 server.xml
> =====================
> <Server port="8007" shutdown="SHUTDOWN" debug="0">
>
> <Listener className="org.apache.catalina.core.JasperListener" debug="0"/>
> <Listener className="org.apache.catalina.mbeans.ServerLifecycleListener"
> debug="0"/>
> <Listener
> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
> debug="0"/>
>
> <GlobalNamingResources>
>
> <Environment name="InstanceName" type="java.lang.String" value="pro2"/>
>
> <Resource name="UserDatabase" auth="Container"
> type="org.apache.catalina.UserDatabase"
> description="User database that can be updated and saved"
> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
> pathname="conf/tomcat-users.xml"/>
> </GlobalNamingResources>
>
> <Service name="Catalina">
>
> <Connector port="8082" protocol="HTTP/1.1" maxHttpHeaderSize="8192"
> emptySessionPath="true"
> maxThreads="150" minSpareThreads="100" maxSpareThreads="300"
> enableLookups="false" redirectPort="82443" acceptCount="1000"
> debug="0" connectionTimeout="20000"
> disableUploadTimeout="true"
> compression="on"
> compressionMinSize="2048"
> noCompressionUserAgents="gozilla, traviata"
> compressableMimeType="text/html,text/xml" />
> <Engine name="Catalina" defaultHost="localhost" debug="0"
> jvmRoute="PR2">
>
> <Cluster
> className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
> channelSendOptions="6">
>
>
> <Manager className="org.apache.catalina.ha.session.DeltaManager"
> expireSessionsOnShutdown="false"
> notifyListenersOnReplication="true"/>
>
> <Channel
> className="org.apache.catalina.tribes.group.GroupChannel">
> <Membership
> className="org.apache.catalina.tribes.membership.McastService"
> address="228.0.0.4"
> port="45564"
> frequency="1000"
> dropTime="30000"/>
> <Receiver
> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
> address="127.0.0.1"
> port="4002"
> autoBind="100"
> selectorTimeout="5000"
> maxThreads="12"/>
>
> <Sender
> className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
> <Transport
> className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
> timeout="60000" keepAliveCount="0"/>
> </Sender>
> <Interceptor
> className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"
> />
> <Interceptor
> className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15In
> terceptor"/>
> <Interceptor
> className="org.apache.catalina.tribes.group.interceptors.ThroughputIntercept
> or"/>
> </Channel>
>
> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
>
> filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.css;.*\.txt;"/>
> <!-- <Valve
> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/> -->
>
> <Deployer
> className="org.apache.catalina.ha.deploy.FarmWarDeployer"
> tempDir="/tmp/war-temp/"
> deployDir="/tmp/war-deploy/"
> watchDir="/tmp/war-listen/"
> watchEnabled="false"/>
>
> <ClusterListener
> className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
> <ClusterListener
> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
> </Cluster>
>
> <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
> resourceName="UserDatabase" debug="0"/>
>
> <Host name="localhost" debug="0" appBase="webapps"
> unpackWARs="true" autoDeploy="true"
> xmlValidation="false" xmlNamespaceAware="false">
>
> <Valve className="org.apache.catalina.valves.RemoteAddrValve"
> allow="10.0.0.*,127.0.0.1,228.0.0.4,111.111.111.111"/>
> </Host>
> </Engine>
> </Service>
> </Server>
> ===============================
>
> -----Mensaje original-----
> De: Filip Hanik - Dev Lists [mailto:devlists@hanik.com]
> Enviado el: jueves, 17 de enero de 2008 19:01
> Para: Tomcat Users List
> Asunto: Re: Tomcat 6 - Cluster error.
>
> already replied to your old thread
>
> ok, it looks like you might have ended up with a rogue socket,
> and what happens is that any message sent to that socket just gets lost
> in the ether, since it doesn't have any interest ops.
> There is a workaround for this, turn off keep alives all together, or
> implement a keep alive timeout
>
> Option 1 - no keep alives at all
>
> <Transport
> className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
> timeout="60000"
> keepAliveCount="0"/>
>
> Option 2 - implement a keep alive timeout
>
> <Transport
> className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
> timeout="60000"
> keepAliveTime="120000"/>
>
> or make a combination of both values
>
> either option should work for you.
>
> On a side note, I'm interested if the scenario you run into is
> reproducible, it keeps happening over and over again, then if possible,
> I'd like to get some debug logs from you
>
> Filip
>
>
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat 6 - Cluster error.
Posted by Raúl García <rg...@emovilia.com>.
Hi Again, once again thanks for your time, but we still have problems,
We applied the "keepAliveCount=0" param. and last Wednesday 23 Jan we
restart both nodes.
Around 11 hour after the startup, node 1 reports a new error, but both nodes
are working perfectly.
I cannot imagine why the member disappear unexpectedly, I repost the error,
and the config files.
INSTANCE 1 - LOG
================
Jan 24, 2008 10:25:54 PM
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Received
memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://loc
alhost:4002,localhost,4002, alive=123412856,id={-31 -91 -122 -60 -58 -5 68
25 -87 13 -20 -12 -100 5 -16 94 }, payload={}, command={}, domain={}, ]]
message. Will verify.
Jan 24, 2008 10:25:54 PM
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Verification complete. Member still
alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://localhost:4002,
localhost,4002, alive=123412856,id={-31 -91 -122 -60 -58 -5 68 25 -87 13 -20
-12 -100 5 -16 94 }, payload={}, command={}, domain={}, ]]
Jan 24, 2008 10:25:54 PM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out(60000
ms.).; Faulty members:tcp://localhost:4002;
at
org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(Paral
lelNioSender.java:97)
at
org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(Po
oledParallelSender.java:53)
at
org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(Repl
icationTransmitter.java:80)
at
org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoord
inator.java:78)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
nterceptorBase.java:75)
at
org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor.sendMess
age(ThroughputInterceptor.java:61)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
nterceptorBase.java:75)
at
org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sen
dMessage(MessageDispatchInterceptor.java:73)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
nterceptorBase.java:75)
at
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage
(TcpFailureDetector.java:87)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
nterceptorBase.java:75)
at
org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
at
org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
at
org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:835)
at
org.apache.catalina.ha.tcp.SimpleTcpCluster.sendClusterDomain(SimpleTcpClust
er.java:814)
at
org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:551)
at
org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.jav
a:535)
at
org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(Re
plicationValve.java:517)
at
org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(Replicati
onValve.java:428)
at
org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:362
)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:263)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http
11Protocol.java:584)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:619)
Jan 24, 2008 10:26:54 PM
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Received memberDisappeared [...] repeats only once again.
Jan 25, 2008 5:37:52 AM
org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor report
INFO: ThroughputInterceptor Report[
Tx Msg:66167 messages
Sent:37.02 MB (total)
Sent:37.02 MB (application)
Time:118.53 seconds
Tx Speed:0.31 MB/sec (total)
TxSpeed:0.31 MB/sec (application)
Error Msg:2
Rx Msg:90000 messages
Rx Speed:0.00 MB/sec (since 1st msg)
Received:41.06 MB]
INSTANCE-1 --- Server.xml
==========================
NOTE:: 111.111.111.111 is the server ip address.
==========================
<Server port="8006" shutdown="SHUTDOWN" debug="0">
<Listener className="org.apache.catalina.core.JasperListener" debug="0"/>
<Listener className="org.apache.catalina.mbeans.ServerLifecycleListener"
debug="0"/>
<Listener
className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
debug="0"/>
<GlobalNamingResources>
<Environment name="InstanceName" type="java.lang.String" value="pro1"/>
<Resource name="UserDatabase" auth="Container"
type="org.apache.catalina.UserDatabase"
description="User database that can be updated and saved"
factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
pathname="conf/tomcat-users.xml" />
</GlobalNamingResources>
<Service name="Catalina">
<Connector port="8081" protocol="HTTP/1.1" maxHttpHeaderSize="8192"
emptySessionPath="true"
maxThreads="150" minSpareThreads="100" maxSpareThreads="300"
enableLookups="false" redirectPort="81443" acceptCount="1000"
debug="0" connectionTimeout="20000"
disableUploadTimeout="true"
compression="on"
compressionMinSize="2048"
noCompressionUserAgents="gozilla, traviata"
compressableMimeType="text/html,text/xml" />
<Engine name="Catalina" defaultHost="localhost" debug="0"
jvmRoute="PR1">
<Cluster
className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
channelSendOptions="6">
<Manager className="org.apache.catalina.ha.session.DeltaManager"
expireSessionsOnShutdown="false"
notifyListenersOnReplication="true"/>
<Channel
className="org.apache.catalina.tribes.group.GroupChannel">
<Membership
className="org.apache.catalina.tribes.membership.McastService"
address="228.0.0.4"
port="45564"
frequency="1000"
dropTime="30000"/>
<Receiver
className="org.apache.catalina.tribes.transport.nio.NioReceiver"
address="127.0.0.1"
port="4001"
autoBind="100"
selectorTimeout="5000"
maxThreads="12"/>
<Sender
className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
<Transport
className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
timeout="60000" keepAliveCount="0"/>
</Sender>
<Interceptor
className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"
/>
<Interceptor
className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15In
terceptor"/>
<Interceptor
className="org.apache.catalina.tribes.group.interceptors.ThroughputIntercept
or"/>
</Channel>
<Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.css;.*\.txt;"/>
<Deployer
className="org.apache.catalina.ha.deploy.FarmWarDeployer"
tempDir="/tmp/war-temp/"
deployDir="/tmp/war-deploy/"
watchDir="/tmp/war-listen/"
watchEnabled="false"/>
<ClusterListener
className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
<ClusterListener
className="org.apache.catalina.ha.session.ClusterSessionListener"/>
</Cluster>
<Realm className="org.apache.catalina.realm.UserDatabaseRealm"
debug="0" resourceName="UserDatabase"/>
<Host name="localhost" debug="0" appBase="webapps"
unpackWARs="true" autoDeploy="true"
xmlValidation="false" xmlNamespaceAware="false">
<Valve className="org.apache.catalina.valves.RemoteAddrValve"
allow="10.0.0.*,127.0.0.1,228.0.0.4,111.111.111.111"/>
</Host>
</Engine>
</Service>
</Server>
==============================================
INSTANCE-2 server.xml
=====================
<Server port="8007" shutdown="SHUTDOWN" debug="0">
<Listener className="org.apache.catalina.core.JasperListener" debug="0"/>
<Listener className="org.apache.catalina.mbeans.ServerLifecycleListener"
debug="0"/>
<Listener
className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
debug="0"/>
<GlobalNamingResources>
<Environment name="InstanceName" type="java.lang.String" value="pro2"/>
<Resource name="UserDatabase" auth="Container"
type="org.apache.catalina.UserDatabase"
description="User database that can be updated and saved"
factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
pathname="conf/tomcat-users.xml"/>
</GlobalNamingResources>
<Service name="Catalina">
<Connector port="8082" protocol="HTTP/1.1" maxHttpHeaderSize="8192"
emptySessionPath="true"
maxThreads="150" minSpareThreads="100" maxSpareThreads="300"
enableLookups="false" redirectPort="82443" acceptCount="1000"
debug="0" connectionTimeout="20000"
disableUploadTimeout="true"
compression="on"
compressionMinSize="2048"
noCompressionUserAgents="gozilla, traviata"
compressableMimeType="text/html,text/xml" />
<Engine name="Catalina" defaultHost="localhost" debug="0"
jvmRoute="PR2">
<Cluster
className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
channelSendOptions="6">
<Manager className="org.apache.catalina.ha.session.DeltaManager"
expireSessionsOnShutdown="false"
notifyListenersOnReplication="true"/>
<Channel
className="org.apache.catalina.tribes.group.GroupChannel">
<Membership
className="org.apache.catalina.tribes.membership.McastService"
address="228.0.0.4"
port="45564"
frequency="1000"
dropTime="30000"/>
<Receiver
className="org.apache.catalina.tribes.transport.nio.NioReceiver"
address="127.0.0.1"
port="4002"
autoBind="100"
selectorTimeout="5000"
maxThreads="12"/>
<Sender
className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
<Transport
className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
timeout="60000" keepAliveCount="0"/>
</Sender>
<Interceptor
className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"
/>
<Interceptor
className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15In
terceptor"/>
<Interceptor
className="org.apache.catalina.tribes.group.interceptors.ThroughputIntercept
or"/>
</Channel>
<Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.css;.*\.txt;"/>
<!-- <Valve
className="org.apache.catalina.ha.session.JvmRouteBinderValve"/> -->
<Deployer
className="org.apache.catalina.ha.deploy.FarmWarDeployer"
tempDir="/tmp/war-temp/"
deployDir="/tmp/war-deploy/"
watchDir="/tmp/war-listen/"
watchEnabled="false"/>
<ClusterListener
className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
<ClusterListener
className="org.apache.catalina.ha.session.ClusterSessionListener"/>
</Cluster>
<Realm className="org.apache.catalina.realm.UserDatabaseRealm"
resourceName="UserDatabase" debug="0"/>
<Host name="localhost" debug="0" appBase="webapps"
unpackWARs="true" autoDeploy="true"
xmlValidation="false" xmlNamespaceAware="false">
<Valve className="org.apache.catalina.valves.RemoteAddrValve"
allow="10.0.0.*,127.0.0.1,228.0.0.4,111.111.111.111"/>
</Host>
</Engine>
</Service>
</Server>
===============================
-----Mensaje original-----
De: Filip Hanik - Dev Lists [mailto:devlists@hanik.com]
Enviado el: jueves, 17 de enero de 2008 19:01
Para: Tomcat Users List
Asunto: Re: Tomcat 6 - Cluster error.
already replied to your old thread
ok, it looks like you might have ended up with a rogue socket,
and what happens is that any message sent to that socket just gets lost
in the ether, since it doesn't have any interest ops.
There is a workaround for this, turn off keep alives all together, or
implement a keep alive timeout
Option 1 - no keep alives at all
<Transport
className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
timeout="60000"
keepAliveCount="0"/>
Option 2 - implement a keep alive timeout
<Transport
className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
timeout="60000"
keepAliveTime="120000"/>
or make a combination of both values
either option should work for you.
On a side note, I'm interested if the scenario you run into is
reproducible, it keeps happening over and over again, then if possible,
I'd like to get some debug logs from you
Filip
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 6 - Cluster error.
Posted by Filip Hanik - Dev Lists <de...@hanik.com>.
already replied to your old thread
ok, it looks like you might have ended up with a rogue socket,
and what happens is that any message sent to that socket just gets lost
in the ether, since it doesn't have any interest ops.
There is a workaround for this, turn off keep alives all together, or
implement a keep alive timeout
Option 1 - no keep alives at all
<Transport
className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
timeout="60000"
keepAliveCount="0"/>
Option 2 - implement a keep alive timeout
<Transport
className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
timeout="60000"
keepAliveTime="120000"/>
or make a combination of both values
either option should work for you.
On a side note, I'm interested if the scenario you run into is
reproducible, it keeps happening over and over again, then if possible,
I'd like to get some debug logs from you
Filip
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org