You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Theparanoidone Theparanoidone <th...@yahoo.com> on 2009/04/12 00:03:02 UTC
Replication Stops Overnight
Greetings,
We are using tomcat6 cluster in two data-centers with replicated sessions. We are not using multicast because multicast broadcast between network subnets/zones is difficult to setup, so instead we are using static members (data center 1: webd101, and webd102 / data center 2: webd201, webd202).
Everything works fine during normal hours and *all 4* servers receive session replication... but if I go home for the night and come back in the morning... data center 1 does not replicate new sessions to data center 2 (and vice versa). For example, a new session will get replicated between: webd101 and webd102 ... but it will not make its way over to webd201 and webd202.
It's like the connection between data centers gets clipped by the ??firewall?? ... and won't automatically pick back up.
The strange thing is... the heartbeats are still flowing so all 4 tomcat instances still know they are all alive.
I'm currently testing explicity setting the firewall to "never" clip connections on port 4000... I have to wait until tomorrow to see if it works.
In the meantime... are there any other server.xml settings I can set to try and get this to pick back up?
I found one error message in catalina.out (only on 1 of 4 clusters):
----------------------------
Apr 11, 2009 4:08:47 AM org.apache.catalina.tribes.transport.nio.NioReplicationTask run
WARNING: IOException in replication worker, unable to drain channel. Probable cause: Keep alive socket closed[Connection reset by peer].
---------------------------
Below is the cluster section of my server.xml:
<!-- Clustering support http://tomcat.apache.org/tomcat-6.0-doc/cluster-howto.html -->
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster">
<!-- Specifies not to replicate session data from files that can't alter sessions -->
<Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.css;.*\.txt;.*\.pdf;.*\.jsp;"/>
<!-- Default High Avialibilty Manager -->
<Manager className="org.apache.catalina.ha.session.DeltaManager"
domainReplication="false"
expireSessionsOnShutdown="false"
notifyListenersOnReplication="true"/>
<!-- Default Send Receive Channels -->
<Channel className="org.apache.catalina.tribes.group.GroupChannel">
<!-- Default Non Blocking Receiver -->
<Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
address="auto"
port="4000"
autoBind="0"
selectorTimeout="100"
maxThreads="6"/>
<!-- Default Sender -->
<Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
<Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
</Sender>
<!-- Only Ping Static Entries (No Multicast) -->
<Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpPingInterceptor"
staticOnly="true" />
<Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
<!-- Static Members -->
<Interceptor className="org.apache.catalina.tribes.group.interceptors.StaticMembershipInterceptor">
<!--
<Member className="org.apache.catalina.tribes.membership.StaticMember"
port="4000"
host="webd101"
uniqueId="{10,101,1,121,0,0,0,0,0,0,0,0,0,0,0,0}"/>
-->
<Member className="org.apache.catalina.tribes.membership.StaticMember"
port="4000"
host="webd102"
uniqueId="{10,101,1,122,0,0,0,0,0,0,0,0,0,0,0,0}"/>
<Member className="org.apache.catalina.tribes.membership.StaticMember"
port="4000"
host="webd201"
uniqueId="{10,102,1,121,0,0,0,0,0,0,0,0,0,0,0,0}"/>
<Member className="org.apache.catalina.tribes.membership.StaticMember"
port="4000"
host="webd202"
uniqueId="{10,102,1,122,0,0,0,0,0,0,0,0,0,0,0,0}"/>
</Interceptor>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
<Interceptor className="com.custom.tomcat.interceptors.DisableMcastInterceptor" />
<Interceptor className="org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor"/>
</Channel>
</Cluster>
Thanks for any help in advance!
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Replication Stops Overnight
Posted by Filip Hanik - Dev Lists <de...@hanik.com>.
sounds like a 'stale connection'.
what I would do first, is to turn off keep alive
look up keepAliveCount and keepAliveTime in
http://tomcat.apache.org/tomcat-6.0-doc/config/cluster-sender.html
Filip
Theparanoidone Theparanoidone wrote:
> Greetings,
>
> We are using tomcat6 cluster in two data-centers with replicated sessions. We are not using multicast because multicast broadcast between network subnets/zones is difficult to setup, so instead we are using static members (data center 1: webd101, and webd102 / data center 2: webd201, webd202).
>
> Everything works fine during normal hours and *all 4* servers receive session replication... but if I go home for the night and come back in the morning... data center 1 does not replicate new sessions to data center 2 (and vice versa). For example, a new session will get replicated between: webd101 and webd102 ... but it will not make its way over to webd201 and webd202.
>
> It's like the connection between data centers gets clipped by the ??firewall?? ... and won't automatically pick back up.
>
> The strange thing is... the heartbeats are still flowing so all 4 tomcat instances still know they are all alive.
>
> I'm currently testing explicity setting the firewall to "never" clip connections on port 4000... I have to wait until tomorrow to see if it works.
>
>
> In the meantime... are there any other server.xml settings I can set to try and get this to pick back up?
>
>
> I found one error message in catalina.out (only on 1 of 4 clusters):
> ----------------------------
> Apr 11, 2009 4:08:47 AM org.apache.catalina.tribes.transport.nio.NioReplicationTask run
> WARNING: IOException in replication worker, unable to drain channel. Probable cause: Keep alive socket closed[Connection reset by peer].
> ---------------------------
>
> Below is the cluster section of my server.xml:
>
>
> <!-- Clustering support http://tomcat.apache.org/tomcat-6.0-doc/cluster-howto.html -->
> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster">
>
> <!-- Specifies not to replicate session data from files that can't alter sessions -->
> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
> filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.css;.*\.txt;.*\.pdf;.*\.jsp;"/>
>
> <!-- Default High Avialibilty Manager -->
> <Manager className="org.apache.catalina.ha.session.DeltaManager"
> domainReplication="false"
> expireSessionsOnShutdown="false"
> notifyListenersOnReplication="true"/>
>
> <!-- Default Send Receive Channels -->
> <Channel className="org.apache.catalina.tribes.group.GroupChannel">
>
> <!-- Default Non Blocking Receiver -->
> <Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
> address="auto"
> port="4000"
> autoBind="0"
> selectorTimeout="100"
> maxThreads="6"/>
>
> <!-- Default Sender -->
> <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
> <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
> </Sender>
>
> <!-- Only Ping Static Entries (No Multicast) -->
> <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpPingInterceptor"
> staticOnly="true" />
> <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
>
> <!-- Static Members -->
> <Interceptor className="org.apache.catalina.tribes.group.interceptors.StaticMembershipInterceptor">
> <!--
> <Member className="org.apache.catalina.tribes.membership.StaticMember"
> port="4000"
> host="webd101"
> uniqueId="{10,101,1,121,0,0,0,0,0,0,0,0,0,0,0,0}"/>
> -->
> <Member className="org.apache.catalina.tribes.membership.StaticMember"
> port="4000"
> host="webd102"
> uniqueId="{10,101,1,122,0,0,0,0,0,0,0,0,0,0,0,0}"/>
> <Member className="org.apache.catalina.tribes.membership.StaticMember"
> port="4000"
> host="webd201"
> uniqueId="{10,102,1,121,0,0,0,0,0,0,0,0,0,0,0,0}"/>
> <Member className="org.apache.catalina.tribes.membership.StaticMember"
> port="4000"
> host="webd202"
> uniqueId="{10,102,1,122,0,0,0,0,0,0,0,0,0,0,0,0}"/>
> </Interceptor>
>
> <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
> <Interceptor className="com.custom.tomcat.interceptors.DisableMcastInterceptor" />
> <Interceptor className="org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor"/>
> </Channel>
> </Cluster>
>
>
>
> Thanks for any help in advance!
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Replication Stops Overnight
Posted by Martin Gainty <mg...@hotmail.com>.
the default dropTime from MCastService is 3 secs e.g.
properties.setProperty("memberDropTime","3000");
would suggest increase memberDropTime property (so MCastService detects a Heartbeat)
Martin
does anyone knows what op means about 'going home'?
______________________________________________
Disclaimer and Confidentiality/Verzicht und Vertraulichkeitanmerkung / Note de déni et de confidentialité
This message is confidential. If you should not be the intended receiver, then we ask politely to report. Each unauthorized forwarding or manufacturing of a copy is inadmissible. This message serves only for the exchange of information and has no legal binding effect. Due to the easy manipulation of emails we cannot take responsibility over the the contents.
Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen.
Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité pour le contenu fourni.
> Date: Sat, 11 Apr 2009 15:03:02 -0700
> From: theparanoidone@yahoo.com
> Subject: Replication Stops Overnight
> To: users@tomcat.apache.org
>
>
> Greetings,
>
> We are using tomcat6 cluster in two data-centers with replicated sessions. We are not using multicast because multicast broadcast between network subnets/zones is difficult to setup, so instead we are using static members (data center 1: webd101, and webd102 / data center 2: webd201, webd202).
>
> Everything works fine during normal hours and *all 4* servers receive session replication... but if I go home for the night and come back in the morning... data center 1 does not replicate new sessions to data center 2 (and vice versa). For example, a new session will get replicated between: webd101 and webd102 ... but it will not make its way over to webd201 and webd202.
>
> It's like the connection between data centers gets clipped by the ??firewall?? ... and won't automatically pick back up.
>
> The strange thing is... the heartbeats are still flowing so all 4 tomcat instances still know they are all alive.
>
> I'm currently testing explicity setting the firewall to "never" clip connections on port 4000... I have to wait until tomorrow to see if it works.
>
>
> In the meantime... are there any other server.xml settings I can set to try and get this to pick back up?
>
>
> I found one error message in catalina.out (only on 1 of 4 clusters):
> ----------------------------
> Apr 11, 2009 4:08:47 AM org.apache.catalina.tribes.transport.nio.NioReplicationTask run
> WARNING: IOException in replication worker, unable to drain channel. Probable cause: Keep alive socket closed[Connection reset by peer].
> ---------------------------
>
> Below is the cluster section of my server.xml:
>
>
> <!-- Clustering support http://tomcat.apache.org/tomcat-6.0-doc/cluster-howto.html -->
> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster">
>
> <!-- Specifies not to replicate session data from files that can't alter sessions -->
> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
> filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.css;.*\.txt;.*\.pdf;.*\.jsp;"/>
>
> <!-- Default High Avialibilty Manager -->
> <Manager className="org.apache.catalina.ha.session.DeltaManager"
> domainReplication="false"
> expireSessionsOnShutdown="false"
> notifyListenersOnReplication="true"/>
>
> <!-- Default Send Receive Channels -->
> <Channel className="org.apache.catalina.tribes.group.GroupChannel">
>
> <!-- Default Non Blocking Receiver -->
> <Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
> address="auto"
> port="4000"
> autoBind="0"
> selectorTimeout="100"
> maxThreads="6"/>
>
> <!-- Default Sender -->
> <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
> <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
> </Sender>
>
> <!-- Only Ping Static Entries (No Multicast) -->
> <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpPingInterceptor"
> staticOnly="true" />
> <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
>
> <!-- Static Members -->
> <Interceptor className="org.apache.catalina.tribes.group.interceptors.StaticMembershipInterceptor">
> <!--
> <Member className="org.apache.catalina.tribes.membership.StaticMember"
> port="4000"
> host="webd101"
> uniqueId="{10,101,1,121,0,0,0,0,0,0,0,0,0,0,0,0}"/>
> -->
> <Member className="org.apache.catalina.tribes.membership.StaticMember"
> port="4000"
> host="webd102"
> uniqueId="{10,101,1,122,0,0,0,0,0,0,0,0,0,0,0,0}"/>
> <Member className="org.apache.catalina.tribes.membership.StaticMember"
> port="4000"
> host="webd201"
> uniqueId="{10,102,1,121,0,0,0,0,0,0,0,0,0,0,0,0}"/>
> <Member className="org.apache.catalina.tribes.membership.StaticMember"
> port="4000"
> host="webd202"
> uniqueId="{10,102,1,122,0,0,0,0,0,0,0,0,0,0,0,0}"/>
> </Interceptor>
>
> <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
> <Interceptor className="com.custom.tomcat.interceptors.DisableMcastInterceptor" />
> <Interceptor className="org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor"/>
> </Channel>
> </Cluster>
>
>
>
> Thanks for any help in advance!
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
_________________________________________________________________
Rediscover Hotmail®: Now available on your iPhone or BlackBerry
http://windowslive.com/RediscoverHotmail?ocid=TXT_TAGLM_WL_HM_Rediscover_Mobile1_042009