You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by vbandaru <vi...@gmail.com> on 2011/06/23 19:00:01 UTC
ActiveMQ Master/Slave dead/zombie connections issue
Never found a similar issue on this forum and hence this post.
I am using AMQ 5.5.0 with pure Master/slave configuration (with and without
persistence).
Master/slave configuration seems to be Okay interms of :
Message replication to slave while master is ON
Failover to slave when master fails
We are deploying our JMS client (implements JMSlistner interface) on two
remote servers (for failover redundancy)
JMS Client Listener1 connects to Master Broker (using
tcp://masterip:61616) always on TEST.FOO Queue
JMS Client Listener2 connects to Slave Broker (using tcp://masterip:61617)
always on TEST.FOO Queue.This starts consuming messages only if Master
fails.
Both of these listeners use Spring's DefaultMEssageListenerContainer
(DMLC) with dynamic scaling. concurrency="1-6" (starts with 1 and
based on load can scale up to 6 MAX).
My Understanding is Master/Slave config, all the state (messages,
connection info etc.) is replicated to slave. So if Listener 1 opens 6
connections/sessions to master broker, they will be replicated to Slave
Broker as well which seems to be causing problem in my case.
So at this point, I killed the master, failover happens to slave
successfully. My Listener2, which keep polling for Slave broker to
accept connections (DMLC does it automatic connection recovery), will obtain
6 connections.
On Webconsole, I verified messages arriving onTEST.FOO and I also noticed
there are 12 connections present for activeconsumer section of
"TEST.FOO".
Table below (on the bottom) , I am printin printing ClientID and sessionId
from Active Consumers for TEST.FOO Webconsole page First 6 connections
correspond to the Listener1 next 6 for Listener 2. Now
If my producer sends 100 messages
to failover://(tcp://masterip:61616,tcp://slaveip:61617)?randomize=false,
on Listern 2 on Message I get: ( i have log statement)
received Message 7
received Message 8
received Message 9
received Message 10
received Message 11
received Message 12
received Message 19
received Message 20
received Message 21
received Message 22
received Message 23
received Message 24
received Message 31
received Message 32
:
:
Seeems like the 6 dead connections (from listener1) consume half of
the messages which resulting in message loss. If I click on one of the 6
Connections from Listener 1, I get "No connection could be found
for ID ID:HDC00020891-2445-1308844599481-1:1".
Is there any way we can remove the connections established to Master Broker
be not replciated to Slave on Master shutdown.
Thanks.
Client ID
Connection ID
Session ID
*ID:HDC00020891-2445-1308844599481-1:1
ID:HDC00020891-2445-1308844599481-0:1*
*1*
*ID:HDC00020891-2445-1308844599481-1:1
ID:HDC00020891-2445-1308844599481-0:1*
*2*
*ID:HDC00020891-2445-1308844599481-1:1
ID:HDC00020891-2445-1308844599481-0:1*
*3*
*ID:HDC00020891-2445-1308844599481-1:1
ID:HDC00020891-2445-1308844599481-0:1*
*4*
*ID:HDC00020891-2445-1308844599481-1:1
ID:HDC00020891-2445-1308844599481-0:1*
*5*
*ID:HDC00020891-2445-1308844599481-1:1
ID:HDC00020891-2445-1308844599481-0:1*
*6*
*ID:HDC00020891-2445-1308844599481-1:1
ID:HDC00020891-2445-1308844599481-0:1*
1
*ID:HDC00020891-2445-1308844599481-1:1
ID:HDC00020891-2445-1308844599481-0:1*
2
*ID:HDC00020891-2445-1308844599481-1:1
ID:HDC00020891-2445-1308844599481-0:1*
3
*ID:HDC00020891-2445-1308844599481-1:1
ID:HDC00020891-2445-1308844599481-0:1*
4
*ID:HDC00020891-2445-1308844599481-1:1
ID:HDC00020891-2445-1308844599481-0:1*
5
*ID:HDC00020891-2445-1308844599481-1:1
ID:HDC00020891-2445-1308844599481-0:1*
6
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-Master-Slave-dead-zombie-connections-issue-tp3620449p3620449.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.