Consumer stops sending messages to DefaultMessageListener

[shaneg <sh...@gmail.com>]

Hi,

We've introduced a HA deployment of active mq 5.9.1 using leveldb and
zookeeper using 3 nodes.  We did a performance run and subsequently our
message consumers have stopped consuming messages but message creation is
working fine.  

We are seeing the dispatched queue count increasing so it appears that the
messages are being delivered to the consumer but are not being delivered to
the listener.  This is happening across all the clients of our queues , its
not local to a single queue.  

I've observed the following in our log files which may or may not be
connected.  The client seems to be continually connecting and reconnecting
using the failover transport (fyi activemq1 is the active node) but I'm not
sure if this is expected behaviour :

Line 25527: 2014-09-23 15:13:22,882 [ActiveMQ Task-1] DEBUG
org.apache.activemq.transport.failover.FailoverTransport - urlList
connectionList:[tcp://activemq3.cert.api.hco.com:61616,
tcp://activemq2.cert.api.hco.com:61616,
tcp://activemq1.cert.api.hco.com:61616], from:
[tcp://activemq1.cert.api.hco.com:61616,
tcp://activemq2.cert.api.hco.com:61616,
tcp://activemq3.cert.api.hco.com:61616]
	Line 25528: 2014-09-23 15:13:22,882 [ActiveMQ Task-1] DEBUG
org.apache.activemq.transport.failover.FailoverTransport - Attempting  0th 
connect to: tcp://activemq3.cert.api.hco.com:61616
	Line 25529: 2014-09-23 15:13:22,884 [ActiveMQ Task-1] DEBUG
org.apache.activemq.transport.failover.FailoverTransport - Connect fail to:
tcp://activemq3.cert.api.hco.com:61616, reason: java.net.ConnectException:
Connection refused
	Line 25530: 2014-09-23 15:13:22,884 [ActiveMQ Task-1] DEBUG
org.apache.activemq.transport.tcp.TcpTransport - Stopping transport
tcp://activemq3.cert.api.hco.com:61616
	Line 25531: 2014-09-23 15:13:22,884 [ActiveMQ Task-1] DEBUG
org.apache.activemq.thread.TaskRunnerFactory - Initialized
TaskRunnerFactory[ActiveMQ Task] using ExecutorService:
java.util.concurrent.ThreadPoolExecutor@75d2eea6[Running, pool size = 0,
active threads = 0, queued tasks = 0, completed tasks = 0]
	Line 25532: 2014-09-23 15:13:22,885 [ActiveMQ Task-1] DEBUG
org.apache.activemq.transport.tcp.TcpTransport - Closed socket
Socket[unconnected]
	Line 25533: 2014-09-23 15:13:22,885 [ActiveMQ Task-1] DEBUG
org.apache.activemq.util.ThreadPoolUtils - Forcing shutdown of
ExecutorService: java.util.concurrent.ThreadPoolExecutor@75d2eea6[Running,
pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 0]
	Line 25534: 2014-09-23 15:13:22,885 [ActiveMQ Task-1] DEBUG
org.apache.activemq.transport.failover.FailoverTransport - Attempting  0th 
connect to: tcp://activemq2.cert.api.hco.com:61616
	Line 25535: 2014-09-23 15:13:22,886 [ActiveMQ Task-1] DEBUG
org.apache.activemq.transport.failover.FailoverTransport - Connect fail to:
tcp://activemq2.cert.api.hco.com:61616, reason: java.net.ConnectException:
Connection refused
	Line 25536: 2014-09-23 15:13:22,887 [ActiveMQ Task-1] DEBUG
org.apache.activemq

This is the failover connection string:

activemq.connectionfactory.brokerurl=failover:(tcp://activemq1.cert.api.hco.com:61616,tcp://activemq2.cert.api.hco.com:61616,tcp://activemq3.cert.api.hco.com:61616)

This is our current spring config:


    <bean id="amqConnectionFactory"
class="org.apache.activemq.ActiveMQConnectionFactory">
        <property name="brokerURL"
value="${activemq.connectionfactory.brokerurl}"/>
        <property name="userName"
value="${activemq.connectionfactory.brokerUser}"/>
        <property name="password"
value="${activemq.connectionfactory.brokerPassword}"/>
        

        <property name="redeliveryPolicy">
              <bean class="org.apache.activemq.RedeliveryPolicy">
                  <property name="initialRedeliveryDelay" value="10000"/>
                   <property name="maximumRedeliveries" value="2" />
               </bean>
        </property>
        
    </bean>



    <bean id="connectionFactory"
class="org.springframework.jms.connection.CachingConnectionFactory">
        <constructor-arg ref="amqConnectionFactory"/>
        <property name="sessionCacheSize" value="100"/>
    </bean>



    <bean id="inboundContainer"
class="org.springframework.jms.listener.DefaultMessageListenerContainer">
        <property name="connectionFactory" ref="connectionFactory"/>
        <property name="destination" ref="inboundQueue"/>
        <property name="messageListener" ref="inboundQueueListener"/>
        <property name="sessionTransacted" value="true"/>
    </bean>
    

I'd be interested if anyone else has seen similar behaviour after
introducing HA.  The config above has been in use for about 6 months without
any issue so my assumption thus far is that the issue is HA related but of
course it may be that its exposing an issue with our config.

Thanks 

Shane



--
View this message in context: http://activemq.2283324.n4.nabble.com/Consumer-stops-sending-messages-to-DefaultMessageListener-tp4685807.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.