Failed to reconnecting to IBM MQ using Flume JMS source

[Kevin Tran <ke...@gmail.com>]

Hi Everyone,
Does anyone get an issue of reconnecting to IBM MQ using Flume JMS source ?

Steps to reproduce:
1. Stop MQ service (I use IBM MQ Docker)
2. Start flume agent with JMS source using MQ .bindings
3. Start MQ service.
4. You will see the issue with Flume JMS could not reconnect when MQ
service come back to work.

Note that Flums JMS works fine when MQ service is ready before starting
Flume agent.

Please give me your ideas on this issue ?

= Logs =
ERROR [PollableSourceRunner-JMSSource-jms]
(org.apache.flume.source.PollableSourceRunner$PollingRunner.run:156)  -
Unhandled exception, logging and sleeping for 5000ms
org.apache.flume.FlumeException: Source had error configuring or starting
        at
org.apache.flume.source.AbstractPollableSource.process(AbstractPollableSource.java:48)
        at
org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:139)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.flume.FlumeException: Could not create connection to
broker
        at
org.apache.flume.source.jms.JMSMessageConsumer.<init>(JMSMessageConsumer.java:76)
        at
org.apache.flume.source.jms.JMSMessageConsumerFactory.create(JMSMessageConsumerFactory.java:32)
        at
org.apache.flume.source.jms.JMSSource.createConsumer(JMSSource.java:317)
        at org.apache.flume.source.jms.JMSSource.doStart(JMSSource.java:299)
        at
org.apache.flume.source.BasicSourceSemantics.start(BasicSourceSemantics.java:83)
        at
org.apache.flume.source.PollableSourceRunner.start(PollableSourceRunner.java:74)
        at
org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        ... 1 more
Caused by: com.ibm.msg.client.jms.DetailedIllegalStateException:
JMSWMQ0018: Failed to connect to queue manager 'QM1' with connection mode
'Client' and host name 'localhost'. Check the queue manager is started and
if running in client mode, check there is a listener running. Please see
the linked exception for more information.
        at
com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:496)
        at
com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:236)
        at
com.ibm.msg.client.wmq.internal.WMQConnection.<init>(WMQConnection.java:421)
        at
com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createV7ProviderConnection(WMQConnectionFactory.java:6807)
        at
com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createProviderConnection(WMQConnectionFactory.java:6204)
        at
com.ibm.msg.client.jms.admin.JmsConnectionFactoryImpl.createConnection(JmsConnectionFactoryImpl.java:278)
        at
com.ibm.mq.jms.MQConnectionFactory.createCommonConnection(MQConnectionFactory.java:6155)
        at
com.ibm.mq.jms.MQConnectionFactory.createConnection(MQConnectionFactory.java:6209)
        at
org.apache.flume.source.jms.JMSMessageConsumer.<init>(JMSMessageConsumer.java:69)
        ... 13 more
Caused by: com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed
with compcode '2' ('MQCC_FAILED') reason '2059'
('MQRC_Q_MGR_NOT_AVAILABLE').
        at
com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:223)
        ... 20 more
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2059;AMQ9204: Connection
to host 'localhost(1414)' rejected.
[1=com.ibm.mq.jmqi.JmqiException[CC=2;RC=2059;AMQ9213: A communications
error for  occurred. [1=java.net.ConnectException[Connection
refused],3=localhost]],3=localhost(1414),5=RemoteTCPConnection.connnectUsingLocalAddress]
        at
com.ibm.mq.jmqi.remote.internal.RemoteFAP.jmqiConnect(RemoteFAP.java:1809)
        at
com.ibm.msg.client.wmq.internal.WMQConnection.<init>(WMQConnection.java:336)
        ... 19 more
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2059;AMQ9213: A
communications error for  occurred. [1=java.net.ConnectException[Connection
refused],3=localhost]
        at
com.ibm.mq.jmqi.remote.internal.RemoteTCPConnection.connnectUsingLocalAddress(RemoteTCPConnection.java:607)
        at
com.ibm.mq.jmqi.remote.internal.RemoteTCPConnection.protocolConnect(RemoteTCPConnection.java:935)
        at
com.ibm.mq.jmqi.remote.internal.system.RemoteConnection.connect(RemoteConnection.java:1075)
        at
com.ibm.mq.jmqi.remote.internal.system.RemoteConnectionPool.getConnection(RemoteConnectionPool.java:338)
        at
com.ibm.mq.jmqi.remote.internal.RemoteFAP.jmqiConnect(RemoteFAP.java:1488)
        ... 20 more
Caused by: java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at java.net.Socket.connect(Socket.java:538)
        at
com.ibm.mq.jmqi.remote.internal.RemoteTCPConnection$2.run(RemoteTCPConnection.java:592)
        at java.security.AccessController.doPrivileged(Native Method)
        at
com.ibm.mq.jmqi.remote.internal.RemoteTCPConnection.connnectUsingLocalAddress(RemoteTCPConnection.java:583)


Thanks,
Kevin.