You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by John Lilley <jo...@redpointglobal.com.INVALID> on 2023/03/23 20:26:35 UTC
disconnect recovery issue
We’ve added a chaos monkey to randomly disconnect Artemis clients from broker.
This mostly works, however, we do get occasional errors that are not retried by the AMQ JMS client.
This manifests with the exception: AMQ219016: Connection failure detected. Unblocking a blocking call that will never get a response.
I don’t know why the AMQ client thinks it will never get a response. Shouldn’t the client reconnect and continue without error? Do we need to retry explicitly?
This is using version 2.28.0. Example stack trace:
2023-03-23T14:19:54.666 [Schedule-Runner] LazyStartPeriodicThread.run:97 [] ERROR - Exception in interval action RpcException: httpCode=500, errorMessage=error sending message: AMQ219016: Connection failure detected. Unblocking a blocking call that will never get a response
errorFrames:
javax.jms.JMSException: AMQ219016: Connection failure detected. Unblocking a blocking call that will never get a response
at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:558)
at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:450)
at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.sendFullMessage(ActiveMQSessionContext.java:583)
at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.sendRegularMessage(ClientProducerImpl.java:305)
at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:277)
at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:147)
at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:129)
at org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.doSendx(ActiveMQMessageProducer.java:483)
at org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:221)
at net.redpoint.ipc.jms.JmsProducerPool.send_(JmsProducerPool.java:351)
at net.redpoint.ipc.jms.JmsProducerPool.sendRequest(JmsProducerPool.java:289)
at net.redpoint.ipc.jms.JmsRpcClientChannel.sendRequest(JmsRpcClientChannel.java:217)
at net.redpoint.ipc.jms.JmsRpcClientChannel.invokeRaw(JmsRpcClientChannel.java:194)
at net.redpoint.ipc.jms.JmsRpcClientChannel.call(JmsRpcClientChannel.java:98)
at net.redpoint.ipc.clients.RpcClientBase._sync(RpcClientBase.java:165)
at net.redpoint.ipc.clients.RpcClientBase._rpc(RpcClientBase.java:231)
at net.redpoint.rpdm.ipc.clients.PermissionsClient.doesUserHavePermission(PermissionsClient.java:178)
at net.redpoint.rpdm.services.SecurityControlImpl.doesUserHavePermission(SecurityControlImpl.java:205)
at net.redpoint.ipc.SecurityControl.checkUserHasPermission(SecurityControl.java:387)
at net.redpoint.rpdm.ipc.servers.SchedulerServer.checkUserHasPermission(SchedulerServer.java:325)
at net.redpoint.rpdm.ipc.servers.SchedulerServer.list(SchedulerServer.java:264)
at net.redpoint.rpdm.scheduler.SchedulerlServerImpl._doSchedule(SchedulerlServerImpl.java:407)
at net.redpoint.rpdm.scheduler.SchedulerlServerImpl.doSchedule(SchedulerlServerImpl.java:360)
at net.redpoint.rpdm.scheduler.SchedulerlServerImpl$Runner.lambda$action$0(SchedulerlServerImpl.java:511)
at net.redpoint.ipc.SecurityControl.runAsNoThrow(SecurityControl.java:315)
at net.redpoint.rpdm.scheduler.SchedulerlServerImpl$Runner.action(SchedulerlServerImpl.java:510)
at net.redpoint.rpdm.services.LazyStartPeriodicThread.run(LazyStartPeriodicThread.java:95)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: ActiveMQUnBlockedException[errorType=UNBLOCKED message=AMQ219016: Connection failure detected. Unblocking a blocking call that will never get a response]
... 28 more
Caused by: ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006: Channel disconnected]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:383)
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$Listener$1.run(NettyConnector.java:1240)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
at net.redpoint.ipc.jms.JmsProducerPool.sendRequest(JmsProducerPool.java:295)
at net.redpoint.ipc.jms.JmsRpcClientChannel.sendRequest(JmsRpcClientChannel.java:217)
at net.redpoint.ipc.jms.JmsRpcClientChannel.invokeRaw(JmsRpcClientChannel.java:194)
at net.redpoint.ipc.jms.JmsRpcClientChannel.call(JmsRpcClientChannel.java:98)
at net.redpoint.ipc.clients.RpcClientBase._sync(RpcClientBase.java:165)
at net.redpoint.ipc.clients.RpcClientBase._rpc(RpcClientBase.java:231)
at net.redpoint.rpdm.ipc.clients.PermissionsClient.doesUserHavePermission(PermissionsClient.java:178)
at net.redpoint.rpdm.services.SecurityControlImpl.doesUserHavePermission(SecurityControlImpl.java:205)
at net.redpoint.ipc.SecurityControl.checkUserHasPermission(SecurityControl.java:387)
at net.redpoint.rpdm.ipc.servers.SchedulerServer.checkUserHasPermission(SchedulerServer.java:325)
at net.redpoint.rpdm.ipc.servers.SchedulerServer.list(SchedulerServer.java:264)
at net.redpoint.rpdm.scheduler.SchedulerlServerImpl._doSchedule(SchedulerlServerImpl.java:407)
at net.redpoint.rpdm.scheduler.SchedulerlServerImpl.doSchedule(SchedulerlServerImpl.java:360)
at net.redpoint.rpdm.scheduler.SchedulerlServerImpl$Runner.lambda$action$0(SchedulerlServerImpl.java:511)
at net.redpoint.ipc.SecurityControl.runAsNoThrow(SecurityControl.java:315)
at net.redpoint.rpdm.scheduler.SchedulerlServerImpl$Runner.action(SchedulerlServerImpl.java:510)
at net.redpoint.rpdm.services.LazyStartPeriodicThread.run(LazyStartPeriodicThread.java:95)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: javax.jms.JMSException: AMQ219016: Connection failure detected. Unblocking a blocking call that will never get a response
at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:558)
at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:450)
at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.sendFullMessage(ActiveMQSessionContext.java:583)
at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.sendRegularMessage(ClientProducerImpl.java:305)
at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:277)
at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:147)
at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:129)
at org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.doSendx(ActiveMQMessageProducer.java:483)
at org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:221)
at net.redpoint.ipc.jms.JmsProducerPool.send_(JmsProducerPool.java:351)
at net.redpoint.ipc.jms.JmsProducerPool.sendRequest(JmsProducerPool.java:289)
... 17 more
Caused by: ActiveMQUnBlockedException[errorType=UNBLOCKED message=AMQ219016: Connection failure detected. Unblocking a blocking call that will never get a response]
... 28 more
Caused by: ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006: Channel disconnected]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:383)
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$Listener$1.run(NettyConnector.java:1240)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
2023-03-23T14:19:54.700 [Thread-20] JmsStaticConnectionPool.onException:78 [] ERROR - Receive error occurred. javax.jms.JMSException: ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006: Channel disconnected]
at org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(ActiveMQConnection.java:714)
at org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(ActiveMQConnection.java:735)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(ClientSessionFactoryImpl.java:764)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(ClientSessionFactoryImpl.java:752)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:707)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:534)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:527)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:390)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
Caused by: ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006: Channel disconnected]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:383)
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$Listener$1.run(NettyConnector.java:1240)
... 6 more
[rg] <https://www.redpointglobal.com/>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
888 Worcester Street, Suite 200 Wellesley, MA 02482
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
Re: disconnect recovery issue
Posted by "prateekjainaa@gmail.com" <pr...@gmail.com>.
Yes, I too faced similar issues while randomly killing artemis instances.
The pattern which I observed was mainly, when you kill live instances in a
cluster w/o running backup node. So, in our case we incorporated *client
side failover* detection. IMO, most of the times; failover happens
transparently when going from live to backup nodes. But at the same time,
we couldn't replicate it everytime. So, to keep things save; it is better
to have a safety net designed.
Regards,
Prateek Jain
--------------------------------------------------------------
EXPECTATION : Causes all troubles......
--------------------------------------------------------------
On Thu, Mar 23, 2023 at 8:27 PM John Lilley
<jo...@redpointglobal.com.invalid> wrote:
> We’ve added a chaos monkey to randomly disconnect Artemis clients from
> broker.
>
> This mostly works, however, we do get occasional errors that are not
> retried by the AMQ JMS client.
>
> This manifests with the exception: AMQ219016: Connection failure detected.
> Unblocking a blocking call that will never get a response.
>
> I don’t know why the AMQ client thinks it will never get a response.
> Shouldn’t the client reconnect and continue without error? Do we need to
> retry explicitly?
>
>
>
> This is using version 2.28.0. Example stack trace:
>
>
>
> 2023-03-23T14:19:54.666 [Schedule-Runner] LazyStartPeriodicThread.run:97
> [] ERROR - Exception in interval action RpcException: httpCode=500,
> errorMessage=error sending message: AMQ219016: Connection failure detected.
> Unblocking a blocking call that will never get a response
>
> errorFrames:
>
> javax.jms.JMSException: AMQ219016: Connection failure detected. Unblocking
> a blocking call that will never get a response
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:558)
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:450)
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.sendFullMessage(ActiveMQSessionContext.java:583)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.sendRegularMessage(ClientProducerImpl.java:305)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:277)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:147)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:129)
>
> at
> org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.doSendx(ActiveMQMessageProducer.java:483)
>
> at
> org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:221)
>
> at net.redpoint.ipc.jms.JmsProducerPool.send_(JmsProducerPool.java:351)
>
> at
> net.redpoint.ipc.jms.JmsProducerPool.sendRequest(JmsProducerPool.java:289)
>
> at
> net.redpoint.ipc.jms.JmsRpcClientChannel.sendRequest(JmsRpcClientChannel.java:217)
>
> at
> net.redpoint.ipc.jms.JmsRpcClientChannel.invokeRaw(JmsRpcClientChannel.java:194)
>
> at
> net.redpoint.ipc.jms.JmsRpcClientChannel.call(JmsRpcClientChannel.java:98)
>
> at net.redpoint.ipc.clients.RpcClientBase._sync(RpcClientBase.java:165)
>
> at net.redpoint.ipc.clients.RpcClientBase._rpc(RpcClientBase.java:231)
>
> at
> net.redpoint.rpdm.ipc.clients.PermissionsClient.doesUserHavePermission(PermissionsClient.java:178)
>
> at
> net.redpoint.rpdm.services.SecurityControlImpl.doesUserHavePermission(SecurityControlImpl.java:205)
>
> at
> net.redpoint.ipc.SecurityControl.checkUserHasPermission(SecurityControl.java:387)
>
> at
> net.redpoint.rpdm.ipc.servers.SchedulerServer.checkUserHasPermission(SchedulerServer.java:325)
>
> at
> net.redpoint.rpdm.ipc.servers.SchedulerServer.list(SchedulerServer.java:264)
>
> at
> net.redpoint.rpdm.scheduler.SchedulerlServerImpl._doSchedule(SchedulerlServerImpl.java:407)
>
> at
> net.redpoint.rpdm.scheduler.SchedulerlServerImpl.doSchedule(SchedulerlServerImpl.java:360)
>
> at
> net.redpoint.rpdm.scheduler.SchedulerlServerImpl$Runner.lambda$action$0(SchedulerlServerImpl.java:511)
>
> at net.redpoint.ipc.SecurityControl.runAsNoThrow(SecurityControl.java:315)
>
> at
> net.redpoint.rpdm.scheduler.SchedulerlServerImpl$Runner.action(SchedulerlServerImpl.java:510)
>
> at
> net.redpoint.rpdm.services.LazyStartPeriodicThread.run(LazyStartPeriodicThread.java:95)
>
> at java.base/java.lang.Thread.run(Thread.java:833)
>
> Caused by: ActiveMQUnBlockedException[errorType=UNBLOCKED
> message=AMQ219016: Connection failure detected. Unblocking a blocking call
> that will never get a response]
>
> ... 28 more
>
> Caused by: ActiveMQNotConnectedException[errorType=NOT_CONNECTED
> message=AMQ219006: Channel disconnected]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:383)
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$Listener$1.run(NettyConnector.java:1240)
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
>
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
>
> at
> net.redpoint.ipc.jms.JmsProducerPool.sendRequest(JmsProducerPool.java:295)
>
> at
> net.redpoint.ipc.jms.JmsRpcClientChannel.sendRequest(JmsRpcClientChannel.java:217)
>
> at
> net.redpoint.ipc.jms.JmsRpcClientChannel.invokeRaw(JmsRpcClientChannel.java:194)
>
> at
> net.redpoint.ipc.jms.JmsRpcClientChannel.call(JmsRpcClientChannel.java:98)
>
> at
> net.redpoint.ipc.clients.RpcClientBase._sync(RpcClientBase.java:165)
>
> at
> net.redpoint.ipc.clients.RpcClientBase._rpc(RpcClientBase.java:231)
>
> at
> net.redpoint.rpdm.ipc.clients.PermissionsClient.doesUserHavePermission(PermissionsClient.java:178)
>
> at
> net.redpoint.rpdm.services.SecurityControlImpl.doesUserHavePermission(SecurityControlImpl.java:205)
>
> at
> net.redpoint.ipc.SecurityControl.checkUserHasPermission(SecurityControl.java:387)
>
> at
> net.redpoint.rpdm.ipc.servers.SchedulerServer.checkUserHasPermission(SchedulerServer.java:325)
>
> at
> net.redpoint.rpdm.ipc.servers.SchedulerServer.list(SchedulerServer.java:264)
>
> at
> net.redpoint.rpdm.scheduler.SchedulerlServerImpl._doSchedule(SchedulerlServerImpl.java:407)
>
> at
> net.redpoint.rpdm.scheduler.SchedulerlServerImpl.doSchedule(SchedulerlServerImpl.java:360)
>
> at
> net.redpoint.rpdm.scheduler.SchedulerlServerImpl$Runner.lambda$action$0(SchedulerlServerImpl.java:511)
>
> at
> net.redpoint.ipc.SecurityControl.runAsNoThrow(SecurityControl.java:315)
>
> at
> net.redpoint.rpdm.scheduler.SchedulerlServerImpl$Runner.action(SchedulerlServerImpl.java:510)
>
> at
> net.redpoint.rpdm.services.LazyStartPeriodicThread.run(LazyStartPeriodicThread.java:95)
>
> at java.base/java.lang.Thread.run(Thread.java:833)
>
> Caused by: javax.jms.JMSException: AMQ219016: Connection failure detected.
> Unblocking a blocking call that will never get a response
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:558)
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:450)
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.sendFullMessage(ActiveMQSessionContext.java:583)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.sendRegularMessage(ClientProducerImpl.java:305)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:277)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:147)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:129)
>
> at
> org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.doSendx(ActiveMQMessageProducer.java:483)
>
> at
> org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:221)
>
> at
> net.redpoint.ipc.jms.JmsProducerPool.send_(JmsProducerPool.java:351)
>
> at
> net.redpoint.ipc.jms.JmsProducerPool.sendRequest(JmsProducerPool.java:289)
>
> ... 17 more
>
> Caused by: ActiveMQUnBlockedException[errorType=UNBLOCKED
> message=AMQ219016: Connection failure detected. Unblocking a blocking call
> that will never get a response]
>
> ... 28 more
>
> Caused by: ActiveMQNotConnectedException[errorType=NOT_CONNECTED
> message=AMQ219006: Channel disconnected]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:383)
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$Listener$1.run(NettyConnector.java:1240)
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
>
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
>
>
>
> 2023-03-23T14:19:54.700 [Thread-20] JmsStaticConnectionPool.onException:78
> [] ERROR - Receive error occurred. javax.jms.JMSException:
> ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006:
> Channel disconnected]
>
> at
> org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(ActiveMQConnection.java:714)
>
> at
> org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(ActiveMQConnection.java:735)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(ClientSessionFactoryImpl.java:764)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(ClientSessionFactoryImpl.java:752)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:707)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:534)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:527)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:390)
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
>
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
>
> Caused by: ActiveMQNotConnectedException[errorType=NOT_CONNECTED
> message=AMQ219006: Channel disconnected]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:383)
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$Listener$1.run(NettyConnector.java:1240)
>
> ... 6 more
>
>
>
> [image: rg] <https://www.redpointglobal.com/>
>
> John Lilley
>
> Data Management Chief Architect, Redpoint Global Inc.
>
> 888 Worcester Street, Suite 200 Wellesley, MA 02482
>
> *M: *+1 7209385761 <+1%207209385761> | john.lilley@redpointglobal.com
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
Re: disconnect recovery issue
Posted by Justin Bertram <jb...@apache.org>.
What you're seeing is the expected behavior in certain circumstances [1].
In your case, you're sending a durable message to the broker and the
connection fails. The problem now is that the client doesn't know what
happened to the message. Did it make it to the broker and get persisted to
disk before the connection failed (i.e. before the ack is sent back to the
producer)? If so, then retrying the send operation will result in a
duplicate message. Or did the connection fail before the message made it to
the broker? If so, then *not* retrying the send operation will mean the
message is lost. Since there's no clear way to proceed it simply throws an
exception, and it's up to the application to decide what to do based on its
use-case. Maybe the consumers are idempotent so duplicates don't matter.
Maybe the producer is using duplicate detection [2] so it can resend the
message without any worries. Maybe a few lost messages don't matter so it
can just ignore the exception. In short, you need to handle such failures
as necessary.
Justin
[1]
https://activemq.apache.org/components/artemis/documentation/latest/ha.html#handling-blocking-calls-during-failover
[2]
https://activemq.apache.org/components/artemis/documentation/latest/duplicate-detection.html
On Thu, Mar 23, 2023 at 3:27 PM John Lilley
<jo...@redpointglobal.com.invalid> wrote:
> We’ve added a chaos monkey to randomly disconnect Artemis clients from
> broker.
>
> This mostly works, however, we do get occasional errors that are not
> retried by the AMQ JMS client.
>
> This manifests with the exception: AMQ219016: Connection failure detected.
> Unblocking a blocking call that will never get a response.
>
> I don’t know why the AMQ client thinks it will never get a response.
> Shouldn’t the client reconnect and continue without error? Do we need to
> retry explicitly?
>
>
>
> This is using version 2.28.0. Example stack trace:
>
>
>
> 2023-03-23T14:19:54.666 [Schedule-Runner] LazyStartPeriodicThread.run:97
> [] ERROR - Exception in interval action RpcException: httpCode=500,
> errorMessage=error sending message: AMQ219016: Connection failure detected.
> Unblocking a blocking call that will never get a response
>
> errorFrames:
>
> javax.jms.JMSException: AMQ219016: Connection failure detected. Unblocking
> a blocking call that will never get a response
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:558)
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:450)
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.sendFullMessage(ActiveMQSessionContext.java:583)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.sendRegularMessage(ClientProducerImpl.java:305)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:277)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:147)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:129)
>
> at
> org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.doSendx(ActiveMQMessageProducer.java:483)
>
> at
> org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:221)
>
> at net.redpoint.ipc.jms.JmsProducerPool.send_(JmsProducerPool.java:351)
>
> at
> net.redpoint.ipc.jms.JmsProducerPool.sendRequest(JmsProducerPool.java:289)
>
> at
> net.redpoint.ipc.jms.JmsRpcClientChannel.sendRequest(JmsRpcClientChannel.java:217)
>
> at
> net.redpoint.ipc.jms.JmsRpcClientChannel.invokeRaw(JmsRpcClientChannel.java:194)
>
> at
> net.redpoint.ipc.jms.JmsRpcClientChannel.call(JmsRpcClientChannel.java:98)
>
> at net.redpoint.ipc.clients.RpcClientBase._sync(RpcClientBase.java:165)
>
> at net.redpoint.ipc.clients.RpcClientBase._rpc(RpcClientBase.java:231)
>
> at
> net.redpoint.rpdm.ipc.clients.PermissionsClient.doesUserHavePermission(PermissionsClient.java:178)
>
> at
> net.redpoint.rpdm.services.SecurityControlImpl.doesUserHavePermission(SecurityControlImpl.java:205)
>
> at
> net.redpoint.ipc.SecurityControl.checkUserHasPermission(SecurityControl.java:387)
>
> at
> net.redpoint.rpdm.ipc.servers.SchedulerServer.checkUserHasPermission(SchedulerServer.java:325)
>
> at
> net.redpoint.rpdm.ipc.servers.SchedulerServer.list(SchedulerServer.java:264)
>
> at
> net.redpoint.rpdm.scheduler.SchedulerlServerImpl._doSchedule(SchedulerlServerImpl.java:407)
>
> at
> net.redpoint.rpdm.scheduler.SchedulerlServerImpl.doSchedule(SchedulerlServerImpl.java:360)
>
> at
> net.redpoint.rpdm.scheduler.SchedulerlServerImpl$Runner.lambda$action$0(SchedulerlServerImpl.java:511)
>
> at net.redpoint.ipc.SecurityControl.runAsNoThrow(SecurityControl.java:315)
>
> at
> net.redpoint.rpdm.scheduler.SchedulerlServerImpl$Runner.action(SchedulerlServerImpl.java:510)
>
> at
> net.redpoint.rpdm.services.LazyStartPeriodicThread.run(LazyStartPeriodicThread.java:95)
>
> at java.base/java.lang.Thread.run(Thread.java:833)
>
> Caused by: ActiveMQUnBlockedException[errorType=UNBLOCKED
> message=AMQ219016: Connection failure detected. Unblocking a blocking call
> that will never get a response]
>
> ... 28 more
>
> Caused by: ActiveMQNotConnectedException[errorType=NOT_CONNECTED
> message=AMQ219006: Channel disconnected]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:383)
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$Listener$1.run(NettyConnector.java:1240)
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
>
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
>
> at
> net.redpoint.ipc.jms.JmsProducerPool.sendRequest(JmsProducerPool.java:295)
>
> at
> net.redpoint.ipc.jms.JmsRpcClientChannel.sendRequest(JmsRpcClientChannel.java:217)
>
> at
> net.redpoint.ipc.jms.JmsRpcClientChannel.invokeRaw(JmsRpcClientChannel.java:194)
>
> at
> net.redpoint.ipc.jms.JmsRpcClientChannel.call(JmsRpcClientChannel.java:98)
>
> at
> net.redpoint.ipc.clients.RpcClientBase._sync(RpcClientBase.java:165)
>
> at
> net.redpoint.ipc.clients.RpcClientBase._rpc(RpcClientBase.java:231)
>
> at
> net.redpoint.rpdm.ipc.clients.PermissionsClient.doesUserHavePermission(PermissionsClient.java:178)
>
> at
> net.redpoint.rpdm.services.SecurityControlImpl.doesUserHavePermission(SecurityControlImpl.java:205)
>
> at
> net.redpoint.ipc.SecurityControl.checkUserHasPermission(SecurityControl.java:387)
>
> at
> net.redpoint.rpdm.ipc.servers.SchedulerServer.checkUserHasPermission(SchedulerServer.java:325)
>
> at
> net.redpoint.rpdm.ipc.servers.SchedulerServer.list(SchedulerServer.java:264)
>
> at
> net.redpoint.rpdm.scheduler.SchedulerlServerImpl._doSchedule(SchedulerlServerImpl.java:407)
>
> at
> net.redpoint.rpdm.scheduler.SchedulerlServerImpl.doSchedule(SchedulerlServerImpl.java:360)
>
> at
> net.redpoint.rpdm.scheduler.SchedulerlServerImpl$Runner.lambda$action$0(SchedulerlServerImpl.java:511)
>
> at
> net.redpoint.ipc.SecurityControl.runAsNoThrow(SecurityControl.java:315)
>
> at
> net.redpoint.rpdm.scheduler.SchedulerlServerImpl$Runner.action(SchedulerlServerImpl.java:510)
>
> at
> net.redpoint.rpdm.services.LazyStartPeriodicThread.run(LazyStartPeriodicThread.java:95)
>
> at java.base/java.lang.Thread.run(Thread.java:833)
>
> Caused by: javax.jms.JMSException: AMQ219016: Connection failure detected.
> Unblocking a blocking call that will never get a response
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:558)
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:450)
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.sendFullMessage(ActiveMQSessionContext.java:583)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.sendRegularMessage(ClientProducerImpl.java:305)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:277)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:147)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:129)
>
> at
> org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.doSendx(ActiveMQMessageProducer.java:483)
>
> at
> org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:221)
>
> at
> net.redpoint.ipc.jms.JmsProducerPool.send_(JmsProducerPool.java:351)
>
> at
> net.redpoint.ipc.jms.JmsProducerPool.sendRequest(JmsProducerPool.java:289)
>
> ... 17 more
>
> Caused by: ActiveMQUnBlockedException[errorType=UNBLOCKED
> message=AMQ219016: Connection failure detected. Unblocking a blocking call
> that will never get a response]
>
> ... 28 more
>
> Caused by: ActiveMQNotConnectedException[errorType=NOT_CONNECTED
> message=AMQ219006: Channel disconnected]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:383)
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$Listener$1.run(NettyConnector.java:1240)
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
>
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
>
>
>
> 2023-03-23T14:19:54.700 [Thread-20] JmsStaticConnectionPool.onException:78
> [] ERROR - Receive error occurred. javax.jms.JMSException:
> ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006:
> Channel disconnected]
>
> at
> org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(ActiveMQConnection.java:714)
>
> at
> org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(ActiveMQConnection.java:735)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(ClientSessionFactoryImpl.java:764)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(ClientSessionFactoryImpl.java:752)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:707)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:534)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:527)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:390)
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
>
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
>
> Caused by: ActiveMQNotConnectedException[errorType=NOT_CONNECTED
> message=AMQ219006: Channel disconnected]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:383)
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$Listener$1.run(NettyConnector.java:1240)
>
> ... 6 more
>
>
>
> [image: rg] <https://www.redpointglobal.com/>
>
> John Lilley
>
> Data Management Chief Architect, Redpoint Global Inc.
>
> 888 Worcester Street, Suite 200 Wellesley, MA 02482
>
> *M: *+1 7209385761 <+1%207209385761> | john.lilley@redpointglobal.com
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>