You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@activemq.apache.org by "Jigar Shah (Jira)" <ji...@apache.org> on 2020/03/23 15:57:00 UTC
[jira] [Updated] (ARTEMIS-2677) Artemis 2.11.0
RejectedExecutionException after successful failover
[ https://issues.apache.org/jira/browse/ARTEMIS-2677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jigar Shah updated ARTEMIS-2677:
--------------------------------
Environment:
*Environment: Artemis 2.11.0 (Master-Slave)*
*SharedStore, FilePing. AWS EFS/NFS.*
was:
*Environment: 2.11.0*
*SharedStore, FilePing. AWS EFS/NFS.*
> Artemis 2.11.0 RejectedExecutionException after successful failover
> -------------------------------------------------------------------
>
> Key: ARTEMIS-2677
> URL: https://issues.apache.org/jira/browse/ARTEMIS-2677
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Environment: *Environment: Artemis 2.11.0 (Master-Slave)*
> *SharedStore, FilePing. AWS EFS/NFS.*
> Reporter: Jigar Shah
> Priority: Major
>
> Observed an issue on master shutdown, the slave became active. Right after slave being active it started printing "RejectedExecutionException". Also, consumers from master to slave was not transferred. And client application stopped processing messages.
> Note RejectedExecutionException had [Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 59059]
> Following are the logs during failover from master to slave:
> Master1:
> 2020-03-18 12:34:39,911 WARN [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up resources for session 26fb51af-690c-11ea-959a-12f8371a8293
> 2020-03-18 12:34:39,912 INFO [org.apache.activemq.artemis.core.server] AMQ221029: stopped bridge $.artemis.internal.sf.my-cluster.2c85cd9e-5f75-11ea-9be1-0af9119190b9
> 2020-03-18 12:34:39,950 WARN [org.apache.activemq.artemis.core.server] AMQ222061: Client connection failed, clearing up resources for session 2655265e-690c-11ea-9d7c-12f7d16c4f81
> 2020-03-18 12:34:39,951 WARN [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up resources for session 2655265e-690c-11ea-9d7c-12f7d16c4f81
> 2020-03-18 12:34:39,951 WARN [org.apache.activemq.artemis.core.server] AMQ222061: Client connection failed, clearing up resources for session 2662e209-690c-11ea-9d7c-12f7d16c4f81
> 2020-03-18 12:34:39,951 WARN [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up resources for session 2662e209-690c-11ea-9d7c-12f7d16c4f81
> 2020-03-18 12:34:39,951 WARN [org.apache.activemq.artemis.core.server] AMQ222061: Client connection failed, clearing up resources for session 2669bfdd-690c-11ea-9d7c-12f7d16c4f81
> 2020-03-18 12:34:39,951 WARN [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up resources for session 2669bfdd-690c-11ea-9d7c-12f7d16c4f81
> 2020-03-18 12:34:39,993 WARN [org.apache.activemq.artemis.core.server] AMQ222061: Client connection failed, clearing up resources for session 26615b64-690c-11ea-9d7c-12f7d16c4f81
> 2020-03-18 12:34:39,994 WARN [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up resources for session 26615b64-690c-11ea-9d7c-12f7d16c4f81
> 2020-03-18 12:34:39,994 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure to artemis1.sl.idsk.com/10.110.0.17:61616 has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
> 2020-03-18 12:34:40,123 FINE [org.jgroups.protocols.FILE_PING] remove persistence-fs
> 2020-03-18 12:34:40,137 FINE [org.jgroups.protocols.FD_SOCK] ip-10-110-0-17-18496: socket to ip-10-110-0-88-23224 was closed gracefully
> 2020-03-18 12:34:40,138 FINE [org.jgroups.protocols.TCP] closing sockets and stopping threads
> 2020-03-18 12:34:40,169 WARN [org.apache.activemq.artemis.core.client] AMQ212004: Failed to connect to server.
> 2020-03-18 12:34:41,336 INFO [io.hawt.web.AuthenticationFilter] Destroying hawtio authentication filter
> 2020-03-18 12:34:41,341 INFO [io.hawt.HawtioContextListener] Destroying hawtio services
> 2020-03-18 12:34:41,464 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Destroyed artemis-plugin plugin
> 2020-03-18 12:34:41,472 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Destroyed activemq-branding plugin
> 2020-03-18 12:34:41,509 INFO [org.apache.activemq.artemis.core.server] AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.11.0 [19ade3bb-5f75-11ea-b327-1216d251b187] stopped, uptime 5 hours 2 minutes
> Slave1:
> 2020-03-18 12:35:01,613 INFO [org.apache.activemq.artemis.core.server] AMQ221010: Backup Server is now live
> 2020-03-18 12:35:01,626 INFO [org.apache.activemq.artemis.core.server] AMQ221027: Bridge ClusterConnectionBridge@5ced3917 [name=$.artemis.internal.sf.my-cluster.2c85cd
> 9e-5f75-11ea-9be1-0af9119190b9, queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.2c85cd9e-5f75-11ea-9be1-0af9119190b9, postOffice=PostOfficeImpl [server=ActiveMQSe
> rverImpl::serverUUID=19ade3bb-5f75-11ea-b327-1216d251b187], temp=false]@58d2a652 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnecti
> onBridge@5ced3917 [name=$.artemis.internal.sf.my-cluster.2c85cd9e-5f75-11ea-9be1-0af9119190b9, queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.2c85cd9e-5f75-11ea-
> 9be1-0af9119190b9, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=19ade3bb-5f75-11ea-b327-1216d251b187], temp=false]@58d2a652 targetConnector=ServerLo
> catorImpl [initialConnectors=[TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=616
> 16&host=artemis2-sl-idsk-c
> om], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1788190971[nodeUUID=19ade3bb-5f75-11ea-b327-1216d251b187, connector=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=artemis2-sl-idsk-com, address=, server=ActiveMQServerImpl::serverUUID=19ade3bb-5f75-11ea-b327-1216d251b187])) [initialConnectors=[TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=artemis2-sl-idsk-com], discoveryGroupConfiguration=null]] is connected
> 2020-03-18 12:35:20,621 WARN [org.apache.activemq.artemis.core.server] AMQ222054: Error on executing IOCallback: java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$25/191327468@479377d0 rejected from org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@291e146[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 59059]
> at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) [rt.jar:1.8.0_212]
> at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) [rt.jar:1.8.0_212]
> at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) [rt.jar:1.8.0_212]
> at org.apache.activemq.artemis.utils.actors.ProcessorBase.onAddedTaskIfNotRunning(ProcessorBase.java:205) [artemis-commons-2.11.0.jar:2.11.0]
> at org.apache.activemq.artemis.utils.actors.ProcessorBase.task(ProcessorBase.java:193) [artemis-commons-2.11.0.jar:2.11.0]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.execute(OrderedExecutor.java:54) [artemis-commons-2.11.0.jar:2.11.0]
> at org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl.execute(OperationContextImpl.java:238) [artemis-server-2.11.0.jar:2.11.0]
> at org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl.checkTasks(OperationContextImpl.java:221) [artemis-server-2.11.0.jar:2.11.0]
> at org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl.done(OperationContextImpl.java:197) [artemis-server-2.11.0.jar:2.11.0]
> at org.apache.activemq.artemis.core.io.IOCallback.lambda$done$0(IOCallback.java:43) [artemis-journal-2.11.0.jar:2.11.0]
> at java.util.ArrayList.forEach(ArrayList.java:1257) [rt.jar:1.8.0_212]
> at org.apache.activemq.artemis.core.io.IOCallback.done(IOCallback.java:41) [artemis-journal-2.11.0.jar:2.11.0]
> at org.apache.activemq.artemis.core.io.DelegateCallback.done(DelegateCallback.java:41) [artemis-journal-2.11.0.jar:2.11.0]
> at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.doInternalWrite(NIOSequentialFile.java:395) [artemis-journal-2.11.0.jar:2.11.0]
> at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.internalWrite(NIOSequentialFile.java:359) [artemis-journal-2.11.0.jar:2.11.0]
> at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.access$100(NIOSequentialFile.java:43) [artemis-journal-2.11.0.jar:2.11.0]
> at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile$SyncLocalBufferObserver.flushBuffer(NIOSequentialFile.java:434) [artemis-journal-2.11.0.jar:2.11.0]
> at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flushBatch(TimedBuffer.java:361) [artemis-journal-2.11.0.jar:2.11.0]
> at org.apache.activemq.artemis.core.io.buffer.TimedBuffer$CheckTimer.run(TimedBuffer.java:457) [artemis-journal-2.11.0.jar:2.11.0]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_212]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)