You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@activemq.apache.org by "Justin Bertram (Jira)" <ji...@apache.org> on 2025/05/20 17:34:00 UTC

[jira] [Assigned] (ARTEMIS-5484) Artemis Critical IO error when persistence over postgresDB

     [ https://issues.apache.org/jira/browse/ARTEMIS-5484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Justin Bertram reassigned ARTEMIS-5484:
---------------------------------------

    Assignee:     (was: Clebert Suconic)

> Artemis Critical IO error when persistence over postgresDB
> ----------------------------------------------------------
>
>                 Key: ARTEMIS-5484
>                 URL: https://issues.apache.org/jira/browse/ARTEMIS-5484
>             Project: ActiveMQ Artemis
>          Issue Type: Improvement
>    Affects Versions: 2.33.0
>            Reporter: Hitesh Jha
>            Priority: Major
>         Attachments: backup-broker.xml, primary-broker.xml
>
>
> Artemis after few days keeps restarting with below IO error. We understand that node manager lock is lost, but it does not seem due to DB disconnection. It seems during high throughput node manager lease lock gets halted. We compared the behavior with oracle DB with same config but never saw any restarts.
> {noformat}
> 2025-01-25 04:05:54,930 ERROR [org.apache.activemq.artemis.core.server.impl.jdbc.ActiveMQScheduledLeaseLock] primary lock successful renew tooks 2008 ms, while is supposed to take <2000 ms
> 2025-01-25 06:51:55,590 ERROR [org.apache.activemq.artemis.core.server.impl.jdbc.ActiveMQScheduledLeaseLock] primary lock successful renew tooks 2005 ms, while is supposed to take <2000 ms
> 2025-01-26 04:18:58,342 ERROR [org.apache.activemq.artemis.core.server.impl.jdbc.ActiveMQScheduledLeaseLock] primary lock successful renew tooks 2005 ms, while is supposed to take <2000 ms
> 2025-01-26 12:10:24,259 ERROR [org.apache.activemq.artemis.core.server.impl.jdbc.ActiveMQScheduledLeaseLock] primary lock successful renew tooks 2076 ms, while is supposed to take <2000 ms
> 2025-01-26 14:10:29,031 ERROR [org.apache.activemq.artemis.core.server] AMQ222010: Critical IO Error, shutting down the server. file=Lost NodeManager lock, message=NULL
> java.io.IOException: lost lock
> 	at org.apache.activemq.artemis.core.server.impl.SharedStorePrimaryActivation.lambda$registerActiveLockListener$0(SharedStorePrimaryActivation.java:124) ~[artemis-server-2.33.0.jar:2.33.0]
> 	at org.apache.activemq.artemis.core.server.NodeManager.lambda$notifyLostLock$0(NodeManager.java:172) ~[artemis-server-2.33.0.jar:2.33.0]
> 	at java.base/java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
> 	at org.apache.activemq.artemis.core.server.NodeManager.notifyLostLock(NodeManager.java:170) ~[artemis-server-2.33.0.jar:2.33.0]
> 	at org.apache.activemq.artemis.core.server.impl.jdbc.JdbcNodeManager.notifyLostLock(JdbcNodeManager.java:158) ~[artemis-server-2.33.0.jar:2.33.0]
> 	at org.apache.activemq.artemis.core.server.impl.jdbc.ActiveMQScheduledLeaseLock.run(ActiveMQScheduledLeaseLock.java:115) ~[artemis-server-2.33.0.jar:2.33.0]
> 	at org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent.runForExecutor(ActiveMQScheduledComponent.java:314) ~[artemis-commons-2.33.0.jar:2.33.0]
> 	at org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent.lambda$bookedRunForScheduler$2(ActiveMQScheduledComponent.java:321) ~[artemis-commons-2.33.0.jar:2.33.0]
> 	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57) ~[artemis-commons-2.33.0.jar:2.33.0]
> 	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32) ~[artemis-commons-2.33.0.jar:2.33.0]
> 	at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68) ~[artemis-commons-2.33.0.jar:2.33.0]
> 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
> 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
> 	at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.33.0.jar:2.33.0]
> 2025-01-26 14:10:29,043 ERROR [org.apache.activemq.artemis.core.server.impl.jdbc.ActiveMQScheduledLeaseLock] primary lock failed renew tooks 2377 ms, while is supposed to take <2000 ms
> 2025-01-26 14:10:29,357 ERROR [org.apache.activemq.artemis.core.server] AMQ224068: Unable to stop component: org.apache.activemq.artemis.core.server.impl.SharedStorePrimaryActivation
> java.lang.IllegalStateException: the node manager is supposed to be started
> 	at org.apache.activemq.artemis.core.server.NodeManager.checkStarted(NodeManager.java:162) ~[artemis-server-2.33.0.jar:2.33.0]
> 	at org.apache.activemq.artemis.core.server.impl.jdbc.JdbcNodeManager.crashPrimaryServer(JdbcNodeManager.java:543) ~[artemis-server-2.33.0.jar:2.33.0]
> 	at org.apache.activemq.artemis.core.server.impl.SharedStorePrimaryActivation.close(SharedStorePrimaryActivation.java:150) ~[artemis-server-2.33.0.jar:2.33.0]
> 	at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stop(ActiveMQServerImpl.java:1474) [artemis-server-2.33.0.jar:2.33.0]
> 	at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stop(ActiveMQServerImpl.java:1267) [artemis-server-2.33.0.jar:2.33.0]
> 	at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stop(ActiveMQServerImpl.java:1260) [artemis-server-2.33.0.jar:2.33.0]
> 	at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5.run(ActiveMQServerImpl.java:1040) [artemis-server-2.33.0.jar:2.33.0]
> 2025-01-26 14:10:29,359 INFO  [org.apache.activemq.artemis] AMQ241005: Stopping embedded web server
> 2025-01-26 14:10:29,401 INFO  [io.hawt.HawtioContextListener] Destroying hawtio services
> 2025-01-26 14:10:29,410 INFO  [org.apache.activemq.hawtio.plugin.PluginContextListener] Destroyed artemis-plugin plugin
> 2025-01-26 14:10:29,413 INFO  [org.apache.activemq.hawtio.branding.PluginContextListener] Destroyed activemq-branding plugin
> 2025-01-26 14:10:29,478 INFO  [org.apache.activemq.artemis] AMQ241006: Stopped embedded web server
> 2025-01-26 14:10:29,478 INFO  [org.apache.activemq.artemis.protocol.amqp.connect.mirror.AckManagerProvider] Removing ActiveMQServerImpl::name=artemis-primary-0
> 2025-01-26 14:10:29,480 INFO  [org.apache.activemq.artemis.core.server] AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.33.0 [3fa9c14f-0f29-11ef-accf-c2fcb3d9bfbe] stopped, uptime 3 days 23 hours{noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@activemq.apache.org
For additional commands, e-mail: issues-help@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact