You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@activemq.apache.org by "Hitesh Jha (Jira)" <ji...@apache.org> on 2025/05/18 16:07:00 UTC
[jira] [Comment Edited] (ARTEMIS-5484) Artemis Critical IO error when persistence over postgresDB
[ https://issues.apache.org/jira/browse/ARTEMIS-5484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17952458#comment-17952458 ]
Hitesh Jha edited comment on ARTEMIS-5484 at 5/18/25 4:06 PM:
--------------------------------------------------------------
We are trying to understand what oracle DB is doing better than Postgres, so we can do the needed tuning accordingly.
Attached are the broker.xml for primary and backup as we are using HA setup with sharedstore.
was (Author: JIRAUSER309715):
We are trying to understand what oracle DB is doing better than Postgres, so we can do the needed tuning accordingly.
> Artemis Critical IO error when persistence over postgresDB
> ----------------------------------------------------------
>
> Key: ARTEMIS-5484
> URL: https://issues.apache.org/jira/browse/ARTEMIS-5484
> Project: ActiveMQ Artemis
> Issue Type: Improvement
> Components: ActiveMQ-Artemis-Native
> Affects Versions: 2.33.0
> Reporter: Hitesh Jha
> Assignee: Clebert Suconic
> Priority: Major
> Attachments: backup-broker.xml, primary-broker.xml
>
>
> Artemis after few days keeps restarting with below IO error. We understand that node manager lock is lost, but it does not seem due to DB disconnection. It seems during high throughput node manager lease lock gets halted. We compared the behavior with oracle DB with same config but never saw any restarts.
> 2025-01-25 04:05:54,930 ERROR [org.apache.activemq.artemis.core.server.impl.jdbc.ActiveMQScheduledLeaseLock] primary lock successful renew tooks 2008 ms, while is supposed to take <2000 ms
> 2025-01-25 06:51:55,590 ERROR [org.apache.activemq.artemis.core.server.impl.jdbc.ActiveMQScheduledLeaseLock] primary lock successful renew tooks 2005 ms, while is supposed to take <2000 ms
> 2025-01-26 04:18:58,342 ERROR [org.apache.activemq.artemis.core.server.impl.jdbc.ActiveMQScheduledLeaseLock] primary lock successful renew tooks 2005 ms, while is supposed to take <2000 ms
> 2025-01-26 12:10:24,259 ERROR [org.apache.activemq.artemis.core.server.impl.jdbc.ActiveMQScheduledLeaseLock] primary lock successful renew tooks 2076 ms, while is supposed to take <2000 ms
> 2025-01-26 14:10:29,031 ERROR [org.apache.activemq.artemis.core.server] AMQ222010: Critical IO Error, shutting down the server. file=Lost NodeManager lock, message=NULL
> java.io.IOException: lost lock
> at org.apache.activemq.artemis.core.server.impl.SharedStorePrimaryActivation.lambda$registerActiveLockListener$0(SharedStorePrimaryActivation.java:124) ~[artemis-server-2.33.0.jar:2.33.0]
> at org.apache.activemq.artemis.core.server.NodeManager.lambda$notifyLostLock$0(NodeManager.java:172) ~[artemis-server-2.33.0.jar:2.33.0]
> at java.base/java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
> at org.apache.activemq.artemis.core.server.NodeManager.notifyLostLock(NodeManager.java:170) ~[artemis-server-2.33.0.jar:2.33.0]
> at org.apache.activemq.artemis.core.server.impl.jdbc.JdbcNodeManager.notifyLostLock(JdbcNodeManager.java:158) ~[artemis-server-2.33.0.jar:2.33.0]
> at org.apache.activemq.artemis.core.server.impl.jdbc.ActiveMQScheduledLeaseLock.run(ActiveMQScheduledLeaseLock.java:115) ~[artemis-server-2.33.0.jar:2.33.0]
> at org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent.runForExecutor(ActiveMQScheduledComponent.java:314) ~[artemis-commons-2.33.0.jar:2.33.0]
> at org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent.lambda$bookedRunForScheduler$2(ActiveMQScheduledComponent.java:321) ~[artemis-commons-2.33.0.jar:2.33.0]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57) ~[artemis-commons-2.33.0.jar:2.33.0]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32) ~[artemis-commons-2.33.0.jar:2.33.0]
> at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68) ~[artemis-commons-2.33.0.jar:2.33.0]
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.33.0.jar:2.33.0]
> 2025-01-26 14:10:29,043 ERROR [org.apache.activemq.artemis.core.server.impl.jdbc.ActiveMQScheduledLeaseLock] primary lock failed renew tooks 2377 ms, while is supposed to take <2000 ms
> 2025-01-26 14:10:29,357 ERROR [org.apache.activemq.artemis.core.server] AMQ224068: Unable to stop component: org.apache.activemq.artemis.core.server.impl.SharedStorePrimaryActivation
> java.lang.IllegalStateException: the node manager is supposed to be started
> at org.apache.activemq.artemis.core.server.NodeManager.checkStarted(NodeManager.java:162) ~[artemis-server-2.33.0.jar:2.33.0]
> at org.apache.activemq.artemis.core.server.impl.jdbc.JdbcNodeManager.crashPrimaryServer(JdbcNodeManager.java:543) ~[artemis-server-2.33.0.jar:2.33.0]
> at org.apache.activemq.artemis.core.server.impl.SharedStorePrimaryActivation.close(SharedStorePrimaryActivation.java:150) ~[artemis-server-2.33.0.jar:2.33.0]
> at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stop(ActiveMQServerImpl.java:1474) [artemis-server-2.33.0.jar:2.33.0]
> at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stop(ActiveMQServerImpl.java:1267) [artemis-server-2.33.0.jar:2.33.0]
> at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stop(ActiveMQServerImpl.java:1260) [artemis-server-2.33.0.jar:2.33.0]
> at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5.run(ActiveMQServerImpl.java:1040) [artemis-server-2.33.0.jar:2.33.0]
> 2025-01-26 14:10:29,359 INFO [org.apache.activemq.artemis] AMQ241005: Stopping embedded web server
> 2025-01-26 14:10:29,401 INFO [io.hawt.HawtioContextListener] Destroying hawtio services
> 2025-01-26 14:10:29,410 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Destroyed artemis-plugin plugin
> 2025-01-26 14:10:29,413 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Destroyed activemq-branding plugin
> 2025-01-26 14:10:29,478 INFO [org.apache.activemq.artemis] AMQ241006: Stopped embedded web server
> 2025-01-26 14:10:29,478 INFO [org.apache.activemq.artemis.protocol.amqp.connect.mirror.AckManagerProvider] Removing ActiveMQServerImpl::name=artemis-primary-0
> 2025-01-26 14:10:29,480 INFO [org.apache.activemq.artemis.core.server] AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.33.0 [3fa9c14f-0f29-11ef-accf-c2fcb3d9bfbe] stopped, uptime 3 days 23 hours
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@activemq.apache.org
For additional commands, e-mail: issues-help@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact