You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@activemq.apache.org by "Justin Bertram (Jira)" <ji...@apache.org> on 2022/09/14 17:57:00 UTC
[jira] [Resolved] (ARTEMIS-3973) High pressure on paging can leave address stuck in paging
[ https://issues.apache.org/jira/browse/ARTEMIS-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Justin Bertram resolved ARTEMIS-3973.
-------------------------------------
Fix Version/s: 3.0.0
2.25.1
Resolution: Fixed
> High pressure on paging can leave address stuck in paging
> ---------------------------------------------------------
>
> Key: ARTEMIS-3973
> URL: https://issues.apache.org/jira/browse/ARTEMIS-3973
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Reporter: Anton Roskvist
> Priority: Major
> Fix For: 3.0.0, 2.25.1
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> When sending a significant load through the paging store, there are occasional issues that causes the address to get stuck in paging and may end up in a generally inconsistent state (incorrect metrics, and unreadable messages for instance).
> I have seen these stack traces associated with this issue:
> {noformat}
> WARN [org.apache.activemq.artemis.core.server] AMQ222225: Sending unexpected exception to the client: java.lang.IllegalStateException: Unable to insert
> at io.netty.util.collection.IntObjectHashMap.put(IntObjectHashMap.java:141) [netty-common-4.1.79.Final.jar:4.1.79.Final]
> at org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl$PageCursorInfo.internalAddACK(PageSubscriptionImpl.java:1192) [artemis-server-2.25.0.jar:2.25.0]
> at org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl$PageCursorInfo.addACK(PageSubscriptionImpl.java:1174) [artemis-server-2.25.0.jar:2.25.0]
> at org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl.processACK(PageSubscriptionImpl.java:961) [artemis-server-2.25.0.jar:2.25.0]
> at org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl$PageCursorTX.afterCommit(PageSubscriptionImpl.java:1260) [artemis-server-2.25.0.jar:2.25.0]
> at org.apache.activemq.artemis.core.transaction.impl.TransactionImpl.afterCommit(TransactionImpl.java:564) [artemis-server-2.25.0.jar:2.25.0]
> at org.apache.activemq.artemis.core.transaction.impl.TransactionImpl$2.done(TransactionImpl.java:305) [artemis-server-2.25.0.jar:2.25.0]
> at org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl.executeOnCompletion(OperationContextImpl.java:195) [artemis-server-2.25.0.jar:2.25.0]
> at org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl.executeOnCompletion(OperationContextImpl.java:141) [artemis-server-2.25.0.jar:2.25.0]
> at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.afterCompleteOperations(AbstractJournalStorageManager.java:338) [artemis-server-2.25.0.jar:2.25.0]
> at org.apache.activemq.artemis.core.transaction.impl.TransactionImpl.commit(TransactionImpl.java:296) [artemis-server-2.25.0.jar:2.25.0]
> at org.apache.activemq.artemis.core.transaction.impl.TransactionImpl.commit(TransactionImpl.java:247) [artemis-server-2.25.0.jar:2.25.0]
> at org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.commit(ServerSessionImpl.java:1320) [artemis-server-2.25.0.jar:2.25.0]
> at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.slowPacketHandler(ServerSessionPacketHandler.java:474) [artemis-server-2.25.0.jar:2.25.0]
> at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.onMessagePacket(ServerSessionPacketHandler.java:298) [artemis-server-2.25.0.jar:2.25.0]
> at org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:33) [artemis-commons-2.25.0.jar:]
> at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:67) [artemis-commons-2.25.0.jar:]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:56) [artemis-commons-2.25.0.jar:]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.25.0.jar:]
> at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:67) [artemis-commons-2.25.0.jar:]
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [java.base:]
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [java.base:]
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.25.0.jar:] {code}
> and
> {code:java}
> WARN [org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl] Index 16384 out of bounds for length 16384: java.lang.ArrayIndexOutOfBoundsException: Index 16384 out of bounds for length 16384
> at io.netty.util.collection.IntObjectHashMap.indexOf(IntObjectHashMap.java:345) [netty-common-4.1.79.Final.jar:4.1.79.Final]
> at io.netty.util.collection.IntObjectHashMap.get(IntObjectHashMap.java:114) [netty-common-4.1.79.Final.jar:4.1.79.Final]
> at org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl$PageCursorInfo.isAck(PageSubscriptionImpl.java:1061) [artemis-server-2.25.0.jar:2.25.0]
> at org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl$CursorIterator.moveNext(PageSubscriptionImpl.java:1407) [artemis-server-2.25.0.jar:2.25.0]
> at org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl$CursorIterator.next(PageSubscriptionImpl.java:1348) [artemis-server-2.25.0.jar:2.25.0]
> at org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl$CursorIterator.tryNext(PageSubscriptionImpl.java:1492) [artemis-server-2.25.0.jar:2.25.0]
> at org.apache.activemq.artemis.core.server.impl.QueueImpl.depage(QueueImpl.java:3294) [artemis-server-2.25.0.jar:2.25.0]
> at org.apache.activemq.artemis.core.server.impl.QueueImpl.lambda$scheduleDepage$7(QueueImpl.java:3266) [artemis-server-2.25.0.jar:2.25.0]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:56) [artemis-commons-2.25.0.jar:]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.25.0.jar:]
> at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:67) [artemis-commons-2.25.0.jar:]
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [java.base:]
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [java.base:]
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.25.0.jar:] {noformat}
> I have yet to reproduce this in a test but it happens quite regularly as the broker approaches it's throughput limit based on CPU-resources, in my case that's somewhere in the neighborhood of 250k messages/s.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)