You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@activemq.apache.org by "Ilkka Virolainen (JIRA)" <ji...@apache.org> on 2018/07/06 12:08:00 UTC
[jira] [Commented] (ARTEMIS-1889) Replication Failure During
Compacting
[ https://issues.apache.org/jira/browse/ARTEMIS-1889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16534755#comment-16534755 ]
Ilkka Virolainen commented on ARTEMIS-1889:
-------------------------------------------
It is also worth noting that when the replication errors start to appear, clients connecting to the broker encounter timeouts. As of yet there is no replicator.
> Replication Failure During Compacting
> -------------------------------------
>
> Key: ARTEMIS-1889
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1889
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: Broker
> Affects Versions: 2.5.0, 2.6.0
> Environment: RHEL 6.2, Windows 7
> Reporter: Ilkka Virolainen
> Priority: Major
>
> It is possible for the broker to corrupt its journal during operations and subsequently fail replication when configured as a replicating master/slave pair. Exceptions on master node:
>
> {noformat}
> 15:00:18,074 ERROR [org.apache.activemq.artemis.core.server] AMQ224016: Caught exception:
> ActiveMQIllegalStateException[errorType=ILLEGAL_STATE message=AMQ119028: Consumer 0 doesn't
> exist on the server]
> at org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.findConsumer(ServerSessionImpl.java:889)
> [artemis-server-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.acknowledge(ServerSessionImpl.java:858)
> [artemis-server-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.onSessionAcknowledge(ServerSessionPacketHandler.java:632)
> [artemis-server-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.onMessagePacket(ServerSessionPacketHandler.java:268)
> [artemis-server-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:33) [artemis-commons-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66)
> [artemis-commons-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
> [artemis-commons-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
> [artemis-commons-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66)
> [artemis-commons-2.5.0.jar:2.5.0]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_162]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_162]
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> [artemis-commons-2.5.0.jar:2.5.0]
> {noformat}
>
> After this both master and slave log these errors:
>
> {noformat}
> 20:54:15,063 WARN [org.apache.activemq.artemis.journal] AMQ142032: Error reading journal
> file: java.lang.IllegalStateException: Inconsistency during compacting: RollbackRecord ID
> = 4452580183 for an already rolled back transaction during compacting
> at org.apache.activemq.artemis.core.journal.impl.JournalCompactor.onReadRollbackRecord(JournalCompactor.java:349)
> [artemis-journal-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.journal.impl.JournalImpl.readJournalFile(JournalImpl.java:733)
> [artemis-journal-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.journal.impl.JournalImpl.compact(JournalImpl.java:1566)
> [artemis-journal-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.journal.impl.JournalImpl$14.run(JournalImpl.java:2111)
> [artemis-journal-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
> [artemis-commons-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
> [artemis-commons-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66)
> [artemis-commons-2.5.0.jar:2.5.0]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_162]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_162]
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> [artemis-commons-2.5.0.jar:2.5.0]
> 20:54:15,067 WARN [org.apache.activemq.artemis.journal] AMQ142011: Error on reading compacting
> for JournalFileImpl: (activemq-data-259.amq id = 355, recordID = 355)
> 20:54:15,104 ERROR [org.apache.activemq.artemis.journal] AMQ144003: Error compacting: java.lang.Exception:
> Error on reading compacting for JournalFileImpl: (activemq-data-259.amq id = 355, recordID
> = 355)
> at org.apache.activemq.artemis.core.journal.impl.JournalImpl.compact(JournalImpl.java:1569)
> [artemis-journal-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.journal.impl.JournalImpl$14.run(JournalImpl.java:2111)
> [artemis-journal-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
> [artemis-commons-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
> [artemis-commons-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66)
> [artemis-commons-2.5.0.jar:2.5.0]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_162]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_162]
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> [artemis-commons-2.5.0.jar:2.5.0]
> Caused by: java.lang.Exception: Inconsistency during compacting: RollbackRecord ID = 4452580183
> for an already rolled back transaction during compacting
> at org.apache.activemq.artemis.core.journal.impl.JournalImpl.readJournalFile(JournalImpl.java:761)
> [artemis-journal-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.journal.impl.JournalImpl.compact(JournalImpl.java:1566)
> [artemis-journal-2.5.0.jar:2.5.0]
> ... 7 more
> Caused by: java.lang.IllegalStateException: Inconsistency during compacting: RollbackRecord
> ID = 4452580183 for an already rolled back transaction during compacting
> at org.apache.activemq.artemis.core.journal.impl.JournalCompactor.onReadRollbackRecord(JournalCompactor.java:349)
> [artemis-journal-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.journal.impl.JournalImpl.readJournalFile(JournalImpl.java:733)
> [artemis-journal-2.5.0.jar:2.5.0]
> ... 8 more{noformat}
>
> This results in HA not working. Restarting the broker logs:
> {noformat}
> 08:58:28,608 WARN [org.apache.activemq.artemis.core.server] AMQ222013: Error when trying
> to start replication: java.lang.RuntimeException: Error during compact, look at the logs
> at org.apache.activemq.artemis.core.journal.impl.JournalImpl.scheduleCompactAndBlock(JournalImpl.java:1481)
> [artemis-journal-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.startReplication(JournalStorageManager.java:553)
> [artemis-server-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation$2.run(SharedNothingLiveActivation.java:178)
> [artemis-server-2.5.0.jar:2.5.0]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_162]{noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)