You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by "Alexey Scherbakov (Jira)" <ji...@apache.org> on 2025/12/09 10:18:00 UTC
[jira] [Updated] (IGNITE-27288) Fix java.lang.AssertionError: Reordering detected on unstable raft topology
[ https://issues.apache.org/jira/browse/IGNITE-27288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexey Scherbakov updated IGNITE-27288:
---------------------------------------
Description:
I've observed this assertion in a scenario, when a partition leader gets temporary segmented from remaining nodes:
A,B,C - raft nodes
B - is leader
B is segmented
A and C choose new leader A
B returns to topology
B and C choose new leader C
{noformat}
2025-09-05 21:24:35:393 +0000 [ERROR][%poc-tester-SERVER-192.168.211.108-id-0%JRaft-FSMCaller-Disruptor_stripe_6-0][StateMachineAdapter] Encountered an error=Status[ESTATEMACHINE<10002>: StateMachine meet critical error when applying one or more tasks since index=1142, Status[ESTATEMACHINE<10002>: Reordering detected: [old=HybridTimestamp [physical=2025-09-05 21:24:34:122 +0000, logical=2, composite=115153795424059394], new=HybridTimestamp [physical=2025-09-05 21:24:34:085 +0000, logical=17, composite=115153795421634577]]]] on StateMachine org.apache.ignite.internal.raft.server.impl.JraftServerImpl$DelegatingStateMachine, it's highly recommended to implement this method as raft stops working since some error occurs, you should figure out the cause and repair or remove this node.
Error [type=ERROR_TYPE_STATE_MACHINE, status=Status[ESTATEMACHINE<10002>: StateMachine meet critical error when applying one or more tasks since index=1142, Status[ESTATEMACHINE<10002>: Reordering detected: [old=HybridTimestamp [physical=2025-09-05 21:24:34:122 +0000, logical=2, composite=115153795424059394], new=HybridTimestamp [physical=2025-09-05 21:24:34:085 +0000, logical=17, composite=115153795421634577]]]]]
at org.apache.ignite.raft.jraft.core.IteratorImpl.getOrCreateError(IteratorImpl.java:168)
at org.apache.ignite.raft.jraft.core.IteratorImpl.setErrorAndRollback(IteratorImpl.java:159)
at org.apache.ignite.raft.jraft.core.IteratorWrapper.setErrorAndRollback(IteratorWrapper.java:74)
at org.apache.ignite.internal.raft.server.impl.JraftServerImpl$DelegatingStateMachine.onApply(JraftServerImpl.java:921)
at org.apache.ignite.raft.jraft.core.FSMCallerImpl.doApplyTasks(FSMCallerImpl.java:570)
at org.apache.ignite.raft.jraft.core.FSMCallerImpl.doCommitted(FSMCallerImpl.java:536)
at org.apache.ignite.raft.jraft.core.FSMCallerImpl.runApplyTask(FSMCallerImpl.java:454)
at org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:123)
at org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:117)
at org.apache.ignite.raft.jraft.disruptor.StripedDisruptor$StripeEntryHandler.onEvent(StripedDisruptor.java:322)
at org.apache.ignite.raft.jraft.disruptor.StripedDisruptor$StripeEntryHandler.onEvent(StripedDisruptor.java:279)
at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:167)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:122)
at java.base/java.lang.Thread.run(Thread.java:829)
{noformat}
Need to get to the bottom of this issue.
Most likely it's related to RAFT leader lease invariant broken (no two leaders exists in intersecting raft leader leases)
was:
I've observed this assertion in a scenario, then a partition leader gets temporary segmented from remaining nodes
{noformat}
2025-09-05 21:24:35:393 +0000 [ERROR][%poc-tester-SERVER-192.168.211.108-id-0%JRaft-FSMCaller-Disruptor_stripe_6-0][StateMachineAdapter] Encountered an error=Status[ESTATEMACHINE<10002>: StateMachine meet critical error when applying one or more tasks since index=1142, Status[ESTATEMACHINE<10002>: Reordering detected: [old=HybridTimestamp [physical=2025-09-05 21:24:34:122 +0000, logical=2, composite=115153795424059394], new=HybridTimestamp [physical=2025-09-05 21:24:34:085 +0000, logical=17, composite=115153795421634577]]]] on StateMachine org.apache.ignite.internal.raft.server.impl.JraftServerImpl$DelegatingStateMachine, it's highly recommended to implement this method as raft stops working since some error occurs, you should figure out the cause and repair or remove this node.
Error [type=ERROR_TYPE_STATE_MACHINE, status=Status[ESTATEMACHINE<10002>: StateMachine meet critical error when applying one or more tasks since index=1142, Status[ESTATEMACHINE<10002>: Reordering detected: [old=HybridTimestamp [physical=2025-09-05 21:24:34:122 +0000, logical=2, composite=115153795424059394], new=HybridTimestamp [physical=2025-09-05 21:24:34:085 +0000, logical=17, composite=115153795421634577]]]]]
at org.apache.ignite.raft.jraft.core.IteratorImpl.getOrCreateError(IteratorImpl.java:168)
at org.apache.ignite.raft.jraft.core.IteratorImpl.setErrorAndRollback(IteratorImpl.java:159)
at org.apache.ignite.raft.jraft.core.IteratorWrapper.setErrorAndRollback(IteratorWrapper.java:74)
at org.apache.ignite.internal.raft.server.impl.JraftServerImpl$DelegatingStateMachine.onApply(JraftServerImpl.java:921)
at org.apache.ignite.raft.jraft.core.FSMCallerImpl.doApplyTasks(FSMCallerImpl.java:570)
at org.apache.ignite.raft.jraft.core.FSMCallerImpl.doCommitted(FSMCallerImpl.java:536)
at org.apache.ignite.raft.jraft.core.FSMCallerImpl.runApplyTask(FSMCallerImpl.java:454)
at org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:123)
at org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:117)
at org.apache.ignite.raft.jraft.disruptor.StripedDisruptor$StripeEntryHandler.onEvent(StripedDisruptor.java:322)
at org.apache.ignite.raft.jraft.disruptor.StripedDisruptor$StripeEntryHandler.onEvent(StripedDisruptor.java:279)
at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:167)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:122)
at java.base/java.lang.Thread.run(Thread.java:829)
{noformat}
Need to get to the bottom of this issue.
Most likely it's related to RAFT leader lease invariant broken (no two leaders exists in intersecting raft leader leases)
> Fix java.lang.AssertionError: Reordering detected on unstable raft topology
> ---------------------------------------------------------------------------
>
> Key: IGNITE-27288
> URL: https://issues.apache.org/jira/browse/IGNITE-27288
> Project: Ignite
> Issue Type: Bug
> Reporter: Alexey Scherbakov
> Assignee: Alexey Scherbakov
> Priority: Major
> Labels: ignite-3
>
> I've observed this assertion in a scenario, when a partition leader gets temporary segmented from remaining nodes:
> A,B,C - raft nodes
> B - is leader
> B is segmented
> A and C choose new leader A
> B returns to topology
> B and C choose new leader C
> {noformat}
> 2025-09-05 21:24:35:393 +0000 [ERROR][%poc-tester-SERVER-192.168.211.108-id-0%JRaft-FSMCaller-Disruptor_stripe_6-0][StateMachineAdapter] Encountered an error=Status[ESTATEMACHINE<10002>: StateMachine meet critical error when applying one or more tasks since index=1142, Status[ESTATEMACHINE<10002>: Reordering detected: [old=HybridTimestamp [physical=2025-09-05 21:24:34:122 +0000, logical=2, composite=115153795424059394], new=HybridTimestamp [physical=2025-09-05 21:24:34:085 +0000, logical=17, composite=115153795421634577]]]] on StateMachine org.apache.ignite.internal.raft.server.impl.JraftServerImpl$DelegatingStateMachine, it's highly recommended to implement this method as raft stops working since some error occurs, you should figure out the cause and repair or remove this node.
> Error [type=ERROR_TYPE_STATE_MACHINE, status=Status[ESTATEMACHINE<10002>: StateMachine meet critical error when applying one or more tasks since index=1142, Status[ESTATEMACHINE<10002>: Reordering detected: [old=HybridTimestamp [physical=2025-09-05 21:24:34:122 +0000, logical=2, composite=115153795424059394], new=HybridTimestamp [physical=2025-09-05 21:24:34:085 +0000, logical=17, composite=115153795421634577]]]]]
> at org.apache.ignite.raft.jraft.core.IteratorImpl.getOrCreateError(IteratorImpl.java:168)
> at org.apache.ignite.raft.jraft.core.IteratorImpl.setErrorAndRollback(IteratorImpl.java:159)
> at org.apache.ignite.raft.jraft.core.IteratorWrapper.setErrorAndRollback(IteratorWrapper.java:74)
> at org.apache.ignite.internal.raft.server.impl.JraftServerImpl$DelegatingStateMachine.onApply(JraftServerImpl.java:921)
> at org.apache.ignite.raft.jraft.core.FSMCallerImpl.doApplyTasks(FSMCallerImpl.java:570)
> at org.apache.ignite.raft.jraft.core.FSMCallerImpl.doCommitted(FSMCallerImpl.java:536)
> at org.apache.ignite.raft.jraft.core.FSMCallerImpl.runApplyTask(FSMCallerImpl.java:454)
> at org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:123)
> at org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:117)
> at org.apache.ignite.raft.jraft.disruptor.StripedDisruptor$StripeEntryHandler.onEvent(StripedDisruptor.java:322)
> at org.apache.ignite.raft.jraft.disruptor.StripedDisruptor$StripeEntryHandler.onEvent(StripedDisruptor.java:279)
> at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:167)
> at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:122)
> at java.base/java.lang.Thread.run(Thread.java:829)
> {noformat}
> Need to get to the bottom of this issue.
> Most likely it's related to RAFT leader lease invariant broken (no two leaders exists in intersecting raft leader leases)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)