You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by "Roman Puchkovskiy (Jira)" <ji...@apache.org> on 2024/12/09 10:40:00 UTC
[jira] [Updated] (IGNITE-23907) Node cannot be restarted due to Metastorage checksum mismatch
[ https://issues.apache.org/jira/browse/IGNITE-23907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Roman Puchkovskiy updated IGNITE-23907:
---------------------------------------
Description:
2024-12-06 18:00:02:905 +0000 [ERROR][%node3%JRaft-FSMCaller-Disruptormetastorage_stripe_0-0][MetaStorageWriteHandler] Unknown error while processing command [commandIndex=59060, commandTerm=1, command=EvictIdempotentCommandsCacheCommandImpl [evictionTimestamp=HybridTimestamp [physical=2024-12-06 17:18:51:759 +0000, logical=55036, composite=113607018529412860], initiatorTime=HybridTimestamp [physical=2024-12-06 17:18:51:760 +0000, logical=1, composite=113607018529423361], safeTime=HybridTimestamp [physical=2024-12-06 17:18:51:760 +0000, logical=3, composite=113607018529423363]]]
org.apache.ignite.internal.metastorage.exceptions.MetaStorageException: IGN-CMN-65535 TraceId:47b327f8-274f-4716-8ee8-97e9c51cb4e9 Metastorage revision checksum differs from a checksum for the same revision saved earlier. This probably means that the Metastorage has diverged. [revision=5915, existingChecksum=-7812799892490681793, newChecksum=866784937340784120]
at org.apache.ignite.internal.metastorage.server.persistence.RocksDbKeyValueStorage.validateNoChecksumConflict(RocksDbKeyValueStorage.java:724)
at org.apache.ignite.internal.metastorage.server.persistence.RocksDbKeyValueStorage.completeAndWriteBatch(RocksDbKeyValueStorage.java:689)
at org.apache.ignite.internal.metastorage.server.persistence.RocksDbKeyValueStorage.removeAll(RocksDbKeyValueStorage.java:806)
at org.apache.ignite.internal.metastorage.server.raft.MetaStorageWriteHandler.evictIdempotentCommandsCache(MetaStorageWriteHandler.java:409)
at org.apache.ignite.internal.metastorage.server.raft.MetaStorageWriteHandler.handleWriteWithTime(MetaStorageWriteHandler.java:229)
at org.apache.ignite.internal.metastorage.server.raft.MetaStorageWriteHandler.handleNonCachedWriteCommand(MetaStorageWriteHandler.java:160)
at org.apache.ignite.internal.metastorage.server.raft.MetaStorageWriteHandler.handleWriteCommand(MetaStorageWriteHandler.java:130)
at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
at org.apache.ignite.internal.metastorage.server.raft.MetaStorageListener.onWrite(MetaStorageListener.java:216)
at org.apache.ignite.internal.raft.server.impl.JraftServerImpl$DelegatingStateMachine.onApply(JraftServerImpl.java:766)
at org.apache.ignite.raft.jraft.core.FSMCallerImpl.doApplyTasks(FSMCallerImpl.java:571)
at org.apache.ignite.raft.jraft.core.FSMCallerImpl.doCommitted(FSMCallerImpl.java:539)
at org.apache.ignite.raft.jraft.core.FSMCallerImpl.runApplyTask(FSMCallerImpl.java:458)
at org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:131)
at org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:125)
at org.apache.ignite.raft.jraft.disruptor.StripedDisruptor$StripeEntryHandler.onEvent(StripedDisruptor.java:326)
at org.apache.ignite.raft.jraft.disruptor.StripedDisruptor$StripeEntryHandler.onEvent(StripedDisruptor.java:283)
at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:167)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:122)
at java.base/java.lang.Thread.run(Thread.java:840)
The problem seems to be caused by the fact that idempotent cache command eviction produces removeAll() calls in a non-repeatable way: that is, two executions of the same EvictIdempotentCommandsCacheCommand against the same underlying storage might calculate different checksums (because they are calculated for different orders of keys-to-be-removed).
> Node cannot be restarted due to Metastorage checksum mismatch
> -------------------------------------------------------------
>
> Key: IGNITE-23907
> URL: https://issues.apache.org/jira/browse/IGNITE-23907
> Project: Ignite
> Issue Type: Bug
> Reporter: Roman Puchkovskiy
> Assignee: Roman Puchkovskiy
> Priority: Major
> Labels: ignite-3
>
> 2024-12-06 18:00:02:905 +0000 [ERROR][%node3%JRaft-FSMCaller-Disruptormetastorage_stripe_0-0][MetaStorageWriteHandler] Unknown error while processing command [commandIndex=59060, commandTerm=1, command=EvictIdempotentCommandsCacheCommandImpl [evictionTimestamp=HybridTimestamp [physical=2024-12-06 17:18:51:759 +0000, logical=55036, composite=113607018529412860], initiatorTime=HybridTimestamp [physical=2024-12-06 17:18:51:760 +0000, logical=1, composite=113607018529423361], safeTime=HybridTimestamp [physical=2024-12-06 17:18:51:760 +0000, logical=3, composite=113607018529423363]]]
> org.apache.ignite.internal.metastorage.exceptions.MetaStorageException: IGN-CMN-65535 TraceId:47b327f8-274f-4716-8ee8-97e9c51cb4e9 Metastorage revision checksum differs from a checksum for the same revision saved earlier. This probably means that the Metastorage has diverged. [revision=5915, existingChecksum=-7812799892490681793, newChecksum=866784937340784120]
> at org.apache.ignite.internal.metastorage.server.persistence.RocksDbKeyValueStorage.validateNoChecksumConflict(RocksDbKeyValueStorage.java:724)
> at org.apache.ignite.internal.metastorage.server.persistence.RocksDbKeyValueStorage.completeAndWriteBatch(RocksDbKeyValueStorage.java:689)
> at org.apache.ignite.internal.metastorage.server.persistence.RocksDbKeyValueStorage.removeAll(RocksDbKeyValueStorage.java:806)
> at org.apache.ignite.internal.metastorage.server.raft.MetaStorageWriteHandler.evictIdempotentCommandsCache(MetaStorageWriteHandler.java:409)
> at org.apache.ignite.internal.metastorage.server.raft.MetaStorageWriteHandler.handleWriteWithTime(MetaStorageWriteHandler.java:229)
> at org.apache.ignite.internal.metastorage.server.raft.MetaStorageWriteHandler.handleNonCachedWriteCommand(MetaStorageWriteHandler.java:160)
> at org.apache.ignite.internal.metastorage.server.raft.MetaStorageWriteHandler.handleWriteCommand(MetaStorageWriteHandler.java:130)
> at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
> at org.apache.ignite.internal.metastorage.server.raft.MetaStorageListener.onWrite(MetaStorageListener.java:216)
> at org.apache.ignite.internal.raft.server.impl.JraftServerImpl$DelegatingStateMachine.onApply(JraftServerImpl.java:766)
> at org.apache.ignite.raft.jraft.core.FSMCallerImpl.doApplyTasks(FSMCallerImpl.java:571)
> at org.apache.ignite.raft.jraft.core.FSMCallerImpl.doCommitted(FSMCallerImpl.java:539)
> at org.apache.ignite.raft.jraft.core.FSMCallerImpl.runApplyTask(FSMCallerImpl.java:458)
> at org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:131)
> at org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:125)
> at org.apache.ignite.raft.jraft.disruptor.StripedDisruptor$StripeEntryHandler.onEvent(StripedDisruptor.java:326)
> at org.apache.ignite.raft.jraft.disruptor.StripedDisruptor$StripeEntryHandler.onEvent(StripedDisruptor.java:283)
> at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:167)
> at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:122)
> at java.base/java.lang.Thread.run(Thread.java:840)
>
> The problem seems to be caused by the fact that idempotent cache command eviction produces removeAll() calls in a non-repeatable way: that is, two executions of the same EvictIdempotentCommandsCacheCommand against the same underlying storage might calculate different checksums (because they are calculated for different orders of keys-to-be-removed).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)