You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by "Isaac Zhu (Jira)" <ji...@apache.org> on 2023/02/06 09:20:00 UTC
[jira] [Created] (IGNITE-18715) B+Tree corruption error caused Ignite cluster crash and not able restart
Isaac Zhu created IGNITE-18715:
----------------------------------
Summary: B+Tree corruption error caused Ignite cluster crash and not able restart
Key: IGNITE-18715
URL: https://issues.apache.org/jira/browse/IGNITE-18715
Project: Ignite
Issue Type: Bug
Components: cache
Affects Versions: 2.14
Reporter: Isaac Zhu
With version 2.14, see this error during doing cache remove & put. And after this happens, the cluster can't be restarted, all data get lost:
[00:48:21,922][SEVERE][sys-stripe-8-#9][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException: B+Tree is corrupted [groupId=828437433, pageIds=[217017202749409008], cacheId=-595580467, cacheName=SQL_PUBLIC_URT_TESTCASE_RESULTS_VCS, indexName=URT_TESTCASE_RESULTS_VCS_STATUS_IDX, groupName=nav_mem_part, msg=Runtime failure on search row: Row@7b8be26c[ key: SQL_PUBLIC_URT_TESTCASE_RESULTS_VCS_070eda86_0aab_4da3_900c_8d3baf08b3a7_KEY [idHash=1221100184, hash=701595465, TEST_CASE_ID=610062, GROUP_ID=497], val: SQL_PUBLIC_URT_TESTCASE_RESULTS_VCS_070eda86_0aab_4da3_900c_8d3baf08b3a7 [idHash=1823360128, hash=-882680090, NEW=null, STATUS=MARK_FOR_DELETION, DIFF_FILE_ID=610, EXEC_TIME=67, TIMED_OUT=null, RECORD_TIME=2023-02-05 20:26:30.234, FILE_MOD_TIME=2023-02-02 01:35:59.0, SINCE=2022-10-22 00:00:00.0, JIRA=, ERROR_TOOL=null, ERROR_CODE=null, BUILD_DATE=2023-02-01 20:00:00.0, PRODUCER_START_TIME=2023-02-03 00:10:41.0] ][ MARK_FOR_DELETION, 497, 610062 ]]]] class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: B+Tree is corrupted [groupId=828437433, pageIds=[217017202749409008], cacheId=-595580467, cacheName=SQL_PUBLIC_URT_TESTCASE_RESULTS_VCS, indexName=URT_TESTCASE_RESULTS_VCS_STATUS_IDX, groupName=nav_mem_part, msg=Runtime failure on search row: Row@7b8be26c[ key: SQL_PUBLIC_URT_TESTCASE_RESULTS_VCS_070eda86_0aab_4da3_900c_8d3baf08b3a7_KEY [idHash=1221100184, hash=701595465, TEST_CASE_ID=610062, GROUP_ID=497], val: SQL_PUBLIC_URT_TESTCASE_RESULTS_VCS_070eda86_0aab_4da3_900c_8d3baf08b3a7 [idHash=1823360128, hash=-882680090, NEW=null, STATUS=MARK_FOR_DELETION, DIFF_FILE_ID=610, EXEC_TIME=67, TIMED_OUT=null, RECORD_TIME=2023-02-05 20:26:30.234, FILE_MOD_TIME=2023-02-02 01:35:59.0, SINCE=2022-10-22 00:00:00.0, JIRA=, ERROR_TOOL=null, ERROR_CODE=null, BUILD_DATE=2023-02-01 20:00:00.0, PRODUCER_START_TIME=2023-02-03 00:10:41.0] ][ MARK_FOR_DELETION, 497, 610062 ]] at org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexTree.corruptedTreeException(InlineIndexTree.java:561) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doRemove(BPlusTree.java:2310) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removex(BPlusTree.java:2079) at org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexImpl.remove(InlineIndexImpl.java:377) at org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexImpl.onUpdate(InlineIndexImpl.java:330) at org.apache.ignite.internal.cache.query.index.IndexProcessor.updateIndex(IndexProcessor.java:465) at org.apache.ignite.internal.cache.query.index.IndexProcessor.updateIndexes(IndexProcessor.java:308) at org.apache.ignite.internal.cache.query.index.IndexProcessor.store(IndexProcessor.java:156) at org.apache.ignite.internal.processors.query.GridQueryProcessor.store(GridQueryProcessor.java:2741) at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.store(GridCacheQueryManager.java:420) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishUpdate(IgniteCacheOffheapManagerImpl.java:2629) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishUpdate(IgniteCacheOffheapManagerImpl.java:2611) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.update(IgniteCacheOffheapManagerImpl.java:2510) at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.update(GridCacheOffheapManager.java:2600) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.update(IgniteCacheOffheapManagerImpl.java:440) at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.applyUpdate(GridCacheDatabaseSharedManager.java:2987) at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.lambda$applyLogicalUpdates$29(GridCacheDatabaseSharedManager.java:2775) at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.lambda$stripedApply$28(GridCacheDatabaseSharedManager.java:2455) at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:637) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTreeRuntimeException: java.lang.IllegalStateException: Item not found: 19 at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.doInitFromLink(CacheDataRowAdapter.java:345) at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:165) at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:136) at org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexTree.createIndexRow(InlineIndexTree.java:360) at org.apache.ignite.internal.cache.query.index.sorted.inline.io.AbstractInlineLeafIO.getLookupRow(AbstractInlineLeafIO.java:129) at org.apache.ignite.internal.cache.query.index.sorted.inline.io.AbstractInlineLeafIO.getLookupRow(AbstractInlineLeafIO.java:37) at org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexTree.getRow(InlineIndexTree.java:403) at org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexTree.getRow(InlineIndexTree.java:72) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.getRow(BPlusTree.java:5693) at org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexTree.compare(InlineIndexTree.java:309) at org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexTree.compare(InlineIndexTree.java:72) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:5680) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:5600) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$1100(BPlusTree.java:162) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:369) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:6216) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run(BPlusTree.java:349) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:6202) at org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:174) at org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:415) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.read(BPlusTree.java:6403) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:2345) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:2364) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:2364) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:2364) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:2364) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doRemove(BPlusTree.java:2272) ... 19 more Caused by: java.lang.IllegalStateException: Item not found: 19 at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:488) at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:596) at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:638) at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncomplete(CacheDataRowAdapter.java:380) at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.doInitFromLink(CacheDataRowAdapter.java:316) ... 45 more
--
This message was sent by Atlassian Jira
(v8.20.10#820010)