You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by "Semyon Zikunov (Jira)" <ji...@apache.org> on 2024/12/19 04:50:00 UTC
[jira] [Commented] (IGNITE-20902) Fix IgniteWalRebalanceTest flakiness
[ https://issues.apache.org/jira/browse/IGNITE-20902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17906920#comment-17906920 ]
Semyon Zikunov commented on IGNITE-20902:
-----------------------------------------
Considering the testWithLocalWalChange() - Success Rate: 3.2%, the first run is always successful, and the next ones fall with the *Invalid version for inner update error* when performing the first insertion in a loop on line 332 of cache.put(k, new IndexedObject(k)). The full stack trace is below:
{code:java}
[2024-12-19T14:32:09,890][ERROR][sys-stripe-0-#1089%wal.IgniteWalRebalanceTest1%][IgniteTestResources] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException: B+Tree is corrupted [groupId=94416770, pageIds=[1125899906842630], msg=Runtime failure on search row: SearchRow [key=KeyCacheObjectImpl [part=0, val=0, hasValBytes=true], hash=0, cacheId=0]]]]
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: B+Tree is corrupted [groupId=94416770, pageIds=[1125899906842630], msg=Runtime failure on search row: SearchRow [key=KeyCacheObjectImpl [part=0, val=0, hasValBytes=true], hash=0, cacheId=0]]
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.corruptedTreeException(BPlusTree.java:6046) [classes/:?]
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:2171) [classes/:?]
at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke0(IgniteCacheOffheapManagerImpl.java:1497) [classes/:?]
at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1480) [classes/:?]
at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:2530) [classes/:?]
at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:395) [classes/:?]
at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:1519) [classes/:?]
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2528) [classes/:?]
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update(GridDhtAtomicCache.java:1988) [classes/:?]
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1807) [classes/:?]
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1680) [classes/:?]
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3170) [classes/:?]
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$3.apply(GridDhtAtomicCache.java:270) [classes/:?]
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$3.apply(GridDhtAtomicCache.java:265) [classes/:?]
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1096) [classes/:?]
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:597) [classes/:?]
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:398) [classes/:?]
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:316) [classes/:?]
at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:306) [classes/:?]
at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1893) [classes/:?]
at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1514) [classes/:?]
at org.apache.ignite.internal.managers.communication.GridIoManager$9.execute(GridIoManager.java:1418) [classes/:?]
at org.apache.ignite.internal.managers.communication.TraceRunnable.run(TraceRunnable.java:55) [classes/:?]
at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:637) [classes/:?]
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125) [classes/:?]
at java.base/java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.lang.AssertionError: Invalid version for inner update [name=cache, topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], isNew=false, entry=GridDhtCacheEntry [rdrs=ReaderId[] [], part=0, super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [part=0, val=0, hasValBytes=true], val=org.apache.ignite.internal.processors.cache.persistence.db.wal.IgniteWalRebalanceTest$IndexedObject [idHash=1819169310, hash=1887600494, payload=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0... and 924 more], iVal=-1], ver=GridCacheVersion [topVer=346062732, order=1734582728645, nodeOrder=2, dataCenterId=0], hash=0, extras=null, flags=0]]], newVer=GridCacheVersion [topVer=346062731, order=1734582729780, nodeOrder=2, dataCenterId=0]]
at org.apache.ignite.internal.processors.cache.GridCacheMapEntry$AtomicCacheUpdateClosure.versionCheck(GridCacheMapEntry.java:5346) ~[classes/:?]
at org.apache.ignite.internal.processors.cache.GridCacheMapEntry$AtomicCacheUpdateClosure.call(GridCacheMapEntry.java:4707) ~[classes/:?]
at org.apache.ignite.internal.processors.cache.GridCacheMapEntry$AtomicCacheUpdateClosure.call(GridCacheMapEntry.java:4470) ~[classes/:?]
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Invoke.invokeClosure(BPlusTree.java:3942) ~[classes/:?]
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invokeDown(BPlusTree.java:2262) ~[classes/:?]
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:2138) [classes/:?]
... 24 more
[2024-12-19T14:32:09,897][ERROR][sys-stripe-0-#1089%wal.IgniteWalRebalanceTest1%][FailureProcessor] A critical problem with persistence data structures was detected. Please make backup of persistence storage and WAL files for further analysis. Persistence storage path: null WAL path: db/wal WAL archive path: db/wal/archive
[2024-12-19T14:32:09,899][ERROR][sys-stripe-0-#1089%wal.IgniteWalRebalanceTest1%][FailureProcessor] No deadlocked threads detected.
{code}
Attaching the startup log on my personal machine: [^testWithLocalWalChange.log]
And on TC:[^tc_stacktrace.log]
> Fix IgniteWalRebalanceTest flakiness
> ------------------------------------
>
> Key: IGNITE-20902
> URL: https://issues.apache.org/jira/browse/IGNITE-20902
> Project: Ignite
> Issue Type: Test
> Reporter: Ilya Shishkov
> Priority: Trivial
> Labels: ise
> Attachments: tc_stacktrace.log, testWithLocalWalChange.log
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Some of {{IgniteWalRebalanceTest}} methods are flaky:
> # {{#testRebalanceCancelOnSupplyError}} - fails often.
> # {{#testWithLocalWalChange}} - fails rarely.
> Tests fails with a foolowing error:
> {code}
> java.lang.AssertionError: WAL rebalance hasn't been invoked.
> at org.apache.ignite.internal.processors.cache.persistence.db.wal.IgniteWalRebalanceTest.afterTest(IgniteWalRebalanceTest.java:197)
> at org.apache.ignite.testframework.junits.GridAbstractTest.runAfterTest(GridAbstractTest.java:760)
> at org.apache.ignite.testframework.junits.GridAbstractTest.cleanUpTestEnviroment(GridAbstractTest.java:737)
> at org.apache.ignite.testframework.junits.GridAbstractTest.runTest(GridAbstractTest.java:2550)
> at org.apache.ignite.testframework.junits.GridAbstractTest$1.evaluate(GridAbstractTest.java:235)
> ...
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)