You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by "Dmitriy Pavlov (JIRA)" <ji...@apache.org> on 2019/01/23 17:12:00 UTC
[jira] [Comment Edited] (IGNITE-11030) Teamcity Bot cache is in
unrecoverable state after bot restart: need to check reasons
[ https://issues.apache.org/jira/browse/IGNITE-11030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16750235#comment-16750235 ]
Dmitriy Pavlov edited comment on IGNITE-11030 at 1/23/19 5:11 PM:
------------------------------------------------------------------
Scanning segment [248190], relIdx [0]
InitNewPageRecord:: pos=FileWALPointer [idx=248190, fileOff=7691736, len=45], rec=InitNewPageRecord [newPageId=00010000000145b4, super=PageDeltaRecord [grpId=482599155, pageId=0101001a000145b4, super=WALRecord [size=45, chainSize=0, pos=FileWALPointer [idx=248190, fileOff=7691736, len=45], type=INIT_NEW_PAGE_RECORD]]], ioType=1, ioVersion=1[2019-01-23 19:31:04,287][ERROR][main][root] Test failed.
has different partitions:
pageId=101001a000145b4
newPageId= 10000000145b4
was (Author: dpavlov):
pageId=101001a000145b4
newPageId= 10000000145b4
> Teamcity Bot cache is in unrecoverable state after bot restart: need to check reasons
> -------------------------------------------------------------------------------------
>
> Key: IGNITE-11030
> URL: https://issues.apache.org/jira/browse/IGNITE-11030
> Project: Ignite
> Issue Type: Task
> Reporter: Dmitriy Pavlov
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> - TC Bot was stopped with kill command (default signal).
> - Shutdown hook calls ignite close
> But after updating to V190122 TC Bot DB was became unrecoverable for "teamcityTestRunHist" cache with exception
> {noformat}
> Failure periodic check failed: javax.cache.CacheException: class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: Runtime failure on lookup row: SearchRow [key=org.apache.ignite.ci.teamcity.ignited.runhist.RunHistKey [idHash=1028871081, hash=1241170874, srvId=1411517106, testOrSuiteName=11924, branch=281], hash=1241170874, cacheId=0]
> com.google.common.util.concurrent.UncheckedExecutionException: javax.cache.CacheException: class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: Runtime failure on lookup row: SearchRow [key=org.apache.ignite.ci.teamcity.ignited.runhist.RunHistKey [idHash=1028871081, hash=1241170874, srvId=1411517106, testOrSuiteName=11924, branch=281], hash=1241170874, cacheId=0]
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2050)
> at com.google.common.cache.LocalCache.get(LocalCache.java:3952)
> at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4871)
> at org.apache.ignite.ci.di.cache.GuavaCachedInterceptor.invoke(GuavaCachedInterceptor.java:59)
> at org.apache.ignite.ci.teamcity.ignited.TeamcityIgnitedImpl.getTestRunHist(TeamcityIgnitedImpl.java:404)
> at org.apache.ignite.ci.di.AutoProfilingInterceptor.invoke(AutoProfilingInterceptor.java:76)
> at org.apache.ignite.ci.web.model.current.SuiteCurrentStatus.lambda$initFromContext$0(SuiteCurrentStatus.java:155)
> at java.util.Comparator.lambda$comparing$77a9974f$1(Comparator.java:469)
> at java.util.Collections$ReverseComparator2.compare(Collections.java:5178)
> at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355)
> at java.util.TimSort.sort(TimSort.java:220)
> at java.util.Arrays.sort(Arrays.java:1512)
> at java.util.ArrayList.sort(ArrayList.java:1462)
> at org.apache.ignite.ci.web.model.current.SuiteCurrentStatus.initFromContext(SuiteCurrentStatus.java:160)
> at org.apache.ignite.ci.web.model.current.ChainAtServerCurrentStatus.lambda$initFromContext$0(ChainAtServerCurrentStatus.java:171)
> at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
> at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
> at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
> at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
> at org.apache.ignite.ci.web.model.current.ChainAtServerCurrentStatus.initFromContext(ChainAtServerCurrentStatus.java:167)
> at org.apache.ignite.ci.tcbot.chain.TrackedBranchChainsProcessor.lambda$getTrackedBranchTestFailures$1(TrackedBranchChainsProcessor.java:121)
> at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
> at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
> at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
> at org.apache.ignite.ci.tcbot.chain.TrackedBranchChainsProcessor.getTrackedBranchTestFailures(TrackedBranchChainsProcessor.java:125)
> at org.apache.ignite.ci.di.AutoProfilingInterceptor.invoke(AutoProfilingInterceptor.java:76)
> at org.apache.ignite.ci.tcbot.issue.IssueDetector.checkFailuresEx(IssueDetector.java:445)
> at org.apache.ignite.ci.di.MonitoredTaskInterceptor.invoke(MonitoredTaskInterceptor.java:111)
> at org.apache.ignite.ci.di.AutoProfilingInterceptor.invoke(AutoProfilingInterceptor.java:76)
> at org.apache.ignite.ci.tcbot.issue.IssueDetector.checkFailures(IssueDetector.java:422)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: javax.cache.CacheException: class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: Runtime failure on lookup row: SearchRow [key=org.apache.ignite.ci.teamcity.ignited.runhist.RunHistKey [idHash=1028871081, hash=1241170874, srvId=1411517106, testOrSuiteName=11924, branch=281], hash=1241170874, cacheId=0]
> at org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1337)
> at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.cacheException(IgniteCacheProxyImpl.java:1758)
> at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.get(IgniteCacheProxyImpl.java:931)
> at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.get(GatewayProtectedCacheProxy.java:640)
> at org.apache.ignite.ci.teamcity.ignited.runhist.RunHistCompactedDao.getTestRunHist(RunHistCompactedDao.java:102)
> at org.apache.ignite.ci.di.cache.GuavaCachedInterceptor$1.call(GuavaCachedInterceptor.java:64)
> at org.apache.ignite.ci.di.cache.GuavaCachedInterceptor$1.call(GuavaCachedInterceptor.java:60)
> at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4876)
> at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3528)
> at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2277)
> at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154)
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044)
> ... 47 common frames omitted
> Caused by: org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: Runtime failure on lookup row: SearchRow [key=org.apache.ignite.ci.teamcity.ignited.runhist.RunHistKey [idHash=1028871081, hash=1241170874, srvId=1411517106, testOrSuiteName=11924, branch=281], hash=1241170874, cacheId=0]
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findOne(BPlusTree.java:1251)
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findOne(BPlusTree.java:1227)
> at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.find(IgniteCacheOffheapManagerImpl.java:2754)
> at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.find(GridCacheOffheapManager.java:1953)
> at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.read(IgniteCacheOffheapManagerImpl.java:670)
> at org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.localGet(GridPartitionedSingleGetFuture.java:395)
> at org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.mapKeyToNode(GridPartitionedSingleGetFuture.java:353)
> at org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.map(GridPartitionedSingleGetFuture.java:224)
> at org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.init(GridPartitionedSingleGetFuture.java:216)
> at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.getAsync0(GridDhtAtomicCache.java:1428)
> at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$1600(GridDhtAtomicCache.java:135)
> at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$16.apply(GridDhtAtomicCache.java:474)
> at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$16.apply(GridDhtAtomicCache.java:472)
> at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.asyncOp(GridDhtAtomicCache.java:761)
> at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.getAsync(GridDhtAtomicCache.java:472)
> at org.apache.ignite.internal.processors.cache.GridCacheAdapter.get0(GridCacheAdapter.java:4716)
> at org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:4697)
> at org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:1415)
> at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.get(IgniteCacheProxyImpl.java:928)
> ... 56 common frames omitted
> Caused by: java.lang.IllegalStateException: Failed to get page IO instance (page content is corrupted)
> at org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forVersion(IOVersions.java:85)
> at org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forPage(IOVersions.java:97)
> at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:154)
> at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:108)
> at org.apache.ignite.internal.processors.cache.tree.DataRow.<init>(DataRow.java:55)
> at org.apache.ignite.internal.processors.cache.tree.CacheDataRowStore.dataRow(CacheDataRowStore.java:92)
> at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:200)
> at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:49)
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetOne.found(BPlusTree.java:2862)
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:298)
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5615)
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run(BPlusTree.java:271)
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5600)
> at org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:159)
> at org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:334)
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findDown(BPlusTree.java:1309)
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findDown(BPlusTree.java:1318)
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findDown(BPlusTree.java:1318)
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doFind(BPlusTree.java:1276)
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findOne(BPlusTree.java:1243)
> ... 74 common frames omitted
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)