You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by SamsonLai <Sa...@Outlook.com> on 2019/06/18 08:32:15 UTC
Random CorruptedTreeException from Apache Ignite
Hello
We are using Apache Ignite 2.7.5 and we encountered the following in our
ignite. After this error occurred, it seems that the cache have been
corrupted and when we try to access this cache, the same errors occurred. We
also encountered this error with Apache Ignite 2.7.0.
The errors seems to occurred for random cache as well. Sometimes it will
happen in QA_Account cache but another times it happened for QA_Order cache,
etc. The only common denominator at the moment are the issues occurred for
the caches in the same data region and there is a schedule job at the
backend which operation involved iterate all records in the cache. (The job
run every 1 minutes). We do not have any idea what caused this as the error
seems to occur at random time for random cache and this is the only error
messages found in the ignite log.
Please kindly help. Thanks.
[15:01:53,699][SEVERE][query-#127][GridCacheDistributedQueryManager]
<QA_Account> Failed to run query [qry=GridCacheQueryInfo [loc=false,
trans=null, rdc=null, qry=GridCacheQueryAdapter [type=SCAN, clsName=null,
clause=null, filter=null, transform=null, part=null, incMeta=false,
metrics=null, pageSize=1024, timeout=0, incBackups=false, forceLocal=false,
dedup=false, prj=null, keepBinary=false,
subjId=c52f26d5-48c6-4b48-9566-41accfe995ff, taskHash=0, mvccSnapshot=null],
locFut=null, sndId=c52f26d5-48c6-4b48-9566-41accfe995ff, reqId=97142,
incMeta=false, all=false], node=b79e29cd-80ad-4191-9001-5a1454b5e207]
class org.apache.ignite.IgniteException: Runtime failure on bounds:
[lower=SearchRow [key=null, hash=0, cacheId=-703495123], upper=SearchRow
[key=null, hash=0, cacheId=-703495123]]
at
org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:48)
at
org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:2996)
at
org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:2965)
at
org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
at
org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
at
org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1266)
at
org.apache.ignite.internal.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:234)
at
org.apache.ignite.internal.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)
at
org.apache.ignite.internal.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: class
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on bounds: [lower=SearchRow [key=null, hash=0,
cacheId=-703495123], upper=SearchRow [key=null, hash=0, cacheId=-703495123]]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1013)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:985)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2919)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2890)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2884)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2872)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.cursor(GridCacheOffheapManager.java:2075)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$6.onHasNext(IgniteCacheOffheapManagerImpl.java:968)
at
org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
at
org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
... 21 more
Caused by: java.lang.IllegalArgumentException: Invalid object type: 107
at
org.apache.ignite.internal.processors.cacheobject.IgniteCacheObjectProcessorImpl.toKeyCacheObject(IgniteCacheObjectProcessorImpl.java:166)
at
org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.toKeyCacheObject(CacheObjectBinaryProcessorImpl.java:865)
at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readFullRow(CacheDataRowAdapter.java:333)
at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:167)
at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:108)
at
org.apache.ignite.internal.processors.cache.tree.DataRow.<init>(DataRow.java:55)
at
org.apache.ignite.internal.processors.cache.tree.CacheDataRowStore.dataRow(CacheDataRowStore.java:92)
at
org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:200)
at
org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:49)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer0(BPlusTree.java:5512)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.fillFromBuffer(BPlusTree.java:5280)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.init(BPlusTree.java:5205)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetCursor.notFound(BPlusTree.java:2899)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:306)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5615)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run(BPlusTree.java:271)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5600)
at
org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:159)
at
org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:334)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findDown(BPlusTree.java:1309)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doFind(BPlusTree.java:1276)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$8700(BPlusTree.java:90)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.find(BPlusTree.java:5289)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1005)
... 30 more
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Random CorruptedTreeException from Apache Ignite
Posted by ihalilaltun <ib...@segmentify.com>.
Hi Maxim,
we are facing the exact same problem :(
Is it ok/safe to remove cacheGroupName from the code that is already been
created on nodes? If so, when we start our applications from the updated
code will we be still have access to same caches or new caches will be
created on nodes?
-----
İbrahim Halil Altun
Senior Software Engineer @ Segmentify
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Random CorruptedTreeException from Apache Ignite
Posted by Maxim Pudov <pu...@gmail.com>.
Here is the ticket: https://issues.apache.org/jira/browse/IGNITE-11953
вт, 2 июл. 2019 г. в 11:17, Maxim Pudov <pu...@gmail.com>:
> There is a known issue with cache groups which lead to
> CorruptedTreeException. Try to remove
> cacheCfg.setGroupName(groupName);
> from your configuration. I'll check whether JIRA ticket was created or not
> for this problem.
>
> ср, 26 июн. 2019 г. в 04:17, SamsonLai <Sa...@outlook.com>:
>
>> Yes they are created using JAVA API. The JAVA version is 1.8 and please
>> find
>> below the code we used to create and get the caches. Thanks.
>>
>> private IgniteCache<?, ?> getIgniteCache(String cacheName, String
>> groupName){
>> CacheConfiguration cacheCfg = new CacheConfiguration();
>> cacheCfg.setName(cacheName);
>> cacheCfg.setBackups(0);
>> cacheCfg.setCacheMode(CacheMode.REPLICATED);
>>
>> cacheCfg.setGroupName(groupName);
>> cacheCfg.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
>>
>> return this.ignite.getOrCreateCache(cacheCfg);
>> }
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>
Re: Random CorruptedTreeException from Apache Ignite
Posted by Maxim Pudov <pu...@gmail.com>.
There is a known issue with cache groups which lead to
CorruptedTreeException. Try to remove
cacheCfg.setGroupName(groupName);
from your configuration. I'll check whether JIRA ticket was created or not
for this problem.
ср, 26 июн. 2019 г. в 04:17, SamsonLai <Sa...@outlook.com>:
> Yes they are created using JAVA API. The JAVA version is 1.8 and please
> find
> below the code we used to create and get the caches. Thanks.
>
> private IgniteCache<?, ?> getIgniteCache(String cacheName, String
> groupName){
> CacheConfiguration cacheCfg = new CacheConfiguration();
> cacheCfg.setName(cacheName);
> cacheCfg.setBackups(0);
> cacheCfg.setCacheMode(CacheMode.REPLICATED);
>
> cacheCfg.setGroupName(groupName);
> cacheCfg.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
>
> return this.ignite.getOrCreateCache(cacheCfg);
> }
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Random CorruptedTreeException from Apache Ignite
Posted by SamsonLai <Sa...@Outlook.com>.
Yes they are created using JAVA API. The JAVA version is 1.8 and please find
below the code we used to create and get the caches. Thanks.
private IgniteCache<?, ?> getIgniteCache(String cacheName, String
groupName){
CacheConfiguration cacheCfg = new CacheConfiguration();
cacheCfg.setName(cacheName);
cacheCfg.setBackups(0);
cacheCfg.setCacheMode(CacheMode.REPLICATED);
cacheCfg.setGroupName(groupName);
cacheCfg.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
return this.ignite.getOrCreateCache(cacheCfg);
}
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Random CorruptedTreeException from Apache Ignite
Posted by "Maxim.Pudov" <pu...@gmail.com>.
I don't see cacheConfiguration in the provided snippet. Do you create it
dynamically? What is your usage pattern? How do you use your cache? SQL,
cache API?
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Random CorruptedTreeException from Apache Ignite
Posted by SamsonLai <Sa...@Outlook.com>.
Please find below the config we used. You mentioned that there is a problem
with the index so I guessed the data are still in the cache. Are there any
way to re-build the index to remedy the error when it occurred? Thanks.
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:util="http://www.springframework.org/schema/util"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/util
http://www.springframework.org/schema/util/spring-util.xsd">
<bean id="grid.cfg"
class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="localPort" value="48500"/>
<property name="localPortRange" value="10"/>
<property name="ipFinder">
<bean
class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
<property name="addresses">
<list>
<value>192.168.3.130127.0.0.1
</property>
</bean>
</property>
</bean>
</property>
<property name="dataStorageConfiguration">
<bean class="org.apache.ignite.configuration.DataStorageConfiguration">
<property name="defaultDataRegionConfiguration">
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="initialSize" value="#{1L * 1024 * 1024 * 1024}"/>
<property name="maxSize" value="#{2L * 1024 * 1024 * 1024}"/>
<property name="persistenceEnabled" value="true"/>
</bean>
</property>
<property name="dataRegionConfigurations">
<list>
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="name" value="CC"/>
<property name="initialSize" value="#{512 * 1024 * 1024}"/>
<property name="maxSize" value="#{1L * 1024 * 1024 * 1024}"/>
<property name="persistenceEnabled" value="false"/>
</bean>
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="name" value="NP"/>
<property name="initialSize" value="#{512 * 1024 * 1024}"/>
<property name="maxSize" value="#{1L * 1024 * 1024 * 1024}"/>
<property name="persistenceEnabled" value="false"/>
</bean>
</list>
</property>
</bean>
</property>
<property name="communicationSpi">
<bean
class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
<property name="slowClientQueueLimit" value="100"/>
<property name="messageQueueLimit" value="1024"/>
</bean>
</property>
<property name="authenticationEnabled" value="false"/>
<property name="sslContextFactory">
<bean class="org.apache.ignite.ssl.SslContextFactory">
<property name="keyStoreFilePath" value="C:/temp/ignite.jks"/>
<property name="keyStorePassword" value="password"/>
<property name="protocol" value="TLS"/>
<property name="trustManagers">
<bean class="org.apache.ignite.ssl.SslContextFactory"
factory-method="getDisabledTrustManager"/>
</property>
</bean>
</property>
<property name="peerClassLoadingEnabled" value="true"/>
</bean>
</beans>
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Random CorruptedTreeException from Apache Ignite
Posted by "Maxim.Pudov" <pu...@gmail.com>.
Hi,
From the stacktrace I can tell that there is a problem with an index in your
cache. It's hard to tell more from the stacktrace itself. Could you share
the cache config as well?
In the next release (probably 2.8) such exceptions will be better explained
in the logs (https://issues.apache.org/jira/browse/IGNITE-11750).
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/