You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Mạnh Tâm Nguyễn <ng...@gmail.com> on 2018/09/19 03:14:25 UTC
ttl-cleanup-worker got "Critical system error detected"
Hi all,
Some time i got this exception and service crash after that. I think that Ignite got exception during
remove expired records. I have few Caches with expiry but only this Cache got exception.
Anyone know how to sort this issue?
Here are exception details and configuration:
StackTrace:
```
[11:27:17,451][SEVERE][ttl-cleanup-worker-#39%TravelInventory%][] Critical system error detected. Will be handled accordingly to configured handler [hnd=class o.a.i.failure.StopNodeOrHaltFailureHandler, failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class o.a.i.IgniteException: Runtime failure on search row: Row@6a4eb016[ key: 8bbcac6f-3219-48e3-a1cf-cae8be90752e, val: GoQuoEngine.Data.Domain.PackageQuery [idHash=793518907, hash=1245385215 .... ]]]
class org.apache.ignite.IgniteException: Runtime failure on search row: Row@6a4eb016[ key: 8bbcac6f-3219-48e3-a1cf-cae8be90752e, val: GoQuoEngine.Data.Domain.PackageQuery [idHash=793518907, hash=1245385215 .... ]]
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doRemove(BPlusTree.java:1800)
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removex(BPlusTree.java:1595)
at org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.removex(H2TreeIndex.java:289)
at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.remove(GridH2Table.java:522)
at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.remove(IgniteH2Indexing.java:703)
at org.apache.ignite.internal.processors.query.GridQueryProcessor.remove(GridQueryProcessor.java:2518)
at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.remove(GridCacheQueryManager.java:457)
at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishRemove(IgniteCacheOffheapManagerImpl.java:1456)
at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.remove(IgniteCacheOffheapManagerImpl.java:1426)
at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.remove(IgniteCacheOffheapManagerImpl.java:377)
at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.removeValue(GridCacheMapEntry.java:3679)
at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:3409)
at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.onTtlExpired(GridCacheMapEntry.java:3341)
at org.apache.ignite.internal.processors.cache.GridCacheTtlManager$1.applyx(GridCacheTtlManager.java:60)
at org.apache.ignite.internal.processors.cache.GridCacheTtlManager$1.applyx(GridCacheTtlManager.java:51)
at org.apache.ignite.internal.util.lang.IgniteInClosure2X.apply(IgniteInClosure2X.java:38)
at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:1049)
at org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:197)
at org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:137)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:748)
```
Configuration:
```XML
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:util="http://www.springframework.org/schema/util"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/util
http://www.springframework.org/schema/util/spring-util.xsd">
<bean class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="igniteInstanceName" value="Sample"/>
<property name="localHost" value="127.0.0.1"/>
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
<property name="addresses">
<list>
<value>127.0.0.1:47500..47510</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
<property name="communicationSpi">
<bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
<property name="localAddress" value="127.0.0.1"/>
</bean>
</property>
<property name="deploymentMode" value="CONTINUOUS"/>
<property name="loadBalancingSpi">
<list>
<bean class="org.apache.ignite.spi.loadbalancing.weightedrandom.WeightedRandomLoadBalancingSpi">
</bean>
</list>
</property>
<property name="dataStorageConfiguration">
<bean class="org.apache.ignite.configuration.DataStorageConfiguration">
<property name="writeThrottlingEnabled" value="true"/>
<property name="dataRegionConfigurations">
<list>
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="name" value="StaticRegion"/>
<property name="maxSize" value="#{2* 1024L * 1024 * 1024}"/>
<property name="initialSize" value="#{1 *1024L * 1024 * 1024}"/>
<property name="pageEvictionMode" value="RANDOM_LRU"/>
<property name="persistenceEnabled" value="true"/>
</bean>
</list>
</property>
<property name="defaultDataRegionConfiguration">
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="name" value="NonPersistentRegion"/>
</bean>
</property>
<property name="walMode" value="BACKGROUND"/>
<property name="fileIOFactory">
<bean class="org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory">
</bean>
</property>
</bean>
</property>
<property name="cacheConfiguration">
<list>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="PackageQuery"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="atomicityMode" value="ATOMIC"/>
<property name="backups" value="0"/>
<property name="readFromBackup" value="False"/>
<property name="dataRegionName" value="NonPersistentRegion"/>
<property name="partitionLossPolicy" value="IGNORE"/>
<property name="rebalanceMode" value="ASYNC"/>
<property name="rebalanceBatchSize" value="1000"/>
<property name="rebalanceBatchesPrefetchCount" value="1000"/>
<property name="rebalanceOrder" value="0"/>
<property name="sqlSchema" value="public"/>
<property name="expiryPolicyFactory">
<bean class="javax.cache.expiry.CreatedExpiryPolicy" factory-method="factoryOf">
<constructor-arg>
<bean class="javax.cache.expiry.Duration">
<constructor-arg value="MINUTES"/>
<constructor-arg value="30"/>
</bean>
</constructor-arg>
</bean>
</property>
<property name="onheapCacheEnabled" value="true"/>
<property name="evictionPolicy">
<bean class="org.apache.ignite.cache.eviction.lru.LruEvictionPolicy">
<property name="maxSize" value="10000"/>
</bean>
</property>
<property name="queryEntities">
<list>
<bean class="org.apache.ignite.cache.QueryEntity">
<property name="keyType" value="java.util.UUID"/>
<property name="valueType" value="GoQuoEngine.Data.Domain.PackageQuery"/>
<property name="keyFields">
<list>
</list>
</property>
<property name="fields">
<map>
<entry key="CabinClass" value="GoQuoEngine.Flights.CabinClass"/>
<entry key="CheckIn" value="java.sql.Timestamp"/>
<entry key="CheckOut" value="java.sql.Timestamp"/>
<entry key="CreatedAt" value="java.sql.Timestamp"/>
<entry key="CultureCode" value="java.lang.String"/>
<entry key="Currency" value="java.lang.String"/>
<entry key="DepartureDate" value="java.sql.Timestamp"/>
<entry key="DirectFlightsOnly" value="java.lang.Boolean"/>
<entry key="From" value="java.lang.String"/>
<entry key="HashingInfo" value="java.lang.String"/>
<entry key="IsInternationalFlight" value="java.lang.Boolean"/>
<entry key="JourneyType" value="GoQuoEngine.Flights.JourneyType"/>
<entry key="PackageType" value="java.lang.String"/>
<entry key="ProductId" value="java.util.UUID"/>
<entry key="PromoCode" value="java.lang.String"/>
<entry key="ReturnDate" value="java.sql.Timestamp"/>
<entry key="SessionId" value="java.lang.String"/>
<entry key="SiteId" value="java.util.UUID"/>
<entry key="To" value="java.lang.String"/>
<entry key="Id" value="java.util.UUID"/>
</map>
</property>
<property name="notNullFields">
<list>
</list>
</property>
<property name="indexes">
<list>
<bean class="org.apache.ignite.cache.QueryIndex">
<property name="name" value="PACKAGEQUERY_PRODUCTID_IDX"/>
<property name="indexType" value="SORTED"/>
<property name="fields">
<map>
<entry key="ProductId" value="True"/>
</map>
</property>
</bean>
<bean class="org.apache.ignite.cache.QueryIndex">
<property name="name" value="PACKAGEQUERY_FROM_IDX"/>
<property name="indexType" value="SORTED"/>
<property name="fields">
<map>
<entry key="From" value="True"/>
</map>
</property>
</bean>
<bean class="org.apache.ignite.cache.QueryIndex">
<property name="name" value="PACKAGEQUERY_TO_IDX"/>
<property name="indexType" value="SORTED"/>
<property name="fields">
<map>
<entry key="To" value="True"/>
</map>
</property>
</bean>
<bean class="org.apache.ignite.cache.QueryIndex">
<property name="name" value="PACKAGEQUERY_DEPARTUREDATE_IDX"/>
<property name="indexType" value="SORTED"/>
<property name="fields">
<map>
<entry key="DepartureDate" value="True"/>
</map>
</property>
</bean>
<bean class="org.apache.ignite.cache.QueryIndex">
<property name="name" value="PACKAGEQUERY_RETURNDATE_IDX"/>
<property name="indexType" value="SORTED"/>
<property name="fields">
<map>
<entry key="ReturnDate" value="True"/>
</map>
</property>
</bean>
<bean class="org.apache.ignite.cache.QueryIndex">
<property name="name" value="PACKAGEQUERY_CHECKIN_IDX"/>
<property name="indexType" value="SORTED"/>
<property name="fields">
<map>
<entry key="CheckIn" value="True"/>
</map>
</property>
</bean>
<bean class="org.apache.ignite.cache.QueryIndex">
<property name="name" value="PACKAGEQUERY_CHECKOUT_IDX"/>
<property name="indexType" value="SORTED"/>
<property name="fields">
<map>
<entry key="CheckOut" value="True"/>
</map>
</property>
</bean>
<bean class="org.apache.ignite.cache.QueryIndex">
<property name="name" value="PACKAGEQUERY_ISINTERNATIONALFLIGHT_IDX"/>
<property name="indexType" value="SORTED"/>
<property name="fields">
<map>
<entry key="IsInternationalFlight" value="True"/>
</map>
</property>
</bean>
<bean class="org.apache.ignite.cache.QueryIndex">
<property name="name" value="PACKAGEQUERY_CURRENCY_IDX"/>
<property name="indexType" value="SORTED"/>
<property name="fields">
<map>
<entry key="Currency" value="True"/>
</map>
</property>
</bean>
<bean class="org.apache.ignite.cache.QueryIndex">
<property name="name" value="PACKAGEQUERY_JOURNEYTYPE_IDX"/>
<property name="indexType" value="SORTED"/>
<property name="fields">
<map>
<entry key="JourneyType" value="True"/>
</map>
</property>
</bean>
<bean class="org.apache.ignite.cache.QueryIndex">
<property name="name" value="PACKAGEQUERY_CABINCLASS_IDX"/>
<property name="indexType" value="SORTED"/>
<property name="fields">
<map>
<entry key="CabinClass" value="True"/>
</map>
</property>
</bean>
<bean class="org.apache.ignite.cache.QueryIndex">
<property name="name" value="PACKAGEQUERY_DIRECTFLIGHTSONLY_IDX"/>
<property name="indexType" value="SORTED"/>
<property name="fields">
<map>
<entry key="DirectFlightsOnly" value="True"/>
</map>
</property>
</bean>
<bean class="org.apache.ignite.cache.QueryIndex">
<property name="name" value="PACKAGEQUERY_PROMOCODE_IDX"/>
<property name="indexType" value="SORTED"/>
<property name="fields">
<map>
<entry key="PromoCode" value="True"/>
</map>
</property>
</bean>
<bean class="org.apache.ignite.cache.QueryIndex">
<property name="name" value="PACKAGEQUERY_HASHINGINFO_IDX"/>
<property name="indexType" value="SORTED"/>
<property name="fields">
<map>
<entry key="HashingInfo" value="True"/>
</map>
</property>
</bean>
<bean class="org.apache.ignite.cache.QueryIndex">
<property name="name" value="PACKAGEQUERY_SESSIONID_IDX"/>
<property name="indexType" value="SORTED"/>
<property name="fields">
<map>
<entry key="SessionId" value="True"/>
</map>
</property>
</bean>
<bean class="org.apache.ignite.cache.QueryIndex">
<property name="name" value="PACKAGEQUERY_CULTURECODE_IDX"/>
<property name="indexType" value="SORTED"/>
<property name="fields">
<map>
<entry key="CultureCode" value="True"/>
</map>
</property>
</bean>
<bean class="org.apache.ignite.cache.QueryIndex">
<property name="name" value="PACKAGEQUERY_CREATEDAT_IDX"/>
<property name="indexType" value="SORTED"/>
<property name="fields">
<map>
<entry key="CreatedAt" value="False"/>
</map>
</property>
</bean>
<bean class="org.apache.ignite.cache.QueryIndex">
<property name="name" value="PACKAGEQUERY_SITEID_IDX"/>
<property name="indexType" value="SORTED"/>
<property name="fields">
<map>
<entry key="SiteId" value="True"/>
</map>
</property>
</bean>
<bean class="org.apache.ignite.cache.QueryIndex">
<property name="name" value="PACKAGEQUERY_PACKAGETYPE_IDX"/>
<property name="indexType" value="SORTED"/>
<property name="fields">
<map>
<entry key="PackageType" value="True"/>
</map>
</property>
</bean>
</list>
</property>
</bean>
</list>
</property>
</bean>
</list>
</property>
</bean>
</beans>
```
Sent from Mail for Windows 10
Re: ttl-cleanup-worker got "Critical system error detected"
Posted by akurbanov <an...@gmail.com>.
Hello,
Would you mind sharing complete log for your nodes? Looks like SQL indexes
got corrupted or operation was interrupted. Could you also point out exact
version of Ignite that you are using?
Regards,
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/