You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Weizhou He <we...@gmail.com> on 2018/10/18 06:12:31 UTC
Adding entry to partition that is concurrently evicted
Hi guys,
I was using ignite in cluster mode with SQL. Yesterday, when I was trying
to add more node into existed cluster nodes, an error occurred and the
whole cluster seems done. Error logs shows below. Any idea what should I do
to avoid this error again? Thanks a lot.
*org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtInvalidPartitionException:
Adding entry to partition that is concurrently evicted
[grp=SQL_PUBLIC_MONITORLABEL, part=113, shouldBeMoving=, belongs=false,
topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1],
curTopVer=AffinityTopologyVersion [topVer=4, minorTopVer=1]]*
* at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtPartitionTopologyImpl.localPartition0(GridDhtPartitionTopologyImpl.java:909)
~[ignite-core-2.6.0.jar!/:2.6.0]*
* at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtPartitionTopologyImpl.localPartition(GridDhtPartitionTopologyImpl.java:784)
~[ignite-core-2.6.0.jar!/:2.6.0]*
* at
org.apache.ignite.internal.processors.cache.distributed.dht.GridCachePartitionedConcurrentMap.localPartition(GridCachePartitionedConcurrentMap.java:69)
~[ignite-core-2.6.0.jar!/:2.6.0]*
* at
org.apache.ignite.internal.processors.cache.distributed.dht.GridCachePartitionedConcurrentMap.putEntryIfObsoleteOrAbsent(GridCachePartitionedConcurrentMap.java:88)
~[ignite-core-2.6.0.jar!/:2.6.0]*
* at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.entryEx(GridCacheAdapter.java:955)
~[ignite-core-2.6.0.jar!/:2.6.0]*
* at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.entryEx(GridDhtCacheAdapter.java:525)
~[ignite-core-2.6.0.jar!/:2.6.0]*
* at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.entryEx(GridCacheAdapter.java:946)
~[ignite-core-2.6.0.jar!/:2.6.0]*
* at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:1049)
~[ignite-core-2.6.0.jar!/:2.6.0]*
* at
org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:197)
~[ignite-core-2.6.0.jar!/:2.6.0]*
* at
org.apache.ignite.internal.processors.cache.GridCacheUtils.unwindEvicts(GridCacheUtils.java:839)
~[ignite-core-2.6.0.jar!/:2.6.0]*
* at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessageProcessed(GridCacheIoManager.java:1093)
[ignite-core-2.6.0.jar!/:2.6.0]*
* at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1066)
[ignite-core-2.6.0.jar!/:2.6.0]*
* at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
[ignite-core-2.6.0.jar!/:2.6.0]*
* at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
[ignite-core-2.6.0.jar!/:2.6.0]*
* at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
[ignite-core-2.6.0.jar!/:2.6.0]*
* at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
[ignite-core-2.6.0.jar!/:2.6.0]*
* at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
[ignite-core-2.6.0.jar!/:2.6.0]*
* at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
[ignite-core-2.6.0.jar!/:2.6.0]*
* at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
[ignite-core-2.6.0.jar!/:2.6.0]*
* at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:125)
[ignite-core-2.6.0.jar!/:2.6.0]*
* at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1091)
[ignite-core-2.6.0.jar!/:2.6.0]*
* at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:511)
[ignite-core-2.6.0.jar!/:2.6.0]*
* at java.lang.Thread.run(Thread.java:745) [na:1.8.0_71]*
Weizhou He
Re: Adding entry to partition that is concurrently evicted
Posted by aealexsandrov <ae...@gmail.com>.
Hi,
This exception shows that one of your partitions was moved to the invalid
state. It's not expected case and for investigation of the reason, we
require log file too. Is it possible to provide it?
BR,
Andrei
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Adding entry to partition that is concurrently evicted
Posted by Weizhou He <we...@gmail.com>.
Hi,
All error log was pasted.
I used spring to launch Ignite, so all config is defined in java. As below,
@Bean("igniteConfiguration")
IgniteConfiguration igniteConfiguration(@Value("${zookeeper.host}")
String zkHost) {
IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
igniteConfiguration.setClientMode(false);
ClientConnectorConfiguration clientConnectorConfiguration = new
ClientConnectorConfiguration();
clientConnectorConfiguration.setPort(10801);
igniteConfiguration.setClientConnectorConfiguration(clientConnectorConfiguration);
// common ignite configuration
igniteConfiguration.setMetricsLogFrequency(0);
igniteConfiguration.setQueryThreadPoolSize(2);
igniteConfiguration.setDataStreamerThreadPoolSize(1);
igniteConfiguration.setManagementThreadPoolSize(2);
igniteConfiguration.setPublicThreadPoolSize(2);
igniteConfiguration.setSystemThreadPoolSize(2);
igniteConfiguration.setRebalanceThreadPoolSize(1);
igniteConfiguration.setAsyncCallbackPoolSize(2);
igniteConfiguration.setPeerClassLoadingEnabled(false);
// memory configuration
DataStorageConfiguration storageCfg = new DataStorageConfiguration();
storageCfg.getDefaultDataRegionConfiguration().setMaxSize(4L *
1024 * 1024 * 1024);
storageCfg.getDefaultDataRegionConfiguration().setPageEvictionMode(DataPageEvictionMode.RANDOM_LRU);
igniteConfiguration.setDataStorageConfiguration(storageCfg);
BinaryConfiguration binaryConfiguration = new BinaryConfiguration();
binaryConfiguration.setCompactFooter(false);
igniteConfiguration.setBinaryConfiguration(binaryConfiguration);
ZookeeperDiscoverySpi zkDiscoSpi = new ZookeeperDiscoverySpi();
zkDiscoSpi.setZkConnectionString(zkHost);
zkDiscoSpi.setSessionTimeout(30_000);
if(debug){
zkDiscoSpi.setZkRootPath("/ignite-debug");
}else{
zkDiscoSpi.setZkRootPath("/ignite");
}
zkDiscoSpi.setJoinTimeout(10_000);
igniteConfiguration.setDiscoverySpi(zkDiscoSpi);
TcpCommunicationSpi tcpCommunicationSpi = new TcpCommunicationSpi();
tcpCommunicationSpi.setLocalPort(48500);
igniteConfiguration.setCommunicationSpi(tcpCommunicationSpi);
return igniteConfiguration;
}
@Bean(destroyMethod = "close")
Ignite labelIgnite(@Qualifier("igniteConfiguration")
IgniteConfiguration igniteConfiguration) {
CacheConfiguration fiveMinutesCacheConfiguration = new CacheConfiguration();
fiveMinutesCacheConfiguration.setName("FiveMinutes");
fiveMinutesCacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
fiveMinutesCacheConfiguration.setBackups(0);
fiveMinutesCacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC);
fiveMinutesCacheConfiguration.setExpiryPolicyFactory(CreatedExpiryPolicy.factoryOf(Duration.FIVE_MINUTES));
CacheConfiguration oneMinuteCacheConfiguration = new CacheConfiguration();
oneMinuteCacheConfiguration.setName("OneMinute");
oneMinuteCacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
oneMinuteCacheConfiguration.setBackups(0);
oneMinuteCacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC);
oneMinuteCacheConfiguration.setExpiryPolicyFactory(CreatedExpiryPolicy.factoryOf(Duration.ONE_MINUTE));
CacheConfiguration oneDayConfiguration = new CacheConfiguration();
oneDayConfiguration.setName("OneDay");
oneDayConfiguration.setCacheMode(CacheMode.PARTITIONED);
oneDayConfiguration.setBackups(0);
oneDayConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC);
oneDayConfiguration.setExpiryPolicyFactory(CreatedExpiryPolicy.factoryOf(Duration.ONE_DAY));
final Ignite start = Ignition.start(igniteConfiguration);
start.addCacheConfiguration(fiveMinutesCacheConfiguration);
start.addCacheConfiguration(oneMinuteCacheConfiguration);
start.addCacheConfiguration(oneDayConfiguration);
return start;
}
And I will try to create table each time I start spring process.
try {
jdbcTemplate.execute("CREATE TABLE IF NOT EXISTS Monitor" +
getType() + " (\n" +
"hostip VARCHAR(32),\n" +
"tid BIGINT,\n" +
"type VARCHAR(32),\n" +
"request VARCHAR(32),\n" +
"parametermap VARCHAR(MAX),\n" +
"PRIMARY KEY(hostip, tid, type, request)\n" +
") WITH \"template=FiveMinutes\"");
} catch (Exception e) {
e.printStackTrace();
}
aealexsandrov <ae...@gmail.com> 于2018年10月18日周四 下午5:15写道：
> Hi,
>
> Could you please attach the configuration of the cluster and full log?
>
> BR,
> Andrei
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Adding entry to partition that is concurrently evicted
Posted by aealexsandrov <ae...@gmail.com>.
Hi,
Could you please attach the configuration of the cluster and full log?
BR,
Andrei
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/