You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by "Ksenia Rybakova (JIRA)" <ji...@apache.org> on 2016/06/30 09:00:21 UTC
[jira] [Created] (IGNITE-3401) "Remote node ID is not as expected"
errors at driver during load test
Ksenia Rybakova created IGNITE-3401:
---------------------------------------
Summary: "Remote node ID is not as expected" errors at driver during load test
Key: IGNITE-3401
URL: https://issues.apache.org/jira/browse/IGNITE-3401
Project: Ignite
Issue Type: Bug
Components: general
Reporter: Ksenia Rybakova
Fix For: 1.7
A lot of errors at some drivers after several minutes after grid is up during load test:
{noformat}
[14:33:42] (err) Failed to execute compound future reducer: Compound future listener []class org.apache.ignite.IgniteCheckedException: Failed to send message (node may have left the grid or TCP connection cannot be established due to fir
at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1129)
at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1193)
at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$Buffer.submit(DataStreamerImpl.java:1378)
at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$Buffer.update(DataStreamerImpl.java:1173)
at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl.load0(DataStreamerImpl.java:756)
at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl.addDataInternal(DataStreamerImpl.java:568)
at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl.addData(DataStreamerImpl.java:604)
at org.apache.ignite.yardstick.cache.load.IgniteCacheRandomOperationBenchmark$1.run(IgniteCacheRandomOperationBenchmark.java:383)
Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to send message to remote node: TcpDiscoveryNode [id=11da20f5-87c5-4321-8163-ffef85f2d54c, addrs=[10.20.0.215, 127.0.0.1], sockAddrs=[/127.0.0.1:47503, fosters-215/10.20.0
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:1986)
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:1926)
at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1124)
... 7 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node still alive?). Make sure that each GridComputeTask and GridCacheTransaction has a timeout set in order to prevent parties from waiting forever
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2489)
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2130)
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2024)
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:1960)
... 9 more
Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to connect to address: fosters-215/10.20.0.215:47103
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2494)
... 12 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to read remote node recovery handshake (connection closed).
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeHandshake(TcpCommunicationSpi.java:2699)
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2361)
... 12 more
Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to connect to address: /127.0.0.1:47103
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2494)
... 12 more
Caused by: class org.apache.ignite.IgniteCheckedException: Remote node ID is not as expected [expected=11da20f5-87c5-4321-8163-ffef85f2d54c, rcvd=03bc7d54-4fec-481d-af24-6f5408b32fe2]
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeHandshake(TcpCommunicationSpi.java:2604)
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2361)
... 12 more
{noformat}
Tested version: 1.5.27
Load config:
- 7 hosts with 4 servers and 3 drivers each (28 servers and 21 drivers in total)
- benchmark: IgniteCacheRandomOperationBenchmark
- preload: 2M
- key range: 4M
- warmup: 30s
- duration: 10h
- operations: PUT PUT_ALL GET GET_ALL INVOKE INVOKE_ALL REMOVE REMOVE_ALL PUT_IF_ABSENT REPLACE
-backup count: 1
- GC: G1
- heap size: 8Gb for servers, 4Gb for drivers
- caches:
{noformat}
<bean name="atomic" class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="atomic"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="atomicityMode" value="ATOMIC"/>
<property name="swapEnabled" value="false"/>
</bean>
<bean name="atomic-evict" class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="atomic-evict"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="atomicityMode" value="ATOMIC"/>
<property name="swapEnabled" value="false"/>
<property name="offHeapMaxMemory" value="#{1024L*1024L*1024L*6L}"/>
<property name="evictionPolicy">
<bean class="org.apache.ignite.cache.eviction.fifo.FifoEvictionPolicy">
<property name="maxSize" value="750000"/>
</bean>
</property>
</bean>
<bean name="atomic-offheap" class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="atomic-offheap"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="atomicityMode" value="ATOMIC"/>
<property name="swapEnabled" value="false"/>
<property name="memoryMode" value="OFFHEAP_TIERED"/>
</bean>
<bean name="atomic-offheap-values" class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="atomic-offheap-values"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="atomicityMode" value="ATOMIC"/>
<property name="swapEnabled" value="false"/>
<property name="memoryMode" value="OFFHEAP_VALUES"/>
</bean>
<bean name="tx" class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="tx"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="atomicityMode" value="TRANSACTIONAL"/>
<property name="swapEnabled" value="false"/>
</bean>
<bean name="tx-evict" class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="tx-evict"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="atomicityMode" value="TRANSACTIONAL"/>
<property name="swapEnabled" value="false"/>
<property name="offHeapMaxMemory" value="#{1024L*1024L*1024L*6L}"/>
<property name="evictionPolicy">
<bean class="org.apache.ignite.cache.eviction.lru.LruEvictionPolicy">
<property name="maxSize" value="750000"/>
</bean>
</property>
</bean>
<bean name="compute" class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="compute"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="atomicityMode" value="TRANSACTIONAL"/>
<property name="swapEnabled" value="false"/>
<property name="offHeapMaxMemory" value="#{1024L*1024L*1024L*6L}"/>
<property name="evictionPolicy">
<bean class="org.apache.ignite.cache.eviction.fifo.FifoEvictionPolicy">
<property name="maxSize" value="750000"/>
</bean>
</property>
</bean>
<bean name="tx-offheap" class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="tx-offheap"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="atomicityMode" value="TRANSACTIONAL"/>
<property name="swapEnabled" value="false"/>
<property name="memoryMode" value="OFFHEAP_TIERED"/>
</bean>
<bean name="tx-offheap-values" class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="tx-offheap-values"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="atomicityMode" value="TRANSACTIONAL"/>
<property name="swapEnabled" value="false"/>
<property name="memoryMode" value="OFFHEAP_VALUES"/>
</bean>
<bean name="atomic-index" class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="atomic-index"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="atomicityMode" value="ATOMIC"/>
<property name="swapEnabled" value="false"/>
<property name="indexedTypes">
<list>
<value>java.lang.Integer</value>
<value>org.apache.ignite.yardstick.cache.model.Person1</value>
</list>
</property>
</bean>
<bean name="query" class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="query"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="atomicityMode" value="ATOMIC"/>
<property name="swapEnabled" value="false"/>
<!-- Configure query entities -->
<property name="queryEntities">
<list>
<bean class="org.apache.ignite.cache.QueryEntity">
<property name="keyType" value="java.lang.Integer"/>
<property name="valueType" value="org.apache.ignite.yardstick.cache.model.Person"/>
<property name="fields">
<map>
<entry key="id" value="java.lang.Integer"/>
<entry key="orgId" value="java.lang.Integer"/>
<entry key="firstName" value="java.lang.String"/>
<entry key="lastName" value="java.lang.String"/>
<entry key="salary" value="java.lang.Double"/>
</map>
</property>
<property name="indexes">
<list>
<bean class="org.apache.ignite.cache.QueryIndex">
<constructor-arg value="id"/>
</bean>
<!-- Group index. -->
<bean class="org.apache.ignite.cache.QueryIndex">
<constructor-arg>
<list>
<value>id</value>
<value>salary</value>
</list>
</constructor-arg>
<constructor-arg value="SORTED"/>
</bean>
</list>
</property>
</bean>
</list>
</property>
</bean>
<bean name="orgCache" class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="orgCache"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="atomicityMode" value="ATOMIC"/>
<property name="swapEnabled" value="false"/>
<!-- Configure query entities -->
<property name="queryEntities">
<list>
<bean class="org.apache.ignite.cache.QueryEntity">
<property name="keyType" value="java.lang.Integer"/>
<property name="valueType" value="org.apache.ignite.yardstick.cache.model.Organization"/>
<property name="fields">
<map>
<entry key="id" value="java.lang.Integer"/>
<entry key="name" value="java.lang.String"/>
</map>
</property>
<property name="indexes">
<list>
<bean class="org.apache.ignite.cache.QueryIndex">
<constructor-arg value="id"/>
</bean>
<bean class="org.apache.ignite.cache.QueryIndex">
<constructor-arg value="name"/>
</bean>
</list>
</property>
</bean>
</list>
</property>
</bean>
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)