[jira] [Created] (IGNITE-7570) Client nodes failed with "Failed to process invalid partitions response" during failover

["Ksenia Rybakova (JIRA)" <ji...@apache.org>]

Ksenia Rybakova created IGNITE-7570:
---------------------------------------

             Summary: Client nodes failed with "Failed to process invalid partitions response" during failover
                 Key: IGNITE-7570
                 URL: https://issues.apache.org/jira/browse/IGNITE-7570
             Project: Ignite
          Issue Type: Bug
    Affects Versions: 2.4
            Reporter: Ksenia Rybakova


Some client nodes fail with "Failed to process invalid partitions response" during failover test:

{noformat}

[2018-01-30 16:27:58,610][INFO ][sys-#190][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=80ebd2ac-1432-4bfc-bab7-d9dbf56cdeb4, resVer=AffinityTopologyVersion [topVer=37, minorTopVer=0]]
[2018-01-30 16:27:58,688][INFO ][sys-#190][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=37, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=37, minorTopVer=0], err=null]
<16:27:58><benchmark-worker-32><yardstick> The benchmark of random operation failed.
javax.cache.CacheException: class org.apache.ignite.IgniteCheckedException: Failed to process invalid partitions response (remote node reported invalid partitions but remote topology version does not differ from local) [topVer=AffinityTopologyVersion [topVer=37, minorTopVer=0], rmtTopVer=AffinityTopologyVersion [topVer=37, minorTopVer=0], part=204, nodeId=80ebd2ac-1432-4bfc-bab7-d9dbf56cdeb4]
 at org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1294)
 at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.cacheException(IgniteCacheProxyImpl.java:1673)
 at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.get(IgniteCacheProxyImpl.java:852)
 at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.get(GatewayProtectedCacheProxy.java:676)
 at org.apache.ignite.yardstick.cache.load.IgniteCacheRandomOperationBenchmark.doGet(IgniteCacheRandomOperationBenchmark.java:776)
 at org.apache.ignite.yardstick.cache.load.IgniteCacheRandomOperationBenchmark.executeRandomOperation(IgniteCacheRandomOperationBenchmark.java:624)
 at org.apache.ignite.yardstick.cache.load.IgniteCacheRandomOperationBenchmark.executeOutOfTx(IgniteCacheRandomOperationBenchmark.java:602)
 at org.apache.ignite.yardstick.cache.load.IgniteCacheRandomOperationBenchmark.test(IgniteCacheRandomOperationBenchmark.java:207)
 at org.yardstickframework.impl.BenchmarkRunner$2.run(BenchmarkRunner.java:178)
 at java.lang.Thread.run(Thread.java:748)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to process invalid partitions response (remote node reported invalid partitions but remote topology version does not differ from local) [topVer=AffinityTopologyVersion [topVer=37, minorTopVer=0], rmtTopVer=AffinityTopologyVersion [topVer=37, minorTopVer=0], part=204, nodeId=80ebd2ac-1432-4bfc-bab7-d9dbf56cdeb4]
 at org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.checkError(GridPartitionedSingleGetFuture.java:596)
 at org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.onResult(GridPartitionedSingleGetFuture.java:505)
 at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.processNearSingleGetResponse(GridDhtCacheAdapter.java:349)
 at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$1400(GridDhtAtomicCache.java:130)
 at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$15.apply(GridDhtAtomicCache.java:422)
 at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$15.apply(GridDhtAtomicCache.java:417)
 at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1060)
 at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
 at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
 at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
 at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
 at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
 at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
 at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
 at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
 at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
 at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
 ... 1 more
Finishing main test [ts=1517318878863, date=Tue Jan 30 16:27:58 MSK 2018]
ERROR: Shutting down benchmark driver to unexpected exception.

{noformat}

Test config:

CacheRandomOperationBenchmark
- 20 server nodes, 10 client nodes at 10 hosts
- 34 caches with different configs with and without PDS, 3 backups
- preload amount 250

- key range 500K

- operations: PUT PUT_ALL GET GET_ALL INVOKE INVOKE_ALL REMOVE REMOVE_ALL PUT_IF_ABSENT REPLACE

- 2 of 20 servers are being restarted every 15 minutes

Complete yardstick configs are attached.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)