[jira] [Resolved] (IGNITE-6491) Race in TopologyValidator.validate() and EVT_NODE_LEFT listener calls (split-brain activator)

["Alexei Scherbakov (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IGNITE-6491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alexei Scherbakov resolved IGNITE-6491.
---------------------------------------
       Resolution: Fixed
    Fix Version/s:     (was: 2.2)
                   2.3

> Race in TopologyValidator.validate() and EVT_NODE_LEFT listener calls (split-brain activator)
> ---------------------------------------------------------------------------------------------
>
>                 Key: IGNITE-6491
>                 URL: https://issues.apache.org/jira/browse/IGNITE-6491
>             Project: Ignite
>          Issue Type: Bug
>          Components: cache, general
>    Affects Versions: 2.1
>            Reporter: Alexandr Kuramshin
>            Assignee: Alexandr Kuramshin
>             Fix For: 2.3
>
>
> The following wrong cache {{validate}}/{{put}} sequence may occur
> On node left {{GridDhtPartitionsExchangeFuture}} will be generated by the {{disco-event-worker}} thread.
> Then the {{exchange-worker}} thread does
> {noformat}
> Split-brain detected [cacheName=test40, activatorTopVer=0, cacheTopVer=14]
> 	at org.apache.ignite.internal.util.IgniteUtils.dumpStack(IgniteUtils.java:1141)
> 	at org.apache.ignite.internal.processors.cache.IgniteTopologyValidatorGridSplitCacheTest$SplitAwareTopologyValidator.validate(IgniteTopologyValidatorGridSplitCacheTest.java:307)
> 	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTopologyFutureAdapter.validateCacheGroup(GridDhtTopologyFutureAdapter.java:64)
> 	at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onDone(GridDhtPartitionsExchangeFuture.java:1456)
> 	at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onDone(GridDhtPartitionsExchangeFuture.java:115)
> 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:450)
> 	at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:668)
> 	at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2278)
> {noformat}
> The result of validation is stored in {{grpValidRes}} with value of {{false}}.
> After some delay the {{disco-event-worker}} thread will do
> {noformat}
> java.lang.Exception: Node is segment activator [cacheName=test40, activatorTopVer=14]
> 	at org.apache.ignite.internal.util.IgniteUtils.dumpStack(IgniteUtils.java:1141)
> 	at org.apache.ignite.internal.processors.cache.IgniteTopologyValidatorGridSplitCacheTest$SplitAwareTopologyValidator$2.apply(IgniteTopologyValidatorGridSplitCacheTest.java:360)
> 	at org.apache.ignite.internal.processors.cache.IgniteTopologyValidatorGridSplitCacheTest$SplitAwareTopologyValidator$2.apply(IgniteTopologyValidatorGridSplitCacheTest.java:349)
> 	at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager$UserListenerWrapper.onEvent(GridEventStorageManager.java:1463)
> 	at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:859)
> 	at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:844)
> 	at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.record0(GridEventStorageManager.java:341)
> 	at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.record(GridEventStorageManager.java:307)
> 	at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.recordEvent(GridDiscoveryManager.java:2478)
> 	at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:2684)
> 	at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:2507)
> {noformat}
> After this invocation the result of {{SplitAwareTopologyValidator.validate}} should be changed to {{true}}, but it was already invoked and the result has been cached in {{grpValidRes}} with the value of {{false}}.
> So any successive calls to {{cache.put}} causes to fail
> {noformat}
> Test failed.
> java.lang.RuntimeException: tryPut() failed [gridName=cache.IgniteTopologyValidatorGridSplitCacheTest0]
> 	at org.apache.ignite.internal.processors.cache.IgniteTopologyValidatorGridSplitCacheTest.tryPut(IgniteTopologyValidatorGridSplitCacheTest.java:262)
> 	at org.apache.ignite.internal.processors.cache.IgniteTopologyValidatorGridSplitCacheTest.testTopologyValidator(IgniteTopologyValidatorGridSplitCacheTest.java:182)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at junit.framework.TestCase.runTest(TestCase.java:176)
> 	at org.apache.ignite.testframework.junits.GridAbstractTest.runTestInternal(GridAbstractTest.java:2000)
> 	at org.apache.ignite.testframework.junits.GridAbstractTest.access$000(GridAbstractTest.java:132)
> 	at org.apache.ignite.testframework.junits.GridAbstractTest$5.run(GridAbstractTest.java:1915)
> 	at java.lang.Thread.run(Thread.java:748)
> Caused by: javax.cache.CacheException: class org.apache.ignite.IgniteCheckedException: Failed to perform cache operation (cache topology is not valid): test40
> 	at org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1327)
> 	at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.cacheException(IgniteCacheProxyImpl.java:1672)
> 	at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1032)
> 	at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:872)
> 	at org.apache.ignite.internal.processors.cache.IgniteTopologyValidatorGridSplitCacheTest.tryPut(IgniteTopologyValidatorGridSplitCacheTest.java:252)
> 	... 10 more
> Caused by: class org.apache.ignite.IgniteCheckedException: Failed to perform cache operation (cache topology is not valid): test40
> 	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTopologyFutureAdapter.validateCache(GridDhtTopologyFutureAdapter.java:112)
> 	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.mapOnTopology(GridNearAtomicSingleUpdateFuture.java:415)
> 	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.map(GridNearAtomicAbstractUpdateFuture.java:248)
> 	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update0(GridDhtAtomicCache.java:1170)
> 	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:659)
> 	at org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2334)
> 	at org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2311)
> 	at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1029)
> 	... 12 more
> {noformat}
> The updated test {{IgniteTopologyValidatorGridSplitCacheTest}} fails frequently on my laptop with 8 nodes and 100 caches.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)