[jira] [Created] (IGNITE-2465) Assertion in load cache closure

["Denis Magda (JIRA)" <ji...@apache.org>]

Denis Magda created IGNITE-2465:
-----------------------------------

             Summary: Assertion in load cache closure
                 Key: IGNITE-2465
                 URL: https://issues.apache.org/jira/browse/IGNITE-2465
             Project: Ignite
          Issue Type: Bug
          Components: cache
    Affects Versions: 1.5.0.final
            Reporter: Denis Magda
            Assignee: Artem Shutak
            Priority: Blocker
             Fix For: 1.6


This is a tricky one. Every once in a while I get an assertion exception due to a null cache instance. It's difficult to reproduce, but the reason is more or less clear.

First, here's the sequence of events:

1) node N0 starts a cache with GridGain's LocalCacheStore configured (see the cache config below).
2) also, N0 registers a listener for the Ignite DISCO_EVENTS.
3) node N1 joins the cluster.
4) N0 receives a discovery event (EVT_NODE_JOINED) and triggers cache loading using IgniteCache.loadCache(null).
5) N1 throws an AssertionException due to Ignite.cache("persistent-cache") call returning a null.

>From the log snippet below you can see that the exception is first reported and then a millisecond later GridCacheProcessor reports that the cache was started. This means that the cache load closure starts executing on node N1 a bit too early while the cache is still being started. I believe Ignite must be able to handle such race properly.

{noformat}
9319 [pub-#212%N1%] ERROR GridJobWorker - Failed to execute job due to unexpected runtime exception [jobId=0af5bad7251-9f7af4ba-6a64-4de4-b5d2-81d59be05303, ses=GridJobSessionImpl [ses=GridTaskSessionImpl [taskName=o.a.i.i.processors.cache.GridCacheAdapter$LoadCacheClosure, dep=LocalDeployment [super=GridDeployment [ts=1453807336443, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@15db9742, clsLdrId=46f5bad7251-9f7af4ba-6a64-4de4-b5d2-81d59be05303, userVer=0, loc=true, sampleClsName=java.lang.String, pendingUndeploy=false, undeployed=false, usage=0]], taskClsName=o.a.i.i.processors.cache.GridCacheAdapter$LoadCacheClosure, sesId=e9f5bad7251-1edbab1e-37bf-424e-a9e1-0c866b95009d, startTime=1453807336874, endTime=9223372036854775807, taskNodeId=1edbab1e-37bf-424e-a9e1-0c866b95009d, clsLdr=sun.misc.Launcher$AppClassLoader@15db9742, closed=false, cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, subjId=1edbab1e-37bf-424e-a9e1-0c866b95009d, mapFut=IgniteFuture [orig=GridFutureAdapter [resFlag=0, res=null, startTime=1453807336894, endTime=0, ignoreInterrupts=false, lsnr=null, state=INIT]]], jobId=0af5bad7251-9f7af4ba-6a64-4de4-b5d2-81d59be05303]]
java.lang.AssertionError: persistent-cache
at org.apache.ignite.internal.processors.cache.GridCacheAdapter$LoadCacheClosure.call(GridCacheAdapter.java:5788)
at org.apache.ignite.internal.processors.cache.GridCacheAdapter$LoadCacheClosure.call(GridCacheAdapter.java:5740)
at org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2.execute(GridClosureProcessor.java:1789)
at org.apache.ignite.internal.processors.job.GridJobWorker$2.call(GridJobWorker.java:509)
at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6397)
at org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:503)
at org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:456)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1166)
at org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:1770)
at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:821)
at org.apache.ignite.internal.managers.communication.GridIoManager.access$1600(GridIoManager.java:103)
at org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:784)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
9320 [Thread-19] INFO GridCacheProcessor - Started cache [name=persistent-cache, mode=REPLICATED]
{noformat}

For reference, here's the code that configures the cache:

{noformat}
CacheConfiguration<K, V> config = new CacheConfiguration<>("persistent-cache");
config.setCacheMode(CacheMode.REPLICATED);
config.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
config.setRebalanceMode(CacheRebalanceMode.SYNC);
config.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_SYNC);
config.setStartSize(1024);
config.setCacheStoreFactory(new LocalCacheStoreFactory(somepath));
config.setWriteThrough(true);
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)