Re: Failed to execute query because cache partition has been lostParts

[Surinder Mehra <re...@gmail.com>]

Hi Stephen,
I tried it again to identify issues. This time I commented out the line
which was activating the cluster everytime client started. It looks to work
properly now.

Thanks for your help.

On Mon, Jan 31, 2022 at 8:58 PM Surinder Mehra <re...@gmail.com> wrote:

> Hi,
> Backups are configured, please recheck cacheconfig. Its set to 1, thats
> why node 3 has backups from node 1 and node two as shown in pic attached to
> my previous reply.
>
> On Mon, Jan 31, 2022, 20:46 Stephen Darlington <
> stephen.darlington@gridgain.com> wrote:
>
>> According to the configuration you shared, you don’t have backups.
>>
>> I would expect to see something like:
>>
>> CacheConfiguration<Integer, Department> deptCacheConfig = new
>> CacheConfiguration<>(DEPT_CACHE);
>> deptCacheConfig.setCacheMode(CacheMode.PARTITIONED);
>> *deptCacheConfig.setBackups(1);*
>> IgniteCache<Integer, Department> deptCache =
>> ignite.getOrCreateCache(deptCacheConfig);
>>
>> The default is zero backups, so loss of a single node results in lost
>> partitions.
>>
>> Note, that since you have persistence enabled, you’ll have to drop and
>> recreate the table for the change to take effect.
>>
>> On 31 Jan 2022, at 11:16, Surinder Mehra <re...@gmail.com> wrote:
>>
>> Hi, thanks for pointing out the problem of why data is being stored on
>> the same machine.
>> The 2nd point:
>> As I mentioned in steps, we are getting the same exception even with
>> backups.
>> Could you please point out other problems with configuration as you
>> mentioned
>>
>>
>> On Mon, Jan 31, 2022 at 4:35 PM Stephen Darlington <
>> stephen.darlington@gridgain.com> wrote:
>>
>>> There’s a lot to unpack here, but there are (at least) two problems with
>>> your configuration.
>>>
>>> First, you should not be activating your cluster in your startup script.
>>> That’s an operation that needs to be completed only once, not every time
>>> the cluster (or node) starts. This is probably why all the data is being
>>> stored on a single machine.
>>>
>>> Second, you create a cache but without any backups. That means that when
>>> a node dies, you lose access to all the data.
>>>
>>> > On 31 Jan 2022, at 10:57, Surinder Mehra <re...@gmail.com> wrote:
>>> >
>>> >
>>> > Hi Guys,
>>> > I observed below behavior with persistence enabled. Could you please
>>> check and confirm if it's a bug or I missed some configuration.
>>> >
>>> > Steps:
>>> > 1. Start ignite node1 and 2 with persistence enabled
>>> > 2. Step 1 will write few entries to ignite cache
>>> > 3. All entries go to one node
>>> > 4. Stop node containing the data
>>> > 5. Run a query to get data, below exception will be thrown. I guess
>>> this is expected because the node containing the data is down
>>> > 6. Restart killed node, exception shouldn’t be thrown because data is
>>> back in caches
>>> > 7. Run a query to get data, below exception will be thrown again. This
>>> looks weird.
>>> > 8. Kill node2 and try to query again, still the same exception
>>> > 9. Restart only the left node(data owning node). After restart, the
>>> query starts working again.
>>> > 10. I checked the cache directory under node1, it didn't have
>>> part0.bin ever. files start from part1 and so on.
>>> > 11. From the exception, it looks like it is trying to read non
>>> existent file
>>> > 12. Tried setting consistent Id for each node, still same exception.
>>> >
>>> >
>>> > Another observation : When backup is enabled, shouldn't the backup
>>> node serve the query. Why it throws same below exception when
>>> node1(primary) is down
>>> >
>>> > This exception is only thrown if data containing node goes down
>>> >
>>> > Caused by: class
>>> org.apache.ignite.internal.processors.cache.CacheInvalidStateException:
>>> Failed to execute query because cache partition has been lostParts
>>> [cacheName=deptCache, part=0]
>>> >
>>> > Node2{
>>> >
>>> > DataStorageConfiguration storageCfg = new DataStorageConfiguration();
>>> >
>>> storageCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(true);
>>> > IgniteConfiguration cfg = new IgniteConfiguration();
>>> > cfg.setDataStorageConfiguration(storageCfg);
>>> >
>>> > try (Ignite ignite = Ignition.start(cfg)) {
>>> >             ignite.cluster().state(ClusterState.ACTIVE);
>>> >             CacheConfiguration<Integer, Department> deptCacheConfig =
>>> new CacheConfiguration<>(DEPT_CACHE);
>>> >             deptCacheConfig.setCacheMode(CacheMode.PARTITIONED);
>>> > IgniteCache<Integer, Department> deptCache =
>>> ignite.getOrCreateCache(deptCacheConfig);
>>> >
>>> > Department d1 = new Department(1, "CS");
>>> > Department d2 = new Department(2, "ECE");
>>> > Department d3 = new Department(3, "CIVIL");
>>> >
>>> > if(deptCache.size(CachePeekMode.ALL) == 0){
>>> > System.out.println("Adding dept data to cache");
>>> > deptCache.put(d1.getDeptId(), d1);
>>> > deptCache.put(d2.getDeptId(), d2);
>>> > deptCache.put(d3.getDeptId(), d3);
>>> > }
>>> >
>>> > }
>>> >
>>> > }
>>> >
>>> > Node1{
>>> >
>>> > DataStorageConfiguration storageCfg = new DataStorageConfiguration();
>>> >
>>> storageCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(true);
>>> > IgniteConfiguration cfg = new IgniteConfiguration();
>>> > cfg.setDataStorageConfiguration(storageCfg);
>>> >
>>> > try (Ignite ignite = Ignition.start(cfg)) {
>>> >             ignite.cluster().state(ClusterState.ACTIVE);
>>> >             CacheConfiguration<Integer, Department> deptCacheConfig =
>>> new CacheConfiguration<>(DEPT_CACHE);
>>> >             deptCacheConfig.setCacheMode(CacheMode.PARTITIONED);
>>> > IgniteCache<Integer, Department> deptCache =
>>> ignite.getOrCreateCache(deptCacheConfig);
>>> >
>>> > Department d1 = new Department(1, "CS");
>>> > Department d2 = new Department(2, "ECE");
>>> > Department d3 = new Department(3, "CIVIL");
>>> >
>>> > if(deptCache.size(CachePeekMode.ALL) == 0){
>>> > System.out.println("Adding dept data to cache");
>>> > deptCache.put(d1.getDeptId(), d1);
>>> > deptCache.put(d2.getDeptId(), d2);
>>> > deptCache.put(d3.getDeptId(), d3);
>>> > }
>>> >
>>> > }
>>> > }
>>> >
>>> > Client{
>>> >
>>> > IgniteConfiguration cfg = new IgniteConfiguration();
>>> > cfg.setDataStorageConfiguration(storageCfg);
>>> >
>>> > try (Ignite ignite = Ignition.start(cfg)) {
>>> >         IgniteCache<Integer, Department> deptCache =
>>> ignite.getOrCreateCache(DEPT_CACHE);
>>> >         List<Cache.Entry<Object, Object>> depts = deptCache.query(new
>>> ScanQuery<>()).getAll();
>>> >         depts.stream().forEach(entry ->
>>> System.out.println(entry.getValue()));
>>> >     }
>>> > }
>>>
>>>
>>