You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Surinder Mehra <re...@gmail.com> on 2022/01/31 10:57:57 UTC
Failed to execute query because cache partition has been lostParts
Hi Guys,
I observed below behavior with persistence enabled. Could you please check
and confirm if it's a bug or I missed some configuration.
Steps:
1. Start ignite node1 and 2 with persistence enabled
2. Step 1 will write few entries to ignite cache
3. All entries go to one node
4. Stop node containing the data
5. Run a query to get data, below exception will be thrown. I guess this is
expected because the node containing the data is down
6. Restart killed node, exception shouldn’t be thrown because data is back
in caches
7. Run a query to get data, below exception will be thrown again. This
looks weird.
8. Kill node2 and try to query again, still the same exception
9. Restart only the left node(data owning node). After restart, the query
starts working again.
10. I checked the cache directory under node1, it didn't have part0.bin
ever. files start from part1 and so on.
11. From the exception, it looks like it is trying to read non existent file
12. Tried setting consistent Id for each node, still same exception.
Another observation : When backup is enabled, shouldn't the backup node
serve the query. Why it throws same below exception when node1(primary) is
down
This exception is only thrown if data containing node goes down
Caused by: class
org.apache.ignite.internal.processors.cache.CacheInvalidStateException:
Failed to execute query because cache partition has been lostParts
[cacheName=deptCache, part=0]
Node2{
DataStorageConfiguration storageCfg = new DataStorageConfiguration();
storageCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(true);
IgniteConfiguration cfg = new IgniteConfiguration();
cfg.setDataStorageConfiguration(storageCfg);
try (Ignite ignite = Ignition.start(cfg)) {
ignite.cluster().state(ClusterState.ACTIVE);
CacheConfiguration<Integer, Department> deptCacheConfig = new
CacheConfiguration<>(DEPT_CACHE);
deptCacheConfig.setCacheMode(CacheMode.PARTITIONED);
IgniteCache<Integer, Department> deptCache =
ignite.getOrCreateCache(deptCacheConfig);
Department d1 = new Department(1, "CS");
Department d2 = new Department(2, "ECE");
Department d3 = new Department(3, "CIVIL");
if(deptCache.size(CachePeekMode.ALL) == 0){
System.out.println("Adding dept data to cache");
deptCache.put(d1.getDeptId(), d1);
deptCache.put(d2.getDeptId(), d2);
deptCache.put(d3.getDeptId(), d3);
}
}
}
Node1{
DataStorageConfiguration storageCfg = new DataStorageConfiguration();
storageCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(true);
IgniteConfiguration cfg = new IgniteConfiguration();
cfg.setDataStorageConfiguration(storageCfg);
try (Ignite ignite = Ignition.start(cfg)) {
ignite.cluster().state(ClusterState.ACTIVE);
CacheConfiguration<Integer, Department> deptCacheConfig = new
CacheConfiguration<>(DEPT_CACHE);
deptCacheConfig.setCacheMode(CacheMode.PARTITIONED);
IgniteCache<Integer, Department> deptCache =
ignite.getOrCreateCache(deptCacheConfig);
Department d1 = new Department(1, "CS");
Department d2 = new Department(2, "ECE");
Department d3 = new Department(3, "CIVIL");
if(deptCache.size(CachePeekMode.ALL) == 0){
System.out.println("Adding dept data to cache");
deptCache.put(d1.getDeptId(), d1);
deptCache.put(d2.getDeptId(), d2);
deptCache.put(d3.getDeptId(), d3);
}
}
}
Client{
IgniteConfiguration cfg = new IgniteConfiguration();
cfg.setDataStorageConfiguration(storageCfg);
try (Ignite ignite = Ignition.start(cfg)) {
IgniteCache<Integer, Department> deptCache =
ignite.getOrCreateCache(DEPT_CACHE);
List<Cache.Entry<Object, Object>> depts = deptCache.query(new
ScanQuery<>()).getAll();
depts.stream().forEach(entry ->
System.out.println(entry.getValue()));
}
}
Re: Failed to execute query because cache partition has been lostParts
Posted by Surinder Mehra <re...@gmail.com>.
New setup :
Node configuration :
<bean class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="peerClassLoadingEnabled" value="true"/>
<property name="deploymentMode" value="CONTINUOUS"/>
<property name="dataStorageConfiguration">
<bean
class="org.apache.ignite.configuration.DataStorageConfiguration">
<property name="defaultDataRegionConfiguration">
<bean
class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="persistenceEnabled" value="true"/>
</bean>
</property>
</bean>
</property>
</bean>
</beans>
Started 3 different nodes with this config :
./ignite.sh ../config/custom-config.xml
Client : It adds data once and queries the cache
cfg.setClientMode(true);
try (Ignite ignite = Ignition.start(cfg)) {
ignite.cluster().state(ClusterState.ACTIVE);
CacheConfiguration<Integer, Department> deptCacheConfig = new
CacheConfiguration<>(DEPT_CACHE);
deptCacheConfig.setCacheMode(CacheMode.PARTITIONED);
deptCacheConfig.setBackups(1);
IgniteCache<Integer, Department> deptCache =
ignite.getOrCreateCache(deptCacheConfig);
Department d1 = new Department(1, "CS");
Department d2 = new Department(2, "ECE");
Department d3 = new Department(3, "CIVIL");
if(deptCache.size(CachePeekMode.ALL) == 0){
System.out.println("Adding dept data to cache");
deptCache.put(d1.getDeptId(), d1);
deptCache.put(d2.getDeptId(), d2);
deptCache.put(d3.getDeptId(), d3);
}
List<Cache.Entry<Object, Object>> depts = deptCache.query(new
ScanQuery<>()).getAll();
depts.stream().forEach(entry -> System.out.println(entry.getValue()));
}
After inserting data, this is how cache data looks like on nodes :
[image: image.png]
Test :
1. first time it prints data
2. Kill node 3, query still works. I believe because node 1 and 2 are
primary nodes for 3 records we inserted.
3. Restart node3, kill node 2. query gets below error.
Caused by: class
org.apache.ignite.internal.processors.cache.CacheInvalidStateException:
Failed to execute query because cache partition has been lostParts
[cacheName=deptCache, part=513]
Base line log :
[17:40:44] Topology snapshot [ver=11, locNode=340bad33, servers=2,
clients=1, state=ACTIVE, CPUs=8, offheap=19.0GB, heap=24.0GB]
[17:40:44] ^-- Baseline [id=0, size=3, online=2, offline=1]
Does it mean for query to run, baseline must have all nodes up ?
On Mon, Jan 31, 2022 at 4:46 PM Surinder Mehra <re...@gmail.com> wrote:
> Hi, thanks for pointing out the problem of why data is being stored on the
> same machine.
> The 2nd point:
> As I mentioned in steps, we are getting the same exception even with
> backups.
> Could you please point out other problems with configuration as you
> mentioned
>
>
> On Mon, Jan 31, 2022 at 4:35 PM Stephen Darlington <
> stephen.darlington@gridgain.com> wrote:
>
>> There’s a lot to unpack here, but there are (at least) two problems with
>> your configuration.
>>
>> First, you should not be activating your cluster in your startup script.
>> That’s an operation that needs to be completed only once, not every time
>> the cluster (or node) starts. This is probably why all the data is being
>> stored on a single machine.
>>
>> Second, you create a cache but without any backups. That means that when
>> a node dies, you lose access to all the data.
>>
>> > On 31 Jan 2022, at 10:57, Surinder Mehra <re...@gmail.com> wrote:
>> >
>> >
>> > Hi Guys,
>> > I observed below behavior with persistence enabled. Could you please
>> check and confirm if it's a bug or I missed some configuration.
>> >
>> > Steps:
>> > 1. Start ignite node1 and 2 with persistence enabled
>> > 2. Step 1 will write few entries to ignite cache
>> > 3. All entries go to one node
>> > 4. Stop node containing the data
>> > 5. Run a query to get data, below exception will be thrown. I guess
>> this is expected because the node containing the data is down
>> > 6. Restart killed node, exception shouldn’t be thrown because data is
>> back in caches
>> > 7. Run a query to get data, below exception will be thrown again. This
>> looks weird.
>> > 8. Kill node2 and try to query again, still the same exception
>> > 9. Restart only the left node(data owning node). After restart, the
>> query starts working again.
>> > 10. I checked the cache directory under node1, it didn't have part0.bin
>> ever. files start from part1 and so on.
>> > 11. From the exception, it looks like it is trying to read non existent
>> file
>> > 12. Tried setting consistent Id for each node, still same exception.
>> >
>> >
>> > Another observation : When backup is enabled, shouldn't the backup node
>> serve the query. Why it throws same below exception when node1(primary) is
>> down
>> >
>> > This exception is only thrown if data containing node goes down
>> >
>> > Caused by: class
>> org.apache.ignite.internal.processors.cache.CacheInvalidStateException:
>> Failed to execute query because cache partition has been lostParts
>> [cacheName=deptCache, part=0]
>> >
>> > Node2{
>> >
>> > DataStorageConfiguration storageCfg = new DataStorageConfiguration();
>> >
>> storageCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(true);
>> > IgniteConfiguration cfg = new IgniteConfiguration();
>> > cfg.setDataStorageConfiguration(storageCfg);
>> >
>> > try (Ignite ignite = Ignition.start(cfg)) {
>> > ignite.cluster().state(ClusterState.ACTIVE);
>> > CacheConfiguration<Integer, Department> deptCacheConfig =
>> new CacheConfiguration<>(DEPT_CACHE);
>> > deptCacheConfig.setCacheMode(CacheMode.PARTITIONED);
>> > IgniteCache<Integer, Department> deptCache =
>> ignite.getOrCreateCache(deptCacheConfig);
>> >
>> > Department d1 = new Department(1, "CS");
>> > Department d2 = new Department(2, "ECE");
>> > Department d3 = new Department(3, "CIVIL");
>> >
>> > if(deptCache.size(CachePeekMode.ALL) == 0){
>> > System.out.println("Adding dept data to cache");
>> > deptCache.put(d1.getDeptId(), d1);
>> > deptCache.put(d2.getDeptId(), d2);
>> > deptCache.put(d3.getDeptId(), d3);
>> > }
>> >
>> > }
>> >
>> > }
>> >
>> > Node1{
>> >
>> > DataStorageConfiguration storageCfg = new DataStorageConfiguration();
>> >
>> storageCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(true);
>> > IgniteConfiguration cfg = new IgniteConfiguration();
>> > cfg.setDataStorageConfiguration(storageCfg);
>> >
>> > try (Ignite ignite = Ignition.start(cfg)) {
>> > ignite.cluster().state(ClusterState.ACTIVE);
>> > CacheConfiguration<Integer, Department> deptCacheConfig =
>> new CacheConfiguration<>(DEPT_CACHE);
>> > deptCacheConfig.setCacheMode(CacheMode.PARTITIONED);
>> > IgniteCache<Integer, Department> deptCache =
>> ignite.getOrCreateCache(deptCacheConfig);
>> >
>> > Department d1 = new Department(1, "CS");
>> > Department d2 = new Department(2, "ECE");
>> > Department d3 = new Department(3, "CIVIL");
>> >
>> > if(deptCache.size(CachePeekMode.ALL) == 0){
>> > System.out.println("Adding dept data to cache");
>> > deptCache.put(d1.getDeptId(), d1);
>> > deptCache.put(d2.getDeptId(), d2);
>> > deptCache.put(d3.getDeptId(), d3);
>> > }
>> >
>> > }
>> > }
>> >
>> > Client{
>> >
>> > IgniteConfiguration cfg = new IgniteConfiguration();
>> > cfg.setDataStorageConfiguration(storageCfg);
>> >
>> > try (Ignite ignite = Ignition.start(cfg)) {
>> > IgniteCache<Integer, Department> deptCache =
>> ignite.getOrCreateCache(DEPT_CACHE);
>> > List<Cache.Entry<Object, Object>> depts = deptCache.query(new
>> ScanQuery<>()).getAll();
>> > depts.stream().forEach(entry ->
>> System.out.println(entry.getValue()));
>> > }
>> > }
>>
>>
Re: Failed to execute query because cache partition has been lostParts
Posted by Surinder Mehra <re...@gmail.com>.
Hi Stephen,
I tried it again to identify issues. This time I commented out the line
which was activating the cluster everytime client started. It looks to work
properly now.
Thanks for your help.
On Mon, Jan 31, 2022 at 8:58 PM Surinder Mehra <re...@gmail.com> wrote:
> Hi,
> Backups are configured, please recheck cacheconfig. Its set to 1, thats
> why node 3 has backups from node 1 and node two as shown in pic attached to
> my previous reply.
>
> On Mon, Jan 31, 2022, 20:46 Stephen Darlington <
> stephen.darlington@gridgain.com> wrote:
>
>> According to the configuration you shared, you don’t have backups.
>>
>> I would expect to see something like:
>>
>> CacheConfiguration<Integer, Department> deptCacheConfig = new
>> CacheConfiguration<>(DEPT_CACHE);
>> deptCacheConfig.setCacheMode(CacheMode.PARTITIONED);
>> *deptCacheConfig.setBackups(1);*
>> IgniteCache<Integer, Department> deptCache =
>> ignite.getOrCreateCache(deptCacheConfig);
>>
>> The default is zero backups, so loss of a single node results in lost
>> partitions.
>>
>> Note, that since you have persistence enabled, you’ll have to drop and
>> recreate the table for the change to take effect.
>>
>> On 31 Jan 2022, at 11:16, Surinder Mehra <re...@gmail.com> wrote:
>>
>> Hi, thanks for pointing out the problem of why data is being stored on
>> the same machine.
>> The 2nd point:
>> As I mentioned in steps, we are getting the same exception even with
>> backups.
>> Could you please point out other problems with configuration as you
>> mentioned
>>
>>
>> On Mon, Jan 31, 2022 at 4:35 PM Stephen Darlington <
>> stephen.darlington@gridgain.com> wrote:
>>
>>> There’s a lot to unpack here, but there are (at least) two problems with
>>> your configuration.
>>>
>>> First, you should not be activating your cluster in your startup script.
>>> That’s an operation that needs to be completed only once, not every time
>>> the cluster (or node) starts. This is probably why all the data is being
>>> stored on a single machine.
>>>
>>> Second, you create a cache but without any backups. That means that when
>>> a node dies, you lose access to all the data.
>>>
>>> > On 31 Jan 2022, at 10:57, Surinder Mehra <re...@gmail.com> wrote:
>>> >
>>> >
>>> > Hi Guys,
>>> > I observed below behavior with persistence enabled. Could you please
>>> check and confirm if it's a bug or I missed some configuration.
>>> >
>>> > Steps:
>>> > 1. Start ignite node1 and 2 with persistence enabled
>>> > 2. Step 1 will write few entries to ignite cache
>>> > 3. All entries go to one node
>>> > 4. Stop node containing the data
>>> > 5. Run a query to get data, below exception will be thrown. I guess
>>> this is expected because the node containing the data is down
>>> > 6. Restart killed node, exception shouldn’t be thrown because data is
>>> back in caches
>>> > 7. Run a query to get data, below exception will be thrown again. This
>>> looks weird.
>>> > 8. Kill node2 and try to query again, still the same exception
>>> > 9. Restart only the left node(data owning node). After restart, the
>>> query starts working again.
>>> > 10. I checked the cache directory under node1, it didn't have
>>> part0.bin ever. files start from part1 and so on.
>>> > 11. From the exception, it looks like it is trying to read non
>>> existent file
>>> > 12. Tried setting consistent Id for each node, still same exception.
>>> >
>>> >
>>> > Another observation : When backup is enabled, shouldn't the backup
>>> node serve the query. Why it throws same below exception when
>>> node1(primary) is down
>>> >
>>> > This exception is only thrown if data containing node goes down
>>> >
>>> > Caused by: class
>>> org.apache.ignite.internal.processors.cache.CacheInvalidStateException:
>>> Failed to execute query because cache partition has been lostParts
>>> [cacheName=deptCache, part=0]
>>> >
>>> > Node2{
>>> >
>>> > DataStorageConfiguration storageCfg = new DataStorageConfiguration();
>>> >
>>> storageCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(true);
>>> > IgniteConfiguration cfg = new IgniteConfiguration();
>>> > cfg.setDataStorageConfiguration(storageCfg);
>>> >
>>> > try (Ignite ignite = Ignition.start(cfg)) {
>>> > ignite.cluster().state(ClusterState.ACTIVE);
>>> > CacheConfiguration<Integer, Department> deptCacheConfig =
>>> new CacheConfiguration<>(DEPT_CACHE);
>>> > deptCacheConfig.setCacheMode(CacheMode.PARTITIONED);
>>> > IgniteCache<Integer, Department> deptCache =
>>> ignite.getOrCreateCache(deptCacheConfig);
>>> >
>>> > Department d1 = new Department(1, "CS");
>>> > Department d2 = new Department(2, "ECE");
>>> > Department d3 = new Department(3, "CIVIL");
>>> >
>>> > if(deptCache.size(CachePeekMode.ALL) == 0){
>>> > System.out.println("Adding dept data to cache");
>>> > deptCache.put(d1.getDeptId(), d1);
>>> > deptCache.put(d2.getDeptId(), d2);
>>> > deptCache.put(d3.getDeptId(), d3);
>>> > }
>>> >
>>> > }
>>> >
>>> > }
>>> >
>>> > Node1{
>>> >
>>> > DataStorageConfiguration storageCfg = new DataStorageConfiguration();
>>> >
>>> storageCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(true);
>>> > IgniteConfiguration cfg = new IgniteConfiguration();
>>> > cfg.setDataStorageConfiguration(storageCfg);
>>> >
>>> > try (Ignite ignite = Ignition.start(cfg)) {
>>> > ignite.cluster().state(ClusterState.ACTIVE);
>>> > CacheConfiguration<Integer, Department> deptCacheConfig =
>>> new CacheConfiguration<>(DEPT_CACHE);
>>> > deptCacheConfig.setCacheMode(CacheMode.PARTITIONED);
>>> > IgniteCache<Integer, Department> deptCache =
>>> ignite.getOrCreateCache(deptCacheConfig);
>>> >
>>> > Department d1 = new Department(1, "CS");
>>> > Department d2 = new Department(2, "ECE");
>>> > Department d3 = new Department(3, "CIVIL");
>>> >
>>> > if(deptCache.size(CachePeekMode.ALL) == 0){
>>> > System.out.println("Adding dept data to cache");
>>> > deptCache.put(d1.getDeptId(), d1);
>>> > deptCache.put(d2.getDeptId(), d2);
>>> > deptCache.put(d3.getDeptId(), d3);
>>> > }
>>> >
>>> > }
>>> > }
>>> >
>>> > Client{
>>> >
>>> > IgniteConfiguration cfg = new IgniteConfiguration();
>>> > cfg.setDataStorageConfiguration(storageCfg);
>>> >
>>> > try (Ignite ignite = Ignition.start(cfg)) {
>>> > IgniteCache<Integer, Department> deptCache =
>>> ignite.getOrCreateCache(DEPT_CACHE);
>>> > List<Cache.Entry<Object, Object>> depts = deptCache.query(new
>>> ScanQuery<>()).getAll();
>>> > depts.stream().forEach(entry ->
>>> System.out.println(entry.getValue()));
>>> > }
>>> > }
>>>
>>>
>>
Re: Failed to execute query because cache partition has been lostParts
Posted by Surinder Mehra <re...@gmail.com>.
Hi,
Backups are configured, please recheck cacheconfig. Its set to 1, thats why
node 3 has backups from node 1 and node two as shown in pic attached to my
previous reply.
On Mon, Jan 31, 2022, 20:46 Stephen Darlington <
stephen.darlington@gridgain.com> wrote:
> According to the configuration you shared, you don’t have backups.
>
> I would expect to see something like:
>
> CacheConfiguration<Integer, Department> deptCacheConfig = new
> CacheConfiguration<>(DEPT_CACHE);
> deptCacheConfig.setCacheMode(CacheMode.PARTITIONED);
> *deptCacheConfig.setBackups(1);*
> IgniteCache<Integer, Department> deptCache =
> ignite.getOrCreateCache(deptCacheConfig);
>
> The default is zero backups, so loss of a single node results in lost
> partitions.
>
> Note, that since you have persistence enabled, you’ll have to drop and
> recreate the table for the change to take effect.
>
> On 31 Jan 2022, at 11:16, Surinder Mehra <re...@gmail.com> wrote:
>
> Hi, thanks for pointing out the problem of why data is being stored on the
> same machine.
> The 2nd point:
> As I mentioned in steps, we are getting the same exception even with
> backups.
> Could you please point out other problems with configuration as you
> mentioned
>
>
> On Mon, Jan 31, 2022 at 4:35 PM Stephen Darlington <
> stephen.darlington@gridgain.com> wrote:
>
>> There’s a lot to unpack here, but there are (at least) two problems with
>> your configuration.
>>
>> First, you should not be activating your cluster in your startup script.
>> That’s an operation that needs to be completed only once, not every time
>> the cluster (or node) starts. This is probably why all the data is being
>> stored on a single machine.
>>
>> Second, you create a cache but without any backups. That means that when
>> a node dies, you lose access to all the data.
>>
>> > On 31 Jan 2022, at 10:57, Surinder Mehra <re...@gmail.com> wrote:
>> >
>> >
>> > Hi Guys,
>> > I observed below behavior with persistence enabled. Could you please
>> check and confirm if it's a bug or I missed some configuration.
>> >
>> > Steps:
>> > 1. Start ignite node1 and 2 with persistence enabled
>> > 2. Step 1 will write few entries to ignite cache
>> > 3. All entries go to one node
>> > 4. Stop node containing the data
>> > 5. Run a query to get data, below exception will be thrown. I guess
>> this is expected because the node containing the data is down
>> > 6. Restart killed node, exception shouldn’t be thrown because data is
>> back in caches
>> > 7. Run a query to get data, below exception will be thrown again. This
>> looks weird.
>> > 8. Kill node2 and try to query again, still the same exception
>> > 9. Restart only the left node(data owning node). After restart, the
>> query starts working again.
>> > 10. I checked the cache directory under node1, it didn't have part0.bin
>> ever. files start from part1 and so on.
>> > 11. From the exception, it looks like it is trying to read non existent
>> file
>> > 12. Tried setting consistent Id for each node, still same exception.
>> >
>> >
>> > Another observation : When backup is enabled, shouldn't the backup node
>> serve the query. Why it throws same below exception when node1(primary) is
>> down
>> >
>> > This exception is only thrown if data containing node goes down
>> >
>> > Caused by: class
>> org.apache.ignite.internal.processors.cache.CacheInvalidStateException:
>> Failed to execute query because cache partition has been lostParts
>> [cacheName=deptCache, part=0]
>> >
>> > Node2{
>> >
>> > DataStorageConfiguration storageCfg = new DataStorageConfiguration();
>> >
>> storageCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(true);
>> > IgniteConfiguration cfg = new IgniteConfiguration();
>> > cfg.setDataStorageConfiguration(storageCfg);
>> >
>> > try (Ignite ignite = Ignition.start(cfg)) {
>> > ignite.cluster().state(ClusterState.ACTIVE);
>> > CacheConfiguration<Integer, Department> deptCacheConfig =
>> new CacheConfiguration<>(DEPT_CACHE);
>> > deptCacheConfig.setCacheMode(CacheMode.PARTITIONED);
>> > IgniteCache<Integer, Department> deptCache =
>> ignite.getOrCreateCache(deptCacheConfig);
>> >
>> > Department d1 = new Department(1, "CS");
>> > Department d2 = new Department(2, "ECE");
>> > Department d3 = new Department(3, "CIVIL");
>> >
>> > if(deptCache.size(CachePeekMode.ALL) == 0){
>> > System.out.println("Adding dept data to cache");
>> > deptCache.put(d1.getDeptId(), d1);
>> > deptCache.put(d2.getDeptId(), d2);
>> > deptCache.put(d3.getDeptId(), d3);
>> > }
>> >
>> > }
>> >
>> > }
>> >
>> > Node1{
>> >
>> > DataStorageConfiguration storageCfg = new DataStorageConfiguration();
>> >
>> storageCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(true);
>> > IgniteConfiguration cfg = new IgniteConfiguration();
>> > cfg.setDataStorageConfiguration(storageCfg);
>> >
>> > try (Ignite ignite = Ignition.start(cfg)) {
>> > ignite.cluster().state(ClusterState.ACTIVE);
>> > CacheConfiguration<Integer, Department> deptCacheConfig =
>> new CacheConfiguration<>(DEPT_CACHE);
>> > deptCacheConfig.setCacheMode(CacheMode.PARTITIONED);
>> > IgniteCache<Integer, Department> deptCache =
>> ignite.getOrCreateCache(deptCacheConfig);
>> >
>> > Department d1 = new Department(1, "CS");
>> > Department d2 = new Department(2, "ECE");
>> > Department d3 = new Department(3, "CIVIL");
>> >
>> > if(deptCache.size(CachePeekMode.ALL) == 0){
>> > System.out.println("Adding dept data to cache");
>> > deptCache.put(d1.getDeptId(), d1);
>> > deptCache.put(d2.getDeptId(), d2);
>> > deptCache.put(d3.getDeptId(), d3);
>> > }
>> >
>> > }
>> > }
>> >
>> > Client{
>> >
>> > IgniteConfiguration cfg = new IgniteConfiguration();
>> > cfg.setDataStorageConfiguration(storageCfg);
>> >
>> > try (Ignite ignite = Ignition.start(cfg)) {
>> > IgniteCache<Integer, Department> deptCache =
>> ignite.getOrCreateCache(DEPT_CACHE);
>> > List<Cache.Entry<Object, Object>> depts = deptCache.query(new
>> ScanQuery<>()).getAll();
>> > depts.stream().forEach(entry ->
>> System.out.println(entry.getValue()));
>> > }
>> > }
>>
>>
>
Re: Failed to execute query because cache partition has been lostParts
Posted by Stephen Darlington <st...@gridgain.com>.
According to the configuration you shared, you don’t have backups.
I would expect to see something like:
CacheConfiguration<Integer, Department> deptCacheConfig = new CacheConfiguration<>(DEPT_CACHE);
deptCacheConfig.setCacheMode(CacheMode.PARTITIONED);
deptCacheConfig.setBackups(1);
IgniteCache<Integer, Department> deptCache = ignite.getOrCreateCache(deptCacheConfig);
The default is zero backups, so loss of a single node results in lost partitions.
Note, that since you have persistence enabled, you’ll have to drop and recreate the table for the change to take effect.
> On 31 Jan 2022, at 11:16, Surinder Mehra <re...@gmail.com> wrote:
>
> Hi, thanks for pointing out the problem of why data is being stored on the same machine.
> The 2nd point:
> As I mentioned in steps, we are getting the same exception even with backups.
> Could you please point out other problems with configuration as you mentioned
>
>
> On Mon, Jan 31, 2022 at 4:35 PM Stephen Darlington <stephen.darlington@gridgain.com <ma...@gridgain.com>> wrote:
> There’s a lot to unpack here, but there are (at least) two problems with your configuration.
>
> First, you should not be activating your cluster in your startup script. That’s an operation that needs to be completed only once, not every time the cluster (or node) starts. This is probably why all the data is being stored on a single machine.
>
> Second, you create a cache but without any backups. That means that when a node dies, you lose access to all the data.
>
> > On 31 Jan 2022, at 10:57, Surinder Mehra <rednirus@gmail.com <ma...@gmail.com>> wrote:
> >
> >
> > Hi Guys,
> > I observed below behavior with persistence enabled. Could you please check and confirm if it's a bug or I missed some configuration.
> >
> > Steps:
> > 1. Start ignite node1 and 2 with persistence enabled
> > 2. Step 1 will write few entries to ignite cache
> > 3. All entries go to one node
> > 4. Stop node containing the data
> > 5. Run a query to get data, below exception will be thrown. I guess this is expected because the node containing the data is down
> > 6. Restart killed node, exception shouldn’t be thrown because data is back in caches
> > 7. Run a query to get data, below exception will be thrown again. This looks weird.
> > 8. Kill node2 and try to query again, still the same exception
> > 9. Restart only the left node(data owning node). After restart, the query starts working again.
> > 10. I checked the cache directory under node1, it didn't have part0.bin ever. files start from part1 and so on.
> > 11. From the exception, it looks like it is trying to read non existent file
> > 12. Tried setting consistent Id for each node, still same exception.
> >
> >
> > Another observation : When backup is enabled, shouldn't the backup node serve the query. Why it throws same below exception when node1(primary) is down
> >
> > This exception is only thrown if data containing node goes down
> >
> > Caused by: class org.apache.ignite.internal.processors.cache.CacheInvalidStateException: Failed to execute query because cache partition has been lostParts [cacheName=deptCache, part=0]
> >
> > Node2{
> >
> > DataStorageConfiguration storageCfg = new DataStorageConfiguration();
> > storageCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(true);
> > IgniteConfiguration cfg = new IgniteConfiguration();
> > cfg.setDataStorageConfiguration(storageCfg);
> >
> > try (Ignite ignite = Ignition.start(cfg)) {
> > ignite.cluster().state(ClusterState.ACTIVE);
> > CacheConfiguration<Integer, Department> deptCacheConfig = new CacheConfiguration<>(DEPT_CACHE);
> > deptCacheConfig.setCacheMode(CacheMode.PARTITIONED);
> > IgniteCache<Integer, Department> deptCache = ignite.getOrCreateCache(deptCacheConfig);
> >
> > Department d1 = new Department(1, "CS");
> > Department d2 = new Department(2, "ECE");
> > Department d3 = new Department(3, "CIVIL");
> >
> > if(deptCache.size(CachePeekMode.ALL) == 0){
> > System.out.println("Adding dept data to cache");
> > deptCache.put(d1.getDeptId(), d1);
> > deptCache.put(d2.getDeptId(), d2);
> > deptCache.put(d3.getDeptId(), d3);
> > }
> >
> > }
> >
> > }
> >
> > Node1{
> >
> > DataStorageConfiguration storageCfg = new DataStorageConfiguration();
> > storageCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(true);
> > IgniteConfiguration cfg = new IgniteConfiguration();
> > cfg.setDataStorageConfiguration(storageCfg);
> >
> > try (Ignite ignite = Ignition.start(cfg)) {
> > ignite.cluster().state(ClusterState.ACTIVE);
> > CacheConfiguration<Integer, Department> deptCacheConfig = new CacheConfiguration<>(DEPT_CACHE);
> > deptCacheConfig.setCacheMode(CacheMode.PARTITIONED);
> > IgniteCache<Integer, Department> deptCache = ignite.getOrCreateCache(deptCacheConfig);
> >
> > Department d1 = new Department(1, "CS");
> > Department d2 = new Department(2, "ECE");
> > Department d3 = new Department(3, "CIVIL");
> >
> > if(deptCache.size(CachePeekMode.ALL) == 0){
> > System.out.println("Adding dept data to cache");
> > deptCache.put(d1.getDeptId(), d1);
> > deptCache.put(d2.getDeptId(), d2);
> > deptCache.put(d3.getDeptId(), d3);
> > }
> >
> > }
> > }
> >
> > Client{
> >
> > IgniteConfiguration cfg = new IgniteConfiguration();
> > cfg.setDataStorageConfiguration(storageCfg);
> >
> > try (Ignite ignite = Ignition.start(cfg)) {
> > IgniteCache<Integer, Department> deptCache = ignite.getOrCreateCache(DEPT_CACHE);
> > List<Cache.Entry<Object, Object>> depts = deptCache.query(new ScanQuery<>()).getAll();
> > depts.stream().forEach(entry -> System.out.println(entry.getValue()));
> > }
> > }
>
Re: Failed to execute query because cache partition has been lostParts
Posted by Surinder Mehra <re...@gmail.com>.
Hi, thanks for pointing out the problem of why data is being stored on the
same machine.
The 2nd point:
As I mentioned in steps, we are getting the same exception even with
backups.
Could you please point out other problems with configuration as you
mentioned
On Mon, Jan 31, 2022 at 4:35 PM Stephen Darlington <
stephen.darlington@gridgain.com> wrote:
> There’s a lot to unpack here, but there are (at least) two problems with
> your configuration.
>
> First, you should not be activating your cluster in your startup script.
> That’s an operation that needs to be completed only once, not every time
> the cluster (or node) starts. This is probably why all the data is being
> stored on a single machine.
>
> Second, you create a cache but without any backups. That means that when a
> node dies, you lose access to all the data.
>
> > On 31 Jan 2022, at 10:57, Surinder Mehra <re...@gmail.com> wrote:
> >
> >
> > Hi Guys,
> > I observed below behavior with persistence enabled. Could you please
> check and confirm if it's a bug or I missed some configuration.
> >
> > Steps:
> > 1. Start ignite node1 and 2 with persistence enabled
> > 2. Step 1 will write few entries to ignite cache
> > 3. All entries go to one node
> > 4. Stop node containing the data
> > 5. Run a query to get data, below exception will be thrown. I guess this
> is expected because the node containing the data is down
> > 6. Restart killed node, exception shouldn’t be thrown because data is
> back in caches
> > 7. Run a query to get data, below exception will be thrown again. This
> looks weird.
> > 8. Kill node2 and try to query again, still the same exception
> > 9. Restart only the left node(data owning node). After restart, the
> query starts working again.
> > 10. I checked the cache directory under node1, it didn't have part0.bin
> ever. files start from part1 and so on.
> > 11. From the exception, it looks like it is trying to read non existent
> file
> > 12. Tried setting consistent Id for each node, still same exception.
> >
> >
> > Another observation : When backup is enabled, shouldn't the backup node
> serve the query. Why it throws same below exception when node1(primary) is
> down
> >
> > This exception is only thrown if data containing node goes down
> >
> > Caused by: class
> org.apache.ignite.internal.processors.cache.CacheInvalidStateException:
> Failed to execute query because cache partition has been lostParts
> [cacheName=deptCache, part=0]
> >
> > Node2{
> >
> > DataStorageConfiguration storageCfg = new DataStorageConfiguration();
> >
> storageCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(true);
> > IgniteConfiguration cfg = new IgniteConfiguration();
> > cfg.setDataStorageConfiguration(storageCfg);
> >
> > try (Ignite ignite = Ignition.start(cfg)) {
> > ignite.cluster().state(ClusterState.ACTIVE);
> > CacheConfiguration<Integer, Department> deptCacheConfig =
> new CacheConfiguration<>(DEPT_CACHE);
> > deptCacheConfig.setCacheMode(CacheMode.PARTITIONED);
> > IgniteCache<Integer, Department> deptCache =
> ignite.getOrCreateCache(deptCacheConfig);
> >
> > Department d1 = new Department(1, "CS");
> > Department d2 = new Department(2, "ECE");
> > Department d3 = new Department(3, "CIVIL");
> >
> > if(deptCache.size(CachePeekMode.ALL) == 0){
> > System.out.println("Adding dept data to cache");
> > deptCache.put(d1.getDeptId(), d1);
> > deptCache.put(d2.getDeptId(), d2);
> > deptCache.put(d3.getDeptId(), d3);
> > }
> >
> > }
> >
> > }
> >
> > Node1{
> >
> > DataStorageConfiguration storageCfg = new DataStorageConfiguration();
> >
> storageCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(true);
> > IgniteConfiguration cfg = new IgniteConfiguration();
> > cfg.setDataStorageConfiguration(storageCfg);
> >
> > try (Ignite ignite = Ignition.start(cfg)) {
> > ignite.cluster().state(ClusterState.ACTIVE);
> > CacheConfiguration<Integer, Department> deptCacheConfig =
> new CacheConfiguration<>(DEPT_CACHE);
> > deptCacheConfig.setCacheMode(CacheMode.PARTITIONED);
> > IgniteCache<Integer, Department> deptCache =
> ignite.getOrCreateCache(deptCacheConfig);
> >
> > Department d1 = new Department(1, "CS");
> > Department d2 = new Department(2, "ECE");
> > Department d3 = new Department(3, "CIVIL");
> >
> > if(deptCache.size(CachePeekMode.ALL) == 0){
> > System.out.println("Adding dept data to cache");
> > deptCache.put(d1.getDeptId(), d1);
> > deptCache.put(d2.getDeptId(), d2);
> > deptCache.put(d3.getDeptId(), d3);
> > }
> >
> > }
> > }
> >
> > Client{
> >
> > IgniteConfiguration cfg = new IgniteConfiguration();
> > cfg.setDataStorageConfiguration(storageCfg);
> >
> > try (Ignite ignite = Ignition.start(cfg)) {
> > IgniteCache<Integer, Department> deptCache =
> ignite.getOrCreateCache(DEPT_CACHE);
> > List<Cache.Entry<Object, Object>> depts = deptCache.query(new
> ScanQuery<>()).getAll();
> > depts.stream().forEach(entry ->
> System.out.println(entry.getValue()));
> > }
> > }
>
>
Re: Failed to execute query because cache partition has been lostParts
Posted by Stephen Darlington <st...@gridgain.com>.
There’s a lot to unpack here, but there are (at least) two problems with your configuration.
First, you should not be activating your cluster in your startup script. That’s an operation that needs to be completed only once, not every time the cluster (or node) starts. This is probably why all the data is being stored on a single machine.
Second, you create a cache but without any backups. That means that when a node dies, you lose access to all the data.
> On 31 Jan 2022, at 10:57, Surinder Mehra <re...@gmail.com> wrote:
>
>
> Hi Guys,
> I observed below behavior with persistence enabled. Could you please check and confirm if it's a bug or I missed some configuration.
>
> Steps:
> 1. Start ignite node1 and 2 with persistence enabled
> 2. Step 1 will write few entries to ignite cache
> 3. All entries go to one node
> 4. Stop node containing the data
> 5. Run a query to get data, below exception will be thrown. I guess this is expected because the node containing the data is down
> 6. Restart killed node, exception shouldn’t be thrown because data is back in caches
> 7. Run a query to get data, below exception will be thrown again. This looks weird.
> 8. Kill node2 and try to query again, still the same exception
> 9. Restart only the left node(data owning node). After restart, the query starts working again.
> 10. I checked the cache directory under node1, it didn't have part0.bin ever. files start from part1 and so on.
> 11. From the exception, it looks like it is trying to read non existent file
> 12. Tried setting consistent Id for each node, still same exception.
>
>
> Another observation : When backup is enabled, shouldn't the backup node serve the query. Why it throws same below exception when node1(primary) is down
>
> This exception is only thrown if data containing node goes down
>
> Caused by: class org.apache.ignite.internal.processors.cache.CacheInvalidStateException: Failed to execute query because cache partition has been lostParts [cacheName=deptCache, part=0]
>
> Node2{
>
> DataStorageConfiguration storageCfg = new DataStorageConfiguration();
> storageCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(true);
> IgniteConfiguration cfg = new IgniteConfiguration();
> cfg.setDataStorageConfiguration(storageCfg);
>
> try (Ignite ignite = Ignition.start(cfg)) {
> ignite.cluster().state(ClusterState.ACTIVE);
> CacheConfiguration<Integer, Department> deptCacheConfig = new CacheConfiguration<>(DEPT_CACHE);
> deptCacheConfig.setCacheMode(CacheMode.PARTITIONED);
> IgniteCache<Integer, Department> deptCache = ignite.getOrCreateCache(deptCacheConfig);
>
> Department d1 = new Department(1, "CS");
> Department d2 = new Department(2, "ECE");
> Department d3 = new Department(3, "CIVIL");
>
> if(deptCache.size(CachePeekMode.ALL) == 0){
> System.out.println("Adding dept data to cache");
> deptCache.put(d1.getDeptId(), d1);
> deptCache.put(d2.getDeptId(), d2);
> deptCache.put(d3.getDeptId(), d3);
> }
>
> }
>
> }
>
> Node1{
>
> DataStorageConfiguration storageCfg = new DataStorageConfiguration();
> storageCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(true);
> IgniteConfiguration cfg = new IgniteConfiguration();
> cfg.setDataStorageConfiguration(storageCfg);
>
> try (Ignite ignite = Ignition.start(cfg)) {
> ignite.cluster().state(ClusterState.ACTIVE);
> CacheConfiguration<Integer, Department> deptCacheConfig = new CacheConfiguration<>(DEPT_CACHE);
> deptCacheConfig.setCacheMode(CacheMode.PARTITIONED);
> IgniteCache<Integer, Department> deptCache = ignite.getOrCreateCache(deptCacheConfig);
>
> Department d1 = new Department(1, "CS");
> Department d2 = new Department(2, "ECE");
> Department d3 = new Department(3, "CIVIL");
>
> if(deptCache.size(CachePeekMode.ALL) == 0){
> System.out.println("Adding dept data to cache");
> deptCache.put(d1.getDeptId(), d1);
> deptCache.put(d2.getDeptId(), d2);
> deptCache.put(d3.getDeptId(), d3);
> }
>
> }
> }
>
> Client{
>
> IgniteConfiguration cfg = new IgniteConfiguration();
> cfg.setDataStorageConfiguration(storageCfg);
>
> try (Ignite ignite = Ignition.start(cfg)) {
> IgniteCache<Integer, Department> deptCache = ignite.getOrCreateCache(DEPT_CACHE);
> List<Cache.Entry<Object, Object>> depts = deptCache.query(new ScanQuery<>()).getAll();
> depts.stream().forEach(entry -> System.out.println(entry.getValue()));
> }
> }