You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by jr...@identy.io on 2024/09/18 09:03:37 UTC
Failed to execute query because cache partition has been lost
Hi.
We are using Apache Ignite in our application, and currently, we are testing
the behaviour of the system when there are system errors.
One of our tests is not working as expected:
* we have got an Ignite cluster with 2 servers and 1 client, and no
backups
* Ignite version 2.16
* We shut down one node server for several minutes
* During this time there is no read nor write to Ignite (we do not use
the DB)
* When we restart the server node, we expect to recover the system
smoothly BUT we have exceptions when we query the data: "Failed to execute
query because cache partition has been lost"
We can resolve the problem resetting the lost partitions, but is this a
normal behaviour of Ignite? I mean, it is a simple case, and the node should
be able to join the cluster without problems.
Thank you.
Re: Failed to execute query because cache partition has been lost
Posted by Pavel Tupitsyn <pt...@apache.org>.
I recommend enabling backups in CacheConfiguration to avoid dealing with
partition loss at all. Is there any reason not to?
On Wed, Sep 18, 2024 at 1:21 PM <jr...@identy.io> wrote:
> After sending the reply, I was thinking… partitionLossPolicy prevents
> that someone tries to modify the DB while a partition is lost (this means
> that a node is not on the cache).
>
> So, if I set partitionLossPolicy = IGNORE the DB can be modified when a
> node is not present.
>
>
>
> Maybe the solution could be, maintain partitionLossPolicy as
> READ_WRITE_SAFE (all reads and writes will be allowed for keys in valid
> partitions), and when the node rejoins to the cluster, reset the lost
> partitions.
>
>
>
> What event do I have to listen? Maybe EVT_NODE_JOINED?
>
>
>
> *From:* jrovira@identy.io <jr...@identy.io>
> *Sent:* miércoles, 18 de septiembre de 2024 11:54
> *To:* user@ignite.apache.org
> *Subject:* RE: Failed to execute query because cache partition has been
> lost
>
>
>
> Ok, thanks, I understand.
>
> But in this case, if someone tries to modify the DB while a node is down,
> does Ignites offers any mechanism to prevent this or should I implement it?
>
>
>
>
>
> *From:* Pavel Tupitsyn <pt...@apache.org>
> *Sent:* miércoles, 18 de septiembre de 2024 11:30
> *To:* user@ignite.apache.org
> *Subject:* Re: Failed to execute query because cache partition has been
> lost
>
>
>
> > 2 servers and 1 client, and no backups
>
> > shut down one node
>
>
>
> There are no backups => any node shutdown leads to partition loss.
>
> If you want to ignore data loss, set partitionLossPolicy = IGNORE [1]
>
>
>
> [1]
> https://ignite.apache.org/docs/latest/configuring-caches/partition-loss-policy
> <https://www.google.com/url?q=https://ignite.apache.org/docs/latest/configuring-caches/partition-loss-policy&source=gmail-imap&ust=1727256685000000&usg=AOvVaw1bBnb3M95kjNcHsDKBEVir>
>
>
>
> On Wed, Sep 18, 2024 at 12:04 PM <jr...@identy.io> wrote:
>
> Hi.
>
>
>
> We are using Apache Ignite in our application, and currently, we are
> testing the behaviour of the system when there are system errors.
>
>
>
> One of our tests is not working as expected:
>
> - we have got an Ignite cluster with 2 servers and 1 client, and no
> backups
> - Ignite version 2.16
> - We shut down one node server for several minutes
> - During this time there is no read nor write to Ignite (we do not use
> the DB)
> - When we restart the server node, we expect to recover the system
> smoothly BUT we have exceptions when we query the data: “Failed to execute
> query because cache partition has been lost”
>
>
>
> We can resolve the problem resetting the lost partitions, but is this a
> normal behaviour of Ignite? I mean, it is a simple case, and the node
> should be able to join the cluster without problems.
>
>
>
> Thank you.
>
>
>
>
RE: Failed to execute query because cache partition has been lost
Posted by jr...@identy.io.
After sending the reply, I was thinking… partitionLossPolicy prevents that someone tries to modify the DB while a partition is lost (this means that a node is not on the cache).
So, if I set partitionLossPolicy = IGNORE the DB can be modified when a node is not present.
Maybe the solution could be, maintain partitionLossPolicy as READ_WRITE_SAFE (all reads and writes will be allowed for keys in valid partitions), and when the node rejoins to the cluster, reset the lost partitions.
What event do I have to listen? Maybe EVT_NODE_JOINED?
From: jrovira@identy.io <jr...@identy.io>
Sent: miércoles, 18 de septiembre de 2024 11:54
To: user@ignite.apache.org
Subject: RE: Failed to execute query because cache partition has been lost
Ok, thanks, I understand.
But in this case, if someone tries to modify the DB while a node is down, does Ignites offers any mechanism to prevent this or should I implement it?
From: Pavel Tupitsyn <ptupitsyn@apache.org <ma...@apache.org> >
Sent: miércoles, 18 de septiembre de 2024 11:30
To: user@ignite.apache.org <ma...@ignite.apache.org>
Subject: Re: Failed to execute query because cache partition has been lost
> 2 servers and 1 client, and no backups
> shut down one node
There are no backups => any node shutdown leads to partition loss.
If you want to ignore data loss, set partitionLossPolicy = IGNORE [1]
[1] https://ignite.apache.org/docs/latest/configuring-caches/partition-loss-policy <https://www.google.com/url?q=https://ignite.apache.org/docs/latest/configuring-caches/partition-loss-policy&source=gmail-imap&ust=1727256685000000&usg=AOvVaw1bBnb3M95kjNcHsDKBEVir>
On Wed, Sep 18, 2024 at 12:04 PM <jrovira@identy.io <ma...@identy.io> > wrote:
Hi.
We are using Apache Ignite in our application, and currently, we are testing the behaviour of the system when there are system errors.
One of our tests is not working as expected:
* we have got an Ignite cluster with 2 servers and 1 client, and no backups
* Ignite version 2.16
* We shut down one node server for several minutes
* During this time there is no read nor write to Ignite (we do not use the DB)
* When we restart the server node, we expect to recover the system smoothly BUT we have exceptions when we query the data: “Failed to execute query because cache partition has been lost”
We can resolve the problem resetting the lost partitions, but is this a normal behaviour of Ignite? I mean, it is a simple case, and the node should be able to join the cluster without problems.
Thank you.
Re: Failed to execute query because cache partition has been lost
Posted by Jeremy McMillan <jt...@gridgain.com>.
If you want to do maintenance, and you want to block access during
maintenance, inactivate the cluster, then do the maintenance, then activate
the cluster.
I recommend that you tell the community what you are trying to do, and then
ask with an open mind how the community would accomplish that goal.
Think carefully, like a database: if you had been tracking changes to data
on a remote node, and that node disappeared and reappeared, is it safe to
assume nothing bad has happened to that node or its data while it was not
available? Is it fair to others for you to assert that data is trustworthy?
Lost partitions must be recovered in the default case. If you want unsafe
behavior, configure the cluster to ignore lost partitions.
On Wed, Sep 18, 2024 at 4:56 AM <jr...@identy.io> wrote:
> Ok, thanks, I understand.
>
> But in this case, if someone tries to modify the DB while a node is down,
> does Ignites offers any mechanism to prevent this or should I implement it?
>
>
>
>
>
> *From:* Pavel Tupitsyn <pt...@apache.org>
> *Sent:* miércoles, 18 de septiembre de 2024 11:30
> *To:* user@ignite.apache.org
> *Subject:* Re: Failed to execute query because cache partition has been
> lost
>
>
>
> > 2 servers and 1 client, and no backups
>
> > shut down one node
>
>
>
> There are no backups => any node shutdown leads to partition loss.
>
> If you want to ignore data loss, set partitionLossPolicy = IGNORE [1]
>
>
>
> [1]
> https://ignite.apache.org/docs/latest/configuring-caches/partition-loss-policy
> <https://www.google.com/url?q=https://ignite.apache.org/docs/latest/configuring-caches/partition-loss-policy&source=gmail-imap&ust=1727256685000000&usg=AOvVaw1bBnb3M95kjNcHsDKBEVir>
>
>
>
> On Wed, Sep 18, 2024 at 12:04 PM <jr...@identy.io> wrote:
>
> Hi.
>
>
>
> We are using Apache Ignite in our application, and currently, we are
> testing the behaviour of the system when there are system errors.
>
>
>
> One of our tests is not working as expected:
>
> - we have got an Ignite cluster with 2 servers and 1 client, and no
> backups
> - Ignite version 2.16
> - We shut down one node server for several minutes
> - During this time there is no read nor write to Ignite (we do not use
> the DB)
> - When we restart the server node, we expect to recover the system
> smoothly BUT we have exceptions when we query the data: “Failed to execute
> query because cache partition has been lost”
>
>
>
> We can resolve the problem resetting the lost partitions, but is this a
> normal behaviour of Ignite? I mean, it is a simple case, and the node
> should be able to join the cluster without problems.
>
>
>
> Thank you.
>
>
>
>
RE: Failed to execute query because cache partition has been lost
Posted by jr...@identy.io.
Ok, thanks, I understand.
But in this case, if someone tries to modify the DB while a node is down, does Ignites offers any mechanism to prevent this or should I implement it?
From: Pavel Tupitsyn <pt...@apache.org>
Sent: miércoles, 18 de septiembre de 2024 11:30
To: user@ignite.apache.org
Subject: Re: Failed to execute query because cache partition has been lost
> 2 servers and 1 client, and no backups
> shut down one node
There are no backups => any node shutdown leads to partition loss.
If you want to ignore data loss, set partitionLossPolicy = IGNORE [1]
[1] https://ignite.apache.org/docs/latest/configuring-caches/partition-loss-policy <https://www.google.com/url?q=https://ignite.apache.org/docs/latest/configuring-caches/partition-loss-policy&source=gmail-imap&ust=1727256685000000&usg=AOvVaw1bBnb3M95kjNcHsDKBEVir>
On Wed, Sep 18, 2024 at 12:04 PM <jrovira@identy.io <ma...@identy.io> > wrote:
Hi.
We are using Apache Ignite in our application, and currently, we are testing the behaviour of the system when there are system errors.
One of our tests is not working as expected:
* we have got an Ignite cluster with 2 servers and 1 client, and no backups
* Ignite version 2.16
* We shut down one node server for several minutes
* During this time there is no read nor write to Ignite (we do not use the DB)
* When we restart the server node, we expect to recover the system smoothly BUT we have exceptions when we query the data: “Failed to execute query because cache partition has been lost”
We can resolve the problem resetting the lost partitions, but is this a normal behaviour of Ignite? I mean, it is a simple case, and the node should be able to join the cluster without problems.
Thank you.
Re: Failed to execute query because cache partition has been lost
Posted by Pavel Tupitsyn <pt...@apache.org>.
> 2 servers and 1 client, and no backups
> shut down one node
There are no backups => any node shutdown leads to partition loss.
If you want to ignore data loss, set partitionLossPolicy = IGNORE [1]
[1]
https://ignite.apache.org/docs/latest/configuring-caches/partition-loss-policy
On Wed, Sep 18, 2024 at 12:04 PM <jr...@identy.io> wrote:
> Hi.
>
>
>
> We are using Apache Ignite in our application, and currently, we are
> testing the behaviour of the system when there are system errors.
>
>
>
> One of our tests is not working as expected:
>
> - we have got an Ignite cluster with 2 servers and 1 client, and no
> backups
> - Ignite version 2.16
> - We shut down one node server for several minutes
> - During this time there is no read nor write to Ignite (we do not use
> the DB)
> - When we restart the server node, we expect to recover the system
> smoothly BUT we have exceptions when we query the data: “Failed to execute
> query because cache partition has been lost”
>
>
>
> We can resolve the problem resetting the lost partitions, but is this a
> normal behaviour of Ignite? I mean, it is a simple case, and the node
> should be able to join the cluster without problems.
>
>
>
> Thank you.
>
>
>