You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Surinder Mehra <re...@gmail.com> on 2022/10/31 13:00:53 UTC
Backup filter in ignite [Multi AZ deployment]
Hi,
As per link attached, to ensure primary and backup partitions are not
stored on same node, We used AWS AZ as backup filter and now I can see if I
start two ignite nodes on the same machine, primary partitions are evenly
distributed but backups are always zero which is expected.
https://www.gridgain.com/docs/latest/installation-guide/aws/multiple-availability-zone-aws
My question is what would happen if AZ-1 has 2 machines and AZ-2 has 1
machine and ignite cluster has only 3 nodes, each machine having one ignite
node.
Node1[AZ1] - keys 1-100
Node2[AZ1] - keys 101-200
Node3[AZ2] - keys 201 -300
In the above scenario, if the backup count is 2, how would back up
partitions be distributed.
1. Would it mean node3 will have 2 backup copies of primary partitions of
node 1 and 2 ?
2. If we have a 4 node cluster with 2 nodes in each AZ, would backup copies
also be placed on different nodes(In other words, does the backup filter
also apply to how backup copies are placed on nodes) ?
Re: Backup filter in ignite [Multi AZ deployment]
Posted by Surinder Mehra <re...@gmail.com>.
Understood thanks !
On Sun, 6 Nov 2022, 21:33 Jeremy McMillan, <je...@gridgain.com>
wrote:
> Think of each AZ as being a massive piece of server hardware running VMs
> or workloads for you. When hardware (or infrastructure maintenance process)
> fails, assume everything on one AZ is lost at the same time.
>
> On Sun, Nov 6, 2022, 09:58 Surinder Mehra <re...@gmail.com> wrote:
>
>> That's partially true. Whole excercise of configuring AZ as backup filter
>> is because we want to handle AZ level failure.
>>
>> Anyway, thanks for inputs. Will figure out further steps
>>
>> On Sun, 6 Nov 2022, 20:55 Jeremy McMillan, <je...@gridgain.com>
>> wrote:
>>
>>> Don't configure 2 backups when you only have two failure domains.
>>>
>>> You're worried about node level failure, but you're telling Ignite to
>>> worry about AZ level failure.
>>>
>>>
>>> On Sat, Nov 5, 2022, 21:57 Surinder Mehra <re...@gmail.com> wrote:
>>>
>>>> Yeah I think there is a misunderstanding. Although I figured out my
>>>> answers from our discussion, I will try one final attempt to clarify my
>>>> point on 2X space for node3
>>>>
>>>> Node setup:
>>>> Node1 and node 2 placed in AZ1
>>>> Node 3 placed in AZ2
>>>>
>>>> Since I am using AZ as backup filter as I mentioned in my first
>>>> message. Back up if node 1 cannot be placed on node2 and back up of node 2
>>>> cannot be placed on node1 as they are in same AZ. This simply means their
>>>> backups would go to node3 which in another AZ. Hence node 3 space =(node3
>>>> primary partitions+node 1 back up partitions+node2 backup partitions)
>>>>
>>>> Wouldn't this mean node 3 need 2X space as compared to node 1 and node2.
>>>> Assuming backup partitions of node 3 would be equally distributed among
>>>> other two nodes. They would need almost same space.
>>>>
>>>>
>>>> On Tue, 1 Nov 2022, 23:30 Jeremy McMillan, <
>>>> jeremy.mcmillan@gridgain.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Tue, Nov 1, 2022 at 10:02 AM Surinder Mehra <re...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Even if we have 2 copies of data and primary and backup copy would be
>>>>>> stored in different AZs. My question remains valid in this case as well.
>>>>>>
>>>>>
>>>>> I think additional backup copies in the same AZ are superfluous if we
>>>>> start with the assumption that multiple concurrent failures are most likely
>>>>> to affect resources in the concurrent AZ. A second node failure, if that's
>>>>> your failure budget, is likely to corrupt all the backup copies in the
>>>>> second AZ.
>>>>>
>>>>> If you only have two AZs available in some data centers/deployments,
>>>>> but you need 3-way redundancy on certain caches/tables, then using AZ node
>>>>> attribute for backup filtering is too coarse grained. Using AZ is a general
>>>>> case best practice which gives your cluster the best chance of surviving
>>>>> multiple hardware failures in AWS because they pool hardware resources in
>>>>> AZs. Maybe you just need three AZs? Maybe AZ isn't the correct failure
>>>>> domain for your use case?
>>>>>
>>>>>
>>>>>> Do we have to ensure nodes in two AZs are always present or does
>>>>>> ignite have a way to indicate it couldn't create backups. Silently killing
>>>>>> backups is not desirable state.
>>>>>>
>>>>>
>>>>> Do you use synchronous or asynchronous backups?
>>>>>
>>>>> https://ignite.apache.org/docs/2.11.1/configuring-caches/configuring-backups#synchronous-and-asynchronous-backups
>>>>>
>>>>> You can periodically poll caches' configurations or hook a cluster
>>>>> state event, and re-compare the cache backup configuration against the
>>>>> enumerated available AZs, and raise an exception or log a message or
>>>>> whatever to detect the issue as soon as AZ count drops below minimum. This
>>>>> way might also be good for fuzzy warning condition detection point for
>>>>> proactive infrastructure operations. If you count all of the nodes in each
>>>>> AZ, you can detect and track AZ load imbalances as the ratio between the
>>>>> smallest AZ node count and the average AZ node count.
>>>>>
>>>>>
>>>>>> 2. In my original message with 2 nodes(node1 and node2) in AZ1, and
>>>>>> 3rdnode in second AZ, backups of node1 and node2 would be placed one node 3
>>>>>> in AZ2. It would mean it need to have 2X space to store backups.
>>>>>> Just trying to ensure my understanding is correct.
>>>>>>
>>>>>
>>>>> If you have three nodes, you divide your total footprint by three to
>>>>> get the minimum node capacity.
>>>>>
>>>>> If you have 2 backups, that is one primary copy plus two more backup
>>>>> copies, so you multiply your total footprint by 3.
>>>>>
>>>>> If you multiply, say 32GB by three for redundancy, that would be 96GB
>>>>> total space needed for the sum of all nodes' footprint.
>>>>>
>>>>> If you divide the 96GB storage commitment among three nodes, then each
>>>>> node must have a minimum of 32GB. That's what we started with as a nominal
>>>>> data footprint, so 1x not 2x. Node 1 will need to accommodate backups from
>>>>> node 2 and node 3. Node 2 will need to accommodate backups from node 1 and
>>>>> node 3. Each node has one primary and two backup partition copies for each
>>>>> partition of each cache with two backups.
>>>>>
>>>>>
>>>>>> Hope my queries are clear to you now
>>>>>>
>>>>>
>>>>> I still don't understand your operational goals, so I feel like we may
>>>>> be dancing around a misunderstanding.
>>>>>
>>>>>
>>>>>> On Tue, 1 Nov 2022, 20:19 Surinder Mehra, <re...@gmail.com> wrote:
>>>>>>
>>>>>>> Thanks for your reply. Let me try to answer your 2 questions below.
>>>>>>> 1. I understand that it sacrifices the backups incase it can't place
>>>>>>> backups appropriately. Question is, is it possible to fail the deployment
>>>>>>> rather than risking single copy of data present in cluster. If this only
>>>>>>> copy goes down, we will have downtime as data won't be present in cluster.
>>>>>>> We should rather throw error if enough hardware is not present than risking
>>>>>>> data unavailability issue during business activity
>>>>>>>
>>>>>>> 2. Why we want 3 copies of data. It's a design choice. We want to
>>>>>>> ensure even if 2 nodes go down, we still have 3rd present to serve the
>>>>>>> data.
>>>>>>>
>>>>>>> Hope I answered your question
>>>>>>>
>>>>>>> On Tue, 1 Nov 2022, 19:40 Jeremy McMillan, <
>>>>>>> jeremy.mcmillan@gridgain.com> wrote:
>>>>>>>
>>>>>>>> This question is a design question.
>>>>>>>>
>>>>>>>> What kids of fault states do you expect to tolerate? What is your
>>>>>>>> failure budget?
>>>>>>>>
>>>>>>>> Why are you trying to make more than 2 copies of the data
>>>>>>>> distribute across only two failure domains?
>>>>>>>>
>>>>>>>> Also "fail fast" means discover your implementation defects faster
>>>>>>>> than your release cycle, not how fast you can cause data loss.
>>>>>>>>
>>>>>>>> On Tue, Nov 1, 2022, 09:01 Surinder Mehra <re...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> gentle reminder.
>>>>>>>>> One additional question: We have observed that if available AZs
>>>>>>>>> are less than backups count, ignite skips creating backups. Is this correct
>>>>>>>>> understanding? If yes, how can we fail fast if backups can not be placed
>>>>>>>>> due to AZ limitation?
>>>>>>>>>
>>>>>>>>> On Mon, Oct 31, 2022 at 6:30 PM Surinder Mehra <re...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>> As per link attached, to ensure primary and backup partitions are
>>>>>>>>>> not stored on same node, We used AWS AZ as backup filter and now I can see
>>>>>>>>>> if I start two ignite nodes on the same machine, primary partitions are
>>>>>>>>>> evenly distributed but backups are always zero which is expected.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> https://www.gridgain.com/docs/latest/installation-guide/aws/multiple-availability-zone-aws
>>>>>>>>>>
>>>>>>>>>> My question is what would happen if AZ-1 has 2 machines and AZ-2
>>>>>>>>>> has 1 machine and ignite cluster has only 3 nodes, each machine having one
>>>>>>>>>> ignite node.
>>>>>>>>>>
>>>>>>>>>> Node1[AZ1] - keys 1-100
>>>>>>>>>> Node2[AZ1] - keys 101-200
>>>>>>>>>> Node3[AZ2] - keys 201 -300
>>>>>>>>>>
>>>>>>>>>> In the above scenario, if the backup count is 2, how would back
>>>>>>>>>> up partitions be distributed.
>>>>>>>>>>
>>>>>>>>>> 1. Would it mean node3 will have 2 backup copies of primary
>>>>>>>>>> partitions of node 1 and 2 ?
>>>>>>>>>> 2. If we have a 4 node cluster with 2 nodes in each AZ, would
>>>>>>>>>> backup copies also be placed on different nodes(In other words, does the
>>>>>>>>>> backup filter also apply to how backup copies are placed on nodes) ?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
Re: Backup filter in ignite [Multi AZ deployment]
Posted by Jeremy McMillan <je...@gridgain.com>.
Think of each AZ as being a massive piece of server hardware running VMs or
workloads for you. When hardware (or infrastructure maintenance process)
fails, assume everything on one AZ is lost at the same time.
On Sun, Nov 6, 2022, 09:58 Surinder Mehra <re...@gmail.com> wrote:
> That's partially true. Whole excercise of configuring AZ as backup filter
> is because we want to handle AZ level failure.
>
> Anyway, thanks for inputs. Will figure out further steps
>
> On Sun, 6 Nov 2022, 20:55 Jeremy McMillan, <je...@gridgain.com>
> wrote:
>
>> Don't configure 2 backups when you only have two failure domains.
>>
>> You're worried about node level failure, but you're telling Ignite to
>> worry about AZ level failure.
>>
>>
>> On Sat, Nov 5, 2022, 21:57 Surinder Mehra <re...@gmail.com> wrote:
>>
>>> Yeah I think there is a misunderstanding. Although I figured out my
>>> answers from our discussion, I will try one final attempt to clarify my
>>> point on 2X space for node3
>>>
>>> Node setup:
>>> Node1 and node 2 placed in AZ1
>>> Node 3 placed in AZ2
>>>
>>> Since I am using AZ as backup filter as I mentioned in my first
>>> message. Back up if node 1 cannot be placed on node2 and back up of node 2
>>> cannot be placed on node1 as they are in same AZ. This simply means their
>>> backups would go to node3 which in another AZ. Hence node 3 space =(node3
>>> primary partitions+node 1 back up partitions+node2 backup partitions)
>>>
>>> Wouldn't this mean node 3 need 2X space as compared to node 1 and node2.
>>> Assuming backup partitions of node 3 would be equally distributed among
>>> other two nodes. They would need almost same space.
>>>
>>>
>>> On Tue, 1 Nov 2022, 23:30 Jeremy McMillan, <je...@gridgain.com>
>>> wrote:
>>>
>>>>
>>>>
>>>> On Tue, Nov 1, 2022 at 10:02 AM Surinder Mehra <re...@gmail.com>
>>>> wrote:
>>>>
>>>>> Even if we have 2 copies of data and primary and backup copy would be
>>>>> stored in different AZs. My question remains valid in this case as well.
>>>>>
>>>>
>>>> I think additional backup copies in the same AZ are superfluous if we
>>>> start with the assumption that multiple concurrent failures are most likely
>>>> to affect resources in the concurrent AZ. A second node failure, if that's
>>>> your failure budget, is likely to corrupt all the backup copies in the
>>>> second AZ.
>>>>
>>>> If you only have two AZs available in some data centers/deployments,
>>>> but you need 3-way redundancy on certain caches/tables, then using AZ node
>>>> attribute for backup filtering is too coarse grained. Using AZ is a general
>>>> case best practice which gives your cluster the best chance of surviving
>>>> multiple hardware failures in AWS because they pool hardware resources in
>>>> AZs. Maybe you just need three AZs? Maybe AZ isn't the correct failure
>>>> domain for your use case?
>>>>
>>>>
>>>>> Do we have to ensure nodes in two AZs are always present or does
>>>>> ignite have a way to indicate it couldn't create backups. Silently killing
>>>>> backups is not desirable state.
>>>>>
>>>>
>>>> Do you use synchronous or asynchronous backups?
>>>>
>>>> https://ignite.apache.org/docs/2.11.1/configuring-caches/configuring-backups#synchronous-and-asynchronous-backups
>>>>
>>>> You can periodically poll caches' configurations or hook a cluster
>>>> state event, and re-compare the cache backup configuration against the
>>>> enumerated available AZs, and raise an exception or log a message or
>>>> whatever to detect the issue as soon as AZ count drops below minimum. This
>>>> way might also be good for fuzzy warning condition detection point for
>>>> proactive infrastructure operations. If you count all of the nodes in each
>>>> AZ, you can detect and track AZ load imbalances as the ratio between the
>>>> smallest AZ node count and the average AZ node count.
>>>>
>>>>
>>>>> 2. In my original message with 2 nodes(node1 and node2) in AZ1, and
>>>>> 3rdnode in second AZ, backups of node1 and node2 would be placed one node 3
>>>>> in AZ2. It would mean it need to have 2X space to store backups.
>>>>> Just trying to ensure my understanding is correct.
>>>>>
>>>>
>>>> If you have three nodes, you divide your total footprint by three to
>>>> get the minimum node capacity.
>>>>
>>>> If you have 2 backups, that is one primary copy plus two more backup
>>>> copies, so you multiply your total footprint by 3.
>>>>
>>>> If you multiply, say 32GB by three for redundancy, that would be 96GB
>>>> total space needed for the sum of all nodes' footprint.
>>>>
>>>> If you divide the 96GB storage commitment among three nodes, then each
>>>> node must have a minimum of 32GB. That's what we started with as a nominal
>>>> data footprint, so 1x not 2x. Node 1 will need to accommodate backups from
>>>> node 2 and node 3. Node 2 will need to accommodate backups from node 1 and
>>>> node 3. Each node has one primary and two backup partition copies for each
>>>> partition of each cache with two backups.
>>>>
>>>>
>>>>> Hope my queries are clear to you now
>>>>>
>>>>
>>>> I still don't understand your operational goals, so I feel like we may
>>>> be dancing around a misunderstanding.
>>>>
>>>>
>>>>> On Tue, 1 Nov 2022, 20:19 Surinder Mehra, <re...@gmail.com> wrote:
>>>>>
>>>>>> Thanks for your reply. Let me try to answer your 2 questions below.
>>>>>> 1. I understand that it sacrifices the backups incase it can't place
>>>>>> backups appropriately. Question is, is it possible to fail the deployment
>>>>>> rather than risking single copy of data present in cluster. If this only
>>>>>> copy goes down, we will have downtime as data won't be present in cluster.
>>>>>> We should rather throw error if enough hardware is not present than risking
>>>>>> data unavailability issue during business activity
>>>>>>
>>>>>> 2. Why we want 3 copies of data. It's a design choice. We want to
>>>>>> ensure even if 2 nodes go down, we still have 3rd present to serve the
>>>>>> data.
>>>>>>
>>>>>> Hope I answered your question
>>>>>>
>>>>>> On Tue, 1 Nov 2022, 19:40 Jeremy McMillan, <
>>>>>> jeremy.mcmillan@gridgain.com> wrote:
>>>>>>
>>>>>>> This question is a design question.
>>>>>>>
>>>>>>> What kids of fault states do you expect to tolerate? What is your
>>>>>>> failure budget?
>>>>>>>
>>>>>>> Why are you trying to make more than 2 copies of the data distribute
>>>>>>> across only two failure domains?
>>>>>>>
>>>>>>> Also "fail fast" means discover your implementation defects faster
>>>>>>> than your release cycle, not how fast you can cause data loss.
>>>>>>>
>>>>>>> On Tue, Nov 1, 2022, 09:01 Surinder Mehra <re...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> gentle reminder.
>>>>>>>> One additional question: We have observed that if available AZs are
>>>>>>>> less than backups count, ignite skips creating backups. Is this correct
>>>>>>>> understanding? If yes, how can we fail fast if backups can not be placed
>>>>>>>> due to AZ limitation?
>>>>>>>>
>>>>>>>> On Mon, Oct 31, 2022 at 6:30 PM Surinder Mehra <re...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>> As per link attached, to ensure primary and backup partitions are
>>>>>>>>> not stored on same node, We used AWS AZ as backup filter and now I can see
>>>>>>>>> if I start two ignite nodes on the same machine, primary partitions are
>>>>>>>>> evenly distributed but backups are always zero which is expected.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> https://www.gridgain.com/docs/latest/installation-guide/aws/multiple-availability-zone-aws
>>>>>>>>>
>>>>>>>>> My question is what would happen if AZ-1 has 2 machines and AZ-2
>>>>>>>>> has 1 machine and ignite cluster has only 3 nodes, each machine having one
>>>>>>>>> ignite node.
>>>>>>>>>
>>>>>>>>> Node1[AZ1] - keys 1-100
>>>>>>>>> Node2[AZ1] - keys 101-200
>>>>>>>>> Node3[AZ2] - keys 201 -300
>>>>>>>>>
>>>>>>>>> In the above scenario, if the backup count is 2, how would back up
>>>>>>>>> partitions be distributed.
>>>>>>>>>
>>>>>>>>> 1. Would it mean node3 will have 2 backup copies of primary
>>>>>>>>> partitions of node 1 and 2 ?
>>>>>>>>> 2. If we have a 4 node cluster with 2 nodes in each AZ, would
>>>>>>>>> backup copies also be placed on different nodes(In other words, does the
>>>>>>>>> backup filter also apply to how backup copies are placed on nodes) ?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
Re: Backup filter in ignite [Multi AZ deployment]
Posted by Surinder Mehra <re...@gmail.com>.
That's partially true. Whole excercise of configuring AZ as backup filter
is because we want to handle AZ level failure.
Anyway, thanks for inputs. Will figure out further steps
On Sun, 6 Nov 2022, 20:55 Jeremy McMillan, <je...@gridgain.com>
wrote:
> Don't configure 2 backups when you only have two failure domains.
>
> You're worried about node level failure, but you're telling Ignite to
> worry about AZ level failure.
>
>
> On Sat, Nov 5, 2022, 21:57 Surinder Mehra <re...@gmail.com> wrote:
>
>> Yeah I think there is a misunderstanding. Although I figured out my
>> answers from our discussion, I will try one final attempt to clarify my
>> point on 2X space for node3
>>
>> Node setup:
>> Node1 and node 2 placed in AZ1
>> Node 3 placed in AZ2
>>
>> Since I am using AZ as backup filter as I mentioned in my first message.
>> Back up if node 1 cannot be placed on node2 and back up of node 2 cannot be
>> placed on node1 as they are in same AZ. This simply means their backups
>> would go to node3 which in another AZ. Hence node 3 space =(node3 primary
>> partitions+node 1 back up partitions+node2 backup partitions)
>>
>> Wouldn't this mean node 3 need 2X space as compared to node 1 and node2.
>> Assuming backup partitions of node 3 would be equally distributed among
>> other two nodes. They would need almost same space.
>>
>>
>> On Tue, 1 Nov 2022, 23:30 Jeremy McMillan, <je...@gridgain.com>
>> wrote:
>>
>>>
>>>
>>> On Tue, Nov 1, 2022 at 10:02 AM Surinder Mehra <re...@gmail.com>
>>> wrote:
>>>
>>>> Even if we have 2 copies of data and primary and backup copy would be
>>>> stored in different AZs. My question remains valid in this case as well.
>>>>
>>>
>>> I think additional backup copies in the same AZ are superfluous if we
>>> start with the assumption that multiple concurrent failures are most likely
>>> to affect resources in the concurrent AZ. A second node failure, if that's
>>> your failure budget, is likely to corrupt all the backup copies in the
>>> second AZ.
>>>
>>> If you only have two AZs available in some data centers/deployments, but
>>> you need 3-way redundancy on certain caches/tables, then using AZ node
>>> attribute for backup filtering is too coarse grained. Using AZ is a general
>>> case best practice which gives your cluster the best chance of surviving
>>> multiple hardware failures in AWS because they pool hardware resources in
>>> AZs. Maybe you just need three AZs? Maybe AZ isn't the correct failure
>>> domain for your use case?
>>>
>>>
>>>> Do we have to ensure nodes in two AZs are always present or does ignite
>>>> have a way to indicate it couldn't create backups. Silently killing backups
>>>> is not desirable state.
>>>>
>>>
>>> Do you use synchronous or asynchronous backups?
>>>
>>> https://ignite.apache.org/docs/2.11.1/configuring-caches/configuring-backups#synchronous-and-asynchronous-backups
>>>
>>> You can periodically poll caches' configurations or hook a cluster state
>>> event, and re-compare the cache backup configuration against the enumerated
>>> available AZs, and raise an exception or log a message or whatever to
>>> detect the issue as soon as AZ count drops below minimum. This way might
>>> also be good for fuzzy warning condition detection point for proactive
>>> infrastructure operations. If you count all of the nodes in each AZ, you
>>> can detect and track AZ load imbalances as the ratio between the smallest
>>> AZ node count and the average AZ node count.
>>>
>>>
>>>> 2. In my original message with 2 nodes(node1 and node2) in AZ1, and
>>>> 3rdnode in second AZ, backups of node1 and node2 would be placed one node 3
>>>> in AZ2. It would mean it need to have 2X space to store backups.
>>>> Just trying to ensure my understanding is correct.
>>>>
>>>
>>> If you have three nodes, you divide your total footprint by three to get
>>> the minimum node capacity.
>>>
>>> If you have 2 backups, that is one primary copy plus two more backup
>>> copies, so you multiply your total footprint by 3.
>>>
>>> If you multiply, say 32GB by three for redundancy, that would be 96GB
>>> total space needed for the sum of all nodes' footprint.
>>>
>>> If you divide the 96GB storage commitment among three nodes, then each
>>> node must have a minimum of 32GB. That's what we started with as a nominal
>>> data footprint, so 1x not 2x. Node 1 will need to accommodate backups from
>>> node 2 and node 3. Node 2 will need to accommodate backups from node 1 and
>>> node 3. Each node has one primary and two backup partition copies for each
>>> partition of each cache with two backups.
>>>
>>>
>>>> Hope my queries are clear to you now
>>>>
>>>
>>> I still don't understand your operational goals, so I feel like we may
>>> be dancing around a misunderstanding.
>>>
>>>
>>>> On Tue, 1 Nov 2022, 20:19 Surinder Mehra, <re...@gmail.com> wrote:
>>>>
>>>>> Thanks for your reply. Let me try to answer your 2 questions below.
>>>>> 1. I understand that it sacrifices the backups incase it can't place
>>>>> backups appropriately. Question is, is it possible to fail the deployment
>>>>> rather than risking single copy of data present in cluster. If this only
>>>>> copy goes down, we will have downtime as data won't be present in cluster.
>>>>> We should rather throw error if enough hardware is not present than risking
>>>>> data unavailability issue during business activity
>>>>>
>>>>> 2. Why we want 3 copies of data. It's a design choice. We want to
>>>>> ensure even if 2 nodes go down, we still have 3rd present to serve the
>>>>> data.
>>>>>
>>>>> Hope I answered your question
>>>>>
>>>>> On Tue, 1 Nov 2022, 19:40 Jeremy McMillan, <
>>>>> jeremy.mcmillan@gridgain.com> wrote:
>>>>>
>>>>>> This question is a design question.
>>>>>>
>>>>>> What kids of fault states do you expect to tolerate? What is your
>>>>>> failure budget?
>>>>>>
>>>>>> Why are you trying to make more than 2 copies of the data distribute
>>>>>> across only two failure domains?
>>>>>>
>>>>>> Also "fail fast" means discover your implementation defects faster
>>>>>> than your release cycle, not how fast you can cause data loss.
>>>>>>
>>>>>> On Tue, Nov 1, 2022, 09:01 Surinder Mehra <re...@gmail.com> wrote:
>>>>>>
>>>>>>> gentle reminder.
>>>>>>> One additional question: We have observed that if available AZs are
>>>>>>> less than backups count, ignite skips creating backups. Is this correct
>>>>>>> understanding? If yes, how can we fail fast if backups can not be placed
>>>>>>> due to AZ limitation?
>>>>>>>
>>>>>>> On Mon, Oct 31, 2022 at 6:30 PM Surinder Mehra <re...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>> As per link attached, to ensure primary and backup partitions are
>>>>>>>> not stored on same node, We used AWS AZ as backup filter and now I can see
>>>>>>>> if I start two ignite nodes on the same machine, primary partitions are
>>>>>>>> evenly distributed but backups are always zero which is expected.
>>>>>>>>
>>>>>>>>
>>>>>>>> https://www.gridgain.com/docs/latest/installation-guide/aws/multiple-availability-zone-aws
>>>>>>>>
>>>>>>>> My question is what would happen if AZ-1 has 2 machines and AZ-2
>>>>>>>> has 1 machine and ignite cluster has only 3 nodes, each machine having one
>>>>>>>> ignite node.
>>>>>>>>
>>>>>>>> Node1[AZ1] - keys 1-100
>>>>>>>> Node2[AZ1] - keys 101-200
>>>>>>>> Node3[AZ2] - keys 201 -300
>>>>>>>>
>>>>>>>> In the above scenario, if the backup count is 2, how would back up
>>>>>>>> partitions be distributed.
>>>>>>>>
>>>>>>>> 1. Would it mean node3 will have 2 backup copies of primary
>>>>>>>> partitions of node 1 and 2 ?
>>>>>>>> 2. If we have a 4 node cluster with 2 nodes in each AZ, would
>>>>>>>> backup copies also be placed on different nodes(In other words, does the
>>>>>>>> backup filter also apply to how backup copies are placed on nodes) ?
>>>>>>>>
>>>>>>>>
>>>>>>>>
Re: Backup filter in ignite [Multi AZ deployment]
Posted by Jeremy McMillan <je...@gridgain.com>.
Don't configure 2 backups when you only have two failure domains.
You're worried about node level failure, but you're telling Ignite to worry
about AZ level failure.
On Sat, Nov 5, 2022, 21:57 Surinder Mehra <re...@gmail.com> wrote:
> Yeah I think there is a misunderstanding. Although I figured out my
> answers from our discussion, I will try one final attempt to clarify my
> point on 2X space for node3
>
> Node setup:
> Node1 and node 2 placed in AZ1
> Node 3 placed in AZ2
>
> Since I am using AZ as backup filter as I mentioned in my first message.
> Back up if node 1 cannot be placed on node2 and back up of node 2 cannot be
> placed on node1 as they are in same AZ. This simply means their backups
> would go to node3 which in another AZ. Hence node 3 space =(node3 primary
> partitions+node 1 back up partitions+node2 backup partitions)
>
> Wouldn't this mean node 3 need 2X space as compared to node 1 and node2.
> Assuming backup partitions of node 3 would be equally distributed among
> other two nodes. They would need almost same space.
>
>
> On Tue, 1 Nov 2022, 23:30 Jeremy McMillan, <je...@gridgain.com>
> wrote:
>
>>
>>
>> On Tue, Nov 1, 2022 at 10:02 AM Surinder Mehra <re...@gmail.com>
>> wrote:
>>
>>> Even if we have 2 copies of data and primary and backup copy would be
>>> stored in different AZs. My question remains valid in this case as well.
>>>
>>
>> I think additional backup copies in the same AZ are superfluous if we
>> start with the assumption that multiple concurrent failures are most likely
>> to affect resources in the concurrent AZ. A second node failure, if that's
>> your failure budget, is likely to corrupt all the backup copies in the
>> second AZ.
>>
>> If you only have two AZs available in some data centers/deployments, but
>> you need 3-way redundancy on certain caches/tables, then using AZ node
>> attribute for backup filtering is too coarse grained. Using AZ is a general
>> case best practice which gives your cluster the best chance of surviving
>> multiple hardware failures in AWS because they pool hardware resources in
>> AZs. Maybe you just need three AZs? Maybe AZ isn't the correct failure
>> domain for your use case?
>>
>>
>>> Do we have to ensure nodes in two AZs are always present or does ignite
>>> have a way to indicate it couldn't create backups. Silently killing backups
>>> is not desirable state.
>>>
>>
>> Do you use synchronous or asynchronous backups?
>>
>> https://ignite.apache.org/docs/2.11.1/configuring-caches/configuring-backups#synchronous-and-asynchronous-backups
>>
>> You can periodically poll caches' configurations or hook a cluster state
>> event, and re-compare the cache backup configuration against the enumerated
>> available AZs, and raise an exception or log a message or whatever to
>> detect the issue as soon as AZ count drops below minimum. This way might
>> also be good for fuzzy warning condition detection point for proactive
>> infrastructure operations. If you count all of the nodes in each AZ, you
>> can detect and track AZ load imbalances as the ratio between the smallest
>> AZ node count and the average AZ node count.
>>
>>
>>> 2. In my original message with 2 nodes(node1 and node2) in AZ1, and
>>> 3rdnode in second AZ, backups of node1 and node2 would be placed one node 3
>>> in AZ2. It would mean it need to have 2X space to store backups.
>>> Just trying to ensure my understanding is correct.
>>>
>>
>> If you have three nodes, you divide your total footprint by three to get
>> the minimum node capacity.
>>
>> If you have 2 backups, that is one primary copy plus two more backup
>> copies, so you multiply your total footprint by 3.
>>
>> If you multiply, say 32GB by three for redundancy, that would be 96GB
>> total space needed for the sum of all nodes' footprint.
>>
>> If you divide the 96GB storage commitment among three nodes, then each
>> node must have a minimum of 32GB. That's what we started with as a nominal
>> data footprint, so 1x not 2x. Node 1 will need to accommodate backups from
>> node 2 and node 3. Node 2 will need to accommodate backups from node 1 and
>> node 3. Each node has one primary and two backup partition copies for each
>> partition of each cache with two backups.
>>
>>
>>> Hope my queries are clear to you now
>>>
>>
>> I still don't understand your operational goals, so I feel like we may be
>> dancing around a misunderstanding.
>>
>>
>>> On Tue, 1 Nov 2022, 20:19 Surinder Mehra, <re...@gmail.com> wrote:
>>>
>>>> Thanks for your reply. Let me try to answer your 2 questions below.
>>>> 1. I understand that it sacrifices the backups incase it can't place
>>>> backups appropriately. Question is, is it possible to fail the deployment
>>>> rather than risking single copy of data present in cluster. If this only
>>>> copy goes down, we will have downtime as data won't be present in cluster.
>>>> We should rather throw error if enough hardware is not present than risking
>>>> data unavailability issue during business activity
>>>>
>>>> 2. Why we want 3 copies of data. It's a design choice. We want to
>>>> ensure even if 2 nodes go down, we still have 3rd present to serve the
>>>> data.
>>>>
>>>> Hope I answered your question
>>>>
>>>> On Tue, 1 Nov 2022, 19:40 Jeremy McMillan, <
>>>> jeremy.mcmillan@gridgain.com> wrote:
>>>>
>>>>> This question is a design question.
>>>>>
>>>>> What kids of fault states do you expect to tolerate? What is your
>>>>> failure budget?
>>>>>
>>>>> Why are you trying to make more than 2 copies of the data distribute
>>>>> across only two failure domains?
>>>>>
>>>>> Also "fail fast" means discover your implementation defects faster
>>>>> than your release cycle, not how fast you can cause data loss.
>>>>>
>>>>> On Tue, Nov 1, 2022, 09:01 Surinder Mehra <re...@gmail.com> wrote:
>>>>>
>>>>>> gentle reminder.
>>>>>> One additional question: We have observed that if available AZs are
>>>>>> less than backups count, ignite skips creating backups. Is this correct
>>>>>> understanding? If yes, how can we fail fast if backups can not be placed
>>>>>> due to AZ limitation?
>>>>>>
>>>>>> On Mon, Oct 31, 2022 at 6:30 PM Surinder Mehra <re...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>> As per link attached, to ensure primary and backup partitions are
>>>>>>> not stored on same node, We used AWS AZ as backup filter and now I can see
>>>>>>> if I start two ignite nodes on the same machine, primary partitions are
>>>>>>> evenly distributed but backups are always zero which is expected.
>>>>>>>
>>>>>>>
>>>>>>> https://www.gridgain.com/docs/latest/installation-guide/aws/multiple-availability-zone-aws
>>>>>>>
>>>>>>> My question is what would happen if AZ-1 has 2 machines and AZ-2 has
>>>>>>> 1 machine and ignite cluster has only 3 nodes, each machine having one
>>>>>>> ignite node.
>>>>>>>
>>>>>>> Node1[AZ1] - keys 1-100
>>>>>>> Node2[AZ1] - keys 101-200
>>>>>>> Node3[AZ2] - keys 201 -300
>>>>>>>
>>>>>>> In the above scenario, if the backup count is 2, how would back up
>>>>>>> partitions be distributed.
>>>>>>>
>>>>>>> 1. Would it mean node3 will have 2 backup copies of primary
>>>>>>> partitions of node 1 and 2 ?
>>>>>>> 2. If we have a 4 node cluster with 2 nodes in each AZ, would backup
>>>>>>> copies also be placed on different nodes(In other words, does the backup
>>>>>>> filter also apply to how backup copies are placed on nodes) ?
>>>>>>>
>>>>>>>
>>>>>>>
Re: Backup filter in ignite [Multi AZ deployment]
Posted by Surinder Mehra <re...@gmail.com>.
Yeah I think there is a misunderstanding. Although I figured out my answers
from our discussion, I will try one final attempt to clarify my point on 2X
space for node3
Node setup:
Node1 and node 2 placed in AZ1
Node 3 placed in AZ2
Since I am using AZ as backup filter as I mentioned in my first message.
Back up if node 1 cannot be placed on node2 and back up of node 2 cannot be
placed on node1 as they are in same AZ. This simply means their backups
would go to node3 which in another AZ. Hence node 3 space =(node3 primary
partitions+node 1 back up partitions+node2 backup partitions)
Wouldn't this mean node 3 need 2X space as compared to node 1 and node2.
Assuming backup partitions of node 3 would be equally distributed among
other two nodes. They would need almost same space.
On Tue, 1 Nov 2022, 23:30 Jeremy McMillan, <je...@gridgain.com>
wrote:
>
>
> On Tue, Nov 1, 2022 at 10:02 AM Surinder Mehra <re...@gmail.com> wrote:
>
>> Even if we have 2 copies of data and primary and backup copy would be
>> stored in different AZs. My question remains valid in this case as well.
>>
>
> I think additional backup copies in the same AZ are superfluous if we
> start with the assumption that multiple concurrent failures are most likely
> to affect resources in the concurrent AZ. A second node failure, if that's
> your failure budget, is likely to corrupt all the backup copies in the
> second AZ.
>
> If you only have two AZs available in some data centers/deployments, but
> you need 3-way redundancy on certain caches/tables, then using AZ node
> attribute for backup filtering is too coarse grained. Using AZ is a general
> case best practice which gives your cluster the best chance of surviving
> multiple hardware failures in AWS because they pool hardware resources in
> AZs. Maybe you just need three AZs? Maybe AZ isn't the correct failure
> domain for your use case?
>
>
>> Do we have to ensure nodes in two AZs are always present or does ignite
>> have a way to indicate it couldn't create backups. Silently killing backups
>> is not desirable state.
>>
>
> Do you use synchronous or asynchronous backups?
>
> https://ignite.apache.org/docs/2.11.1/configuring-caches/configuring-backups#synchronous-and-asynchronous-backups
>
> You can periodically poll caches' configurations or hook a cluster state
> event, and re-compare the cache backup configuration against the enumerated
> available AZs, and raise an exception or log a message or whatever to
> detect the issue as soon as AZ count drops below minimum. This way might
> also be good for fuzzy warning condition detection point for proactive
> infrastructure operations. If you count all of the nodes in each AZ, you
> can detect and track AZ load imbalances as the ratio between the smallest
> AZ node count and the average AZ node count.
>
>
>> 2. In my original message with 2 nodes(node1 and node2) in AZ1, and
>> 3rdnode in second AZ, backups of node1 and node2 would be placed one node 3
>> in AZ2. It would mean it need to have 2X space to store backups.
>> Just trying to ensure my understanding is correct.
>>
>
> If you have three nodes, you divide your total footprint by three to get
> the minimum node capacity.
>
> If you have 2 backups, that is one primary copy plus two more backup
> copies, so you multiply your total footprint by 3.
>
> If you multiply, say 32GB by three for redundancy, that would be 96GB
> total space needed for the sum of all nodes' footprint.
>
> If you divide the 96GB storage commitment among three nodes, then each
> node must have a minimum of 32GB. That's what we started with as a nominal
> data footprint, so 1x not 2x. Node 1 will need to accommodate backups from
> node 2 and node 3. Node 2 will need to accommodate backups from node 1 and
> node 3. Each node has one primary and two backup partition copies for each
> partition of each cache with two backups.
>
>
>> Hope my queries are clear to you now
>>
>
> I still don't understand your operational goals, so I feel like we may be
> dancing around a misunderstanding.
>
>
>> On Tue, 1 Nov 2022, 20:19 Surinder Mehra, <re...@gmail.com> wrote:
>>
>>> Thanks for your reply. Let me try to answer your 2 questions below.
>>> 1. I understand that it sacrifices the backups incase it can't place
>>> backups appropriately. Question is, is it possible to fail the deployment
>>> rather than risking single copy of data present in cluster. If this only
>>> copy goes down, we will have downtime as data won't be present in cluster.
>>> We should rather throw error if enough hardware is not present than risking
>>> data unavailability issue during business activity
>>>
>>> 2. Why we want 3 copies of data. It's a design choice. We want to ensure
>>> even if 2 nodes go down, we still have 3rd present to serve the data.
>>>
>>> Hope I answered your question
>>>
>>> On Tue, 1 Nov 2022, 19:40 Jeremy McMillan, <je...@gridgain.com>
>>> wrote:
>>>
>>>> This question is a design question.
>>>>
>>>> What kids of fault states do you expect to tolerate? What is your
>>>> failure budget?
>>>>
>>>> Why are you trying to make more than 2 copies of the data distribute
>>>> across only two failure domains?
>>>>
>>>> Also "fail fast" means discover your implementation defects faster than
>>>> your release cycle, not how fast you can cause data loss.
>>>>
>>>> On Tue, Nov 1, 2022, 09:01 Surinder Mehra <re...@gmail.com> wrote:
>>>>
>>>>> gentle reminder.
>>>>> One additional question: We have observed that if available AZs are
>>>>> less than backups count, ignite skips creating backups. Is this correct
>>>>> understanding? If yes, how can we fail fast if backups can not be placed
>>>>> due to AZ limitation?
>>>>>
>>>>> On Mon, Oct 31, 2022 at 6:30 PM Surinder Mehra <re...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>> As per link attached, to ensure primary and backup partitions are not
>>>>>> stored on same node, We used AWS AZ as backup filter and now I can see if I
>>>>>> start two ignite nodes on the same machine, primary partitions are evenly
>>>>>> distributed but backups are always zero which is expected.
>>>>>>
>>>>>>
>>>>>> https://www.gridgain.com/docs/latest/installation-guide/aws/multiple-availability-zone-aws
>>>>>>
>>>>>> My question is what would happen if AZ-1 has 2 machines and AZ-2 has
>>>>>> 1 machine and ignite cluster has only 3 nodes, each machine having one
>>>>>> ignite node.
>>>>>>
>>>>>> Node1[AZ1] - keys 1-100
>>>>>> Node2[AZ1] - keys 101-200
>>>>>> Node3[AZ2] - keys 201 -300
>>>>>>
>>>>>> In the above scenario, if the backup count is 2, how would back up
>>>>>> partitions be distributed.
>>>>>>
>>>>>> 1. Would it mean node3 will have 2 backup copies of primary
>>>>>> partitions of node 1 and 2 ?
>>>>>> 2. If we have a 4 node cluster with 2 nodes in each AZ, would backup
>>>>>> copies also be placed on different nodes(In other words, does the backup
>>>>>> filter also apply to how backup copies are placed on nodes) ?
>>>>>>
>>>>>>
>>>>>>
Re: Backup filter in ignite [Multi AZ deployment]
Posted by Jeremy McMillan <je...@gridgain.com>.
On Tue, Nov 1, 2022 at 10:02 AM Surinder Mehra <re...@gmail.com> wrote:
> Even if we have 2 copies of data and primary and backup copy would be
> stored in different AZs. My question remains valid in this case as well.
>
I think additional backup copies in the same AZ are superfluous if we start
with the assumption that multiple concurrent failures are most likely to
affect resources in the concurrent AZ. A second node failure, if that's
your failure budget, is likely to corrupt all the backup copies in the
second AZ.
If you only have two AZs available in some data centers/deployments, but
you need 3-way redundancy on certain caches/tables, then using AZ node
attribute for backup filtering is too coarse grained. Using AZ is a general
case best practice which gives your cluster the best chance of surviving
multiple hardware failures in AWS because they pool hardware resources in
AZs. Maybe you just need three AZs? Maybe AZ isn't the correct failure
domain for your use case?
> Do we have to ensure nodes in two AZs are always present or does ignite
> have a way to indicate it couldn't create backups. Silently killing backups
> is not desirable state.
>
Do you use synchronous or asynchronous backups?
https://ignite.apache.org/docs/2.11.1/configuring-caches/configuring-backups#synchronous-and-asynchronous-backups
You can periodically poll caches' configurations or hook a cluster state
event, and re-compare the cache backup configuration against the enumerated
available AZs, and raise an exception or log a message or whatever to
detect the issue as soon as AZ count drops below minimum. This way might
also be good for fuzzy warning condition detection point for proactive
infrastructure operations. If you count all of the nodes in each AZ, you
can detect and track AZ load imbalances as the ratio between the smallest
AZ node count and the average AZ node count.
> 2. In my original message with 2 nodes(node1 and node2) in AZ1, and
> 3rdnode in second AZ, backups of node1 and node2 would be placed one node 3
> in AZ2. It would mean it need to have 2X space to store backups.
> Just trying to ensure my understanding is correct.
>
If you have three nodes, you divide your total footprint by three to get
the minimum node capacity.
If you have 2 backups, that is one primary copy plus two more backup
copies, so you multiply your total footprint by 3.
If you multiply, say 32GB by three for redundancy, that would be 96GB total
space needed for the sum of all nodes' footprint.
If you divide the 96GB storage commitment among three nodes, then each node
must have a minimum of 32GB. That's what we started with as a nominal data
footprint, so 1x not 2x. Node 1 will need to accommodate backups from node
2 and node 3. Node 2 will need to accommodate backups from node 1 and node
3. Each node has one primary and two backup partition copies for each
partition of each cache with two backups.
> Hope my queries are clear to you now
>
I still don't understand your operational goals, so I feel like we may be
dancing around a misunderstanding.
> On Tue, 1 Nov 2022, 20:19 Surinder Mehra, <re...@gmail.com> wrote:
>
>> Thanks for your reply. Let me try to answer your 2 questions below.
>> 1. I understand that it sacrifices the backups incase it can't place
>> backups appropriately. Question is, is it possible to fail the deployment
>> rather than risking single copy of data present in cluster. If this only
>> copy goes down, we will have downtime as data won't be present in cluster.
>> We should rather throw error if enough hardware is not present than risking
>> data unavailability issue during business activity
>>
>> 2. Why we want 3 copies of data. It's a design choice. We want to ensure
>> even if 2 nodes go down, we still have 3rd present to serve the data.
>>
>> Hope I answered your question
>>
>> On Tue, 1 Nov 2022, 19:40 Jeremy McMillan, <je...@gridgain.com>
>> wrote:
>>
>>> This question is a design question.
>>>
>>> What kids of fault states do you expect to tolerate? What is your
>>> failure budget?
>>>
>>> Why are you trying to make more than 2 copies of the data distribute
>>> across only two failure domains?
>>>
>>> Also "fail fast" means discover your implementation defects faster than
>>> your release cycle, not how fast you can cause data loss.
>>>
>>> On Tue, Nov 1, 2022, 09:01 Surinder Mehra <re...@gmail.com> wrote:
>>>
>>>> gentle reminder.
>>>> One additional question: We have observed that if available AZs are
>>>> less than backups count, ignite skips creating backups. Is this correct
>>>> understanding? If yes, how can we fail fast if backups can not be placed
>>>> due to AZ limitation?
>>>>
>>>> On Mon, Oct 31, 2022 at 6:30 PM Surinder Mehra <re...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>> As per link attached, to ensure primary and backup partitions are not
>>>>> stored on same node, We used AWS AZ as backup filter and now I can see if I
>>>>> start two ignite nodes on the same machine, primary partitions are evenly
>>>>> distributed but backups are always zero which is expected.
>>>>>
>>>>>
>>>>> https://www.gridgain.com/docs/latest/installation-guide/aws/multiple-availability-zone-aws
>>>>>
>>>>> My question is what would happen if AZ-1 has 2 machines and AZ-2 has 1
>>>>> machine and ignite cluster has only 3 nodes, each machine having one ignite
>>>>> node.
>>>>>
>>>>> Node1[AZ1] - keys 1-100
>>>>> Node2[AZ1] - keys 101-200
>>>>> Node3[AZ2] - keys 201 -300
>>>>>
>>>>> In the above scenario, if the backup count is 2, how would back up
>>>>> partitions be distributed.
>>>>>
>>>>> 1. Would it mean node3 will have 2 backup copies of primary partitions
>>>>> of node 1 and 2 ?
>>>>> 2. If we have a 4 node cluster with 2 nodes in each AZ, would backup
>>>>> copies also be placed on different nodes(In other words, does the backup
>>>>> filter also apply to how backup copies are placed on nodes) ?
>>>>>
>>>>>
>>>>>
Re: Backup filter in ignite [Multi AZ deployment]
Posted by Stephen Darlington <st...@gridgain.com>.
You can use a Topology Validator to define when a cache is valid.
> On 1 Nov 2022, at 15:02, Surinder Mehra <re...@gmail.com> wrote:
>
> Even if we have 2 copies of data and primary and backup copy would be stored in different AZs. My question remains valid in this case as well.
>
> Do we have to ensure nodes in two AZs are always present or does ignite have a way to indicate it couldn't create backups. Silently killing backups is not desirable state.
>
> 2. In my original message with 2 nodes(node1 and node2) in AZ1, and 3rdnode in second AZ, backups of node1 and node2 would be placed one node 3 in AZ2. It would mean it need to have 2X space to store backups.
> Just trying to ensure my understanding is correct.
>
> Hope my queries are clear to you now
>
> On Tue, 1 Nov 2022, 20:19 Surinder Mehra, <rednirus@gmail.com <ma...@gmail.com>> wrote:
> Thanks for your reply. Let me try to answer your 2 questions below.
> 1. I understand that it sacrifices the backups incase it can't place backups appropriately. Question is, is it possible to fail the deployment rather than risking single copy of data present in cluster. If this only copy goes down, we will have downtime as data won't be present in cluster. We should rather throw error if enough hardware is not present than risking data unavailability issue during business activity
>
> 2. Why we want 3 copies of data. It's a design choice. We want to ensure even if 2 nodes go down, we still have 3rd present to serve the data.
>
> Hope I answered your question
>
> On Tue, 1 Nov 2022, 19:40 Jeremy McMillan, <jeremy.mcmillan@gridgain.com <ma...@gridgain.com>> wrote:
> This question is a design question.
>
> What kids of fault states do you expect to tolerate? What is your failure budget?
>
> Why are you trying to make more than 2 copies of the data distribute across only two failure domains?
>
> Also "fail fast" means discover your implementation defects faster than your release cycle, not how fast you can cause data loss.
>
> On Tue, Nov 1, 2022, 09:01 Surinder Mehra <rednirus@gmail.com <ma...@gmail.com>> wrote:
> gentle reminder.
> One additional question: We have observed that if available AZs are less than backups count, ignite skips creating backups. Is this correct understanding? If yes, how can we fail fast if backups can not be placed due to AZ limitation?
>
> On Mon, Oct 31, 2022 at 6:30 PM Surinder Mehra <rednirus@gmail.com <ma...@gmail.com>> wrote:
> Hi,
> As per link attached, to ensure primary and backup partitions are not stored on same node, We used AWS AZ as backup filter and now I can see if I start two ignite nodes on the same machine, primary partitions are evenly distributed but backups are always zero which is expected.
>
> https://www.gridgain.com/docs/latest/installation-guide/aws/multiple-availability-zone-aws <https://www.gridgain.com/docs/latest/installation-guide/aws/multiple-availability-zone-aws>
>
> My question is what would happen if AZ-1 has 2 machines and AZ-2 has 1 machine and ignite cluster has only 3 nodes, each machine having one ignite node.
>
> Node1[AZ1] - keys 1-100
> Node2[AZ1] - keys 101-200
> Node3[AZ2] - keys 201 -300
>
> In the above scenario, if the backup count is 2, how would back up partitions be distributed.
>
> 1. Would it mean node3 will have 2 backup copies of primary partitions of node 1 and 2 ?
> 2. If we have a 4 node cluster with 2 nodes in each AZ, would backup copies also be placed on different nodes(In other words, does the backup filter also apply to how backup copies are placed on nodes) ?
>
>
Re: Backup filter in ignite [Multi AZ deployment]
Posted by Surinder Mehra <re...@gmail.com>.
Even if we have 2 copies of data and primary and backup copy would be
stored in different AZs. My question remains valid in this case as well.
Do we have to ensure nodes in two AZs are always present or does ignite
have a way to indicate it couldn't create backups. Silently killing backups
is not desirable state.
2. In my original message with 2 nodes(node1 and node2) in AZ1, and 3rdnode
in second AZ, backups of node1 and node2 would be placed one node 3 in AZ2.
It would mean it need to have 2X space to store backups.
Just trying to ensure my understanding is correct.
Hope my queries are clear to you now
On Tue, 1 Nov 2022, 20:19 Surinder Mehra, <re...@gmail.com> wrote:
> Thanks for your reply. Let me try to answer your 2 questions below.
> 1. I understand that it sacrifices the backups incase it can't place
> backups appropriately. Question is, is it possible to fail the deployment
> rather than risking single copy of data present in cluster. If this only
> copy goes down, we will have downtime as data won't be present in cluster.
> We should rather throw error if enough hardware is not present than risking
> data unavailability issue during business activity
>
> 2. Why we want 3 copies of data. It's a design choice. We want to ensure
> even if 2 nodes go down, we still have 3rd present to serve the data.
>
> Hope I answered your question
>
> On Tue, 1 Nov 2022, 19:40 Jeremy McMillan, <je...@gridgain.com>
> wrote:
>
>> This question is a design question.
>>
>> What kids of fault states do you expect to tolerate? What is your failure
>> budget?
>>
>> Why are you trying to make more than 2 copies of the data distribute
>> across only two failure domains?
>>
>> Also "fail fast" means discover your implementation defects faster than
>> your release cycle, not how fast you can cause data loss.
>>
>> On Tue, Nov 1, 2022, 09:01 Surinder Mehra <re...@gmail.com> wrote:
>>
>>> gentle reminder.
>>> One additional question: We have observed that if available AZs are less
>>> than backups count, ignite skips creating backups. Is this correct
>>> understanding? If yes, how can we fail fast if backups can not be placed
>>> due to AZ limitation?
>>>
>>> On Mon, Oct 31, 2022 at 6:30 PM Surinder Mehra <re...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>> As per link attached, to ensure primary and backup partitions are not
>>>> stored on same node, We used AWS AZ as backup filter and now I can see if I
>>>> start two ignite nodes on the same machine, primary partitions are evenly
>>>> distributed but backups are always zero which is expected.
>>>>
>>>>
>>>> https://www.gridgain.com/docs/latest/installation-guide/aws/multiple-availability-zone-aws
>>>>
>>>> My question is what would happen if AZ-1 has 2 machines and AZ-2 has 1
>>>> machine and ignite cluster has only 3 nodes, each machine having one ignite
>>>> node.
>>>>
>>>> Node1[AZ1] - keys 1-100
>>>> Node2[AZ1] - keys 101-200
>>>> Node3[AZ2] - keys 201 -300
>>>>
>>>> In the above scenario, if the backup count is 2, how would back up
>>>> partitions be distributed.
>>>>
>>>> 1. Would it mean node3 will have 2 backup copies of primary partitions
>>>> of node 1 and 2 ?
>>>> 2. If we have a 4 node cluster with 2 nodes in each AZ, would backup
>>>> copies also be placed on different nodes(In other words, does the backup
>>>> filter also apply to how backup copies are placed on nodes) ?
>>>>
>>>>
>>>>
Re: Backup filter in ignite [Multi AZ deployment]
Posted by Surinder Mehra <re...@gmail.com>.
Thanks for suggestions. Will try to ensure infra as suggested and will
explore topology validator if this can be used.
On Tue, 1 Nov 2022, 21:51 Jeremy McMillan, <je...@gridgain.com>
wrote:
> Can you tell two stories which start out all nodes in the intended cluster
> configuration are down, one story resulting in a successful cluster
> startup, but the other detecting an invalid configuration, and refusing to
> start?
>
> I can anticipate problems understanding what to do when the first node
> attempts to start, but only has its own AZ represented in the topology. How
> can this first node know whether future nodes will be able to fulfill the
> condition backup_replicas + 1 >= AZ_count? The general case, allowing
> elastic deployment, requires individual Ignite nodes to work in a
> best-effort capacity.
>
> I would approach this from a DevOps perspective, and just validate the
> deployment before starting up any infrastructure. Look at all of the
> relevant config files which would be deployed. Enumerate a projection of
> deployed nodes and their AZs. Compare this against the desired backup
> filter configuration and fail before starting any Ignite nodes with a
> deployment automation tool exception.
>
> On Tue, Nov 1, 2022 at 9:49 AM Surinder Mehra <re...@gmail.com> wrote:
>
>> Thanks for your reply. Let me try to answer your 2 questions below.
>> 1. I understand that it sacrifices the backups incase it can't place
>> backups appropriately. Question is, is it possible to fail the deployment
>> rather than risking single copy of data present in cluster. If this only
>> copy goes down, we will have downtime as data won't be present in cluster.
>> We should rather throw error if enough hardware is not present than risking
>> data unavailability issue during business activity
>>
>> 2. Why we want 3 copies of data. It's a design choice. We want to ensure
>> even if 2 nodes go down, we still have 3rd present to serve the data.
>>
>> Hope I answered your question
>>
>> On Tue, 1 Nov 2022, 19:40 Jeremy McMillan, <je...@gridgain.com>
>> wrote:
>>
>>> This question is a design question.
>>>
>>> What kids of fault states do you expect to tolerate? What is your
>>> failure budget?
>>>
>>> Why are you trying to make more than 2 copies of the data distribute
>>> across only two failure domains?
>>>
>>> Also "fail fast" means discover your implementation defects faster than
>>> your release cycle, not how fast you can cause data loss.
>>>
>>> On Tue, Nov 1, 2022, 09:01 Surinder Mehra <re...@gmail.com> wrote:
>>>
>>>> gentle reminder.
>>>> One additional question: We have observed that if available AZs are
>>>> less than backups count, ignite skips creating backups. Is this correct
>>>> understanding? If yes, how can we fail fast if backups can not be placed
>>>> due to AZ limitation?
>>>>
>>>> On Mon, Oct 31, 2022 at 6:30 PM Surinder Mehra <re...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>> As per link attached, to ensure primary and backup partitions are not
>>>>> stored on same node, We used AWS AZ as backup filter and now I can see if I
>>>>> start two ignite nodes on the same machine, primary partitions are evenly
>>>>> distributed but backups are always zero which is expected.
>>>>>
>>>>>
>>>>> https://www.gridgain.com/docs/latest/installation-guide/aws/multiple-availability-zone-aws
>>>>>
>>>>> My question is what would happen if AZ-1 has 2 machines and AZ-2 has 1
>>>>> machine and ignite cluster has only 3 nodes, each machine having one ignite
>>>>> node.
>>>>>
>>>>> Node1[AZ1] - keys 1-100
>>>>> Node2[AZ1] - keys 101-200
>>>>> Node3[AZ2] - keys 201 -300
>>>>>
>>>>> In the above scenario, if the backup count is 2, how would back up
>>>>> partitions be distributed.
>>>>>
>>>>> 1. Would it mean node3 will have 2 backup copies of primary partitions
>>>>> of node 1 and 2 ?
>>>>> 2. If we have a 4 node cluster with 2 nodes in each AZ, would backup
>>>>> copies also be placed on different nodes(In other words, does the backup
>>>>> filter also apply to how backup copies are placed on nodes) ?
>>>>>
>>>>>
>>>>>
Re: Backup filter in ignite [Multi AZ deployment]
Posted by Jeremy McMillan <je...@gridgain.com>.
Can you tell two stories which start out all nodes in the intended cluster
configuration are down, one story resulting in a successful cluster
startup, but the other detecting an invalid configuration, and refusing to
start?
I can anticipate problems understanding what to do when the first node
attempts to start, but only has its own AZ represented in the topology. How
can this first node know whether future nodes will be able to fulfill the
condition backup_replicas + 1 >= AZ_count? The general case, allowing
elastic deployment, requires individual Ignite nodes to work in a
best-effort capacity.
I would approach this from a DevOps perspective, and just validate the
deployment before starting up any infrastructure. Look at all of the
relevant config files which would be deployed. Enumerate a projection of
deployed nodes and their AZs. Compare this against the desired backup
filter configuration and fail before starting any Ignite nodes with a
deployment automation tool exception.
On Tue, Nov 1, 2022 at 9:49 AM Surinder Mehra <re...@gmail.com> wrote:
> Thanks for your reply. Let me try to answer your 2 questions below.
> 1. I understand that it sacrifices the backups incase it can't place
> backups appropriately. Question is, is it possible to fail the deployment
> rather than risking single copy of data present in cluster. If this only
> copy goes down, we will have downtime as data won't be present in cluster.
> We should rather throw error if enough hardware is not present than risking
> data unavailability issue during business activity
>
> 2. Why we want 3 copies of data. It's a design choice. We want to ensure
> even if 2 nodes go down, we still have 3rd present to serve the data.
>
> Hope I answered your question
>
> On Tue, 1 Nov 2022, 19:40 Jeremy McMillan, <je...@gridgain.com>
> wrote:
>
>> This question is a design question.
>>
>> What kids of fault states do you expect to tolerate? What is your failure
>> budget?
>>
>> Why are you trying to make more than 2 copies of the data distribute
>> across only two failure domains?
>>
>> Also "fail fast" means discover your implementation defects faster than
>> your release cycle, not how fast you can cause data loss.
>>
>> On Tue, Nov 1, 2022, 09:01 Surinder Mehra <re...@gmail.com> wrote:
>>
>>> gentle reminder.
>>> One additional question: We have observed that if available AZs are less
>>> than backups count, ignite skips creating backups. Is this correct
>>> understanding? If yes, how can we fail fast if backups can not be placed
>>> due to AZ limitation?
>>>
>>> On Mon, Oct 31, 2022 at 6:30 PM Surinder Mehra <re...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>> As per link attached, to ensure primary and backup partitions are not
>>>> stored on same node, We used AWS AZ as backup filter and now I can see if I
>>>> start two ignite nodes on the same machine, primary partitions are evenly
>>>> distributed but backups are always zero which is expected.
>>>>
>>>>
>>>> https://www.gridgain.com/docs/latest/installation-guide/aws/multiple-availability-zone-aws
>>>>
>>>> My question is what would happen if AZ-1 has 2 machines and AZ-2 has 1
>>>> machine and ignite cluster has only 3 nodes, each machine having one ignite
>>>> node.
>>>>
>>>> Node1[AZ1] - keys 1-100
>>>> Node2[AZ1] - keys 101-200
>>>> Node3[AZ2] - keys 201 -300
>>>>
>>>> In the above scenario, if the backup count is 2, how would back up
>>>> partitions be distributed.
>>>>
>>>> 1. Would it mean node3 will have 2 backup copies of primary partitions
>>>> of node 1 and 2 ?
>>>> 2. If we have a 4 node cluster with 2 nodes in each AZ, would backup
>>>> copies also be placed on different nodes(In other words, does the backup
>>>> filter also apply to how backup copies are placed on nodes) ?
>>>>
>>>>
>>>>
Re: Backup filter in ignite [Multi AZ deployment]
Posted by Surinder Mehra <re...@gmail.com>.
Thanks for your reply. Let me try to answer your 2 questions below.
1. I understand that it sacrifices the backups incase it can't place
backups appropriately. Question is, is it possible to fail the deployment
rather than risking single copy of data present in cluster. If this only
copy goes down, we will have downtime as data won't be present in cluster.
We should rather throw error if enough hardware is not present than risking
data unavailability issue during business activity
2. Why we want 3 copies of data. It's a design choice. We want to ensure
even if 2 nodes go down, we still have 3rd present to serve the data.
Hope I answered your question
On Tue, 1 Nov 2022, 19:40 Jeremy McMillan, <je...@gridgain.com>
wrote:
> This question is a design question.
>
> What kids of fault states do you expect to tolerate? What is your failure
> budget?
>
> Why are you trying to make more than 2 copies of the data distribute
> across only two failure domains?
>
> Also "fail fast" means discover your implementation defects faster than
> your release cycle, not how fast you can cause data loss.
>
> On Tue, Nov 1, 2022, 09:01 Surinder Mehra <re...@gmail.com> wrote:
>
>> gentle reminder.
>> One additional question: We have observed that if available AZs are less
>> than backups count, ignite skips creating backups. Is this correct
>> understanding? If yes, how can we fail fast if backups can not be placed
>> due to AZ limitation?
>>
>> On Mon, Oct 31, 2022 at 6:30 PM Surinder Mehra <re...@gmail.com>
>> wrote:
>>
>>> Hi,
>>> As per link attached, to ensure primary and backup partitions are not
>>> stored on same node, We used AWS AZ as backup filter and now I can see if I
>>> start two ignite nodes on the same machine, primary partitions are evenly
>>> distributed but backups are always zero which is expected.
>>>
>>>
>>> https://www.gridgain.com/docs/latest/installation-guide/aws/multiple-availability-zone-aws
>>>
>>> My question is what would happen if AZ-1 has 2 machines and AZ-2 has 1
>>> machine and ignite cluster has only 3 nodes, each machine having one ignite
>>> node.
>>>
>>> Node1[AZ1] - keys 1-100
>>> Node2[AZ1] - keys 101-200
>>> Node3[AZ2] - keys 201 -300
>>>
>>> In the above scenario, if the backup count is 2, how would back up
>>> partitions be distributed.
>>>
>>> 1. Would it mean node3 will have 2 backup copies of primary partitions
>>> of node 1 and 2 ?
>>> 2. If we have a 4 node cluster with 2 nodes in each AZ, would backup
>>> copies also be placed on different nodes(In other words, does the backup
>>> filter also apply to how backup copies are placed on nodes) ?
>>>
>>>
>>>
Re: Backup filter in ignite [Multi AZ deployment]
Posted by Jeremy McMillan <je...@gridgain.com>.
This question is a design question.
What kids of fault states do you expect to tolerate? What is your failure
budget?
Why are you trying to make more than 2 copies of the data distribute across
only two failure domains?
Also "fail fast" means discover your implementation defects faster than
your release cycle, not how fast you can cause data loss.
On Tue, Nov 1, 2022, 09:01 Surinder Mehra <re...@gmail.com> wrote:
> gentle reminder.
> One additional question: We have observed that if available AZs are less
> than backups count, ignite skips creating backups. Is this correct
> understanding? If yes, how can we fail fast if backups can not be placed
> due to AZ limitation?
>
> On Mon, Oct 31, 2022 at 6:30 PM Surinder Mehra <re...@gmail.com> wrote:
>
>> Hi,
>> As per link attached, to ensure primary and backup partitions are not
>> stored on same node, We used AWS AZ as backup filter and now I can see if I
>> start two ignite nodes on the same machine, primary partitions are evenly
>> distributed but backups are always zero which is expected.
>>
>>
>> https://www.gridgain.com/docs/latest/installation-guide/aws/multiple-availability-zone-aws
>>
>> My question is what would happen if AZ-1 has 2 machines and AZ-2 has 1
>> machine and ignite cluster has only 3 nodes, each machine having one ignite
>> node.
>>
>> Node1[AZ1] - keys 1-100
>> Node2[AZ1] - keys 101-200
>> Node3[AZ2] - keys 201 -300
>>
>> In the above scenario, if the backup count is 2, how would back up
>> partitions be distributed.
>>
>> 1. Would it mean node3 will have 2 backup copies of primary partitions of
>> node 1 and 2 ?
>> 2. If we have a 4 node cluster with 2 nodes in each AZ, would backup
>> copies also be placed on different nodes(In other words, does the backup
>> filter also apply to how backup copies are placed on nodes) ?
>>
>>
>>
Re: Backup filter in ignite [Multi AZ deployment]
Posted by Surinder Mehra <re...@gmail.com>.
gentle reminder.
One additional question: We have observed that if available AZs are less
than backups count, ignite skips creating backups. Is this correct
understanding? If yes, how can we fail fast if backups can not be placed
due to AZ limitation?
On Mon, Oct 31, 2022 at 6:30 PM Surinder Mehra <re...@gmail.com> wrote:
> Hi,
> As per link attached, to ensure primary and backup partitions are not
> stored on same node, We used AWS AZ as backup filter and now I can see if I
> start two ignite nodes on the same machine, primary partitions are evenly
> distributed but backups are always zero which is expected.
>
>
> https://www.gridgain.com/docs/latest/installation-guide/aws/multiple-availability-zone-aws
>
> My question is what would happen if AZ-1 has 2 machines and AZ-2 has 1
> machine and ignite cluster has only 3 nodes, each machine having one ignite
> node.
>
> Node1[AZ1] - keys 1-100
> Node2[AZ1] - keys 101-200
> Node3[AZ2] - keys 201 -300
>
> In the above scenario, if the backup count is 2, how would back up
> partitions be distributed.
>
> 1. Would it mean node3 will have 2 backup copies of primary partitions of
> node 1 and 2 ?
> 2. If we have a 4 node cluster with 2 nodes in each AZ, would backup
> copies also be placed on different nodes(In other words, does the backup
> filter also apply to how backup copies are placed on nodes) ?
>
>
>
Re: Backup filter in ignite [Multi AZ deployment]
Posted by Jeremy McMillan <je...@gridgain.com>.
Using the AWS tutorial will get you a backup filter using this
implementation: ClusterNodeAttributeAffinityBackupFilter
There is logic to prevent a cascade of backup data onto survivor nodes in
case of multiple concurrent failures if you read the documentation.
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/affinity/rendezvous/ClusterNodeAttributeAffinityBackupFilter.html
"This implementation will discard backups rather than place multiple on the
same set of nodes. This avoids trying to cram more data onto remaining
nodes when some have failed."
If you don't have enough nodes to support backups, this implementation will
sacrifice backups to keep the cluster operating.
It isn't clear if this question is a hypothetical understanding of features
and functionality, or whether you have a design problem you need to solve.
On Mon, Oct 31, 2022 at 8:01 AM Surinder Mehra <re...@gmail.com> wrote:
> Hi,
> As per link attached, to ensure primary and backup partitions are not
> stored on same node, We used AWS AZ as backup filter and now I can see if I
> start two ignite nodes on the same machine, primary partitions are evenly
> distributed but backups are always zero which is expected.
>
>
> https://www.gridgain.com/docs/latest/installation-guide/aws/multiple-availability-zone-aws
>
> My question is what would happen if AZ-1 has 2 machines and AZ-2 has 1
> machine and ignite cluster has only 3 nodes, each machine having one ignite
> node.
>
> Node1[AZ1] - keys 1-100
> Node2[AZ1] - keys 101-200
> Node3[AZ2] - keys 201 -300
>
> In the above scenario, if the backup count is 2, how would back up
> partitions be distributed.
>
> 1. Would it mean node3 will have 2 backup copies of primary partitions of
> node 1 and 2 ?
> 2. If we have a 4 node cluster with 2 nodes in each AZ, would backup
> copies also be placed on different nodes(In other words, does the backup
> filter also apply to how backup copies are placed on nodes) ?
>
>
>