You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Fd Habash <fm...@gmail.com> on 2019/05/02 14:54:15 UTC
CL=LQ, RF=3: Can a Write be Lost If Two Nodes ACK'ing it Die
C*: 2.2.8
Write CL = LQ
Kspace RF = 3
Three racks
A write gets received by node 1 in rack 1 at above specs. Node 1 (rack1) & node 2 (rack2) acknowledge it to the client.
Within some unit of time, node 1 & 2 die. Either ….
- Scenario 1: C* process death: Row did not make it to sstable (it is in commit log & was in memtable)
- Scenario 2: Node death: row may be have made to sstable, but nodes are gone (will have to bootstrap to replace).
Scenario 1: Row is not lost because once C* is restarted, commit log should replay the mutation.
Scenario 2: row is gone forever? If these two nodes are replaced via bootstrapping, will they ever get the row back from node 3 (rack3) if the write ever made it there?
----------------
Thank you
Re: CL=LQ, RF=3: Can a Write be Lost If Two Nodes ACK'ing it Die
Posted by Ben Slater <be...@instaclustr.com>.
In the normal, happy case the replica would be written to the third node at
the time of the write. However, if they third node happened to be down or
very overloaded at the time of the write (your step 3) the write would
still be reported to the client as successful. Even if the 3rd node is up
again before nodes 1 and 2 die, hints may have expired by that time or may
not finish replaying either due to load (which is sort of the scenario you
outlined) or just not enough time. You’re only really guaranteed all three
replicas are there if a repair runs successful between the initial write
and the two nodes dieing (although it’s very likely there will be three
replicas from the start if the cluster is in a healthy state at the time of
the write).
Cheers
Ben
---
*Ben Slater**Chief Product Officer*
<https://www.instaclustr.com/platform/>
<https://www.facebook.com/instaclustr> <https://twitter.com/instaclustr>
<https://www.linkedin.com/company/instaclustr>
Read our latest technical blog posts here
<https://www.instaclustr.com/blog/>.
This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
and Instaclustr Inc (USA).
This email and any attachments may contain confidential and legally
privileged information. If you are not the intended recipient, do not copy
or disclose its content, but please reply to this email immediately and
highlight the error to the sender and then immediately delete the message.
On Fri, 3 May 2019 at 23:19, Fred Habash <fm...@gmail.com> wrote:
> Thank you all.
>
> So, please, bear with me for a second. I'm trying to figure out how can
> data be totally lost under the above circumstances when nodes die in two
> out of three racks.
>
> You stated ....
> "the replica may or many not have made its way to the third node '. Why
> 'may not'?
>
> This is what I came up with ...
>
> 1. Write goes to coordinator in rac1
> 2. Local coordinator submits RF = 3 writes to all racks
> 3. Two nodes in rack 1 and 2 ack the write. Client is happy
> 4. Nodes massacre happens in rack 1 & 2 (infrastructure event)
> 5. Nodes in rack3 witness in increased in load as a result of cluster
> shrinking
> 6. Coordinator in rack1 stores HH for the row for rack3 (either
> coordinator slows down or rack3 node is overloaded).
> 7. Eventually, coordinator in rack1 dies and HH's are lost.
> 8. The row that was once ack'd to the app, is now gone gone.
>
> Plausible?
>
>
> On Thu, May 2, 2019 at 8:23 PM Avinash Mandava <av...@vorstella.com>
> wrote:
>
>> Good catch, misread the detail.
>>
>> On Thu, May 2, 2019 at 4:56 PM Ben Slater <be...@instaclustr.com>
>> wrote:
>>
>>> Reading more carefully, it could actually be either way: quorum requires
>>> that a majority of nodes complete and ack the write but still aims to write
>>> to RF nodes (with the last replicate either written immediately or
>>> eventually via hints or repairs). So, in the scenario outlined the replica
>>> may or many not have made its way to the third node by the time the first
>>> two replicas are lost. If there is a replica on the third node it can be
>>> recovered to the other two nodes by either rebuild (actually replace) or
>>> repair.
>>>
>>> Cheers
>>> Ben
>>>
>>> ---
>>>
>>>
>>> *Ben Slater**Chief Product Officer*
>>>
>>> <https://www.instaclustr.com/platform/>
>>>
>>> <https://www.facebook.com/instaclustr>
>>> <https://twitter.com/instaclustr>
>>> <https://www.linkedin.com/company/instaclustr>
>>>
>>> Read our latest technical blog posts here
>>> <https://www.instaclustr.com/blog/>.
>>>
>>> This email has been sent on behalf of Instaclustr Pty. Limited
>>> (Australia) and Instaclustr Inc (USA).
>>>
>>> This email and any attachments may contain confidential and legally
>>> privileged information. If you are not the intended recipient, do not copy
>>> or disclose its content, but please reply to this email immediately and
>>> highlight the error to the sender and then immediately delete the message.
>>>
>>>
>>> On Fri, 3 May 2019 at 09:33, Avinash Mandava <av...@vorstella.com>
>>> wrote:
>>>
>>>> In scenario 2 it's lost, if both nodes die and get replaced entirely
>>>> there's no history anywhere that the write ever happened, as it wouldn't be
>>>> in commitlog, memtable, or sstable in node 3. Surviving that failure
>>>> scenario of two nodes with same data simultaneously failing requires upping
>>>> CL or RF, or spreading across 3 racks, if the situation you're trying to
>>>> avoid is rack failure (which im guessing it is from the question setup)
>>>>
>>>> On Thu, May 2, 2019 at 2:25 PM Ben Slater <be...@instaclustr.com>
>>>> wrote:
>>>>
>>>>> In scenario 2, if the row has been written to node 3 it will be
>>>>> replaced on the other nodes via rebuild or repair.
>>>>>
>>>>> ---
>>>>>
>>>>>
>>>>> *Ben Slater**Chief Product Officer*
>>>>>
>>>>> <https://www.instaclustr.com/platform/>
>>>>>
>>>>> <https://www.facebook.com/instaclustr>
>>>>> <https://twitter.com/instaclustr>
>>>>> <https://www.linkedin.com/company/instaclustr>
>>>>>
>>>>> Read our latest technical blog posts here
>>>>> <https://www.instaclustr.com/blog/>.
>>>>>
>>>>> This email has been sent on behalf of Instaclustr Pty. Limited
>>>>> (Australia) and Instaclustr Inc (USA).
>>>>>
>>>>> This email and any attachments may contain confidential and legally
>>>>> privileged information. If you are not the intended recipient, do not copy
>>>>> or disclose its content, but please reply to this email immediately and
>>>>> highlight the error to the sender and then immediately delete the message.
>>>>>
>>>>>
>>>>> On Fri, 3 May 2019 at 00:54, Fd Habash <fm...@gmail.com> wrote:
>>>>>
>>>>>> C*: 2.2.8
>>>>>>
>>>>>> Write CL = LQ
>>>>>>
>>>>>> Kspace RF = 3
>>>>>>
>>>>>> Three racks
>>>>>>
>>>>>>
>>>>>>
>>>>>> A write gets received by node 1 in rack 1 at above specs. Node 1
>>>>>> (rack1) & node 2 (rack2) acknowledge it to the client.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Within some unit of time, node 1 & 2 die. Either ….
>>>>>>
>>>>>> - Scenario 1: C* process death: Row did not make it to sstable
>>>>>> (it is in commit log & was in memtable)
>>>>>> - Scenario 2: Node death: row may be have made to sstable, but
>>>>>> nodes are gone (will have to bootstrap to replace).
>>>>>>
>>>>>>
>>>>>>
>>>>>> Scenario 1: Row is not lost because once C* is restarted, commit log
>>>>>> should replay the mutation.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Scenario 2: row is gone forever? If these two nodes are replaced via
>>>>>> bootstrapping, will they ever get the row back from node 3 (rack3) if the
>>>>>> write ever made it there?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ----------------
>>>>>> Thank you
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> --
>>>> www.vorstella.com
>>>> 408 691 8402
>>>>
>>>
>>
>> --
>> www.vorstella.com
>> 408 691 8402
>>
>
>
> --
>
> ----------------------------------------
> Thank you
>
>
>
Re: CL=LQ, RF=3: Can a Write be Lost If Two Nodes ACK'ing it Die
Posted by Fred Habash <fm...@gmail.com>.
Thank you all.
So, please, bear with me for a second. I'm trying to figure out how can
data be totally lost under the above circumstances when nodes die in two
out of three racks.
You stated ....
"the replica may or many not have made its way to the third node '. Why
'may not'?
This is what I came up with ...
1. Write goes to coordinator in rac1
2. Local coordinator submits RF = 3 writes to all racks
3. Two nodes in rack 1 and 2 ack the write. Client is happy
4. Nodes massacre happens in rack 1 & 2 (infrastructure event)
5. Nodes in rack3 witness in increased in load as a result of cluster
shrinking
6. Coordinator in rack1 stores HH for the row for rack3 (either coordinator
slows down or rack3 node is overloaded).
7. Eventually, coordinator in rack1 dies and HH's are lost.
8. The row that was once ack'd to the app, is now gone gone.
Plausible?
On Thu, May 2, 2019 at 8:23 PM Avinash Mandava <av...@vorstella.com>
wrote:
> Good catch, misread the detail.
>
> On Thu, May 2, 2019 at 4:56 PM Ben Slater <be...@instaclustr.com>
> wrote:
>
>> Reading more carefully, it could actually be either way: quorum requires
>> that a majority of nodes complete and ack the write but still aims to write
>> to RF nodes (with the last replicate either written immediately or
>> eventually via hints or repairs). So, in the scenario outlined the replica
>> may or many not have made its way to the third node by the time the first
>> two replicas are lost. If there is a replica on the third node it can be
>> recovered to the other two nodes by either rebuild (actually replace) or
>> repair.
>>
>> Cheers
>> Ben
>>
>> ---
>>
>>
>> *Ben Slater**Chief Product Officer*
>>
>> <https://www.instaclustr.com/platform/>
>>
>> <https://www.facebook.com/instaclustr>
>> <https://twitter.com/instaclustr>
>> <https://www.linkedin.com/company/instaclustr>
>>
>> Read our latest technical blog posts here
>> <https://www.instaclustr.com/blog/>.
>>
>> This email has been sent on behalf of Instaclustr Pty. Limited
>> (Australia) and Instaclustr Inc (USA).
>>
>> This email and any attachments may contain confidential and legally
>> privileged information. If you are not the intended recipient, do not copy
>> or disclose its content, but please reply to this email immediately and
>> highlight the error to the sender and then immediately delete the message.
>>
>>
>> On Fri, 3 May 2019 at 09:33, Avinash Mandava <av...@vorstella.com>
>> wrote:
>>
>>> In scenario 2 it's lost, if both nodes die and get replaced entirely
>>> there's no history anywhere that the write ever happened, as it wouldn't be
>>> in commitlog, memtable, or sstable in node 3. Surviving that failure
>>> scenario of two nodes with same data simultaneously failing requires upping
>>> CL or RF, or spreading across 3 racks, if the situation you're trying to
>>> avoid is rack failure (which im guessing it is from the question setup)
>>>
>>> On Thu, May 2, 2019 at 2:25 PM Ben Slater <be...@instaclustr.com>
>>> wrote:
>>>
>>>> In scenario 2, if the row has been written to node 3 it will be
>>>> replaced on the other nodes via rebuild or repair.
>>>>
>>>> ---
>>>>
>>>>
>>>> *Ben Slater**Chief Product Officer*
>>>>
>>>> <https://www.instaclustr.com/platform/>
>>>>
>>>> <https://www.facebook.com/instaclustr>
>>>> <https://twitter.com/instaclustr>
>>>> <https://www.linkedin.com/company/instaclustr>
>>>>
>>>> Read our latest technical blog posts here
>>>> <https://www.instaclustr.com/blog/>.
>>>>
>>>> This email has been sent on behalf of Instaclustr Pty. Limited
>>>> (Australia) and Instaclustr Inc (USA).
>>>>
>>>> This email and any attachments may contain confidential and legally
>>>> privileged information. If you are not the intended recipient, do not copy
>>>> or disclose its content, but please reply to this email immediately and
>>>> highlight the error to the sender and then immediately delete the message.
>>>>
>>>>
>>>> On Fri, 3 May 2019 at 00:54, Fd Habash <fm...@gmail.com> wrote:
>>>>
>>>>> C*: 2.2.8
>>>>>
>>>>> Write CL = LQ
>>>>>
>>>>> Kspace RF = 3
>>>>>
>>>>> Three racks
>>>>>
>>>>>
>>>>>
>>>>> A write gets received by node 1 in rack 1 at above specs. Node 1
>>>>> (rack1) & node 2 (rack2) acknowledge it to the client.
>>>>>
>>>>>
>>>>>
>>>>> Within some unit of time, node 1 & 2 die. Either ….
>>>>>
>>>>> - Scenario 1: C* process death: Row did not make it to sstable (it
>>>>> is in commit log & was in memtable)
>>>>> - Scenario 2: Node death: row may be have made to sstable, but
>>>>> nodes are gone (will have to bootstrap to replace).
>>>>>
>>>>>
>>>>>
>>>>> Scenario 1: Row is not lost because once C* is restarted, commit log
>>>>> should replay the mutation.
>>>>>
>>>>>
>>>>>
>>>>> Scenario 2: row is gone forever? If these two nodes are replaced via
>>>>> bootstrapping, will they ever get the row back from node 3 (rack3) if the
>>>>> write ever made it there?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ----------------
>>>>> Thank you
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>> --
>>> www.vorstella.com
>>> 408 691 8402
>>>
>>
>
> --
> www.vorstella.com
> 408 691 8402
>
--
----------------------------------------
Thank you
Re: CL=LQ, RF=3: Can a Write be Lost If Two Nodes ACK'ing it Die
Posted by Avinash Mandava <av...@vorstella.com>.
Good catch, misread the detail.
On Thu, May 2, 2019 at 4:56 PM Ben Slater <be...@instaclustr.com>
wrote:
> Reading more carefully, it could actually be either way: quorum requires
> that a majority of nodes complete and ack the write but still aims to write
> to RF nodes (with the last replicate either written immediately or
> eventually via hints or repairs). So, in the scenario outlined the replica
> may or many not have made its way to the third node by the time the first
> two replicas are lost. If there is a replica on the third node it can be
> recovered to the other two nodes by either rebuild (actually replace) or
> repair.
>
> Cheers
> Ben
>
> ---
>
>
> *Ben Slater**Chief Product Officer*
>
> <https://www.instaclustr.com/platform/>
>
> <https://www.facebook.com/instaclustr> <https://twitter.com/instaclustr>
> <https://www.linkedin.com/company/instaclustr>
>
> Read our latest technical blog posts here
> <https://www.instaclustr.com/blog/>.
>
> This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
> and Instaclustr Inc (USA).
>
> This email and any attachments may contain confidential and legally
> privileged information. If you are not the intended recipient, do not copy
> or disclose its content, but please reply to this email immediately and
> highlight the error to the sender and then immediately delete the message.
>
>
> On Fri, 3 May 2019 at 09:33, Avinash Mandava <av...@vorstella.com>
> wrote:
>
>> In scenario 2 it's lost, if both nodes die and get replaced entirely
>> there's no history anywhere that the write ever happened, as it wouldn't be
>> in commitlog, memtable, or sstable in node 3. Surviving that failure
>> scenario of two nodes with same data simultaneously failing requires upping
>> CL or RF, or spreading across 3 racks, if the situation you're trying to
>> avoid is rack failure (which im guessing it is from the question setup)
>>
>> On Thu, May 2, 2019 at 2:25 PM Ben Slater <be...@instaclustr.com>
>> wrote:
>>
>>> In scenario 2, if the row has been written to node 3 it will be replaced
>>> on the other nodes via rebuild or repair.
>>>
>>> ---
>>>
>>>
>>> *Ben Slater**Chief Product Officer*
>>>
>>> <https://www.instaclustr.com/platform/>
>>>
>>> <https://www.facebook.com/instaclustr>
>>> <https://twitter.com/instaclustr>
>>> <https://www.linkedin.com/company/instaclustr>
>>>
>>> Read our latest technical blog posts here
>>> <https://www.instaclustr.com/blog/>.
>>>
>>> This email has been sent on behalf of Instaclustr Pty. Limited
>>> (Australia) and Instaclustr Inc (USA).
>>>
>>> This email and any attachments may contain confidential and legally
>>> privileged information. If you are not the intended recipient, do not copy
>>> or disclose its content, but please reply to this email immediately and
>>> highlight the error to the sender and then immediately delete the message.
>>>
>>>
>>> On Fri, 3 May 2019 at 00:54, Fd Habash <fm...@gmail.com> wrote:
>>>
>>>> C*: 2.2.8
>>>>
>>>> Write CL = LQ
>>>>
>>>> Kspace RF = 3
>>>>
>>>> Three racks
>>>>
>>>>
>>>>
>>>> A write gets received by node 1 in rack 1 at above specs. Node 1
>>>> (rack1) & node 2 (rack2) acknowledge it to the client.
>>>>
>>>>
>>>>
>>>> Within some unit of time, node 1 & 2 die. Either ….
>>>>
>>>> - Scenario 1: C* process death: Row did not make it to sstable (it
>>>> is in commit log & was in memtable)
>>>> - Scenario 2: Node death: row may be have made to sstable, but
>>>> nodes are gone (will have to bootstrap to replace).
>>>>
>>>>
>>>>
>>>> Scenario 1: Row is not lost because once C* is restarted, commit log
>>>> should replay the mutation.
>>>>
>>>>
>>>>
>>>> Scenario 2: row is gone forever? If these two nodes are replaced via
>>>> bootstrapping, will they ever get the row back from node 3 (rack3) if the
>>>> write ever made it there?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ----------------
>>>> Thank you
>>>>
>>>>
>>>>
>>>
>>
>> --
>> www.vorstella.com
>> 408 691 8402
>>
>
--
www.vorstella.com
408 691 8402
Re: CL=LQ, RF=3: Can a Write be Lost If Two Nodes ACK'ing it Die
Posted by Ben Slater <be...@instaclustr.com>.
Reading more carefully, it could actually be either way: quorum requires
that a majority of nodes complete and ack the write but still aims to write
to RF nodes (with the last replicate either written immediately or
eventually via hints or repairs). So, in the scenario outlined the replica
may or many not have made its way to the third node by the time the first
two replicas are lost. If there is a replica on the third node it can be
recovered to the other two nodes by either rebuild (actually replace) or
repair.
Cheers
Ben
---
*Ben Slater**Chief Product Officer*
<https://www.instaclustr.com/platform/>
<https://www.facebook.com/instaclustr> <https://twitter.com/instaclustr>
<https://www.linkedin.com/company/instaclustr>
Read our latest technical blog posts here
<https://www.instaclustr.com/blog/>.
This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
and Instaclustr Inc (USA).
This email and any attachments may contain confidential and legally
privileged information. If you are not the intended recipient, do not copy
or disclose its content, but please reply to this email immediately and
highlight the error to the sender and then immediately delete the message.
On Fri, 3 May 2019 at 09:33, Avinash Mandava <av...@vorstella.com> wrote:
> In scenario 2 it's lost, if both nodes die and get replaced entirely
> there's no history anywhere that the write ever happened, as it wouldn't be
> in commitlog, memtable, or sstable in node 3. Surviving that failure
> scenario of two nodes with same data simultaneously failing requires upping
> CL or RF, or spreading across 3 racks, if the situation you're trying to
> avoid is rack failure (which im guessing it is from the question setup)
>
> On Thu, May 2, 2019 at 2:25 PM Ben Slater <be...@instaclustr.com>
> wrote:
>
>> In scenario 2, if the row has been written to node 3 it will be replaced
>> on the other nodes via rebuild or repair.
>>
>> ---
>>
>>
>> *Ben Slater**Chief Product Officer*
>>
>> <https://www.instaclustr.com/platform/>
>>
>> <https://www.facebook.com/instaclustr>
>> <https://twitter.com/instaclustr>
>> <https://www.linkedin.com/company/instaclustr>
>>
>> Read our latest technical blog posts here
>> <https://www.instaclustr.com/blog/>.
>>
>> This email has been sent on behalf of Instaclustr Pty. Limited
>> (Australia) and Instaclustr Inc (USA).
>>
>> This email and any attachments may contain confidential and legally
>> privileged information. If you are not the intended recipient, do not copy
>> or disclose its content, but please reply to this email immediately and
>> highlight the error to the sender and then immediately delete the message.
>>
>>
>> On Fri, 3 May 2019 at 00:54, Fd Habash <fm...@gmail.com> wrote:
>>
>>> C*: 2.2.8
>>>
>>> Write CL = LQ
>>>
>>> Kspace RF = 3
>>>
>>> Three racks
>>>
>>>
>>>
>>> A write gets received by node 1 in rack 1 at above specs. Node 1 (rack1)
>>> & node 2 (rack2) acknowledge it to the client.
>>>
>>>
>>>
>>> Within some unit of time, node 1 & 2 die. Either ….
>>>
>>> - Scenario 1: C* process death: Row did not make it to sstable (it
>>> is in commit log & was in memtable)
>>> - Scenario 2: Node death: row may be have made to sstable, but nodes
>>> are gone (will have to bootstrap to replace).
>>>
>>>
>>>
>>> Scenario 1: Row is not lost because once C* is restarted, commit log
>>> should replay the mutation.
>>>
>>>
>>>
>>> Scenario 2: row is gone forever? If these two nodes are replaced via
>>> bootstrapping, will they ever get the row back from node 3 (rack3) if the
>>> write ever made it there?
>>>
>>>
>>>
>>>
>>>
>>> ----------------
>>> Thank you
>>>
>>>
>>>
>>
>
> --
> www.vorstella.com
> 408 691 8402
>
Re: CL=LQ, RF=3: Can a Write be Lost If Two Nodes ACK'ing it Die
Posted by Avinash Mandava <av...@vorstella.com>.
In scenario 2 it's lost, if both nodes die and get replaced entirely
there's no history anywhere that the write ever happened, as it wouldn't be
in commitlog, memtable, or sstable in node 3. Surviving that failure
scenario of two nodes with same data simultaneously failing requires upping
CL or RF, or spreading across 3 racks, if the situation you're trying to
avoid is rack failure (which im guessing it is from the question setup)
On Thu, May 2, 2019 at 2:25 PM Ben Slater <be...@instaclustr.com>
wrote:
> In scenario 2, if the row has been written to node 3 it will be replaced
> on the other nodes via rebuild or repair.
>
> ---
>
>
> *Ben Slater**Chief Product Officer*
>
> <https://www.instaclustr.com/platform/>
>
> <https://www.facebook.com/instaclustr> <https://twitter.com/instaclustr>
> <https://www.linkedin.com/company/instaclustr>
>
> Read our latest technical blog posts here
> <https://www.instaclustr.com/blog/>.
>
> This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
> and Instaclustr Inc (USA).
>
> This email and any attachments may contain confidential and legally
> privileged information. If you are not the intended recipient, do not copy
> or disclose its content, but please reply to this email immediately and
> highlight the error to the sender and then immediately delete the message.
>
>
> On Fri, 3 May 2019 at 00:54, Fd Habash <fm...@gmail.com> wrote:
>
>> C*: 2.2.8
>>
>> Write CL = LQ
>>
>> Kspace RF = 3
>>
>> Three racks
>>
>>
>>
>> A write gets received by node 1 in rack 1 at above specs. Node 1 (rack1)
>> & node 2 (rack2) acknowledge it to the client.
>>
>>
>>
>> Within some unit of time, node 1 & 2 die. Either ….
>>
>> - Scenario 1: C* process death: Row did not make it to sstable (it is
>> in commit log & was in memtable)
>> - Scenario 2: Node death: row may be have made to sstable, but nodes
>> are gone (will have to bootstrap to replace).
>>
>>
>>
>> Scenario 1: Row is not lost because once C* is restarted, commit log
>> should replay the mutation.
>>
>>
>>
>> Scenario 2: row is gone forever? If these two nodes are replaced via
>> bootstrapping, will they ever get the row back from node 3 (rack3) if the
>> write ever made it there?
>>
>>
>>
>>
>>
>> ----------------
>> Thank you
>>
>>
>>
>
--
www.vorstella.com
408 691 8402
Re: CL=LQ, RF=3: Can a Write be Lost If Two Nodes ACK'ing it Die
Posted by Ben Slater <be...@instaclustr.com>.
In scenario 2, if the row has been written to node 3 it will be replaced on
the other nodes via rebuild or repair.
---
*Ben Slater**Chief Product Officer*
<https://www.instaclustr.com/platform/>
<https://www.facebook.com/instaclustr> <https://twitter.com/instaclustr>
<https://www.linkedin.com/company/instaclustr>
Read our latest technical blog posts here
<https://www.instaclustr.com/blog/>.
This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
and Instaclustr Inc (USA).
This email and any attachments may contain confidential and legally
privileged information. If you are not the intended recipient, do not copy
or disclose its content, but please reply to this email immediately and
highlight the error to the sender and then immediately delete the message.
On Fri, 3 May 2019 at 00:54, Fd Habash <fm...@gmail.com> wrote:
> C*: 2.2.8
>
> Write CL = LQ
>
> Kspace RF = 3
>
> Three racks
>
>
>
> A write gets received by node 1 in rack 1 at above specs. Node 1 (rack1) &
> node 2 (rack2) acknowledge it to the client.
>
>
>
> Within some unit of time, node 1 & 2 die. Either ….
>
> - Scenario 1: C* process death: Row did not make it to sstable (it is
> in commit log & was in memtable)
> - Scenario 2: Node death: row may be have made to sstable, but nodes
> are gone (will have to bootstrap to replace).
>
>
>
> Scenario 1: Row is not lost because once C* is restarted, commit log
> should replay the mutation.
>
>
>
> Scenario 2: row is gone forever? If these two nodes are replaced via
> bootstrapping, will they ever get the row back from node 3 (rack3) if the
> write ever made it there?
>
>
>
>
>
> ----------------
> Thank you
>
>
>