You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ratis.apache.org by Asad Awadia <as...@gmail.com> on 2022/07/14 13:57:34 UTC
Split brain
Hello,
How does ratis handle split brain? (Using the term counter?)
That is two nodes are not aware of each other are both taking writes and
now are merged into one cluster with one leader.
How are the writes reconciled? Whose work is overwritten? Or is the cluster
now in a bad state in need of manual intervention?
Regards,
Asad
Re: Split brain
Posted by Tsz Wo Sze <sz...@gmail.com>.
I see your question now. In this case, Server 1 and Server 2 were from two
different raft groups. They cannot be merged into a single group. Such an
operation is unsupported.
Tsz-Wo
On Thu, Jul 14, 2022 at 10:56 AM Asad Awadia <as...@gmail.com> wrote:
> Maybe I am not understanding.
>
> Server 1 starts up as a single server raft cluster taking writes. (N/2 +1)
> is 1 so it has majority and can commit writes.
>
> Server 2 starts up as a single server raft cluster taking writes. (N/2 +1)
> is 1 so it has majority and can commit writes.
>
> Both servers are unaware of each other.
>
> Then we merge the two servers into a single raft cluster.
>
> What happens in this scenario?
>
> On Thu., Jul. 14, 2022, 1:30 p.m. Tsz Wo Sze, <sz...@gmail.com> wrote:
>
>> When the number of nodes n is even, a majority requires (n/2 + 1) nodes.
>> When there are two nodes, it requires two nodes in order to have a majority.
>>
>> Tsz-Wo
>>
>> On Thu, Jul 14, 2022 at 10:07 AM Asad Awadia <as...@gmail.com>
>> wrote:
>>
>>> So that's the thing. Both have a majority. Since there are two
>>> 'clusters' of 1 node each. There are only 2 nodes. ( i know should be odd)
>>>
>>> So both sides think they have majority right? What then?
>>>
>>> Regards,
>>> Asad
>>>
>>>
>>> On Thu., Jul. 14, 2022, 11:49 a.m. Tsz Wo Sze, <sz...@gmail.com>
>>> wrote:
>>>
>>>> Hi Asad,
>>>>
>>>> Raft uses majority for committing transactions. When two nodes are not
>>>> aware of each other are both taking writes, at most one of them can get a
>>>> majority. The one(s) without majority cannot commit any transactions.
>>>> When these two nodes are merged, if one of them had the majority
>>>> previously, the one without majority won't be voted as the leader so that
>>>> its (non-committed) transactions will be dropped.
>>>>
>>>> For the details, please see the Raft paper
>>>> https://raft.github.io/raft.pdf . It should be discussed in Section
>>>> 5.4 Safety.
>>>>
>>>> Tsz-Wo
>>>>
>>>>
>>>>
>>>> On Thu, Jul 14, 2022 at 6:57 AM Asad Awadia <as...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> How does ratis handle split brain? (Using the term counter?)
>>>>>
>>>>> That is two nodes are not aware of each other are both taking writes
>>>>> and now are merged into one cluster with one leader.
>>>>>
>>>>> How are the writes reconciled? Whose work is overwritten? Or is the
>>>>> cluster now in a bad state in need of manual intervention?
>>>>>
>>>>> Regards,
>>>>> Asad
>>>>>
>>>>
Re: Split brain
Posted by Asad Awadia <as...@gmail.com>.
Maybe I am not understanding.
Server 1 starts up as a single server raft cluster taking writes. (N/2 +1)
is 1 so it has majority and can commit writes.
Server 2 starts up as a single server raft cluster taking writes. (N/2 +1)
is 1 so it has majority and can commit writes.
Both servers are unaware of each other.
Then we merge the two servers into a single raft cluster.
What happens in this scenario?
On Thu., Jul. 14, 2022, 1:30 p.m. Tsz Wo Sze, <sz...@gmail.com> wrote:
> When the number of nodes n is even, a majority requires (n/2 + 1) nodes.
> When there are two nodes, it requires two nodes in order to have a majority.
>
> Tsz-Wo
>
> On Thu, Jul 14, 2022 at 10:07 AM Asad Awadia <as...@gmail.com> wrote:
>
>> So that's the thing. Both have a majority. Since there are two 'clusters'
>> of 1 node each. There are only 2 nodes. ( i know should be odd)
>>
>> So both sides think they have majority right? What then?
>>
>> Regards,
>> Asad
>>
>>
>> On Thu., Jul. 14, 2022, 11:49 a.m. Tsz Wo Sze, <sz...@gmail.com>
>> wrote:
>>
>>> Hi Asad,
>>>
>>> Raft uses majority for committing transactions. When two nodes are not
>>> aware of each other are both taking writes, at most one of them can get a
>>> majority. The one(s) without majority cannot commit any transactions.
>>> When these two nodes are merged, if one of them had the majority
>>> previously, the one without majority won't be voted as the leader so that
>>> its (non-committed) transactions will be dropped.
>>>
>>> For the details, please see the Raft paper
>>> https://raft.github.io/raft.pdf . It should be discussed in Section
>>> 5.4 Safety.
>>>
>>> Tsz-Wo
>>>
>>>
>>>
>>> On Thu, Jul 14, 2022 at 6:57 AM Asad Awadia <as...@gmail.com>
>>> wrote:
>>>
>>>> Hello,
>>>>
>>>> How does ratis handle split brain? (Using the term counter?)
>>>>
>>>> That is two nodes are not aware of each other are both taking writes
>>>> and now are merged into one cluster with one leader.
>>>>
>>>> How are the writes reconciled? Whose work is overwritten? Or is the
>>>> cluster now in a bad state in need of manual intervention?
>>>>
>>>> Regards,
>>>> Asad
>>>>
>>>
Re: Split brain
Posted by Tsz Wo Sze <sz...@gmail.com>.
When the number of nodes n is even, a majority requires (n/2 + 1) nodes.
When there are two nodes, it requires two nodes in order to have a majority.
Tsz-Wo
On Thu, Jul 14, 2022 at 10:07 AM Asad Awadia <as...@gmail.com> wrote:
> So that's the thing. Both have a majority. Since there are two 'clusters'
> of 1 node each. There are only 2 nodes. ( i know should be odd)
>
> So both sides think they have majority right? What then?
>
> Regards,
> Asad
>
>
> On Thu., Jul. 14, 2022, 11:49 a.m. Tsz Wo Sze, <sz...@gmail.com> wrote:
>
>> Hi Asad,
>>
>> Raft uses majority for committing transactions. When two nodes are not
>> aware of each other are both taking writes, at most one of them can get a
>> majority. The one(s) without majority cannot commit any transactions.
>> When these two nodes are merged, if one of them had the majority
>> previously, the one without majority won't be voted as the leader so that
>> its (non-committed) transactions will be dropped.
>>
>> For the details, please see the Raft paper
>> https://raft.github.io/raft.pdf . It should be discussed in Section 5.4
>> Safety.
>>
>> Tsz-Wo
>>
>>
>>
>> On Thu, Jul 14, 2022 at 6:57 AM Asad Awadia <as...@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> How does ratis handle split brain? (Using the term counter?)
>>>
>>> That is two nodes are not aware of each other are both taking writes and
>>> now are merged into one cluster with one leader.
>>>
>>> How are the writes reconciled? Whose work is overwritten? Or is the
>>> cluster now in a bad state in need of manual intervention?
>>>
>>> Regards,
>>> Asad
>>>
>>
Re: Split brain
Posted by Asad Awadia <as...@gmail.com>.
So that's the thing. Both have a majority. Since there are two 'clusters'
of 1 node each. There are only 2 nodes. ( i know should be odd)
So both sides think they have majority right? What then?
Regards,
Asad
On Thu., Jul. 14, 2022, 11:49 a.m. Tsz Wo Sze, <sz...@gmail.com> wrote:
> Hi Asad,
>
> Raft uses majority for committing transactions. When two nodes are not
> aware of each other are both taking writes, at most one of them can get a
> majority. The one(s) without majority cannot commit any transactions.
> When these two nodes are merged, if one of them had the majority
> previously, the one without majority won't be voted as the leader so that
> its (non-committed) transactions will be dropped.
>
> For the details, please see the Raft paper https://raft.github.io/raft.pdf
> . It should be discussed in Section 5.4 Safety.
>
> Tsz-Wo
>
>
>
> On Thu, Jul 14, 2022 at 6:57 AM Asad Awadia <as...@gmail.com> wrote:
>
>> Hello,
>>
>> How does ratis handle split brain? (Using the term counter?)
>>
>> That is two nodes are not aware of each other are both taking writes and
>> now are merged into one cluster with one leader.
>>
>> How are the writes reconciled? Whose work is overwritten? Or is the
>> cluster now in a bad state in need of manual intervention?
>>
>> Regards,
>> Asad
>>
>
Re: Split brain
Posted by Tsz Wo Sze <sz...@gmail.com>.
Hi Asad,
Raft uses majority for committing transactions. When two nodes are not
aware of each other are both taking writes, at most one of them can get a
majority. The one(s) without majority cannot commit any transactions.
When these two nodes are merged, if one of them had the majority
previously, the one without majority won't be voted as the leader so that
its (non-committed) transactions will be dropped.
For the details, please see the Raft paper https://raft.github.io/raft.pdf
. It should be discussed in Section 5.4 Safety.
Tsz-Wo
On Thu, Jul 14, 2022 at 6:57 AM Asad Awadia <as...@gmail.com> wrote:
> Hello,
>
> How does ratis handle split brain? (Using the term counter?)
>
> That is two nodes are not aware of each other are both taking writes and
> now are merged into one cluster with one leader.
>
> How are the writes reconciled? Whose work is overwritten? Or is the
> cluster now in a bad state in need of manual intervention?
>
> Regards,
> Asad
>