You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ratis.apache.org by Asad Awadia <as...@gmail.com> on 2023/06/20 15:44:58 UTC
Raft with 2 nodes
A big part of raft is running an odd number of servers
But we are scaling nodes 1 at a time and there are a lot of times where we
have only 2 nodes in the cluster
If the new node goes down and comes back up is it possible it can become
the leader even though it has less transactions than the existing node?
Since there are only 2 nodes - is there a guarentee that always the node
that has the highest transaction id will be elected the leader?
We are seeing
0. Exisiting 1 node cluster with node 1 as the leader
1. New node 2 gets added
2. Node 1 and 2 are healthy
3. Node 2 goes down
4. Node 1 is in candidate state
5. Node 2 comes back
6. Node 2 becomes leader
Shouldn't node 1 be always the leader?
Re: Raft with 2 nodes
Posted by Tsz Wo Sze <sz...@gmail.com>.
> Is it possible for the new node to somehow have a higher term counter but
lower log id and thus then it becomes the leader?
>
> > ..., B could update the term and then become the leader. ...
Oops, I was wrong. B can keep updating the term in requestVote (say, the
term originally was 3. It started multiple leader elections and updated
the term to 10) but the term in the last log entry of B remains unchanged
(still 3). So A won't vote for B if A has more log entries. B cannot
become the leader unless (1) the logs of both A and B are the same and (2)
B starts a leader election at that moment.
Tsz-Wo
On Thu, Jun 22, 2023 at 10:18 PM Asad Awadia <as...@gmail.com> wrote:
> During a network partition between Node A and B
>
> Can node B increase its term so much that when the partition ends - it
> becomes the leader even though its actual log is behind node A ?
>
> On Wed, Jun 21, 2023 at 1:14 PM Tsz Wo Sze <sz...@gmail.com> wrote:
>
>> Hi Asad,
>>
>> You are right that, after the new node B joined the group, B could
>> update the term and then become the leader. When the old leader A joins
>> the new leader B, A will truncate the uncommitted transactions.
>>
>> Tsz-Wo
>>
>> On Wed, Jun 21, 2023 at 11:20 PM Asad Awadia <as...@gmail.com>
>> wrote:
>>
>>> Is it possible for the new node to somehow have a higher term counter
>>> but lower log id and thus then it becomes the leader?
>>>
>>> On Tue, Jun 20, 2023, 11:03 p.m. Tsz Wo Sze <sz...@gmail.com> wrote:
>>>
>>>> > If the new node goes down and comes back up is it possible it can
>>>> become the leader even though it has less transactions than the
>>>> existing node?
>>>>
>>>> No, it cannot become the leader since the other node won't vote for it.
>>>>
>>>> > ... is there a guarentee that always the node that has the highest
>>>> transaction id will be elected the leader?
>>>>
>>>> For two nodes, yes. A vote is granted only if the candidate has an
>>>> equal or higher transaction index. The node with a higher index won't vote
>>>> for the other node.
>>>>
>>>> A group with two nodes means that any transactions must be replicated
>>>> to both nodes before it is committed. The other node could become the
>>>> leader only if they both have the same transaction index.
>>>>
>>>> For more than two nodes, the node with the highest transaction index is
>>>> not necessarily elected the leader -- A previous leader may have extra
>>>> uncommitted transactions. In the next election, any node with a
>>>> transaction index higher or equal than the majority of the nodes can become
>>>> the leader since the nodes with a lower or equal index will vote for it.
>>>> In such a case, the previous leader will truncate the uncommitted
>>>> transactions when it joins the new leader.
>>>>
>>>> Thanks for your questions!
>>>>
>>>> Tsz-Wo
>>>>
>>>>
>>>> On Tue, Jun 20, 2023 at 11:45 PM Asad Awadia <as...@gmail.com>
>>>> wrote:
>>>>
>>>>> A big part of raft is running an odd number of servers
>>>>>
>>>>> But we are scaling nodes 1 at a time and there are a lot of times
>>>>> where we have only 2 nodes in the cluster
>>>>>
>>>>> If the new node goes down and comes back up is it possible it can
>>>>> become the leader even though it has less transactions than the
>>>>> existing node?
>>>>>
>>>>> Since there are only 2 nodes - is there a guarentee that always the
>>>>> node that has the highest transaction id will be elected the leader?
>>>>>
>>>>> We are seeing
>>>>> 0. Exisiting 1 node cluster with node 1 as the leader
>>>>> 1. New node 2 gets added
>>>>> 2. Node 1 and 2 are healthy
>>>>> 3. Node 2 goes down
>>>>> 4. Node 1 is in candidate state
>>>>> 5. Node 2 comes back
>>>>> 6. Node 2 becomes leader
>>>>>
>>>>> Shouldn't node 1 be always the leader?
>>>>>
>>>>
Re: Raft with 2 nodes
Posted by Asad Awadia <as...@gmail.com>.
During a network partition between Node A and B
Can node B increase its term so much that when the partition ends - it
becomes the leader even though its actual log is behind node A ?
On Wed, Jun 21, 2023 at 1:14 PM Tsz Wo Sze <sz...@gmail.com> wrote:
> Hi Asad,
>
> You are right that, after the new node B joined the group, B could
> update the term and then become the leader. When the old leader A joins
> the new leader B, A will truncate the uncommitted transactions.
>
> Tsz-Wo
>
> On Wed, Jun 21, 2023 at 11:20 PM Asad Awadia <as...@gmail.com> wrote:
>
>> Is it possible for the new node to somehow have a higher term counter but
>> lower log id and thus then it becomes the leader?
>>
>> On Tue, Jun 20, 2023, 11:03 p.m. Tsz Wo Sze <sz...@gmail.com> wrote:
>>
>>> > If the new node goes down and comes back up is it possible it can
>>> become the leader even though it has less transactions than the
>>> existing node?
>>>
>>> No, it cannot become the leader since the other node won't vote for it.
>>>
>>> > ... is there a guarentee that always the node that has the highest
>>> transaction id will be elected the leader?
>>>
>>> For two nodes, yes. A vote is granted only if the candidate has an
>>> equal or higher transaction index. The node with a higher index won't vote
>>> for the other node.
>>>
>>> A group with two nodes means that any transactions must be replicated to
>>> both nodes before it is committed. The other node could become the leader
>>> only if they both have the same transaction index.
>>>
>>> For more than two nodes, the node with the highest transaction index is
>>> not necessarily elected the leader -- A previous leader may have extra
>>> uncommitted transactions. In the next election, any node with a
>>> transaction index higher or equal than the majority of the nodes can become
>>> the leader since the nodes with a lower or equal index will vote for it.
>>> In such a case, the previous leader will truncate the uncommitted
>>> transactions when it joins the new leader.
>>>
>>> Thanks for your questions!
>>>
>>> Tsz-Wo
>>>
>>>
>>> On Tue, Jun 20, 2023 at 11:45 PM Asad Awadia <as...@gmail.com>
>>> wrote:
>>>
>>>> A big part of raft is running an odd number of servers
>>>>
>>>> But we are scaling nodes 1 at a time and there are a lot of times where
>>>> we have only 2 nodes in the cluster
>>>>
>>>> If the new node goes down and comes back up is it possible it can
>>>> become the leader even though it has less transactions than the
>>>> existing node?
>>>>
>>>> Since there are only 2 nodes - is there a guarentee that always the
>>>> node that has the highest transaction id will be elected the leader?
>>>>
>>>> We are seeing
>>>> 0. Exisiting 1 node cluster with node 1 as the leader
>>>> 1. New node 2 gets added
>>>> 2. Node 1 and 2 are healthy
>>>> 3. Node 2 goes down
>>>> 4. Node 1 is in candidate state
>>>> 5. Node 2 comes back
>>>> 6. Node 2 becomes leader
>>>>
>>>> Shouldn't node 1 be always the leader?
>>>>
>>>
Re: Raft with 2 nodes
Posted by Tsz Wo Sze <sz...@gmail.com>.
Hi Asad,
You are right that, after the new node B joined the group, B could
update the term and then become the leader. When the old leader A joins
the new leader B, A will truncate the uncommitted transactions.
Tsz-Wo
On Wed, Jun 21, 2023 at 11:20 PM Asad Awadia <as...@gmail.com> wrote:
> Is it possible for the new node to somehow have a higher term counter but
> lower log id and thus then it becomes the leader?
>
> On Tue, Jun 20, 2023, 11:03 p.m. Tsz Wo Sze <sz...@gmail.com> wrote:
>
>> > If the new node goes down and comes back up is it possible it can
>> become the leader even though it has less transactions than the
>> existing node?
>>
>> No, it cannot become the leader since the other node won't vote for it.
>>
>> > ... is there a guarentee that always the node that has the highest
>> transaction id will be elected the leader?
>>
>> For two nodes, yes. A vote is granted only if the candidate has an equal
>> or higher transaction index. The node with a higher index won't vote for
>> the other node.
>>
>> A group with two nodes means that any transactions must be replicated to
>> both nodes before it is committed. The other node could become the leader
>> only if they both have the same transaction index.
>>
>> For more than two nodes, the node with the highest transaction index is
>> not necessarily elected the leader -- A previous leader may have extra
>> uncommitted transactions. In the next election, any node with a
>> transaction index higher or equal than the majority of the nodes can become
>> the leader since the nodes with a lower or equal index will vote for it.
>> In such a case, the previous leader will truncate the uncommitted
>> transactions when it joins the new leader.
>>
>> Thanks for your questions!
>>
>> Tsz-Wo
>>
>>
>> On Tue, Jun 20, 2023 at 11:45 PM Asad Awadia <as...@gmail.com>
>> wrote:
>>
>>> A big part of raft is running an odd number of servers
>>>
>>> But we are scaling nodes 1 at a time and there are a lot of times where
>>> we have only 2 nodes in the cluster
>>>
>>> If the new node goes down and comes back up is it possible it can become
>>> the leader even though it has less transactions than the existing node?
>>>
>>> Since there are only 2 nodes - is there a guarentee that always the node
>>> that has the highest transaction id will be elected the leader?
>>>
>>> We are seeing
>>> 0. Exisiting 1 node cluster with node 1 as the leader
>>> 1. New node 2 gets added
>>> 2. Node 1 and 2 are healthy
>>> 3. Node 2 goes down
>>> 4. Node 1 is in candidate state
>>> 5. Node 2 comes back
>>> 6. Node 2 becomes leader
>>>
>>> Shouldn't node 1 be always the leader?
>>>
>>
Re: Raft with 2 nodes
Posted by Asad Awadia <as...@gmail.com>.
Is it possible for the new node to somehow have a higher term counter but
lower log id and thus then it becomes the leader?
On Tue, Jun 20, 2023, 11:03 p.m. Tsz Wo Sze <sz...@gmail.com> wrote:
> > If the new node goes down and comes back up is it possible it can become
> the leader even though it has less transactions than the existing node?
>
> No, it cannot become the leader since the other node won't vote for it.
>
> > ... is there a guarentee that always the node that has the highest
> transaction id will be elected the leader?
>
> For two nodes, yes. A vote is granted only if the candidate has an equal
> or higher transaction index. The node with a higher index won't vote for
> the other node.
>
> A group with two nodes means that any transactions must be replicated to
> both nodes before it is committed. The other node could become the leader
> only if they both have the same transaction index.
>
> For more than two nodes, the node with the highest transaction index is
> not necessarily elected the leader -- A previous leader may have extra
> uncommitted transactions. In the next election, any node with a
> transaction index higher or equal than the majority of the nodes can become
> the leader since the nodes with a lower or equal index will vote for it.
> In such a case, the previous leader will truncate the uncommitted
> transactions when it joins the new leader.
>
> Thanks for your questions!
>
> Tsz-Wo
>
>
> On Tue, Jun 20, 2023 at 11:45 PM Asad Awadia <as...@gmail.com> wrote:
>
>> A big part of raft is running an odd number of servers
>>
>> But we are scaling nodes 1 at a time and there are a lot of times where
>> we have only 2 nodes in the cluster
>>
>> If the new node goes down and comes back up is it possible it can become
>> the leader even though it has less transactions than the existing node?
>>
>> Since there are only 2 nodes - is there a guarentee that always the node
>> that has the highest transaction id will be elected the leader?
>>
>> We are seeing
>> 0. Exisiting 1 node cluster with node 1 as the leader
>> 1. New node 2 gets added
>> 2. Node 1 and 2 are healthy
>> 3. Node 2 goes down
>> 4. Node 1 is in candidate state
>> 5. Node 2 comes back
>> 6. Node 2 becomes leader
>>
>> Shouldn't node 1 be always the leader?
>>
>
Re: Raft with 2 nodes
Posted by Tsz Wo Sze <sz...@gmail.com>.
> If the new node goes down and comes back up is it possible it can become
the leader even though it has less transactions than the existing node?
No, it cannot become the leader since the other node won't vote for it.
> ... is there a guarentee that always the node that has the highest
transaction id will be elected the leader?
For two nodes, yes. A vote is granted only if the candidate has an equal
or higher transaction index. The node with a higher index won't vote for
the other node.
A group with two nodes means that any transactions must be replicated to
both nodes before it is committed. The other node could become the leader
only if they both have the same transaction index.
For more than two nodes, the node with the highest transaction index is not
necessarily elected the leader -- A previous leader may have extra
uncommitted transactions. In the next election, any node with a
transaction index higher or equal than the majority of the nodes can become
the leader since the nodes with a lower or equal index will vote for it.
In such a case, the previous leader will truncate the uncommitted
transactions when it joins the new leader.
Thanks for your questions!
Tsz-Wo
On Tue, Jun 20, 2023 at 11:45 PM Asad Awadia <as...@gmail.com> wrote:
> A big part of raft is running an odd number of servers
>
> But we are scaling nodes 1 at a time and there are a lot of times where we
> have only 2 nodes in the cluster
>
> If the new node goes down and comes back up is it possible it can become
> the leader even though it has less transactions than the existing node?
>
> Since there are only 2 nodes - is there a guarentee that always the node
> that has the highest transaction id will be elected the leader?
>
> We are seeing
> 0. Exisiting 1 node cluster with node 1 as the leader
> 1. New node 2 gets added
> 2. Node 1 and 2 are healthy
> 3. Node 2 goes down
> 4. Node 1 is in candidate state
> 5. Node 2 comes back
> 6. Node 2 becomes leader
>
> Shouldn't node 1 be always the leader?
>