[jira] [Assigned] (KAFKA-20895) Different controller election behavior depending on node count

["Suman Pal (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-20895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suman Pal reassigned KAFKA-20895:
---------------------------------

    Assignee: Suman Pal

> Different controller election behavior depending on node count
> --------------------------------------------------------------
>
>                 Key: KAFKA-20895
>                 URL: https://issues.apache.org/jira/browse/KAFKA-20895
>             Project: Kafka
>          Issue Type: Improvement
>            Reporter: Mickael Maison
>            Assignee: Suman Pal
>            Priority: Major
>
> On a clean shutdown, the leader computes a preferred successors list. This list is order by log end offsets and by node.id for followers with the same log end offsets.
> With a 3 node quorum, when the leader enters the resigned state, each follower will wait a specific time based on its position in the successors list before initiating a vote. As the first entry waits the least amount of time, it starts a vote first. It will receive its own vote as well as the vote from the resigned leader. This is 2 votes out of 3, a majority, thus becomes the new leader. The 3rd node won't vote for it however. It reaches FollowerState.canGrantVote(), but will return false because hasFetchedFromLeader will be true.
> With a 4 node quorum, it turns out that the 1st entry in the preferred successors list cannot win the vote. Like in the 3 node scenario, it will get its own vote as well as the vote from the resigned leader, but this isn't a majority. The other 2 nodes won't vote for it because FollowerState.canGrantVote() will return false because hasFetchedFromLeader will be true. So the 1st node in the list will fail the vote and return to Follower state. The 2nd node in the list will then start a new vote, it will receive its own vote, the vote from the resigned leader. The first node will also vote for it, as it's FollowerState.canGrantVote() will return true because hasFetchedFromLeader is false. This is a majority (3 out of 4), so this node will become the new leader. Again the 4th node won't vote for it as FollowerState.canGrantVote() will return false because hasFetchedFromLeader will be true.
> With a 5 node quorum, we get a similar behavior than 4 nodes, the 2nd node in the preferred successors typically win the vote as the first node cannot win.
> The patterns seems to repeat. With 6 or 7 nodes, the 3rd entry in the preferred successors should win, with 8 or 9, the 4th entry, etc.
> I would expect the first entry in the preferred successors list to typically win, but with more than 3 nodes it's not possible. Even if there's logic to avoid electing a node with a shorter metadata log, I think it still means more election rounds so potentially degraded availability for quorum with more than 3 controllers.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)