[jira] [Updated] (RATIS-1770) Yield leader to higher priority peer by TransferLeadership

["Kaijie Chen (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/RATIS-1770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kaijie Chen updated RATIS-1770:
-------------------------------
    Description: 
Followup RATIS-1762.

There might be race conditions between priority-based YieldLeadership and user-requested TransferLeadership. For example:

||Node||Role||Priority||
|Peer 1|Leader|0|
|Peer 2|Follower|1|
|Peer 3|Follower|1|

If user requested TransferLeadership to peer 3, while the YieldLeadership found peer 2 has higher priority than current leader.
Peer 1 will send StartLeaderElection to both peer 2 and peer 3, and there might be a race condition (although it's benign).

One immediate thought is to use the new TransferLeadership to yield leadership to higher priority peer.
But it may cause following problems as quoted:

{quote}If the higher priority peer lags behind a lot, it may take some time to catch up the latest transaction. If the prior leader reject client requests, then the service may be unavailable for a long time.
{quote}

To solve this problem, the old leader should only start TransferLeadership *iff* the higher priority peer is up-to-date.

  was:
Followup RATIS-1762.

There might be race conditions between priority-based YieldLeadership and user-requested TransferLeadership. For example:
||Node||Role||Priority||
|Peer 1|Leader|0|
|Peer 2|Follower|1|
|Peer 3|Follower|1|

If user requested TransferLeadership to peer 3, while the YieldLeadership found peer 2 has higher priority than current leader.
Peer 1 will send StartLeaderElection to both peer 2 and peer 3, and there might be a race condition (although it's benign).

One immediate thought is to use the new TransferLeadership to yield leadership to higher priority peer.
But it may cause following problems as quoted:
{quote}If the higher priority peer lags behind a lot, it may take some time to catch up the latest transaction. If the prior leader reject client requests, then the service may be unavailable for a long time.
{quote}
To solve this problem, we can introduce a non-blocking TransferLeadership (which doesn't block client request).
||TransferLeadership||Requested by||Blocks client request?||Has a timeout?||Can be aborted?||
|Blocking|Client (shell)|Yes|Yes|No|
|Non-blocking|Priority checking|No|No|Yes, by another TransferLeadership (blocking or non-blocking), or explicitly.|

If the current leader regain the highest priority (through a config update), the previous non-blocking request should be cancelled.


> Yield leader to higher priority peer by TransferLeadership
> ----------------------------------------------------------
>
>                 Key: RATIS-1770
>                 URL: https://issues.apache.org/jira/browse/RATIS-1770
>             Project: Ratis
>          Issue Type: Sub-task
>            Reporter: Kaijie Chen
>            Priority: Minor
>          Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Followup RATIS-1762.
> There might be race conditions between priority-based YieldLeadership and user-requested TransferLeadership. For example:
> ||Node||Role||Priority||
> |Peer 1|Leader|0|
> |Peer 2|Follower|1|
> |Peer 3|Follower|1|
> If user requested TransferLeadership to peer 3, while the YieldLeadership found peer 2 has higher priority than current leader.
> Peer 1 will send StartLeaderElection to both peer 2 and peer 3, and there might be a race condition (although it's benign).
> One immediate thought is to use the new TransferLeadership to yield leadership to higher priority peer.
> But it may cause following problems as quoted:
> {quote}If the higher priority peer lags behind a lot, it may take some time to catch up the latest transaction. If the prior leader reject client requests, then the service may be unavailable for a long time.
> {quote}
> To solve this problem, the old leader should only start TransferLeadership *iff* the higher priority peer is up-to-date.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)