[jira] [Commented] (KUDU-2452) Prevent follower from causing pre-elections when UpdateConsensus is slow

["Todd Lipcon (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KUDU-2452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16484697#comment-16484697 ] 

Todd Lipcon commented on KUDU-2452:
-----------------------------------

I think we already do stop the failure detector during UpdateReplica, at least while we're waiting on the log, don't we?

The issue I've seen more is that enough tablets are blocked in UpdateConsensus that other (unrelated) tablets can't get their heartbeats processed due to queue overflows. Those _other_ victim tablets then end up calling pre-elections, which only contribute more to the load.

I think the best starting point for this would be KUDU-1707, which would allow simple liveness heartbeats to continue to get through even when tablets are holding up the threads.

> Prevent follower from causing pre-elections when UpdateConsensus is slow
> ------------------------------------------------------------------------
>
>                 Key: KUDU-2452
>                 URL: https://issues.apache.org/jira/browse/KUDU-2452
>             Project: Kudu
>          Issue Type: Improvement
>    Affects Versions: 1.7.0
>            Reporter: Will Berkeley
>            Priority: Major
>
> Thanks to pre-elections (KUDU-1365), slow UpdateConsensus calls on a single follower don't disturb the whole tablet by calling elections. However, sometimes I see situations where one or more followers are constantly calling pre-elections, and only rarely, if ever, overflowing their service queues. Occasionally, in 3x replicated tablets, the followers will get "lucky" and detect a leader failure at around the same time, and an election will happen.
> This background instability has caused bugs like KUDU-2343 that should be rare to occur pretty frequently, plus the extra RequestConsensusVote RPCs add a little more stress on the consensus service and on replicas' consensus locks. It also spams the logs, since there's no generally no exponential backoff for these pre-elections because there's a successful heartbeat in between them.
> It seems like we can get into the situation where the average number of in-flight consensus requests is constant over time, so on average we are processing each heartbeat in less than the heartbeat interval, however some heartbeats take longer. Since UpdateConsensus calls to a replica are serialized, a few of these in a row trigger the failure detector, despite the follower receiving every heartbeat in a timely manner and responding successfully eventually (and on average in a timely manner).
> It'd be nice to prevent these worthless pre-elections. A couple of ideas:
> 1. Separately calculate a backoff for failed pre-elections, and reset it when a pre-election succeeds or more generally when there's an election.
> 2. Don't count the time the follower is executing UpdateConsensus against the failure detector. [~mpercy] suggested stopping the failure detector during UpdateReplica() and resuming it when the function returns.
> 3. Move leader failure detection out-of-band of UpdateConsensus entirely.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)