You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by "Mikhail Efremov (Jira)" <ji...@apache.org> on 2024/11/13 08:41:00 UTC
[jira] [Updated] (IGNITE-23587) Make PlacementDriver able to choose lease candidate from pendings
[ https://issues.apache.org/jira/browse/IGNITE-23587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mikhail Efremov updated IGNITE-23587:
-------------------------------------
Reviewer: Vladislav Pyatkov (was: Vladislav Pyatkov)
> Make PlacementDriver able to choose lease candidate from pendings
> -----------------------------------------------------------------
>
> Key: IGNITE-23587
> URL: https://issues.apache.org/jira/browse/IGNITE-23587
> Project: Ignite
> Issue Type: Improvement
> Reporter: Mikhail Efremov
> Assignee: Mikhail Efremov
> Priority: Blocker
> Labels: ignite-3, lease
> Time Spent: 10m
> Remaining Estimate: 0h
>
> *Description*
> While working under IGNITE-22036 there were found 2 exceptional scenarios:
> _Recovery case_
> {code:java}
> nodes = [A, B]
> parts = 1
> replicas = 1
> | starting nodes
> v
> 13_part_0 stable = [B]
> B hosts replicaGrpId=13_part_0
> B becomes PR
> |
> v
> A starts
> | <nothing happens>
> v
> B is shutdowned
> | <do reset on pending = [A]>
> v
> A::TableManager::handlePending fails (A not a primary)
> A starts voting and became a leader term=2
> | ReplicaStateManager fails (A not a primary)
> v
> LeaseUpdater#updateBatchLeaseInternal <---------------------------------------\
> | iterate over assignments == [B] |
> v |
> topologyTracker#nodeByConsistentId |
> | returns null |
> v |
> leaseUpdateStatistics#onLeaseWithoutCandidate(); |
> | continue |
> v |
> LOG.debug("No leases to update found.") // don't see in log because debug lvl |
> | |
> \-----------------------------------------------------------------------------/
> {code}
> _Stable dies, change peers wasn't done_
> {code:java}
> nodes = [A, B]
> parts = 1
> replicas = 1
> | starting nodes
> v
> 13_part_0 stable = [B]
> B hosts replicaGrpId=13_part_0
> B becomes PR
> |
> v
> A starts
> | <filter={A}>
> v
> A::TM#handleChangePendingsAssignmentsEvent
> A::ReplicaManager#startReplica
> A::RaftGroupService#updatePeersAndLearners
> | <ready for catch up>
> v
> B::TM#handleChangePendingsAssignmentsEvent
> B::TM#sendChangePeersAndLearnersReplicaRequest
> B::ReplicaPartitionListener#processChangePeersAndLearnersReplicaRequest
> B::RaftGroupService#changePeersAndLearnersAsync
> | <B dies>
> v
> <No primary => no one would process failover>
> <All TM#handleChangePendingsAssignmentsEvent are finished previously =>
> => no one would send ChangePeersAndLearnersReplicaRequest>
> |
> v
> LeaseUpdater#updateBatchLeaseInternal <---------------------------------------\
> | iterate over assignments == [B] |
> v |
> topologyTracker#nodeByConsistentId |
> | returns null |
> v |
> leaseUpdateStatistics#onLeaseWithoutCandidate(); |
> | continue |
> v |
> LOG.debug("No leases to update found.") // don't see in log because debug lvl |
> | |
> \-----------------------------------------------------------------------------/
> {code}
> As we can see, in both cases we have a situation where we can't re-grant lease and then neither send {{ChangePeersAndLearnersReplicaRequest}} nor trigger {{onLeaderElected}} callback that was hooked after {{PRIMARY_REPLICA_ELECTED}}. And if in the first scenario there is possible workaround through sending the replica request with ignoring of primary replica nod colocation, but in the second scenario all possible pending handlers are already finished and wouldn't be called further.
> This situation leads us to an idea that we may take a closer look on lease granting process. For now we're checking only stable assignments for a new lease holder candidate. So, while we didn't finished assignments stable switch (call of {{onNewPeersConfigurationApplied}} after full RAFT rebalabce is done), we will nave {{null}} candidate and stucks in live lock.
> A possible solution is to check pendings assignments too in case if on stable assignments we haven't got any possibe lease candidate. But there we have an assumption that granting lease for a candidate from pending assignments won't be a problem, because we may accept lease only if candidate replica can retrieve replication group leader (see {{ReplicaImpl#processLeaseGrantedMessage}}, mostly the first action is to get the leader), then we're almost able to process requests: at least we will propose leader-colocated replica or (see the 2nd case) will waiting if leader will be elected.
> There should be a note about about the 2nd case: if B node dies before catch up is done and before RAFT group de-facto changed a configuration and then A didn't and unable to elect itself as a leader, we have a situation as now when partition is broken because the only replica is unavailable.
> *Motivation*
> The main motivation there is to make IGNITE-22036 and a whole epic IGNITE-22313 be able to handle rebalance even if we have no any active node in stable assignments because assignments stable switch unable to be called.
> *Definition of done*
> # {{AssignmentsTracker}} is able to track both stable and pending assignments.
> # While {{LeaseUpdater#Updater#updateLeaseBatchInternal}} if on lease expiration {{LeaseUpdater#nextLeaseHolder}} returns {{null}}, then it now should gets pending assignments from {{AssignmentsTracker}} and choose a lease candidate among them.
> # Before IGNITE-22036 is done this changes shouldn't break any test.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)