You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2023/04/12 16:35:00 UTC
[jira] [Updated] (HDDS-8336) ReplicationManager: RatisUnderReplicationHandler should partially recover the container if not enough nodes
[ https://issues.apache.org/jira/browse/HDDS-8336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated HDDS-8336:
---------------------------------
Labels: pull-request-available (was: )
> ReplicationManager: RatisUnderReplicationHandler should partially recover the container if not enough nodes
> -----------------------------------------------------------------------------------------------------------
>
> Key: HDDS-8336
> URL: https://issues.apache.org/jira/browse/HDDS-8336
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: SCM
> Reporter: Stephen O'Donnell
> Assignee: Stephen O'Donnell
> Priority: Major
> Labels: pull-request-available
>
> In RatisUnderReplicationHandler, if the container is under-replicated by more than 1 replica, then we need to select 2 or more new copies. In the code getTargets():
> {code}
> private List<DatanodeDetails> getTargets(
> RatisContainerReplicaCount replicaCount,
> List<ContainerReplicaOp> pendingOps) throws IOException {
> // DNs that already have replicas cannot be targets and should be excluded
> final List<DatanodeDetails> excludeList =
> replicaCount.getReplicas().stream()
> .map(ContainerReplica::getDatanodeDetails)
> .collect(Collectors.toList());
> // DNs that are already waiting to receive replicas cannot be targets
> final List<DatanodeDetails> pendingReplication =
> pendingOps.stream()
> .filter(containerReplicaOp -> containerReplicaOp.getOpType() ==
> ContainerReplicaOp.PendingOpType.ADD)
> .map(ContainerReplicaOp::getTarget)
> .collect(Collectors.toList());
> excludeList.addAll(pendingReplication);
> /*
> Ensure that target datanodes have enough space to hold a complete
> container.
> */
> final long dataSizeRequired =
> Math.max(replicaCount.getContainer().getUsedBytes(),
> currentContainerSize);
> return placementPolicy.chooseDatanodes(excludeList, null,
> replicaCount.additionalReplicaNeeded(), 0, dataSizeRequired);
> }
> {code}
> We ask the placement policy for the required number of nodes. If it cannot provide the required number (eg we want 2, but it can only find 1 spare node), it will throw a SCMException. In that case, we should try again requesting one less node to see if we can partially recover. If that fails, try again with another less node until we reach zero.
> If we successfully send a command for some but not all new copies, then we should throw an exception at the end so the container is re-queued on the under-rep queue to be tried again in a short while.
> Note the class `MisReplicationHandler.getTargetDatanodes()` currently has logic like this, but it does not throw at the end to trigger a re-queue - HDDS-8337
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org