You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Stephen O'Donnell (Jira)" <ji...@apache.org> on 2023/03/31 08:26:00 UTC
[jira] [Updated] (HDDS-8336) ReplicationManager: RatisUnderReplicationHandler should partially recover the container if not enough nodes
[ https://issues.apache.org/jira/browse/HDDS-8336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stephen O'Donnell updated HDDS-8336:
------------------------------------
Description:
In RatisUnderReplicationHandler, if the container is under-replicated by more than 1 replica, then we need to select 2 or more new copies. In the code getTargets():
{code}
private List<DatanodeDetails> getTargets(
RatisContainerReplicaCount replicaCount,
List<ContainerReplicaOp> pendingOps) throws IOException {
// DNs that already have replicas cannot be targets and should be excluded
final List<DatanodeDetails> excludeList =
replicaCount.getReplicas().stream()
.map(ContainerReplica::getDatanodeDetails)
.collect(Collectors.toList());
// DNs that are already waiting to receive replicas cannot be targets
final List<DatanodeDetails> pendingReplication =
pendingOps.stream()
.filter(containerReplicaOp -> containerReplicaOp.getOpType() ==
ContainerReplicaOp.PendingOpType.ADD)
.map(ContainerReplicaOp::getTarget)
.collect(Collectors.toList());
excludeList.addAll(pendingReplication);
/*
Ensure that target datanodes have enough space to hold a complete
container.
*/
final long dataSizeRequired =
Math.max(replicaCount.getContainer().getUsedBytes(),
currentContainerSize);
return placementPolicy.chooseDatanodes(excludeList, null,
replicaCount.additionalReplicaNeeded(), 0, dataSizeRequired);
}
{code}
We ask the placement policy for the required number of nodes. If it cannot provide the required number (eg we want 2, but it can only find 1 spare node), it will throw a SCMException. In that case, we should try again requesting one less node to see if we can partially recover. If that fails, try again with another less node until we reach zero.
If we successfully send a command for some but not all new copies, then we should throw an exception at the end so the container is re-queued on the under-rep queue to be tried again in a short while.
Note the class `MisReplicationHandler.getTargetDatanodes()` currently has logic like this, but it does not throw at the end to trigger a re-queue - HDDS-8337
was:
In RatisUnderReplicationHandler, if the container is under-replicated by more than 1 replica, then we need to select 2 or more new copies. In the code getTargets():
{code}
private List<DatanodeDetails> getTargets(
RatisContainerReplicaCount replicaCount,
List<ContainerReplicaOp> pendingOps) throws IOException {
// DNs that already have replicas cannot be targets and should be excluded
final List<DatanodeDetails> excludeList =
replicaCount.getReplicas().stream()
.map(ContainerReplica::getDatanodeDetails)
.collect(Collectors.toList());
// DNs that are already waiting to receive replicas cannot be targets
final List<DatanodeDetails> pendingReplication =
pendingOps.stream()
.filter(containerReplicaOp -> containerReplicaOp.getOpType() ==
ContainerReplicaOp.PendingOpType.ADD)
.map(ContainerReplicaOp::getTarget)
.collect(Collectors.toList());
excludeList.addAll(pendingReplication);
/*
Ensure that target datanodes have enough space to hold a complete
container.
*/
final long dataSizeRequired =
Math.max(replicaCount.getContainer().getUsedBytes(),
currentContainerSize);
return placementPolicy.chooseDatanodes(excludeList, null,
replicaCount.additionalReplicaNeeded(), 0, dataSizeRequired);
}
{code}
We ask the placement policy for the required number of nodes. If it cannot provide the required number (eg we want 2, but it can only find 1 spare node), it will throw a SCMException. In that case, we should try again requesting one less node to see if we can partially recover. If that fails, try again with another less node until we reach zero.
If we successfully send a command for some but not all new copies, then we should throw an exception at the end so the container is re-queued on the under-rep queue to be tried again in a short while.
Note the class `MisReplicationHandler.getTargetDatanodes()` currently has logic like this, but it does not throw at the end to trigger a re-queue - JIRA_TO_FIX
> ReplicationManager: RatisUnderReplicationHandler should partially recover the container if not enough nodes
> -----------------------------------------------------------------------------------------------------------
>
> Key: HDDS-8336
> URL: https://issues.apache.org/jira/browse/HDDS-8336
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: SCM
> Reporter: Stephen O'Donnell
> Priority: Major
>
> In RatisUnderReplicationHandler, if the container is under-replicated by more than 1 replica, then we need to select 2 or more new copies. In the code getTargets():
> {code}
> private List<DatanodeDetails> getTargets(
> RatisContainerReplicaCount replicaCount,
> List<ContainerReplicaOp> pendingOps) throws IOException {
> // DNs that already have replicas cannot be targets and should be excluded
> final List<DatanodeDetails> excludeList =
> replicaCount.getReplicas().stream()
> .map(ContainerReplica::getDatanodeDetails)
> .collect(Collectors.toList());
> // DNs that are already waiting to receive replicas cannot be targets
> final List<DatanodeDetails> pendingReplication =
> pendingOps.stream()
> .filter(containerReplicaOp -> containerReplicaOp.getOpType() ==
> ContainerReplicaOp.PendingOpType.ADD)
> .map(ContainerReplicaOp::getTarget)
> .collect(Collectors.toList());
> excludeList.addAll(pendingReplication);
> /*
> Ensure that target datanodes have enough space to hold a complete
> container.
> */
> final long dataSizeRequired =
> Math.max(replicaCount.getContainer().getUsedBytes(),
> currentContainerSize);
> return placementPolicy.chooseDatanodes(excludeList, null,
> replicaCount.additionalReplicaNeeded(), 0, dataSizeRequired);
> }
> {code}
> We ask the placement policy for the required number of nodes. If it cannot provide the required number (eg we want 2, but it can only find 1 spare node), it will throw a SCMException. In that case, we should try again requesting one less node to see if we can partially recover. If that fails, try again with another less node until we reach zero.
> If we successfully send a command for some but not all new copies, then we should throw an exception at the end so the container is re-queued on the under-rep queue to be tried again in a short while.
> Note the class `MisReplicationHandler.getTargetDatanodes()` currently has logic like this, but it does not throw at the end to trigger a re-queue - HDDS-8337
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org