You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "swamirishi (via GitHub)" <gi...@apache.org> on 2023/07/21 07:00:56 UTC

[GitHub] [ozone] swamirishi commented on a diff in pull request #5097: HDDS-9011. EC: Unhealthy replica not replaced with a healthy replica in a rack-scatter environment

swamirishi commented on code in PR #5097:
URL: https://github.com/apache/ozone/pull/5097#discussion_r1270282206


##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/SCMContainerPlacementRackScatter.java:
##########
@@ -290,24 +314,45 @@ protected List<DatanodeDetails> chooseDatanodesInternal(
       unavailableNodes.addAll(excludedNodes);
     }
 
+    LOG.debug("Available racks excluding racks with used nodes: {}.", racks);
     if (racks.size() < additionalRacksRequired) {

Review Comment:
   Curious to know when this condition being satisfied.



##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/SCMContainerPlacementRackScatter.java:
##########
@@ -290,24 +314,45 @@ protected List<DatanodeDetails> chooseDatanodesInternal(
       unavailableNodes.addAll(excludedNodes);
     }
 
+    LOG.debug("Available racks excluding racks with used nodes: {}.", racks);
     if (racks.size() < additionalRacksRequired) {
       String reason = "Number of existing racks: " + racks.size()
               + "is less than additional required number of racks to choose: "
               + additionalRacksRequired + " do not match.";
       LOG.warn("Placement policy cannot choose the enough racks. {}"
                       + "Total number of Required Racks: {} Used Racks Count:" +
-                      " {}, Required Nodes count: {}",
+                      " {}, Required Nodes count: {}, fallback: {}.",
               reason, numberOfRacksRequired, usedRacksCntMap.size(),
-              nodesRequired);
-      throw new SCMException(reason,
-              FAILED_TO_FIND_SUITABLE_NODE);
+              nodesRequired, fallback);
+
+      /*
+      The number of racks which aren't already used by existing replicas
+      ("racks") and hence are available for new replicas are less than the
+      additional number of racks that replicas need to be on
+      ("additionalRacksRequired").
+      */
+      if (!fallback) {

Review Comment:
   if we have fallback enabled shouldn't we just change the list available racks as the total number of racks required and changeMaxReplicasPerRack value instead? It would be simpler right?



##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/SCMContainerPlacementRackScatter.java:
##########
@@ -317,31 +362,48 @@ protected List<DatanodeDetails> chooseDatanodesInternal(
                       "available racks. {} Available racks count: {},"
                       + " Excluded nodes count: {}, UsedNodes count: {}",
               reason, racks.size(), excludedNodesCount, usedNodesCount);
-      throw new SCMException(reason,
-              SCMException.ResultCodes.FAILED_TO_FIND_HEALTHY_NODES);
+
+      /*
+      Couldn't choose enough racks to satisfy the rack scatter policy. If
+      fallback is not allowed, then throw an exception.
+      */
+      if (!fallback) {
+        throw new SCMException(reason,
+            SCMException.ResultCodes.FAILED_TO_FIND_HEALTHY_NODES);
+      }
     }
 
     if (chosenNodes.size() < nodesRequired) {
+      /* If fallback is allowed, adjust maxReplicasToMove such that
+      replication factor number of replicas are allowed per rack.
+      */
+      if (fallback) {
+        maxReplicasPerRack = requiredReplicationFactor;
+      }
       racks.addAll(usedRacksCntMap.keySet());
       racks = sortRackWithExcludedNodes(racks, excludedNodes, usedRacksCntMap);
       racks.addAll(usedRacksCntMap.keySet());
+      LOG.debug("Available racks considering racks with used and exclude " +
+              "nodes: {}.", racks);
       chosenNodes.addAll(chooseNodesFromRacks(racks, unavailableNodes,
               mutableFavoredNodes, nodesRequired - chosenNodes.size(),
               metadataSizeRequired, dataSizeRequired,
-              Integer.MAX_VALUE, usedRacksCntMap, maxReplicasPerRack));
+              nodesRequired - chosenNodes.size(),

Review Comment:
   Agreed



##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/SCMContainerPlacementRackScatter.java:
##########
@@ -290,24 +314,45 @@ protected List<DatanodeDetails> chooseDatanodesInternal(
       unavailableNodes.addAll(excludedNodes);
     }
 
+    LOG.debug("Available racks excluding racks with used nodes: {}.", racks);
     if (racks.size() < additionalRacksRequired) {
       String reason = "Number of existing racks: " + racks.size()
               + "is less than additional required number of racks to choose: "
               + additionalRacksRequired + " do not match.";
       LOG.warn("Placement policy cannot choose the enough racks. {}"
                       + "Total number of Required Racks: {} Used Racks Count:" +
-                      " {}, Required Nodes count: {}",
+                      " {}, Required Nodes count: {}, fallback: {}.",
               reason, numberOfRacksRequired, usedRacksCntMap.size(),
-              nodesRequired);
-      throw new SCMException(reason,
-              FAILED_TO_FIND_SUITABLE_NODE);
+              nodesRequired, fallback);
+
+      /*
+      The number of racks which aren't already used by existing replicas
+      ("racks") and hence are available for new replicas are less than the
+      additional number of racks that replicas need to be on
+      ("additionalRacksRequired").
+      */
+      if (!fallback) {
+        throw new SCMException(reason,
+            FAILED_TO_FIND_SUITABLE_NODE);
+      }
     }
 
+    /*
+    If fallback is false, and we reach here, then nodesRequired is greater
+    than or equal to additionalRacksRequired. So we need to ask for
+    nodesRequired number of nodes. If fallback is true, then nodesRequired
+    could be less than additionalRacksRequired. We still need to ask for
+    nodesRequired number of more nodes, even if that won't satisfy the
+    requirement of being on additionalRacksRequired more racks.
+
+    So, in both cases, we want to ask for nodesRequired number of nodes.
+    */
+    int maxOuterLoopIterations = Math.min(nodesRequired, maxReplicasPerRack);
     Set<DatanodeDetails> chosenNodes = new LinkedHashSet<>(
         chooseNodesFromRacks(racks, unavailableNodes,
-            mutableFavoredNodes, additionalRacksRequired,
-            metadataSizeRequired, dataSizeRequired, 1,
-            usedRacksCntMap, maxReplicasPerRack));
+            mutableFavoredNodes, nodesRequired, metadataSizeRequired,
+            dataSizeRequired, maxOuterLoopIterations, usedRacksCntMap,

Review Comment:
   We shouldn't change the numberOfNodesRequired from additionalRacksRequired to nodesRequired. Since this function call is for spreading placement to as many racks as possible & choose racks only from the unused racks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org