You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Russole (via GitHub)" <gi...@apache.org> on 2026/01/10 16:04:43 UTC

[PR] HDDS-14371. ContainerManagerImpl.getOpenContainerCountPerPipeline cou… [ozone]

Russole opened a new pull request, #9621:
URL: https://github.com/apache/ozone/pull/9621

   ## What changes were proposed in this pull request?
   
   * Division-by-zero bug: The value of minPipelineCountPerDn above could be zero since SCMNodeManager.minPipelineLimit(..) could return zero.
   * Another minor performance problem: the minPipelineLimit(..) and minHealthyVolumeNum(..) methods in SCMNodeManager should not create a list for computing min.
   
   ## What is the link to the Apache JIRA
   https://issues.apache.org/jira/browse/HDDS-14371
   
   ## How was this patch tested?
   All CI checks passed.
   https://github.com/Russole/ozone/actions/runs/20872429764
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14371. ContainerManagerImpl.getOpenContainerCountPerPipeline could have division by zero [ozone]

Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #9621:
URL: https://github.com/apache/ozone/pull/9621#discussion_r2679989970


##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java:
##########
@@ -1564,23 +1564,48 @@ public String getLabel() {
   }
 
   /**
-   * Returns the min of no healthy volumes reported out of the set
-   * of datanodes constituting the pipeline.
+   * Returns the open container limit for a pipeline based on the given
+   * datanodes and containers-per-volume configuration.
    */
   @Override
-  public int minHealthyVolumeNum(List<DatanodeDetails> dnList) {
-    List<Integer> volumeCountList = new ArrayList<>(dnList.size());
+  public int openContainerLimit(List<DatanodeDetails> dnList,
+      int numContainerPerVolume) {
+    Preconditions.checkArgument(dnList != null && !dnList.isEmpty(),
+        "dnList must not be empty");
+
+    int min = Integer.MAX_VALUE;
+
     for (DatanodeDetails dn : dnList) {
-      try {
-        volumeCountList.add(nodeStateManager.getNode(dn).
-                getHealthyVolumeCount());
-      } catch (NodeNotFoundException e) {
-        LOG.warn("Cannot generate NodeStat, datanode {} not found.",
-                dn.getID());
+      int perDn = openContainerLimit(dn, numContainerPerVolume);
+      min = Math.min(min, perDn);
+    }
+
+    return min == Integer.MAX_VALUE ? 0 : min;
+  }
+
+  /**
+   * Returns the open container limit contributed by a single datanode.
+   */
+  private int openContainerLimit(DatanodeDetails dn,
+      int numContainerPerVolume) {
+    try {
+      int healthy = nodeStateManager.getNode(dn).getHealthyVolumeCount();
+      if (healthy <= 0) {
+        return 0;
       }
+
+      int capacity = numContainerPerVolume * healthy;
+
+      int pipelineLimit = pipelineLimit(dn); // SCMNodeManager already has this
+      int denom = Math.max(1, pipelineLimit); // avoid division-by-zero

Review Comment:
   When pipelineLimit is zero, openContainerLimit(..) should return zero since it won't be able to create any containers.
   
   The code in https://issues.apache.org/jira/secure/attachment/13080186/9621_review.patch is incorrect.  It should be:
   ```java
    public int openContainerLimit(List<DatanodeDetails> datanodes) {
      int min = Integer.MAX_VALUE;
      for (DatanodeDetails dn : datanodes) {
        final int pipelineLimit = pipelineLimit(dn);
        if (pipelineLimit <= 0) {
          return 0;
        }
   
        final int containerLimit = 1 + (numContainerPerVolume * getHealthyVolumeCount(dn) - 1) / pipelineLimit;
        if (containerLimit < min) {
          min = containerLimit;
        }
      }
      return min;
    }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14371. ContainerManagerImpl.getOpenContainerCountPerPipeline could have division by zero [ozone]

Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on PR #9621:
URL: https://github.com/apache/ozone/pull/9621#issuecomment-3739643830

   @rnblough 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14371. ContainerManagerImpl.getOpenContainerCountPerPipeline could have division by zero [ozone]

Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo merged PR #9621:
URL: https://github.com/apache/ozone/pull/9621


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14371. ContainerManagerImpl.getOpenContainerCountPerPipeline could have division by zero [ozone]

Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #9621:
URL: https://github.com/apache/ozone/pull/9621#discussion_r2678966231


##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerManagerImpl.java:
##########
@@ -387,6 +387,10 @@ private int getOpenContainerCountPerPipeline(Pipeline pipeline) {
     int minContainerCountPerDn = numContainerPerVolume *
         pipelineManager.minHealthyVolumeNum(pipeline);
     int minPipelineCountPerDn = pipelineManager.minPipelineLimit(pipeline);
+    // Defensive guard: avoid division by zero
+    if (minPipelineCountPerDn <= 0) {
+      minPipelineCountPerDn = 1;
+    }

Review Comment:
   The calculation of this getOpenContainerCountPerPipeline(..) method actually is incorrect -- it should not divide two mins:   minContainerCountPerDn / minPipelineCountPerDn.
   
   Example: Suppose numContainerPerVolume = 1.
   - dn1 has 4 healthy volume and pipelineLimit = 2, so it allows 2 open containers
   - dn2 has 5 healthy volume and pipelineLimit = 1, so it allows 5 open containers
   
   Then, they together should allow 2 open containers (i.e. the min of them).  However, using minContainerCountPerDn / minPipelineCountPerDn would allow 4/1 = 4 containers
   
   Let's add a new openContainerLimit(..) method and 
   - remove this getOpenContainerCountPerPipeline(..) method,
   - remove both the minHealthyVolumeNum(..) and minPipelineLimit(..) methods.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14371. ContainerManagerImpl.getOpenContainerCountPerPipeline could have division by zero [ozone]

Posted by "Russole (via GitHub)" <gi...@apache.org>.
Russole commented on PR #9621:
URL: https://github.com/apache/ozone/pull/9621#issuecomment-3734078883

   Thanks @szetszwo for the review. I’ll make further updates based on the comments.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] ContainerManagerImpl.getOpenContainerCountPerPipeline could have division by zero [ozone]

Posted by "Gargi-jais11 (via GitHub)" <gi...@apache.org>.
Gargi-jais11 commented on PR #9621:
URL: https://github.com/apache/ozone/pull/9621#issuecomment-3733272379

   @Russole Please change the title to `HDDS-14371. ContainerManagerImpl.getOpenContainerCountPerPipeline could have division by zero`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14371. ContainerManagerImpl.getOpenContainerCountPerPipeline could have division by zero [ozone]

Posted by "Russole (via GitHub)" <gi...@apache.org>.
Russole commented on PR #9621:
URL: https://github.com/apache/ozone/pull/9621#issuecomment-3733288545

   Thanks @Gargi-jais11 for pointing this out.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org