You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "symious (via GitHub)" <gi...@apache.org> on 2025/11/12 03:44:36 UTC

[PR] HDDS-13914. Filter replicas before checking replicaHealth [ozone]

symious opened a new pull request, #9284:
URL: https://github.com/apache/ozone/pull/9284

   ## What changes were proposed in this pull request?
   
   A issue found when there is decommissioning/ed datanodes in the cluster.
   
   A container has 4 replicas, 1 on the to be removed DN. In current check logic, the container's health result is OK, but one replica sits on the to be removed DN, and it's not in the dnList to be checked with, so this container is marked skipped.
   
   This issue will cause SCM skipping all the containers resides on the decommissioned DNs, slowing down the deletion speed.
   
   This ticket is update the logic of this check, checking the health based on the filter result.
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-13914
   
   ## How was this patch tested?
   
   Update logic, original tests should keep basic logic correct.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-13914. Filter replicas before checking replicaHealth [ozone]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9284:
URL: https://github.com/apache/ozone/pull/9284#issuecomment-3649953290

   Thank you for your contribution. This PR is being closed due to inactivity. If needed, feel free to reopen it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-13914. Filter replicas before checking replicaHealth [ozone]

Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on PR #9284:
URL: https://github.com/apache/ozone/pull/9284#issuecomment-3530880434

   > Could you be more specific on this?
   
   Ok, after revisiting the block deletion flow, I think this patch makes sense.
   
   Below are my reasoning, I might be wrong since block deletion flow is quite complex.
   
   Taking from https://ozone.apache.org/assets/ApacheOzoneBestPracticesAtDidi.pdf (Efficient Deletion), it's quoted that
   
   > Upon analyzing the code, we identified a critical detail. When SCM issues deletion requests for
   block replicas, it does not dispatch all requests to the DataNodes (DNs) at once. For example, in
   the case of EC data using the EC-6-3-1024K format, deletion requires removing 9 replicas.
   However, SCM often only issues deletion requests for 5 or 6 of them. As a result, even if the DNs
   successfully delete those replicas, the remaining 3 are not marked as deleted. These replicas are
   not cleared in time and remain until the system times out and re-initiates the deletion process.
   
   HDDS-11498 argues that deletion needs to be sent to all replicas or not at all so that `SCMDeletedBlockTransactionStatusManager#commitTransactions` will be able to remove the delete transaction from DB. However, this means that for container with any replicas DNs that are either
   1. Not `IN_SERVICE` and `HEALTHY`,
   2. SCM or DN Command queue not full (`SCMBlockDeletingService#getDatanodesWithinCommandLimit`, see HDDS-8888)
   The deletion will not be sent and therefore stays in the delete transactions DB. This causes the issue that we faced (where container with replicas in decommissioned datanodes will be skipped). The things to note is that for condition 1 (unhealthy DNs), it should already be handled by `ReplicationManager#getContainerReplicationHealth` even before HDDS-11498. Therefore I think HDDS-11498 is primarily aimed to handle condition 2 (SCM or DN command queue is full) .
   
   In this patch, we will exclude these non healthy DNs when calculating the inadequate replicas. So if we have container with 3 HEALTHY replicas and 1 DECOMMISSIONED replicas [DN1 (HEALTHY), DN2 (HEALTHY), DN3 (HEALTHY), DN4 (DECOMMISSIONED)], we will filter the DN list to [DN1, DN2, and DN3] since `includedDnSet` does not contain the decommissioned DN and since the  `ReplicationManager#getContainerReplicationHealth` result should be healthy and the container is considered to have enough replicas, we will still send the deletion commands to all the DNs (including the DECOMMISSIONED DN) and deletion is not blocked. However, this should still work for HDDS-11498 since if the some DNs hosting HEALTHY replicas command queue are full, they will be excluded and `getContainerReplicationHealth` will now return `UNDER_REPLICATED` and will be deletion will be skipped.
   
   That said, I think it's still good to write some tests for both HDDS-11498 (@slfan1989 can raise a ticket for writing this test) and this (HDDS-13914). 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-13914. Filter replicas before checking replicaHealth [ozone]

Posted by "symious (via GitHub)" <gi...@apache.org>.
symious commented on code in PR #9284:
URL: https://github.com/apache/ozone/pull/9284#discussion_r2521898987


##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/block/DeletedBlockLogImpl.java:
##########
@@ -271,32 +272,35 @@ private void getTransaction(DeletedBlocksTransaction tx,
   private Boolean checkInadequateReplica(Set<ContainerReplica> replicas,
       DeletedBlocksTransaction txn,
       Set<DatanodeDetails> dnList) throws ContainerNotFoundException {

Review Comment:
   Updated checkInadequateReplica's variable name, getTransactions kept unchanged.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-13914. Filter replicas before checking replicaHealth [ozone]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9284:
URL: https://github.com/apache/ozone/pull/9284#issuecomment-3837938848

   Thank you for your contribution. This PR is being closed due to inactivity. If needed, feel free to reopen it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-13914. Filter replicas before checking replicaHealth [ozone]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #9284: HDDS-13914. Filter replicas before checking replicaHealth
URL: https://github.com/apache/ozone/pull/9284


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-13914. Filter replicas before checking replicaHealth [ozone]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #9284: HDDS-13914. Filter replicas before checking replicaHealth
URL: https://github.com/apache/ozone/pull/9284


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-13914. Filter replicas before checking replicaHealth [ozone]

Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #9284:
URL: https://github.com/apache/ozone/pull/9284#discussion_r2521020928


##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/block/DeletedBlockLogImpl.java:
##########
@@ -271,32 +272,35 @@ private void getTransaction(DeletedBlocksTransaction tx,
   private Boolean checkInadequateReplica(Set<ContainerReplica> replicas,
       DeletedBlocksTransaction txn,
       Set<DatanodeDetails> dnList) throws ContainerNotFoundException {

Review Comment:
   Nit: Let's rename `dnList` to `healthyAndInServiceDnSet` or `includedDnSet` to be more explicit. Also can do the same `DeletedBlockLogImpl#getTransactions` `dnList` argument.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-13914. Filter replicas before checking replicaHealth [ozone]

Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on PR #9284:
URL: https://github.com/apache/ozone/pull/9284#issuecomment-3520256765

   @slfan1989 Could you help take a look?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-13914. Filter replicas before checking replicaHealth [ozone]

Posted by "symious (via GitHub)" <gi...@apache.org>.
symious commented on PR #9284:
URL: https://github.com/apache/ozone/pull/9284#issuecomment-3525859900

   > after this patch checkInadequateReplica will most likely return false (i.e. the replicas are adequate).
   
   Could you be more specific on this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-13914. Filter replicas before checking replicaHealth [ozone]

Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on PR #9284:
URL: https://github.com/apache/ozone/pull/9284#issuecomment-3525891685

   @ivandika3 @symious Thanks for inviting me to review this PR. I’ll take a look later today and leave my comments afterward.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-13914. Filter replicas before checking replicaHealth [ozone]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9284:
URL: https://github.com/apache/ozone/pull/9284#issuecomment-3802378780

   This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-13914. Filter replicas before checking replicaHealth [ozone]

Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #9284:
URL: https://github.com/apache/ozone/pull/9284#discussion_r2525673144


##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/block/DeletedBlockLogImpl.java:
##########
@@ -270,33 +271,36 @@ private void getTransaction(DeletedBlocksTransaction tx,
 
   private Boolean checkInadequateReplica(Set<ContainerReplica> replicas,
       DeletedBlocksTransaction txn,
-      Set<DatanodeDetails> dnList) throws ContainerNotFoundException {
+      Set<DatanodeDetails> includedDnSet) throws ContainerNotFoundException {
+    long containerId = txn.getContainerID();
     ContainerInfo containerInfo = containerManager
         .getContainer(ContainerID.valueOf(txn.getContainerID()));
     ReplicationManager replicationManager =
         scmContext.getScm().getReplicationManager();
-    ContainerHealthResult result = replicationManager
-        .getContainerReplicationHealth(containerInfo, replicas);
-
-    // We have made an improvement here, and we expect that all replicas
-    // of the Container being sent will be included in the dnList.
-    // This change benefits ACK confirmation and improves deletion speed.
-    // The principle behind it is that
-    // DN can receive the command to delete a certain Container at the same time and provide
-    // feedback to SCM at roughly the same time.
-    // This avoids the issue of deletion blocking,
-    // where some replicas of a Container are deleted while others do not receive the delete command.
-    long containerId = txn.getContainerID();
-    for (ContainerReplica replica : replicas) {
-      DatanodeDetails datanodeDetails = replica.getDatanodeDetails();
-      if (!dnList.contains(datanodeDetails)) {
-        DatanodeDetails dnDetail = replica.getDatanodeDetails();
-        LOG.debug("Skip Container = {}, because DN = {} is not in dnList.",
-            containerId, dnDetail);
-        return true;
+
+    // Filter replicas based on dnList, only consider replicas located on target DNs
+    Set<ContainerReplica> filtered = new HashSet<>(replicas.size());
+    for (ContainerReplica r : replicas) {
+      DatanodeDetails dn = r.getDatanodeDetails();
+      if (includedDnSet.contains(dn)) {
+        filtered.add(r);
+      } else {
+        LOG.debug("Filter out replica of Container {} on DN {} not in dnList.",
+            containerId, dn);
       }
     }
 
+    // If there are no replicas on the target DNs, treat the container as inadequate
+    // (or skip it, depending on the higher-level logic)
+    if (filtered.isEmpty()) {
+      LOG.debug("Container {} has no replicas on target dnList; treat as inadequate.",
+          containerId);
+      return true;
+    }

Review Comment:
   I think we can remove this check and let it be handled in `getContainerReplicationHealth` to unify the logic (no special cases).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-13914. Filter replicas before checking replicaHealth [ozone]

Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #9284:
URL: https://github.com/apache/ozone/pull/9284#discussion_r2525673144


##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/block/DeletedBlockLogImpl.java:
##########
@@ -270,33 +271,36 @@ private void getTransaction(DeletedBlocksTransaction tx,
 
   private Boolean checkInadequateReplica(Set<ContainerReplica> replicas,
       DeletedBlocksTransaction txn,
-      Set<DatanodeDetails> dnList) throws ContainerNotFoundException {
+      Set<DatanodeDetails> includedDnSet) throws ContainerNotFoundException {
+    long containerId = txn.getContainerID();
     ContainerInfo containerInfo = containerManager
         .getContainer(ContainerID.valueOf(txn.getContainerID()));
     ReplicationManager replicationManager =
         scmContext.getScm().getReplicationManager();
-    ContainerHealthResult result = replicationManager
-        .getContainerReplicationHealth(containerInfo, replicas);
-
-    // We have made an improvement here, and we expect that all replicas
-    // of the Container being sent will be included in the dnList.
-    // This change benefits ACK confirmation and improves deletion speed.
-    // The principle behind it is that
-    // DN can receive the command to delete a certain Container at the same time and provide
-    // feedback to SCM at roughly the same time.
-    // This avoids the issue of deletion blocking,
-    // where some replicas of a Container are deleted while others do not receive the delete command.
-    long containerId = txn.getContainerID();
-    for (ContainerReplica replica : replicas) {
-      DatanodeDetails datanodeDetails = replica.getDatanodeDetails();
-      if (!dnList.contains(datanodeDetails)) {
-        DatanodeDetails dnDetail = replica.getDatanodeDetails();
-        LOG.debug("Skip Container = {}, because DN = {} is not in dnList.",
-            containerId, dnDetail);
-        return true;
+
+    // Filter replicas based on dnList, only consider replicas located on target DNs
+    Set<ContainerReplica> filtered = new HashSet<>(replicas.size());
+    for (ContainerReplica r : replicas) {
+      DatanodeDetails dn = r.getDatanodeDetails();
+      if (includedDnSet.contains(dn)) {
+        filtered.add(r);
+      } else {
+        LOG.debug("Filter out replica of Container {} on DN {} not in dnList.",
+            containerId, dn);
       }
     }
 
+    // If there are no replicas on the target DNs, treat the container as inadequate
+    // (or skip it, depending on the higher-level logic)
+    if (filtered.isEmpty()) {
+      LOG.debug("Container {} has no replicas on target dnList; treat as inadequate.",
+          containerId);
+      return true;
+    }

Review Comment:
   I think we can remove this check and let it be handled in `getContainerReplicationHealth` to unify the logic (no special cases handling outside `getContainerReplicationHealth`).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-13914. Filter replicas before checking replicaHealth [ozone]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9284:
URL: https://github.com/apache/ozone/pull/9284#issuecomment-3619001229

   This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-13914. Filter replicas before checking replicaHealth [ozone]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9284:
URL: https://github.com/apache/ozone/pull/9284#issuecomment-3708555802

   This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org