You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "sodonnel (via GitHub)" <gi...@apache.org> on 2023/09/07 15:20:31 UTC

[GitHub] [ozone] sodonnel opened a new pull request, #5255: HDDS-8536. ReplicationManager: Unhealthy replicas could block Ratis containers being replicated

sodonnel opened a new pull request, #5255:
URL: https://github.com/apache/ozone/pull/5255

   ## What changes were proposed in this pull request?
   
   
   This PR changes the flow so that if RM cannot find a spare node to replicate to, it will check look for quasi_closed or unhealthy replicas.
   
   Provided there are at least 3 replicas available AND there are no pending deletes on the container AND there is at least one CLOSED replica it will:
   
   1. Try to delete a single quasi_closed replica. If there is more than 1, it will delete the one with the lowest seqID.
   2. If there are no quasi-closed, it will delete an unhealthy one
   3. If there are no unhealthy or quasi-closed, it will do nothing (same functionality as before).
   
   The container will then be requeued to be processed again later when the delete has hopefully completed OK.
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-8536
   
   ## How was this patch tested?
   
   New unit tests added to test the new flow and all scenarios.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] errose28 commented on a diff in pull request #5255: HDDS-8536. ReplicationManager: Unhealthy replicas could block Ratis containers being replicated

Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #5255:
URL: https://github.com/apache/ozone/pull/5255#discussion_r1322165405


##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/ReplicationManagerUtil.java:
##########
@@ -188,4 +195,111 @@ public List<DatanodeDetails> getUsedNodes() {
     }
   }
 
+  /**
+   * This is intended to be call when a container is under replicated, but there
+   * are no spare nodes to create new replicas on, due to having too many
+   * unhealthy replicas or quasi-closed replicas which cannot be closed due to
+   * having a lagging sequence ID. The logic here will select a replica to
+   * delete, or return null if there are none which can be safely deleted.
+   * @param containerInfo The container to select a replica to delete from
+   * @param replicas The list of replicas for the container
+   * @param pendingOps The list of pending replica operations for the container
+   * @return A replica to delete, or null if there are none which can be safely
+   *         deleted.
+   */
+  public static ContainerReplica selectUnhealthyReplicaForDelete(
+      ContainerInfo containerInfo, Set<ContainerReplica> replicas,
+      List<ContainerReplicaOp> pendingOps,
+      Function<DatanodeDetails, NodeStatus> nodeStatusFn) {
+
+    for (ContainerReplicaOp op : pendingOps) {
+      if (op.getOpType() == ContainerReplicaOp.PendingOpType.DELETE) {
+        // There is at least one pending delete which will free up a node.
+        // Therefore we do nothing until that delete completes or times out.
+        LOG.debug("Container {} has pending deletes which will free nodes",
+            pendingOps);
+        return null;
+      }
+    }
+
+    if (replicas.size() <= 2) {
+      // We never remove replicas if it will leave us with less than 2 replicas
+      LOG.debug("There are only {} replicas for container {} so no more will " +
+          "be deleted", replicas.size(), containerInfo);
+      return null;
+    }
+
+    boolean foundMatchingReplica = false;
+    for (ContainerReplica replica : replicas) {
+      if (compareState(containerInfo.getState(), replica.getState())) {
+        foundMatchingReplica = true;
+        break;
+      }
+    }
+    if (!foundMatchingReplica) {
+      LOG.debug("No matching replica found for container {} with replicas " +
+              "{}. No unhealthy replicas can be safely delete.",
+          containerInfo, replicas);
+      return null;
+    }
+
+    List<ContainerReplica> deleteCandidates = new ArrayList<>();
+    for (ContainerReplica r : replicas) {
+      NodeStatus nodeStatus = nodeStatusFn.apply(r.getDatanodeDetails());
+      if (nodeStatus == null || !nodeStatus.isHealthy()
+          || !nodeStatus.isInService()) {
+        continue;
+      }
+
+      if (containerInfo.getState() != HddsProtos.LifeCycleState.QUASI_CLOSED &&
+          r.getState() == ContainerReplicaProto.State.QUASI_CLOSED) {
+        // a quasi_closed replica is a candidate only if its seq id is less
+        // than the container's
+        if (r.getSequenceId() < containerInfo.getSequenceId()) {
+          deleteCandidates.add(r);
+        }
+      } else if (r.getState() == ContainerReplicaProto.State.UNHEALTHY) {
+        deleteCandidates.add(r);
+      }
+    }
+
+    deleteCandidates.sort(
+        Comparator.comparingLong(ContainerReplica::getSequenceId));
+    if (containerInfo.getState() == HddsProtos.LifeCycleState.CLOSED) {
+      return deleteCandidates.size() > 0 ? deleteCandidates.get(0) : null;
+    }
+
+    if (containerInfo.getState() == HddsProtos.LifeCycleState.QUASI_CLOSED) {
+      List<ContainerReplica> nonUniqueOrigins =
+          findNonUniqueDeleteCandidates(replicas, deleteCandidates);
+      return nonUniqueOrigins.size() > 0 ? nonUniqueOrigins.get(0) : null;

Review Comment:
   This can still get stuck in a 4 node cluster. For example, the following would remain under replicated in a 4 node cluster, or block decommissioning the 4th node:
   ```
   SCM state: Quasi-closed
   Replica states: quasi(origin 1) quasi(origin 1) unhealthy(origin 2) unhealthy(origin 3)
   ```
   I think we can handle this in a follow-up jira, especially since we are proposing changing #5261 to use this util method as well. The one follow-up will fix it in both places.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] siddhantsangwan merged pull request #5255: HDDS-8536. ReplicationManager: Unhealthy replicas could block Ratis containers being replicated

Posted by "siddhantsangwan (via GitHub)" <gi...@apache.org>.
siddhantsangwan merged PR #5255:
URL: https://github.com/apache/ozone/pull/5255


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] sodonnel commented on a diff in pull request #5255: HDDS-8536. ReplicationManager: Unhealthy replicas could block Ratis containers being replicated

Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on code in PR #5255:
URL: https://github.com/apache/ozone/pull/5255#discussion_r1319568204


##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/RatisUnderReplicationHandler.java:
##########
@@ -137,6 +150,73 @@ public int processAndSendCommands(
     return commandsSent;
   }
 
+  private void removeUnhealthyReplicaIfPossible(ContainerInfo containerInfo,
+      Set<ContainerReplica> replicas, List<ContainerReplicaOp> pendingOps)
+      throws NotLeaderException {
+    for (ContainerReplicaOp op : pendingOps) {
+      if (op.getOpType() == ContainerReplicaOp.PendingOpType.DELETE) {
+        // There is at least one pending delete which will free up a node.
+        // Therefore we do nothing until that delete completes or times out.
+        LOG.debug("Container {} has pending deletes which will free nodes",
+            pendingOps);
+        return;
+      }
+    }
+
+    if (replicas.size() <= 2) {
+      // We never remove replicas if it will leave us with less than 2 replicas
+      LOG.debug("There are only {} replicas for container {} so no more will " +
+          "be deleted", replicas.size(), containerInfo);
+      return;
+    }
+
+    // If there are no healthy containers, we do nothing as this should be
+    // handled elsewhere.
+    boolean hasHealthy = false;
+    for (ContainerReplica r : replicas) {
+      if (r.getState() == State.CLOSED) {

Review Comment:
   I don't think I understand what you are suggesting. The code only gets here if we cannot find a spare node, so that cover the "more space needed" check.
   
   Then we prefer deleting any quasi-closed with a lagging sequence if any exists.
   
   Only then do we delete a single unhealthy if there are no quasi-closed to remove.
   
   Are you saying that we should check for the CONTAINER being quasi-closed, and if the container is quasi-closed, we should delete unhealthy over quasi-closed?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] sodonnel commented on pull request #5255: HDDS-8536. ReplicationManager: Unhealthy replicas could block Ratis containers being replicated

Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on PR #5255:
URL: https://github.com/apache/ozone/pull/5255#issuecomment-1711435378

   Checking through TestReplicationManager, we seem to have pretty good coverage of the unhealthy / quasi replicas flowing through the RM chain onto the under-rep queue. Then in the over / under rep handlers, those scenarios will be tested, so I think we are good there. The tests in ReplicationManager should really test the flow through the check phase and check if the scenario results in a message on the under or over rep queue or nothing.
   
   Then the processing of the over / under rep queue is tested by the unit tests for the over / under rep handlers.
   
   We have a small number of full integration robot tests too that checks the full flow for the happy path.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] errose28 commented on a diff in pull request #5255: HDDS-8536. ReplicationManager: Unhealthy replicas could block Ratis containers being replicated

Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #5255:
URL: https://github.com/apache/ozone/pull/5255#discussion_r1320456448


##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/RatisUnderReplicationHandler.java:
##########
@@ -137,6 +150,73 @@ public int processAndSendCommands(
     return commandsSent;
   }
 
+  private void removeUnhealthyReplicaIfPossible(ContainerInfo containerInfo,
+      Set<ContainerReplica> replicas, List<ContainerReplicaOp> pendingOps)
+      throws NotLeaderException {
+    for (ContainerReplicaOp op : pendingOps) {
+      if (op.getOpType() == ContainerReplicaOp.PendingOpType.DELETE) {
+        // There is at least one pending delete which will free up a node.
+        // Therefore we do nothing until that delete completes or times out.
+        LOG.debug("Container {} has pending deletes which will free nodes",
+            pendingOps);
+        return;
+      }
+    }
+
+    if (replicas.size() <= 2) {
+      // We never remove replicas if it will leave us with less than 2 replicas
+      LOG.debug("There are only {} replicas for container {} so no more will " +
+          "be deleted", replicas.size(), containerInfo);
+      return;
+    }
+
+    // If there are no healthy containers, we do nothing as this should be
+    // handled elsewhere.
+    boolean hasHealthy = false;
+    for (ContainerReplica r : replicas) {
+      if (r.getState() == State.CLOSED) {

Review Comment:
   > Are you saying that we should check for the CONTAINER being quasi-closed, and if the container is quasi-closed, we should delete unhealthy over quasi-closed?
   
   Yes. We can be in the following state in a 4 node cluster:
   SCM: Quasi-closed
   DNs: quasi (origin 1), unhealthy (origin 2), unhealthy (origin 3), quasi (origin 1)
   Ideally we would keep the unhealthy containers because they have unique origin nodes. This is what the new and old RMs currently do. However in this case we need to remove one of the unhealthy replicas to restore the quasi-closed container to full replication.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] sodonnel commented on a diff in pull request #5255: HDDS-8536. ReplicationManager: Unhealthy replicas could block Ratis containers being replicated

Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on code in PR #5255:
URL: https://github.com/apache/ozone/pull/5255#discussion_r1319550002


##########
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/replication/TestRatisUnderReplicationHandler.java:
##########
@@ -247,6 +252,131 @@ public void testInsufficientTargetsFoundBecauseOfPlacementPolicy() {
     Assert.assertEquals(1, metrics.getPartialReplicationTotal());
   }
 
+  @Test
+  public void testNoTargetsFoundBecauseOfPlacementPolicyRemoveNone() {
+    policy = ReplicationTestUtil.getNoNodesTestPlacementPolicy(nodeManager,
+        conf);
+    RatisUnderReplicationHandler handler =
+        new RatisUnderReplicationHandler(policy, conf, replicationManager);
+
+    Set<ContainerReplica> replicas
+        = createReplicas(container.containerID(), State.CLOSED, 0);
+
+    ContainerReplica shouldDelete = createContainerReplica(
+        container.containerID(), 0, IN_SERVICE, State.UNHEALTHY);
+    replicas.add(shouldDelete);
+
+    Assert.assertThrows(IOException.class,
+        () -> handler.processAndSendCommands(replicas,
+            Collections.emptyList(), getUnderReplicatedHealthResult(), 2));
+    // No commands send, as there are only 2 replicas available.
+    Assert.assertEquals(0, commandsSent.size());
+  }
+
+  @Test
+  public void testNoTargetsFoundBecauseOfPlacementPolicyNoneHealthy() {
+    policy = ReplicationTestUtil.getNoNodesTestPlacementPolicy(nodeManager,
+        conf);
+    RatisUnderReplicationHandler handler =
+        new RatisUnderReplicationHandler(policy, conf, replicationManager);
+
+    // All replicas UNHEALTHY so we do nothing.
+    Set<ContainerReplica> replicas
+        = createReplicas(container.containerID(), State.UNHEALTHY, 0, 0);
+
+    Assert.assertThrows(IOException.class,
+        () -> handler.processAndSendCommands(replicas,
+            Collections.emptyList(), getUnderReplicatedHealthResult(), 2));
+    // No commands send, as no CLOSED replicas available.
+    Assert.assertEquals(0, commandsSent.size());
+  }
+
+  @Test
+  public void testNoTargetsFoundBecauseOfPlacementPolicyRemoveUnhealthy() {
+    policy = ReplicationTestUtil.getNoNodesTestPlacementPolicy(nodeManager,
+        conf);
+    RatisUnderReplicationHandler handler =
+        new RatisUnderReplicationHandler(policy, conf, replicationManager);
+
+    Set<ContainerReplica> replicas
+        = createReplicas(container.containerID(), State.CLOSED, 0, 0);
+
+    ContainerReplica shouldDelete = createContainerReplica(
+        container.containerID(), 0, IN_SERVICE, State.UNHEALTHY);
+    replicas.add(shouldDelete);
+
+    Assert.assertThrows(IOException.class,
+        () -> handler.processAndSendCommands(replicas,
+            Collections.emptyList(), getUnderReplicatedHealthResult(), 2));
+    Assert.assertEquals(1, commandsSent.size());
+    Pair<DatanodeDetails, SCMCommand<?>> cmd = commandsSent.iterator().next();
+    Assert.assertEquals(shouldDelete.getDatanodeDetails(), cmd.getKey());
+    Assert.assertEquals(StorageContainerDatanodeProtocolProtos.SCMCommandProto
+        .Type.deleteContainerCommand, cmd.getValue().getType());

Review Comment:
   That should be tested elsewhere. In these unit tests, we are mocking the delete - it doesn't actually delete anything. The delete command is created and added to the commands set, rather than actually deleting anything. There should be other existing tests that check the replication of the correct sources.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] siddhantsangwan commented on a diff in pull request #5255: HDDS-8536. ReplicationManager: Unhealthy replicas could block Ratis containers being replicated

Posted by "siddhantsangwan (via GitHub)" <gi...@apache.org>.
siddhantsangwan commented on code in PR #5255:
URL: https://github.com/apache/ozone/pull/5255#discussion_r1321808509


##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/ReplicationManagerUtil.java:
##########
@@ -188,4 +195,105 @@ public List<DatanodeDetails> getUsedNodes() {
     }
   }
 
+  /**
+   * This is intended to be call when a container is under replicated, but there
+   * are no spare nodes to create new replicas on, due to having too many
+   * unhealthy replicas or quasi-closed replicas which cannot be closed due to
+   * having a lagging sequence ID. The logic here will select a replica to
+   * delete, or return null if there are none which can be safely deleted.
+   * @param containerInfo The container to select a replica to delete from
+   * @param replicas The list of replicas for the container
+   * @param pendingOps The list of pending replica operations for the container
+   * @return A replica to delete, or null if there are none which can be safely
+   *         deleted.
+   */
+  public static ContainerReplica selectUnhealthyReplicaForDelete(
+      ContainerInfo containerInfo, Set<ContainerReplica> replicas,
+      List<ContainerReplicaOp> pendingOps,
+      Function<DatanodeDetails, NodeStatus> nodeStatusFn) {
+
+    for (ContainerReplicaOp op : pendingOps) {
+      if (op.getOpType() == ContainerReplicaOp.PendingOpType.DELETE) {
+        // There is at least one pending delete which will free up a node.
+        // Therefore we do nothing until that delete completes or times out.
+        LOG.debug("Container {} has pending deletes which will free nodes",
+            pendingOps);
+        return null;
+      }
+    }
+
+    if (replicas.size() <= 2) {
+      // We never remove replicas if it will leave us with less than 2 replicas
+      LOG.debug("There are only {} replicas for container {} so no more will " +
+          "be deleted", replicas.size(), containerInfo);
+      return null;
+    }
+
+    boolean foundMatchingReplica = false;
+    for (ContainerReplica replica : replicas) {
+      if (compareState(containerInfo.getState(), replica.getState())) {
+        foundMatchingReplica = true;
+        break;
+      }
+    }
+    if (!foundMatchingReplica) {
+      LOG.debug("No matching replica found for container {} with replicas " +
+              "{}. No unhealthy replicas can be safely delete.",
+          containerInfo, replicas);
+      return null;
+    }
+
+    List<ContainerReplica> deleteCandidates = new ArrayList<>();
+    for (ContainerReplica r : replicas) {
+      NodeStatus nodeStatus = nodeStatusFn.apply(r.getDatanodeDetails());
+      if (nodeStatus == null || !nodeStatus.isHealthy()
+          || !nodeStatus.isInService()) {
+        continue;
+      }
+      // If the container is closed, then quasi-closed and unhealthy replicas
+      // will get added to this list. If the container is quasi-closed, then
+      // only unhealthy replicas will get added here.
+      if (!compareState(containerInfo.getState(), r.getState())) {

Review Comment:
   ```
   When we call this method, we are in an under replicated state - surely CLOSING, QUASI_CLOSED with the same sequence id etc would not trigger under replication, and hence we would never get here?
   ```
   We could have something like CLOSED, CLOSING, UNHEALTHY, which is under replication. Legacy will send a close command for that CLOSING replica and also try to fix under rep.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] sodonnel commented on a diff in pull request #5255: HDDS-8536. ReplicationManager: Unhealthy replicas could block Ratis containers being replicated

Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on code in PR #5255:
URL: https://github.com/apache/ozone/pull/5255#discussion_r1321803870


##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/ReplicationManagerUtil.java:
##########
@@ -188,4 +195,105 @@ public List<DatanodeDetails> getUsedNodes() {
     }
   }
 
+  /**
+   * This is intended to be call when a container is under replicated, but there
+   * are no spare nodes to create new replicas on, due to having too many
+   * unhealthy replicas or quasi-closed replicas which cannot be closed due to
+   * having a lagging sequence ID. The logic here will select a replica to
+   * delete, or return null if there are none which can be safely deleted.
+   * @param containerInfo The container to select a replica to delete from
+   * @param replicas The list of replicas for the container
+   * @param pendingOps The list of pending replica operations for the container
+   * @return A replica to delete, or null if there are none which can be safely
+   *         deleted.
+   */
+  public static ContainerReplica selectUnhealthyReplicaForDelete(
+      ContainerInfo containerInfo, Set<ContainerReplica> replicas,
+      List<ContainerReplicaOp> pendingOps,
+      Function<DatanodeDetails, NodeStatus> nodeStatusFn) {
+
+    for (ContainerReplicaOp op : pendingOps) {
+      if (op.getOpType() == ContainerReplicaOp.PendingOpType.DELETE) {
+        // There is at least one pending delete which will free up a node.
+        // Therefore we do nothing until that delete completes or times out.
+        LOG.debug("Container {} has pending deletes which will free nodes",
+            pendingOps);
+        return null;
+      }
+    }
+
+    if (replicas.size() <= 2) {
+      // We never remove replicas if it will leave us with less than 2 replicas
+      LOG.debug("There are only {} replicas for container {} so no more will " +
+          "be deleted", replicas.size(), containerInfo);
+      return null;
+    }
+
+    boolean foundMatchingReplica = false;
+    for (ContainerReplica replica : replicas) {
+      if (compareState(containerInfo.getState(), replica.getState())) {
+        foundMatchingReplica = true;
+        break;
+      }
+    }
+    if (!foundMatchingReplica) {
+      LOG.debug("No matching replica found for container {} with replicas " +
+              "{}. No unhealthy replicas can be safely delete.",
+          containerInfo, replicas);
+      return null;
+    }
+
+    List<ContainerReplica> deleteCandidates = new ArrayList<>();
+    for (ContainerReplica r : replicas) {
+      NodeStatus nodeStatus = nodeStatusFn.apply(r.getDatanodeDetails());
+      if (nodeStatus == null || !nodeStatus.isHealthy()
+          || !nodeStatus.isInService()) {
+        continue;
+      }
+      // If the container is closed, then quasi-closed and unhealthy replicas
+      // will get added to this list. If the container is quasi-closed, then
+      // only unhealthy replicas will get added here.
+      if (!compareState(containerInfo.getState(), r.getState())) {

Review Comment:
   I pushed a new commit that copies the new logic from #5261. Please have another look.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] sodonnel commented on pull request #5255: HDDS-8536. ReplicationManager: Unhealthy replicas could block Ratis containers being replicated

Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on PR #5255:
URL: https://github.com/apache/ozone/pull/5255#issuecomment-1711318632

   > Should we also add tests in `TestReplicationManager` for end to end flow? We could make sure the whole RM with over and under replication handlers works to go from states like closed,quasi,quasi,quasi on a 4 node cluster to closed,closed,closed,none.
   
   I could add some tests at the RM level, but I don't think we should cover all scenarios there. Otherwise will end up with a lot duplication of tests. Tests at the RM level are still mocked, as there are no datanodes, so even there we are not testing the full flow


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] sodonnel commented on a diff in pull request #5255: HDDS-8536. ReplicationManager: Unhealthy replicas could block Ratis containers being replicated

Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on code in PR #5255:
URL: https://github.com/apache/ozone/pull/5255#discussion_r1321786685


##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/ReplicationManagerUtil.java:
##########
@@ -188,4 +195,105 @@ public List<DatanodeDetails> getUsedNodes() {
     }
   }
 
+  /**
+   * This is intended to be call when a container is under replicated, but there
+   * are no spare nodes to create new replicas on, due to having too many
+   * unhealthy replicas or quasi-closed replicas which cannot be closed due to
+   * having a lagging sequence ID. The logic here will select a replica to
+   * delete, or return null if there are none which can be safely deleted.
+   * @param containerInfo The container to select a replica to delete from
+   * @param replicas The list of replicas for the container
+   * @param pendingOps The list of pending replica operations for the container
+   * @return A replica to delete, or null if there are none which can be safely
+   *         deleted.
+   */
+  public static ContainerReplica selectUnhealthyReplicaForDelete(
+      ContainerInfo containerInfo, Set<ContainerReplica> replicas,
+      List<ContainerReplicaOp> pendingOps,
+      Function<DatanodeDetails, NodeStatus> nodeStatusFn) {
+
+    for (ContainerReplicaOp op : pendingOps) {
+      if (op.getOpType() == ContainerReplicaOp.PendingOpType.DELETE) {
+        // There is at least one pending delete which will free up a node.
+        // Therefore we do nothing until that delete completes or times out.
+        LOG.debug("Container {} has pending deletes which will free nodes",
+            pendingOps);
+        return null;
+      }
+    }
+
+    if (replicas.size() <= 2) {
+      // We never remove replicas if it will leave us with less than 2 replicas
+      LOG.debug("There are only {} replicas for container {} so no more will " +
+          "be deleted", replicas.size(), containerInfo);
+      return null;
+    }
+
+    boolean foundMatchingReplica = false;
+    for (ContainerReplica replica : replicas) {
+      if (compareState(containerInfo.getState(), replica.getState())) {
+        foundMatchingReplica = true;
+        break;
+      }
+    }
+    if (!foundMatchingReplica) {
+      LOG.debug("No matching replica found for container {} with replicas " +
+              "{}. No unhealthy replicas can be safely delete.",
+          containerInfo, replicas);
+      return null;
+    }
+
+    List<ContainerReplica> deleteCandidates = new ArrayList<>();
+    for (ContainerReplica r : replicas) {
+      NodeStatus nodeStatus = nodeStatusFn.apply(r.getDatanodeDetails());
+      if (nodeStatus == null || !nodeStatus.isHealthy()
+          || !nodeStatus.isInService()) {
+        continue;
+      }
+      // If the container is closed, then quasi-closed and unhealthy replicas
+      // will get added to this list. If the container is quasi-closed, then
+      // only unhealthy replicas will get added here.
+      if (!compareState(containerInfo.getState(), r.getState())) {

Review Comment:
   When we call this method, we are in an under replicated state - surely CLOSING, QUASI_CLOSED with the same sequence id etc would not trigger under replication, and hence we would never get here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] siddhantsangwan commented on pull request #5255: HDDS-8536. ReplicationManager: Unhealthy replicas could block Ratis containers being replicated

Posted by "siddhantsangwan (via GitHub)" <gi...@apache.org>.
siddhantsangwan commented on PR #5255:
URL: https://github.com/apache/ozone/pull/5255#issuecomment-1715011681

   Thanks @sodonnel and @errose28. Merging this to master.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] sodonnel commented on pull request #5255: HDDS-8536. ReplicationManager: Unhealthy replicas could block Ratis containers being replicated

Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on PR #5255:
URL: https://github.com/apache/ozone/pull/5255#issuecomment-1713989794

   @errose28 @siddhantsangwan I have updated this PR to move the logic around selecting the replica to remove into ReplicationManagerUtil, and therefore both Legacy and the current RM can share the same logic. I have tried to make this PR match the logic in #5261 so if we can get to agreement on the selection strategy here, it will let us move the other PR forward quite quickly I think.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] errose28 commented on a diff in pull request #5255: HDDS-8536. ReplicationManager: Unhealthy replicas could block Ratis containers being replicated

Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #5255:
URL: https://github.com/apache/ozone/pull/5255#discussion_r1319223354


##########
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/replication/TestRatisUnderReplicationHandler.java:
##########
@@ -247,6 +252,131 @@ public void testInsufficientTargetsFoundBecauseOfPlacementPolicy() {
     Assert.assertEquals(1, metrics.getPartialReplicationTotal());
   }
 
+  @Test
+  public void testNoTargetsFoundBecauseOfPlacementPolicyRemoveNone() {
+    policy = ReplicationTestUtil.getNoNodesTestPlacementPolicy(nodeManager,
+        conf);
+    RatisUnderReplicationHandler handler =
+        new RatisUnderReplicationHandler(policy, conf, replicationManager);
+
+    Set<ContainerReplica> replicas
+        = createReplicas(container.containerID(), State.CLOSED, 0);
+
+    ContainerReplica shouldDelete = createContainerReplica(
+        container.containerID(), 0, IN_SERVICE, State.UNHEALTHY);
+    replicas.add(shouldDelete);
+
+    Assert.assertThrows(IOException.class,
+        () -> handler.processAndSendCommands(replicas,
+            Collections.emptyList(), getUnderReplicatedHealthResult(), 2));
+    // No commands send, as there are only 2 replicas available.
+    Assert.assertEquals(0, commandsSent.size());
+  }
+
+  @Test
+  public void testNoTargetsFoundBecauseOfPlacementPolicyNoneHealthy() {
+    policy = ReplicationTestUtil.getNoNodesTestPlacementPolicy(nodeManager,
+        conf);
+    RatisUnderReplicationHandler handler =
+        new RatisUnderReplicationHandler(policy, conf, replicationManager);
+
+    // All replicas UNHEALTHY so we do nothing.
+    Set<ContainerReplica> replicas
+        = createReplicas(container.containerID(), State.UNHEALTHY, 0, 0);
+
+    Assert.assertThrows(IOException.class,
+        () -> handler.processAndSendCommands(replicas,
+            Collections.emptyList(), getUnderReplicatedHealthResult(), 2));
+    // No commands send, as no CLOSED replicas available.
+    Assert.assertEquals(0, commandsSent.size());
+  }
+
+  @Test
+  public void testNoTargetsFoundBecauseOfPlacementPolicyRemoveUnhealthy() {
+    policy = ReplicationTestUtil.getNoNodesTestPlacementPolicy(nodeManager,
+        conf);
+    RatisUnderReplicationHandler handler =
+        new RatisUnderReplicationHandler(policy, conf, replicationManager);
+
+    Set<ContainerReplica> replicas
+        = createReplicas(container.containerID(), State.CLOSED, 0, 0);
+
+    ContainerReplica shouldDelete = createContainerReplica(
+        container.containerID(), 0, IN_SERVICE, State.UNHEALTHY);
+    replicas.add(shouldDelete);
+
+    Assert.assertThrows(IOException.class,
+        () -> handler.processAndSendCommands(replicas,
+            Collections.emptyList(), getUnderReplicatedHealthResult(), 2));
+    Assert.assertEquals(1, commandsSent.size());
+    Pair<DatanodeDetails, SCMCommand<?>> cmd = commandsSent.iterator().next();
+    Assert.assertEquals(shouldDelete.getDatanodeDetails(), cmd.getKey());
+    Assert.assertEquals(StorageContainerDatanodeProtocolProtos.SCMCommandProto
+        .Type.deleteContainerCommand, cmd.getValue().getType());

Review Comment:
   Once the delete is sent, can we do another run of the under replication handler to check that the correct replica is copied after space is available?



##########
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/replication/TestRatisUnderReplicationHandler.java:
##########
@@ -247,6 +252,131 @@ public void testInsufficientTargetsFoundBecauseOfPlacementPolicy() {
     Assert.assertEquals(1, metrics.getPartialReplicationTotal());
   }
 
+  @Test
+  public void testNoTargetsFoundBecauseOfPlacementPolicyRemoveNone() {
+    policy = ReplicationTestUtil.getNoNodesTestPlacementPolicy(nodeManager,
+        conf);
+    RatisUnderReplicationHandler handler =
+        new RatisUnderReplicationHandler(policy, conf, replicationManager);
+
+    Set<ContainerReplica> replicas
+        = createReplicas(container.containerID(), State.CLOSED, 0);
+
+    ContainerReplica shouldDelete = createContainerReplica(
+        container.containerID(), 0, IN_SERVICE, State.UNHEALTHY);
+    replicas.add(shouldDelete);
+
+    Assert.assertThrows(IOException.class,
+        () -> handler.processAndSendCommands(replicas,
+            Collections.emptyList(), getUnderReplicatedHealthResult(), 2));
+    // No commands send, as there are only 2 replicas available.
+    Assert.assertEquals(0, commandsSent.size());

Review Comment:
   Shouldn't the under replication handler make a copy of the closed one in this case, or does it think that there are only two nodes in the cluster total because of the test setup?



##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/RatisUnderReplicationHandler.java:
##########
@@ -137,6 +150,73 @@ public int processAndSendCommands(
     return commandsSent;
   }
 
+  private void removeUnhealthyReplicaIfPossible(ContainerInfo containerInfo,
+      Set<ContainerReplica> replicas, List<ContainerReplicaOp> pendingOps)
+      throws NotLeaderException {
+    for (ContainerReplicaOp op : pendingOps) {
+      if (op.getOpType() == ContainerReplicaOp.PendingOpType.DELETE) {
+        // There is at least one pending delete which will free up a node.
+        // Therefore we do nothing until that delete completes or times out.
+        LOG.debug("Container {} has pending deletes which will free nodes",
+            pendingOps);
+        return;
+      }
+    }
+
+    if (replicas.size() <= 2) {
+      // We never remove replicas if it will leave us with less than 2 replicas
+      LOG.debug("There are only {} replicas for container {} so no more will " +
+          "be deleted", replicas.size(), containerInfo);
+      return;
+    }
+
+    // If there are no healthy containers, we do nothing as this should be
+    // handled elsewhere.
+    boolean hasHealthy = false;
+    for (ContainerReplica r : replicas) {
+      if (r.getState() == State.CLOSED) {

Review Comment:
   We can also have a container in quasi closed state with 1 or 2 quasi closed replicas and the rest unhealthy. Right now RM will keep the unhealthy ones if they have unique origin node IDs and replicate the quasi-closed ones (`TestReplicationManager#testQuasiClosedContainerWithUnhealthyReplicaOnUniqueOrigin`). It still may run out of space to do this in a 4 node cluster (or a 5 node cluster with a decommissioning node). I think we need an additional check to only delete the unhealthy replicas if more space is needed. With the unhealthy replicas present it might be possible to close the container properly after HDDS-9244, which I don't think is too far fetched.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] siddhantsangwan commented on a diff in pull request #5255: HDDS-8536. ReplicationManager: Unhealthy replicas could block Ratis containers being replicated

Posted by "siddhantsangwan (via GitHub)" <gi...@apache.org>.
siddhantsangwan commented on code in PR #5255:
URL: https://github.com/apache/ozone/pull/5255#discussion_r1321771510


##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/ReplicationManagerUtil.java:
##########
@@ -188,4 +195,105 @@ public List<DatanodeDetails> getUsedNodes() {
     }
   }
 
+  /**
+   * This is intended to be call when a container is under replicated, but there
+   * are no spare nodes to create new replicas on, due to having too many
+   * unhealthy replicas or quasi-closed replicas which cannot be closed due to
+   * having a lagging sequence ID. The logic here will select a replica to
+   * delete, or return null if there are none which can be safely deleted.
+   * @param containerInfo The container to select a replica to delete from
+   * @param replicas The list of replicas for the container
+   * @param pendingOps The list of pending replica operations for the container
+   * @return A replica to delete, or null if there are none which can be safely
+   *         deleted.
+   */
+  public static ContainerReplica selectUnhealthyReplicaForDelete(
+      ContainerInfo containerInfo, Set<ContainerReplica> replicas,
+      List<ContainerReplicaOp> pendingOps,
+      Function<DatanodeDetails, NodeStatus> nodeStatusFn) {
+
+    for (ContainerReplicaOp op : pendingOps) {
+      if (op.getOpType() == ContainerReplicaOp.PendingOpType.DELETE) {
+        // There is at least one pending delete which will free up a node.
+        // Therefore we do nothing until that delete completes or times out.
+        LOG.debug("Container {} has pending deletes which will free nodes",
+            pendingOps);
+        return null;
+      }
+    }
+
+    if (replicas.size() <= 2) {
+      // We never remove replicas if it will leave us with less than 2 replicas
+      LOG.debug("There are only {} replicas for container {} so no more will " +
+          "be deleted", replicas.size(), containerInfo);
+      return null;
+    }
+
+    boolean foundMatchingReplica = false;
+    for (ContainerReplica replica : replicas) {
+      if (compareState(containerInfo.getState(), replica.getState())) {
+        foundMatchingReplica = true;
+        break;
+      }
+    }
+    if (!foundMatchingReplica) {
+      LOG.debug("No matching replica found for container {} with replicas " +
+              "{}. No unhealthy replicas can be safely delete.",
+          containerInfo, replicas);
+      return null;
+    }
+
+    List<ContainerReplica> deleteCandidates = new ArrayList<>();
+    for (ContainerReplica r : replicas) {
+      NodeStatus nodeStatus = nodeStatusFn.apply(r.getDatanodeDetails());
+      if (nodeStatus == null || !nodeStatus.isHealthy()
+          || !nodeStatus.isInService()) {
+        continue;
+      }
+      // If the container is closed, then quasi-closed and unhealthy replicas
+      // will get added to this list. If the container is quasi-closed, then
+      // only unhealthy replicas will get added here.
+      if (!compareState(containerInfo.getState(), r.getState())) {
+        deleteCandidates.add(r);
+      }
+    }
+    deleteCandidates.sort(
+        Comparator.comparingLong(ContainerReplica::getSequenceId));
+    if (containerInfo.getState() == HddsProtos.LifeCycleState.CLOSED) {
+      return deleteCandidates.size() > 0 ? deleteCandidates.get(0) : null;

Review Comment:
   We don't want to delete a quasi closed replica with the same seq id as the container. The logic from my comment above or some variation should take care of this.



##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/ReplicationManagerUtil.java:
##########
@@ -188,4 +195,105 @@ public List<DatanodeDetails> getUsedNodes() {
     }
   }
 
+  /**
+   * This is intended to be call when a container is under replicated, but there
+   * are no spare nodes to create new replicas on, due to having too many
+   * unhealthy replicas or quasi-closed replicas which cannot be closed due to
+   * having a lagging sequence ID. The logic here will select a replica to
+   * delete, or return null if there are none which can be safely deleted.
+   * @param containerInfo The container to select a replica to delete from
+   * @param replicas The list of replicas for the container
+   * @param pendingOps The list of pending replica operations for the container
+   * @return A replica to delete, or null if there are none which can be safely
+   *         deleted.
+   */
+  public static ContainerReplica selectUnhealthyReplicaForDelete(
+      ContainerInfo containerInfo, Set<ContainerReplica> replicas,
+      List<ContainerReplicaOp> pendingOps,
+      Function<DatanodeDetails, NodeStatus> nodeStatusFn) {
+
+    for (ContainerReplicaOp op : pendingOps) {
+      if (op.getOpType() == ContainerReplicaOp.PendingOpType.DELETE) {
+        // There is at least one pending delete which will free up a node.
+        // Therefore we do nothing until that delete completes or times out.
+        LOG.debug("Container {} has pending deletes which will free nodes",
+            pendingOps);
+        return null;
+      }
+    }
+
+    if (replicas.size() <= 2) {
+      // We never remove replicas if it will leave us with less than 2 replicas
+      LOG.debug("There are only {} replicas for container {} so no more will " +
+          "be deleted", replicas.size(), containerInfo);
+      return null;
+    }
+
+    boolean foundMatchingReplica = false;
+    for (ContainerReplica replica : replicas) {
+      if (compareState(containerInfo.getState(), replica.getState())) {
+        foundMatchingReplica = true;
+        break;
+      }
+    }
+    if (!foundMatchingReplica) {
+      LOG.debug("No matching replica found for container {} with replicas " +
+              "{}. No unhealthy replicas can be safely delete.",
+          containerInfo, replicas);
+      return null;
+    }
+
+    List<ContainerReplica> deleteCandidates = new ArrayList<>();
+    for (ContainerReplica r : replicas) {
+      NodeStatus nodeStatus = nodeStatusFn.apply(r.getDatanodeDetails());
+      if (nodeStatus == null || !nodeStatus.isHealthy()
+          || !nodeStatus.isInService()) {
+        continue;
+      }
+      // If the container is closed, then quasi-closed and unhealthy replicas
+      // will get added to this list. If the container is quasi-closed, then
+      // only unhealthy replicas will get added here.
+      if (!compareState(containerInfo.getState(), r.getState())) {

Review Comment:
   This will evaluate to true even for `CLOSED` containers with replica in a state like `CLOSING`, `QUASI_CLOSED` replica with the same sequence ID as this container etc. We want to close such replicas as they're ok, instead of trying to delete them. Similar reasoning for `QUASI_CLOSED` containers - it'll evaluate to true for `CLOSING` replicas, `CLOSED` replicas (weird and shouldn't happen, but still mentioning here).
   
   There's some updated logic in #5261 which we can try here, instead. It should work.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] siddhantsangwan commented on pull request #5255: HDDS-8536. ReplicationManager: Unhealthy replicas could block Ratis containers being replicated

Posted by "siddhantsangwan (via GitHub)" <gi...@apache.org>.
siddhantsangwan commented on PR #5255:
URL: https://github.com/apache/ozone/pull/5255#issuecomment-1714231257

   @sodonnel The updated logic in ReplicationManagerUtil looks good to me.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] sodonnel commented on a diff in pull request #5255: HDDS-8536. ReplicationManager: Unhealthy replicas could block Ratis containers being replicated

Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on code in PR #5255:
URL: https://github.com/apache/ozone/pull/5255#discussion_r1319548112


##########
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/replication/TestRatisUnderReplicationHandler.java:
##########
@@ -247,6 +252,131 @@ public void testInsufficientTargetsFoundBecauseOfPlacementPolicy() {
     Assert.assertEquals(1, metrics.getPartialReplicationTotal());
   }
 
+  @Test
+  public void testNoTargetsFoundBecauseOfPlacementPolicyRemoveNone() {
+    policy = ReplicationTestUtil.getNoNodesTestPlacementPolicy(nodeManager,
+        conf);
+    RatisUnderReplicationHandler handler =
+        new RatisUnderReplicationHandler(policy, conf, replicationManager);
+
+    Set<ContainerReplica> replicas
+        = createReplicas(container.containerID(), State.CLOSED, 0);
+
+    ContainerReplica shouldDelete = createContainerReplica(
+        container.containerID(), 0, IN_SERVICE, State.UNHEALTHY);
+    replicas.add(shouldDelete);
+
+    Assert.assertThrows(IOException.class,
+        () -> handler.processAndSendCommands(replicas,
+            Collections.emptyList(), getUnderReplicatedHealthResult(), 2));
+    // No commands send, as there are only 2 replicas available.
+    Assert.assertEquals(0, commandsSent.size());

Review Comment:
   The placement policy is mocked to always fail to find a new node to push the code into the `removeUnhealthyReplicaIfPossible` method. So this test is there to prove we don't go from 2 replicas to only 1 replica, as that would cause a higher risk of dataloss.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] errose28 commented on a diff in pull request #5255: HDDS-8536. ReplicationManager: Unhealthy replicas could block Ratis containers being replicated

Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #5255:
URL: https://github.com/apache/ozone/pull/5255#discussion_r1324939422


##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/ReplicationManagerUtil.java:
##########
@@ -188,4 +195,111 @@ public List<DatanodeDetails> getUsedNodes() {
     }
   }
 
+  /**
+   * This is intended to be call when a container is under replicated, but there
+   * are no spare nodes to create new replicas on, due to having too many
+   * unhealthy replicas or quasi-closed replicas which cannot be closed due to
+   * having a lagging sequence ID. The logic here will select a replica to
+   * delete, or return null if there are none which can be safely deleted.
+   * @param containerInfo The container to select a replica to delete from
+   * @param replicas The list of replicas for the container
+   * @param pendingOps The list of pending replica operations for the container
+   * @return A replica to delete, or null if there are none which can be safely
+   *         deleted.
+   */
+  public static ContainerReplica selectUnhealthyReplicaForDelete(
+      ContainerInfo containerInfo, Set<ContainerReplica> replicas,
+      List<ContainerReplicaOp> pendingOps,
+      Function<DatanodeDetails, NodeStatus> nodeStatusFn) {
+
+    for (ContainerReplicaOp op : pendingOps) {
+      if (op.getOpType() == ContainerReplicaOp.PendingOpType.DELETE) {
+        // There is at least one pending delete which will free up a node.
+        // Therefore we do nothing until that delete completes or times out.
+        LOG.debug("Container {} has pending deletes which will free nodes",
+            pendingOps);
+        return null;
+      }
+    }
+
+    if (replicas.size() <= 2) {
+      // We never remove replicas if it will leave us with less than 2 replicas
+      LOG.debug("There are only {} replicas for container {} so no more will " +
+          "be deleted", replicas.size(), containerInfo);
+      return null;
+    }
+
+    boolean foundMatchingReplica = false;
+    for (ContainerReplica replica : replicas) {
+      if (compareState(containerInfo.getState(), replica.getState())) {
+        foundMatchingReplica = true;
+        break;
+      }
+    }
+    if (!foundMatchingReplica) {
+      LOG.debug("No matching replica found for container {} with replicas " +
+              "{}. No unhealthy replicas can be safely delete.",
+          containerInfo, replicas);
+      return null;
+    }
+
+    List<ContainerReplica> deleteCandidates = new ArrayList<>();
+    for (ContainerReplica r : replicas) {
+      NodeStatus nodeStatus = nodeStatusFn.apply(r.getDatanodeDetails());
+      if (nodeStatus == null || !nodeStatus.isHealthy()
+          || !nodeStatus.isInService()) {
+        continue;
+      }
+
+      if (containerInfo.getState() != HddsProtos.LifeCycleState.QUASI_CLOSED &&
+          r.getState() == ContainerReplicaProto.State.QUASI_CLOSED) {
+        // a quasi_closed replica is a candidate only if its seq id is less
+        // than the container's
+        if (r.getSequenceId() < containerInfo.getSequenceId()) {
+          deleteCandidates.add(r);
+        }
+      } else if (r.getState() == ContainerReplicaProto.State.UNHEALTHY) {
+        deleteCandidates.add(r);
+      }
+    }
+
+    deleteCandidates.sort(
+        Comparator.comparingLong(ContainerReplica::getSequenceId));
+    if (containerInfo.getState() == HddsProtos.LifeCycleState.CLOSED) {
+      return deleteCandidates.size() > 0 ? deleteCandidates.get(0) : null;
+    }
+
+    if (containerInfo.getState() == HddsProtos.LifeCycleState.QUASI_CLOSED) {
+      List<ContainerReplica> nonUniqueOrigins =
+          findNonUniqueDeleteCandidates(replicas, deleteCandidates);
+      return nonUniqueOrigins.size() > 0 ? nonUniqueOrigins.get(0) : null;

Review Comment:
   A similar situation could also block a 5 node cluster from decommissioning one of its nodes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] errose28 commented on a diff in pull request #5255: HDDS-8536. ReplicationManager: Unhealthy replicas could block Ratis containers being replicated

Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #5255:
URL: https://github.com/apache/ozone/pull/5255#discussion_r1322165405


##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/ReplicationManagerUtil.java:
##########
@@ -188,4 +195,111 @@ public List<DatanodeDetails> getUsedNodes() {
     }
   }
 
+  /**
+   * This is intended to be call when a container is under replicated, but there
+   * are no spare nodes to create new replicas on, due to having too many
+   * unhealthy replicas or quasi-closed replicas which cannot be closed due to
+   * having a lagging sequence ID. The logic here will select a replica to
+   * delete, or return null if there are none which can be safely deleted.
+   * @param containerInfo The container to select a replica to delete from
+   * @param replicas The list of replicas for the container
+   * @param pendingOps The list of pending replica operations for the container
+   * @return A replica to delete, or null if there are none which can be safely
+   *         deleted.
+   */
+  public static ContainerReplica selectUnhealthyReplicaForDelete(
+      ContainerInfo containerInfo, Set<ContainerReplica> replicas,
+      List<ContainerReplicaOp> pendingOps,
+      Function<DatanodeDetails, NodeStatus> nodeStatusFn) {
+
+    for (ContainerReplicaOp op : pendingOps) {
+      if (op.getOpType() == ContainerReplicaOp.PendingOpType.DELETE) {
+        // There is at least one pending delete which will free up a node.
+        // Therefore we do nothing until that delete completes or times out.
+        LOG.debug("Container {} has pending deletes which will free nodes",
+            pendingOps);
+        return null;
+      }
+    }
+
+    if (replicas.size() <= 2) {
+      // We never remove replicas if it will leave us with less than 2 replicas
+      LOG.debug("There are only {} replicas for container {} so no more will " +
+          "be deleted", replicas.size(), containerInfo);
+      return null;
+    }
+
+    boolean foundMatchingReplica = false;
+    for (ContainerReplica replica : replicas) {
+      if (compareState(containerInfo.getState(), replica.getState())) {
+        foundMatchingReplica = true;
+        break;
+      }
+    }
+    if (!foundMatchingReplica) {
+      LOG.debug("No matching replica found for container {} with replicas " +
+              "{}. No unhealthy replicas can be safely delete.",
+          containerInfo, replicas);
+      return null;
+    }
+
+    List<ContainerReplica> deleteCandidates = new ArrayList<>();
+    for (ContainerReplica r : replicas) {
+      NodeStatus nodeStatus = nodeStatusFn.apply(r.getDatanodeDetails());
+      if (nodeStatus == null || !nodeStatus.isHealthy()
+          || !nodeStatus.isInService()) {
+        continue;
+      }
+
+      if (containerInfo.getState() != HddsProtos.LifeCycleState.QUASI_CLOSED &&
+          r.getState() == ContainerReplicaProto.State.QUASI_CLOSED) {
+        // a quasi_closed replica is a candidate only if its seq id is less
+        // than the container's
+        if (r.getSequenceId() < containerInfo.getSequenceId()) {
+          deleteCandidates.add(r);
+        }
+      } else if (r.getState() == ContainerReplicaProto.State.UNHEALTHY) {
+        deleteCandidates.add(r);
+      }
+    }
+
+    deleteCandidates.sort(
+        Comparator.comparingLong(ContainerReplica::getSequenceId));
+    if (containerInfo.getState() == HddsProtos.LifeCycleState.CLOSED) {
+      return deleteCandidates.size() > 0 ? deleteCandidates.get(0) : null;
+    }
+
+    if (containerInfo.getState() == HddsProtos.LifeCycleState.QUASI_CLOSED) {
+      List<ContainerReplica> nonUniqueOrigins =
+          findNonUniqueDeleteCandidates(replicas, deleteCandidates);
+      return nonUniqueOrigins.size() > 0 ? nonUniqueOrigins.get(0) : null;

Review Comment:
   This can still get stuck in a 4 node cluster. For example, the following would remain under replicated in a 4 node cluster, or block decommissioning the 4th node:
   ```
   SCM state: Quasi-closed
   Replica states: quasi(origin 1) quasi(origin 1) unhealthy(origin 2) unhealthy(origin 3)
   ```
   I think we can handle this in a follow-up jira, especially since we are proposing changing #5261 to use this util method as well.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org