You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "sodonnel (via GitHub)" <gi...@apache.org> on 2025/02/13 14:49:47 UTC
[PR] HDDS-12326. Allow Quasi_Closed to Closed if there is an unhealthy replica <= highest BCSID [ozone]
sodonnel opened a new pull request, #7869:
URL: https://github.com/apache/ozone/pull/7869
## What changes were proposed in this pull request?
In [HDDS-12232](https://issues.apache.org/jira/browse/HDDS-12232) we changed the logic to move a container from QC -> CLOSED to require 3 replicas from unique origins rather than 2.
A further scenario is where a contain has replicas:
```
QC-BCSID10-Origin1
QC-BCSID10-Origin2
UNHEALTHY-BCSID10-Origin3
```
In this case, it can still close - we have seen all origins and the BCSID from them all.
In this case, the unhealthy replica somehow has a higher BCSID than the others, so we cannot close it:
```
QC-BCSID10-Origin1
QC-BCSID10-Origin2
UNHEALTHY-BCSID11-Origin3
```
Ultimately, when the Reconciler is added to the system, it would repair the replicas so they are all identical and then it would close.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-12326
## How was this patch tested?
New unit and scenario tests added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12326. Allow Quasi_Closed to Closed if there is an unhealthy replica <= highest BCSID [ozone]
Posted by "nandakumar131 (via GitHub)" <gi...@apache.org>.
nandakumar131 commented on code in PR #7869:
URL: https://github.com/apache/ozone/pull/7869#discussion_r1954943628
##########
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/replication/health/TestQuasiClosedContainerHandler.java:
##########
@@ -177,6 +177,91 @@ public void testQuasiClosedWithAllUniqueOriginSendsForceClose() {
.sendCloseContainerReplicaCommand(any(), any(), anyBoolean());
}
+ /**
+ * When a container is QUASI_CLOSED with some unhealthy and all 3 are reported with unique
+ * origins, it should be forced closed.
+ */
+ @Test
+ public void testQuasiClosedWithAllUniqueOriginAndUnhealthySendsForceClose() {
+ ContainerInfo containerInfo = ReplicationTestUtil.createContainerInfo(
+ ratisReplicationConfig, 1, QUASI_CLOSED);
+ // These 3 replicas will have the same BCSID and unique origin node ids
+ Set<ContainerReplica> containerReplicas = ReplicationTestUtil
+ .createReplicas(containerInfo.containerID(),
+ State.QUASI_CLOSED, 0, 0);
+ containerReplicas.addAll(ReplicationTestUtil
+ .createReplicas(containerInfo.containerID(),
+ State.UNHEALTHY, 0));
+ ContainerCheckRequest request = new ContainerCheckRequest.Builder()
+ .setPendingOps(Collections.emptyList())
+ .setReport(new ReplicationManagerReport())
+ .setContainerInfo(containerInfo)
+ .setContainerReplicas(containerReplicas)
+ .build();
+ ContainerCheckRequest readRequest = new ContainerCheckRequest.Builder()
+ .setPendingOps(Collections.emptyList())
+ .setReport(new ReplicationManagerReport())
+ .setContainerInfo(containerInfo)
+ .setContainerReplicas(containerReplicas)
+ .setReadOnly(true)
+ .build();
+
+ assertFalse(quasiClosedContainerHandler.handle(request));
+ assertFalse(quasiClosedContainerHandler.handle(readRequest));
+ verify(replicationManager, times(2))
+ .sendCloseContainerReplicaCommand(any(), any(), anyBoolean());
+ }
+
+ /**
+ * If it's possible to force close replicas then only replicas with the
+ * highest Sequence ID (also known as BCSID) should be closed.
+ */
+ @Test
+ public void testQuasiClosedWithUnhealthyHavingHighestSeq() {
+ final ContainerInfo containerInfo =
+ getContainer(HddsProtos.LifeCycleState.QUASI_CLOSED);
+ containerInfo.setUsedBytes(99);
+ final ContainerID id = containerInfo.containerID();
+
+ // create replicas with unique origin DNs
+ DatanodeDetails dnOne = randomDatanodeDetails();
+ DatanodeDetails dnTwo = randomDatanodeDetails();
+ DatanodeDetails dnThree = randomDatanodeDetails();
+
+ // 1001 is the highest sequence id
+ final ContainerReplica replicaOne = getReplicas(
+ id, State.QUASI_CLOSED, 1000L, dnOne.getUuid(), dnOne);
+ final ContainerReplica replicaTwo = getReplicas(
+ id, State.QUASI_CLOSED, 1000L, dnTwo.getUuid(), dnTwo);
+ final ContainerReplica replicaThree = getReplicas(
+ id, State.UNHEALTHY, 1001L, dnThree.getUuid(), dnThree);
+ Set<ContainerReplica> containerReplicas = new HashSet<>();
+ containerReplicas.add(replicaOne);
+ containerReplicas.add(replicaTwo);
+ containerReplicas.add(replicaThree);
+
+ ContainerCheckRequest request = new ContainerCheckRequest.Builder()
+ .setPendingOps(Collections.emptyList())
+ .setReport(new ReplicationManagerReport())
+ .setContainerInfo(containerInfo)
+ .setContainerReplicas(containerReplicas)
+ .build();
+ ContainerCheckRequest readRequest = new ContainerCheckRequest.Builder()
+ .setPendingOps(Collections.emptyList())
+ .setReport(new ReplicationManagerReport())
+ .setContainerInfo(containerInfo)
+ .setContainerReplicas(containerReplicas)
+ .setReadOnly(true)
+ .build();
+
+ assertFalse(quasiClosedContainerHandler.handle(request));
+ assertFalse(quasiClosedContainerHandler.handle(readRequest));
+ // verify close command was sent for replicas with sequence ID 1001, that
+ // is dnTwo and dnThree
Review Comment:
This comment seems incorrect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12326. Allow Quasi_Closed to Closed if there is an unhealthy replica <= highest BCSID [ozone]
Posted by "nandakumar131 (via GitHub)" <gi...@apache.org>.
nandakumar131 merged PR #7869:
URL: https://github.com/apache/ozone/pull/7869
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12326. Allow Quasi_Closed to Closed if there is an unhealthy replica <= highest BCSID [ozone]
Posted by "nandakumar131 (via GitHub)" <gi...@apache.org>.
nandakumar131 commented on PR #7869:
URL: https://github.com/apache/ozone/pull/7869#issuecomment-2659879346
Thanks @sodonnel for the contribution!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12326. Allow Quasi_Closed to Closed if there is an unhealthy replica <= highest BCSID [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on code in PR #7869:
URL: https://github.com/apache/ozone/pull/7869#discussion_r1955107096
##########
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/replication/health/TestQuasiClosedContainerHandler.java:
##########
@@ -177,6 +177,91 @@ public void testQuasiClosedWithAllUniqueOriginSendsForceClose() {
.sendCloseContainerReplicaCommand(any(), any(), anyBoolean());
}
+ /**
+ * When a container is QUASI_CLOSED with some unhealthy and all 3 are reported with unique
+ * origins, it should be forced closed.
+ */
+ @Test
+ public void testQuasiClosedWithAllUniqueOriginAndUnhealthySendsForceClose() {
+ ContainerInfo containerInfo = ReplicationTestUtil.createContainerInfo(
+ ratisReplicationConfig, 1, QUASI_CLOSED);
+ // These 3 replicas will have the same BCSID and unique origin node ids
+ Set<ContainerReplica> containerReplicas = ReplicationTestUtil
+ .createReplicas(containerInfo.containerID(),
+ State.QUASI_CLOSED, 0, 0);
+ containerReplicas.addAll(ReplicationTestUtil
+ .createReplicas(containerInfo.containerID(),
+ State.UNHEALTHY, 0));
+ ContainerCheckRequest request = new ContainerCheckRequest.Builder()
+ .setPendingOps(Collections.emptyList())
+ .setReport(new ReplicationManagerReport())
+ .setContainerInfo(containerInfo)
+ .setContainerReplicas(containerReplicas)
+ .build();
+ ContainerCheckRequest readRequest = new ContainerCheckRequest.Builder()
+ .setPendingOps(Collections.emptyList())
+ .setReport(new ReplicationManagerReport())
+ .setContainerInfo(containerInfo)
+ .setContainerReplicas(containerReplicas)
+ .setReadOnly(true)
+ .build();
+
+ assertFalse(quasiClosedContainerHandler.handle(request));
+ assertFalse(quasiClosedContainerHandler.handle(readRequest));
+ verify(replicationManager, times(2))
+ .sendCloseContainerReplicaCommand(any(), any(), anyBoolean());
+ }
+
+ /**
+ * If it's possible to force close replicas then only replicas with the
+ * highest Sequence ID (also known as BCSID) should be closed.
+ */
+ @Test
+ public void testQuasiClosedWithUnhealthyHavingHighestSeq() {
+ final ContainerInfo containerInfo =
+ getContainer(HddsProtos.LifeCycleState.QUASI_CLOSED);
+ containerInfo.setUsedBytes(99);
+ final ContainerID id = containerInfo.containerID();
+
+ // create replicas with unique origin DNs
+ DatanodeDetails dnOne = randomDatanodeDetails();
+ DatanodeDetails dnTwo = randomDatanodeDetails();
+ DatanodeDetails dnThree = randomDatanodeDetails();
+
+ // 1001 is the highest sequence id
+ final ContainerReplica replicaOne = getReplicas(
+ id, State.QUASI_CLOSED, 1000L, dnOne.getUuid(), dnOne);
+ final ContainerReplica replicaTwo = getReplicas(
+ id, State.QUASI_CLOSED, 1000L, dnTwo.getUuid(), dnTwo);
+ final ContainerReplica replicaThree = getReplicas(
+ id, State.UNHEALTHY, 1001L, dnThree.getUuid(), dnThree);
+ Set<ContainerReplica> containerReplicas = new HashSet<>();
+ containerReplicas.add(replicaOne);
+ containerReplicas.add(replicaTwo);
+ containerReplicas.add(replicaThree);
+
+ ContainerCheckRequest request = new ContainerCheckRequest.Builder()
+ .setPendingOps(Collections.emptyList())
+ .setReport(new ReplicationManagerReport())
+ .setContainerInfo(containerInfo)
+ .setContainerReplicas(containerReplicas)
+ .build();
+ ContainerCheckRequest readRequest = new ContainerCheckRequest.Builder()
+ .setPendingOps(Collections.emptyList())
+ .setReport(new ReplicationManagerReport())
+ .setContainerInfo(containerInfo)
+ .setContainerReplicas(containerReplicas)
+ .setReadOnly(true)
+ .build();
+
+ assertFalse(quasiClosedContainerHandler.handle(request));
+ assertFalse(quasiClosedContainerHandler.handle(readRequest));
+ // verify close command was sent for replicas with sequence ID 1001, that
+ // is dnTwo and dnThree
Review Comment:
That was carried over from copy and paste. I have fixed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org