You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "sodonnel (via GitHub)" <gi...@apache.org> on 2023/11/01 16:46:50 UTC
[PR] HDDS-9602. Backport HDDS-9550 to legacy RM (missing containers which are empty) [ozone]
sodonnel opened a new pull request, #5527:
URL: https://github.com/apache/ozone/pull/5527
## What changes were proposed in this pull request?
This change mirrors #5523 for the legacy RM.
If a container appears empty to SCM (ie key count is zero in the containerInfo) and there are no replicas, it could be a container which failed to be created on the datanodes, and no replicas will ever be reported.
However there isn't enough information to know if the container can be safely deleted or not, as the DN could have gone offline or missed some heartbeats etc.
As things stand, such as container will appear as missing in the container report. In this PR, we change that to leave the container as CLOSED but count it as EMPTY without triggering the delete process.
That means if replicas later appear, it will be OK.
However if the replicas never appear, the container will be stuck in this state forever. Additionally, if there should be replicas, as the client successfully wrote to the DNs and updated OM, there is no way to know this on SCM and the container should really be missing despite it being logged as empty, hiding the problem.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-9602
## How was this patch tested?
New unit test added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9602. Backport HDDS-9550 to legacy RM (missing containers which are empty) [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #5527:
URL: https://github.com/apache/ozone/pull/5527#discussion_r1380407897
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/LegacyReplicationManager.java:
##########
@@ -499,6 +499,19 @@ protected void processContainer(ContainerInfo container,
return;
}
+ // If the container is empty and has no replicas, it is possible it was
+ // a container which stuck in the closing state which never got any
+ // replicas created on the datanodes. In this case, we don't have enough
+ // information to delete the container, so we just log it as EMPTY,
+ // leaving it as CLOSED and return true, otherwise, it will end up
+ // marked as missing by the under replication handling.
+ if (replicas.isEmpty()
+ && container.getState() == LifeCycleState.CLOSED
+ && container.getNumberOfKeys() == 0) {
+ report.incrementAndSample(HealthState.EMPTY, container.containerID());
Review Comment:
Can we add a log here just like the one in the https://github.com/apache/ozone/pull/5523
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9602. Backport HDDS-9550 to legacy RM (missing containers which are empty) [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on code in PR #5527:
URL: https://github.com/apache/ozone/pull/5527#discussion_r1380684438
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/LegacyReplicationManager.java:
##########
@@ -499,6 +499,19 @@ protected void processContainer(ContainerInfo container,
return;
}
+ // If the container is empty and has no replicas, it is possible it was
+ // a container which stuck in the closing state which never got any
+ // replicas created on the datanodes. In this case, we don't have enough
+ // information to delete the container, so we just log it as EMPTY,
+ // leaving it as CLOSED and return true, otherwise, it will end up
+ // marked as missing by the under replication handling.
+ if (replicas.isEmpty()
+ && container.getState() == LifeCycleState.CLOSED
+ && container.getNumberOfKeys() == 0) {
+ report.incrementAndSample(HealthState.EMPTY, container.containerID());
Review Comment:
I added the debug message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9602. Backport HDDS-9550 to legacy RM (missing containers which are empty) [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel merged PR #5527:
URL: https://github.com/apache/ozone/pull/5527
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org