You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "sumitagrawl (via GitHub)" <gi...@apache.org> on 2026/05/12 15:56:45 UTC
[PR] HDDS-15238. ContainerSafeModeRule containers list refresh with only removing deleted containers [ozone]
sumitagrawl opened a new pull request, #10249:
URL: https://github.com/apache/ozone/pull/10249
## What changes were proposed in this pull request?
Refresh for (RatisContainerSafeModeRule / ECContainerSafeModeRule) can result in adding newly created containers. But DN which has already registered, will not notify again newly created container to Safemode Rule.
So this can result in stuck of safemode rule if 99% of containers are not reported by DN.
As fix,
- keep same containers list in Safemode rule as initialized during startup
- remove deleted containers from the list and update reduced total containers, cutoff, threshold
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-15238
## How was this patch tested?
- TODO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15238. ContainerSafeModeRule containers list refresh with only removing deleted containers [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl merged PR #10249:
URL: https://github.com/apache/ozone/pull/10249
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15238. ContainerSafeModeRule containers list refresh with only removing deleted containers [ozone]
Posted by "sadanand48 (via GitHub)" <gi...@apache.org>.
sadanand48 commented on code in PR #10249:
URL: https://github.com/apache/ozone/pull/10249#discussion_r3246803648
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/AbstractContainerSafeModeRule.java:
##########
@@ -80,6 +81,27 @@ protected void initializeRule() {
totalContainers.set(containers.size());
final long cutOff = (long) Math.ceil(getTotalNumberOfContainers() * getSafeModeCutoff());
getSafeModeMetrics().setNumContainerReportedThreshold(getContainerType(), cutOff);
+ SCMSafeModeManager.getLogger().info("Initialized {} Containers threshold count to {}.", getContainerType(), cutOff);
+ }
+
+ protected void reinitializeRule() {
+ // Remove closed containers that are moved to deleted state as DN will not report those containers during
+ // registration. Update totalContainers, cutoff and threshold based on reduced containers.
+ // Since ContainerSafeModeRule is updated with container list notified during DN registration only,
+ // So its not required to add newly created container after DN registration.
+ int oldContainerCount = containers.size();
+ Set<ContainerID> containerInfoSet = containerManager.getContainers(getContainerType()).stream()
+ .filter(this::isClosed)
+ .filter(c -> c.getNumberOfKeys() > 0)
+ .filter(c -> containers.containsKey(ContainerID.valueOf(c.getContainerID())))
+ .map(c -> ContainerID.valueOf(c.getContainerID()))
+ .collect(Collectors.toSet());
+ // remove deleted containers from containers list
+ containers.keySet().removeIf(c -> !containerInfoSet.contains(c));
+ // update new total with reducing removed containers
+ totalContainers.set(totalContainers.get() - (oldContainerCount - containers.size()));
Review Comment:
I think this can be simply set to
totalContainers.set(containers.size())
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15238. ContainerSafeModeRule containers list refresh with only removing deleted containers [ozone]
Posted by "sadanand48 (via GitHub)" <gi...@apache.org>.
sadanand48 commented on code in PR #10249:
URL: https://github.com/apache/ozone/pull/10249#discussion_r3247205647
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/AbstractContainerSafeModeRule.java:
##########
@@ -80,6 +81,27 @@ protected void initializeRule() {
totalContainers.set(containers.size());
final long cutOff = (long) Math.ceil(getTotalNumberOfContainers() * getSafeModeCutoff());
getSafeModeMetrics().setNumContainerReportedThreshold(getContainerType(), cutOff);
+ SCMSafeModeManager.getLogger().info("Initialized {} Containers threshold count to {}.", getContainerType(), cutOff);
+ }
+
+ protected void reinitializeRule() {
+ // Remove closed containers that are moved to deleted state as DN will not report those containers during
+ // registration. Update totalContainers, cutoff and threshold based on reduced containers.
+ // Since ContainerSafeModeRule is updated with container list notified during DN registration only,
+ // So its not required to add newly created container after DN registration.
+ int oldContainerCount = containers.size();
+ Set<ContainerID> containerInfoSet = containerManager.getContainers(getContainerType()).stream()
+ .filter(this::isClosed)
+ .filter(c -> c.getNumberOfKeys() > 0)
+ .filter(c -> containers.containsKey(ContainerID.valueOf(c.getContainerID())))
+ .map(c -> ContainerID.valueOf(c.getContainerID()))
+ .collect(Collectors.toSet());
+ // remove deleted containers from containers list
+ containers.keySet().removeIf(c -> !containerInfoSet.contains(c));
+ // update new total with reducing removed containers
+ totalContainers.set(totalContainers.get() - (oldContainerCount - containers.size()));
Review Comment:
Based on discussion:
> No, containers will be having only not-notified containers, not the total containers. So need to update with diff value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org