You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "devmadhuu (via GitHub)" <gi...@apache.org> on 2025/12/10 08:16:43 UTC
[PR] HDDS-14119. Ozone SCM - Capture all container replication status in container info. [ozone]
devmadhuu opened a new pull request, #9472:
URL: https://github.com/apache/ozone/pull/9472
## What changes were proposed in this pull request?
This PR updates existing SCM `ReplicationManager` to capture all container replication statuses in existing `ContainerInfo` object. The implementation uses a simple enum-based approach with short values, enabling SCM to track both individual health states (e.g., `UNDER_REPLICATED`, `MISSING`) and common combinations (e.g., `UNDER_REPLICATED_MIS_REPLICATED`) efficiently.
**Key Features**
1. `ContainerHealthState` Enum
- Individual states (values 0-99): 11 health states including `HEALTHY`, `UNDER_REPLICATED`, `MIS_REPLICATED`, `OVER_REPLICATED`, `MISSING`, `UNHEALTHY`, `EMPTY`, `OPEN_UNHEALTHY`, `QUASI_CLOSED_STUCK`,
`OPEN_WITHOUT_PIPELINE`, and `REPLICA_MISMATCH`
- Combination states (values 100+): 8 common combinations like `MISSING_EMPTY`, `UNDER_REPLICATED_MIS_REPLICATED`, etc.
2. `HealthStateFilter` API
- Simple filtering mechanism for querying containers by exact health state match
- Factory methods: `HealthStateFilter.of(state)` for single state or multiple states
- Clean matches() method for container filtering
3. `ContainerInfo` Integration
- New healthState field (short, 2 bytes per container)
- New healthStateUpdateTime field for tracking when health state changed
- Automatic timestamp updates when health state is modified
- Protobuf serialization support
4. `ContainerManager` Query API
```
List<ContainerInfo> getContainersByHealthState(HealthStateFilter filter);
Map<ContainerInfo, ContainerHealthState> getContainersWithHealthState(HealthStateFilter filter);
```
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-14119
## How was this patch tested?
This patch is tested using new test cases:
`TestContainerHealthState` covering:
- All individual states and combinations
- Conversion from ReplicationManager states
- Protobuf serialization/deserialization
- Health state filtering and querying
- Edge cases and invalid states
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14119. Capture all container replication status in SCM container info [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #9472:
URL: https://github.com/apache/ozone/pull/9472#discussion_r2621913754
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerManagerImpl.java:
##########
@@ -173,6 +175,46 @@ public int getContainerStateCount(final LifeCycleState state) {
return containerStateManager.getContainerCount(state);
}
+ @Override
+ public List<ContainerInfo> getContainersByHealthState(
+ HealthStateFilter filter) {
+ scmContainerManagerMetrics.incNumListContainersOps();
+ List<ContainerInfo> result = new ArrayList<>();
+
+ // Iterate through all containers using a large count
+ List<ContainerInfo> allContainers =
+ containerStateManager.getContainerInfos(ContainerID.MIN, Integer.MAX_VALUE);
+
+ for (ContainerInfo containerInfo : allContainers) {
+ if (containerInfo.getHealthState() != null &&
+ filter.matches(containerInfo.getHealthState())) {
+ result.add(containerInfo);
+ }
+ }
+
+ return result;
+ }
+
+ @Override
+ public Map<ContainerInfo, ContainerHealthState> getContainersWithHealthState(
Review Comment:
we can remove unused methods, can be implemented with optimized way
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/ReplicationManager.java:
##########
@@ -878,7 +879,15 @@ protected boolean processContainer(ContainerInfo containerInfo,
if (!handled) {
Review Comment:
We can set state inside Report about container, initially healthy. And once finished, overwrite the actual state at below logic
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/health/ClosedWithUnhealthyReplicasHandler.java:
##########
@@ -109,9 +109,9 @@ public boolean handle(ContainerCheckRequest request) {
// some unhealthy replicas were found so the container must be
// over replicated due to unhealthy replicas.
if (foundUnhealthy) {
- request.getReport().incrementAndSample(
- ReplicationManagerReport.HealthState.OVER_REPLICATED,
- containerInfo.containerID());
+ ContainerHealthState healthState = ContainerHealthState.OVER_REPLICATED;
+ request.getReport().incrementAndSample(healthState, containerInfo.containerID());
+ containerInfo.setHealthState(healthState);
Review Comment:
move this inside incrementAndSample()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14119. Capture all container replication status in SCM container info [ozone]
Posted by "devmadhuu (via GitHub) via issues" <is...@ozone.apache.org>.
devmadhuu commented on PR #9472:
URL: https://github.com/apache/ozone/pull/9472#issuecomment-6604756097
> > ReplicationManagerReport.HealthState
>
> Could you point out the code? I cannot find the HealthState class.
>
> > ContainerHealthResult.HealthState represented ...
>
> We should redesign ContainerHealthResult. HEALTHY and UNHEALTHY are ill-defined states.
>
> > QUASI_CLOSED already exists in HddsProtos.LifeCycleState
>
> You are right that we should reuse LifeCycleState. Then, we don't need "Open/close: OPEN_WITH_PIPELINE, OPEN_WITHOUT_PIPELINE, CLOSE, QUASI_CLOSED". We could have a
>
> * Pipeline: HAS_PIPELINE, NO_PIPELINE
>
>
> Your suggestion sounds great! Thanks.
Apologies for responding late !
On _Could you point out the code? I cannot find the HealthState class._
This was a enum in pre-PR code [here](https://github.com/apache/ozone/pull/9472/changes#diff-a4a0cc308d982e1596a0921cac768f8f0c1fe94f4f603f2ab76a54dd6f1532beL273).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14119. Capture all container replication status in SCM container info [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #9472:
URL: https://github.com/apache/ozone/pull/9472#discussion_r2626112957
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerInfo.java:
##########
@@ -118,8 +125,7 @@ public static ContainerInfo fromProtobuf(HddsProtos.ContainerInfoProto info) {
.setContainerID(info.getContainerID())
.setDeleteTransactionId(info.getDeleteTransactionId())
.setReplicationConfig(config)
- .setSequenceId(info.getSequenceId())
- .build();
Review Comment:
This is a nice find, but it should be fixed separately so that we can backport it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14119. Capture all container replication status in SCM container info [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on PR #9472:
URL: https://github.com/apache/ozone/pull/9472#issuecomment-3695395449
Thanks @errose28 and @sodonnel for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14119. Capture all container replication status in SCM container info [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on PR #9472:
URL: https://github.com/apache/ozone/pull/9472#issuecomment-3675902009
There is something about storing the health state in ContainerInfo that doesn't feel correct to me. The state is captured at a point in time and then its stale soon afterwards. It doesn't get updated until the next run of RM. The thing about the report object, was that it captured the stats of a complete run of RM, but in the new way, the container infos get changed as RM runs. I guess it should be pretty fast, but it could lead to kind of unstable numbers.
I cannot really come up with a concrete reason as to why I think using container Info for this is wrong, aside from it only being updated by RM with its periodic runs. Perhaps I am over thinking it and its fine.
I think there are also some places that use RM in a read only mode to check container states (decommission maybe), so that may update the containerInfo states between RM runs. I am not sure if that is a problem or not. Probably not as it can only make the state more current.
Aside from the above scanning the PR quickly, the thing I am not sure about is multiplying up the states - like under_replicated, unhealthy_under_replicate, qc_under_replicated ... It leads to a lot more states that may just be more confusing than helpful.
In the RM report, we tried to only have a container in a single state, but it can be unhealthy and under / over replicated. It can be missing and under-replicated I think. Missing is kind of an extreme version of under-replicated. The only way to capture these "double states" with a single field is to multiple up the states I guess.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14119. Capture all container replication status in SCM container info [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on PR #9472:
URL: https://github.com/apache/ozone/pull/9472#issuecomment-6477345275
> It sounds great. Please do the follow-up.
@devmadhuu , Actually, if you don't mind, we should redo it. We should avoid ambiguous terms like Healthy and Stuck.
We could have serval dimensions like below. In each dimension, the states are mutually exclusive.
- Replication: CORRECTLY_REPLICATED, UNDER_REPLICATED, OVER_REPLICATED, INSUFFICIENT_RACKS, NO_ONLINE_REPLICA
- Open/close: OPEN_WITH_PIPELINE, OPEN_WITHOUT_PIPELINE, CLOSE, QUASI_CLOSED
- Consistency: CONSISTENT, INCONSISTENT
- Recoverability: AUTO_RECOVERABLE, UNRECOVERABLE
What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14119. Capture all container replication status in SCM container info [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on PR #9472:
URL: https://github.com/apache/ozone/pull/9472#issuecomment-6297151678
@devmadhuu , some questions:
- It seems that the long description in the PR are mostly invalid. It particular "storing a 2-byte health state" and all the "memory efficient" calculations are invalid since the ContainerInfo is using the enum directly but not a short. If yes, could you revise the description?
```java
// ContainerInfo
private ContainerHealthState healthState;
```
- Also, the ContainerInfo.value actually is unused -- it is only used in tests. If yes, could we remove it?
- BTW, how the value is assigned?
- Define `ContainerHealthState.HEALTHY`. The description "Container is healthy" does not explain what does it mean.
- UNDER_REPLICATED as an example, what are the difference between the cases below?
1. UNDER_REPLICATED
2. UNHEALTHY_UNDER_REPLICATED
3. MISSING_UNDER_REPLICATED
4. QUASI_CLOSED_STUCK_UNDER_REPLICATED
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14119. Capture all container replication status in SCM container info [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on PR #9472:
URL: https://github.com/apache/ozone/pull/9472#issuecomment-6489742940
> > It sounds great. Please do the follow-up.
>
> @devmadhuu , Actually, if you don't mind, we should redo it. We should avoid ambiguous terms like Healthy and Stuck.
>
> We could have serval dimensions like below. In each dimension, the states are mutually exclusive.
>
> * Replication: CORRECTLY_REPLICATED, UNDER_REPLICATED, OVER_REPLICATED, NO_ONLINE_REPLICA
>
> * Racks: INSUFFICIENT_RACKS, ENOUGH_RACKS
>
> * Open/close: OPEN_WITH_PIPELINE, OPEN_WITHOUT_PIPELINE, CLOSE, QUASI_CLOSED
>
> * Consistency: CONSISTENT, INCONSISTENT
>
> * Corruption: CORRUPTION_DETECTED, CHECKSUM_MATCHED
>
> * Recoverability: AUTO_RECOVERABLE, MANUAL_RECOVERY_REQUIRED
>
>
> What do you think?
Thanks @szetszwo for your proposal. On a broader context, it makes sense, but proposed states you mentioned and used in this PR were not created or redefined. During the development of ReplicationManager feature, some limited health check handlers for containers were being developed, so now they are not just a renaming exercise or defining the precise definitions, we need appropriate handlers in `ReplicationManager` with thought process on impacts, compatibility and existing QE and dev test cases. This needs a design discussion on broader area.
Do you think we should create a separate tracker JIRA to revamp various health states being reported as part of RM as per your above proposal where we can have broader discussion on impact ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14119. Capture all container replication status in SCM container info [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #9472:
URL: https://github.com/apache/ozone/pull/9472#discussion_r2626139551
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerInfo.java:
##########
@@ -118,8 +125,7 @@ public static ContainerInfo fromProtobuf(HddsProtos.ContainerInfoProto info) {
.setContainerID(info.getContainerID())
.setDeleteTransactionId(info.getDeleteTransactionId())
.setReplicationConfig(config)
- .setSequenceId(info.getSequenceId())
- .build();
Review Comment:
Created HDDS-14196.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14119. Capture all container replication status in SCM container info [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on PR #9472:
URL: https://github.com/apache/ozone/pull/9472#issuecomment-3712024658
> There is something about storing the health state in ContainerInfo that doesn't feel correct to me. The state is captured at a point in time and then its stale soon afterwards. It doesn't get updated until the next run of RM. The thing about the report object, was that it captured the stats of a complete run of RM, but in the new way, the container infos get changed as RM runs. I guess it should be pretty fast, but it could lead to kind of unstable numbers.
This is true of almost all in-memory state that SCM has. A lot of the metrics we are tracking to follow the cluster state are derived from container report information. Same with Recon which also has an API to list all unhealthy containers. I think the container report still has merit as a point in time snapshot of the replication manager and we should leave it as is, but I don't think that should exclude us from adding additional functionality like querying SCM's current in-memory view of the cluster.
If there's a different place to maintain this information which still supports querying containers by health state we can use that instead, but to me the in-memory `ContainerInfo` object looks like the best place to store it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14119. Capture all container replication status in SCM container info [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on PR #9472:
URL: https://github.com/apache/ozone/pull/9472#issuecomment-6491084898
> I suggest creating a tracker JIRA for these definitions and the reporting model, with a separate small follow-up for the unused numeric machinery. Would that separation work for you? Created this JIRA : [HDDS-17556](https://issues.apache.org/jira/browse/HDDS-17556) to track
Created a follow up JIRA: [HDDS-17557](https://issues.apache.org/jira/browse/HDDS-17557)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14119. Capture all container replication status in SCM container info [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #9472:
URL: https://github.com/apache/ozone/pull/9472#issuecomment-3670358534
@sodonnel please take a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14119. Capture all container replication status in SCM container info [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #9472:
URL: https://github.com/apache/ozone/pull/9472#discussion_r2629595063
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/health/ClosedWithUnhealthyReplicasHandler.java:
##########
@@ -109,9 +109,7 @@ public boolean handle(ContainerCheckRequest request) {
// some unhealthy replicas were found so the container must be
// over replicated due to unhealthy replicas.
if (foundUnhealthy) {
- request.getReport().incrementAndSample(
- ReplicationManagerReport.HealthState.OVER_REPLICATED,
- containerInfo.containerID());
+ request.getReport().incrementAndSample(ContainerHealthState.OVER_REPLICATED, containerInfo);
Review Comment:
this should be OVER_REPLICATED_UNHEALTHY
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/container/ReplicationManagerReport.java:
##########
@@ -80,16 +80,17 @@ public int getSampleLimit() {
return sampleLimit;
}
- public void increment(HealthState stat) {
- increment(stat.toString());
+ public void increment(ContainerHealthState stat) {
+ increment(stat.name());
}
- public void increment(HddsProtos.LifeCycleState stat) {
- increment(stat.toString());
+ public void incrementAndSample(ContainerHealthState stat, ContainerInfo containerInfo) {
+ incrementAndSample(stat.name(), containerInfo.containerID());
+ containerInfo.setHealthState(stat);
Review Comment:
We can do this at caller in RepliationManager after `boolean handled = containerCheckChain.handleChain(checkRequest);`. This we can set in local variable in this report, initially as HEALTHY in local variable and keep updating same. When task is finished, it can be updated with final value.
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerHealthState.java:
##########
@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.container;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Enum representing container health states.
+ *
+ * <p>Each health state has a unique short value (2 bytes), description, and metric name.
+ * This enum provides named constants for:
+ * - Individual health states (e.g., UNDER_REPLICATED)
+ * - Actual combinations that occur in replication handlers
+ *
+ * <p>Individual states use values 0-99, combinations use values 100+.
+ *
+ * <p>This enum replaces ReplicationManagerReport.HealthState and is used both for
+ * tracking container health in ContainerInfo and for metrics/reporting.
+ */
+public enum ContainerHealthState {
+
+ // ========== Individual Health States (0-99) ==========
+
+ /**
+ * Container is healthy with no issues.
+ */
+ HEALTHY((short) 0,
+ "Container is healthy",
+ "HealthyContainers"),
+
+ /**
+ * Container has insufficient replicas.
+ */
+ UNDER_REPLICATED((short) 1,
+ "Containers with insufficient replicas",
+ "UnderReplicatedContainers"),
+
+ /**
+ * Container violates placement policy.
+ */
+ MIS_REPLICATED((short) 2,
+ "Containers with insufficient racks",
+ "MisReplicatedContainers"),
+
+ /**
+ * Container has excess replicas.
+ */
+ OVER_REPLICATED((short) 3,
+ "Containers with more replicas than required",
+ "OverReplicatedContainers"),
+
+ /**
+ * Critical: No online replicas available.
+ */
+ MISSING((short) 4,
+ "Containers with no online replicas",
+ "MissingContainers"),
+
+ /**
+ * Closed/Quasi-Closed with inconsistent replica states.
+ */
+ UNHEALTHY((short) 5,
+ "Containers Closed or Quasi_Closed having some replicas in a different state",
+ "UnhealthyContainers"),
+
+ /**
+ * Container has no blocks (metadata only).
+ */
+ EMPTY((short) 6,
+ "Containers having no blocks",
+ "EmptyContainers"),
+
+ /**
+ * Open container with inconsistent replica states.
+ */
+ OPEN_UNHEALTHY((short) 7,
+ "Containers open and having replicas with different states",
+ "OpenUnhealthyContainers"),
+
+ /**
+ * Container stuck in QUASI_CLOSED state.
+ */
+ QUASI_CLOSED_STUCK((short) 8,
+ "Containers QuasiClosed with insufficient datanode origins",
+ "StuckQuasiClosedContainers"),
+
+ /**
+ * Open container without healthy pipeline.
+ */
+ OPEN_WITHOUT_PIPELINE((short) 9,
+ "Containers in OPEN state without any healthy Pipeline",
+ "OpenContainersWithoutPipeline"),
+
+ // ========== Actual Combinations Found in Code (100+) ==========
+
+ /**
+ * Container is unhealthy AND under-replicated.
+ * Occurs in RatisUnhealthyReplicationCheckHandler when container has only
+ * unhealthy replicas and needs more replicas.
+ */
+ UNHEALTHY_UNDER_REPLICATED((short) 100,
+ "Inconsistent states with insufficient replicas",
+ "UnhealthyUnderReplicatedContainers"),
+
+ /**
+ * Container is unhealthy AND over-replicated.
+ * Occurs in RatisUnhealthyReplicationCheckHandler when container has only
+ * unhealthy replicas and has too many replicas.
+ */
+ UNHEALTHY_OVER_REPLICATED((short) 101,
+ "Inconsistent states with excess replicas",
+ "UnhealthyOverReplicatedContainers"),
+
+ /**
+ * Container is missing AND under-replicated.
+ * Occurs in ECReplicationCheckHandler when EC container is unrecoverable (missing)
+ * and also has unreplicated offline indexes needing replication.
+ */
+ MISSING_UNDER_REPLICATED((short) 102,
+ "No online replicas with offline indexes needing replication",
+ "MissingUnderReplicatedContainers"),
+
+ /**
+ * Container is quasi-closed-stuck AND under-replicated.
+ * Occurs in QuasiClosedStuckReplicationCheck when container is stuck in QUASI_CLOSED
+ * and also needs more replicas.
+ */
+ QUASI_CLOSED_STUCK_UNDER_REPLICATED((short) 103,
+ "Stuck in quasi-closed state with insufficient replicas",
+ "QuasiClosedStuckUnderReplicatedContainers"),
+
+ /**
+ * Container is quasi-closed-stuck AND over-replicated.
+ * Occurs in QuasiClosedStuckReplicationCheck when container is stuck in QUASI_CLOSED
+ * and also has excess replicas.
+ */
+ QUASI_CLOSED_STUCK_OVER_REPLICATED((short) 104,
+ "Stuck in quasi-closed state with excess replicas",
+ "QuasiClosedStuckOverReplicatedContainers");
+
+ // ========== Enum Fields ==========
+
+ private final short value;
+ private final String description;
+ private final String metricName;
+
+ // Static lookup map for efficient fromValue()
+ private static final Map<Short, ContainerHealthState> VALUE_MAP = new HashMap<>();
+
+ static {
+ for (ContainerHealthState state : values()) {
+ VALUE_MAP.put(state.value, state);
+ }
+ }
+
+ // Constructor for all states
+ ContainerHealthState(short value, String description, String metricName) {
+ this.value = value;
+ this.description = description;
+ this.metricName = metricName;
+ }
+
+ public short getValue() {
+ return value;
+ }
+
+ public String getDescription() {
+ return description;
+ }
+
+ /**
+ * Get the metric name for this health state.
+ * Used for metrics reporting in ReplicationManager.
+ *
+ * @return Metric name string
+ */
+ public String getMetricName() {
+ return metricName;
+ }
+
+ /**
+ * Check if container is healthy (no issues).
+ *
+ * @return true if this is HEALTHY
+ */
+ public boolean isHealthy() {
Review Comment:
This is not required
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/health/VulnerableUnhealthyReplicasHandler.java:
##########
@@ -83,7 +84,7 @@ public boolean handle(ContainerCheckRequest request) {
if (!vulnerableUnhealthy.isEmpty()) {
LOG.info("Found vulnerable UNHEALTHY replicas {} for container {}.", vulnerableUnhealthy, container);
ReplicationManagerReport report = request.getReport();
- report.incrementAndSample(ReplicationManagerReport.HealthState.UNDER_REPLICATED, container.containerID());
+ report.incrementAndSample(ContainerHealthState.UNDER_REPLICATED, container);
Review Comment:
should be UNDER_REPLICATED_UNHEALTHY
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/health/QuasiClosedStuckReplicationCheck.java:
##########
@@ -65,8 +65,7 @@ public boolean handle(ContainerCheckRequest request) {
if (request.getContainerReplicas().isEmpty()) {
// If there are no replicas, then mark as missing and return.
- request.getReport().incrementAndSample(
- ReplicationManagerReport.HealthState.MISSING, request.getContainerInfo().containerID());
+ request.getReport().incrementAndSample(ContainerHealthState.MISSING, request.getContainerInfo());
Review Comment:
need set QUASI_CLOSED_STUCK_MISSING
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14119. Capture all container replication status in SCM container info [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on PR #9472:
URL: https://github.com/apache/ozone/pull/9472#issuecomment-3736870724
@errose28 can you pls re-review ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14119. Capture all container replication status in SCM container info [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on code in PR #9472:
URL: https://github.com/apache/ozone/pull/9472#discussion_r2626174867
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerInfo.java:
##########
@@ -118,8 +125,7 @@ public static ContainerInfo fromProtobuf(HddsProtos.ContainerInfoProto info) {
.setContainerID(info.getContainerID())
.setDeleteTransactionId(info.getDeleteTransactionId())
.setReplicationConfig(config)
- .setSequenceId(info.getSequenceId())
- .build();
Review Comment:
Ok sure, I have reverted to original code, Will push it separately. Thank you for your review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14119. Capture all container replication status in SCM container info [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on PR #9472:
URL: https://github.com/apache/ozone/pull/9472#issuecomment-3695400905
> There is something about storing the health state in ContainerInfo that doesn't feel correct to me. The state is captured at a point in time and then its stale soon afterwards. It doesn't get updated until the next run of RM. The thing about the report object, was that it captured the stats of a complete run of RM, but in the new way, the container infos get changed as RM runs. I guess it should be pretty fast, but it could lead to kind of unstable numbers.
>
> I cannot really come up with a concrete reason as to why I think using container Info for this is wrong, aside from it only being updated by RM with its periodic runs. Perhaps I am over thinking it and its fine.
>
> I think there are also some places that use RM in a read only mode to check container states (decommission maybe), so that may update the containerInfo states between RM runs. I am not sure if that is a problem or not. Probably not as it can only make the state more current.
>
> Aside from the above scanning the PR quickly, the thing I am not sure about is multiplying up the states - like under_replicated, unhealthy_under_replicate, qc_under_replicated ... It leads to a lot more states that may just be more confusing than helpful.
>
> In the RM report, we tried to only have a container in a single state, but it can be unhealthy and under / over replicated. It can be missing and under-replicated I think. Missing is kind of an extreme version of under-replicated. The only way to capture these "double states" with a single field is to multiple up the states I guess.
Yes , with this new way, ContainerInfo object will hold its state (multiple or just single) and there is a possibility of changing it between two different RM runs. But in real time, that may be good also as it will reflect the current state even in read only mode. Could you please summarize your points to get better understanding how the current behavior may be an issue, As per my understanding, that should not be an issue, but still looking for deeper understanding from your contextual thinking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14119. Capture all container replication status in SCM container info [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on PR #9472:
URL: https://github.com/apache/ozone/pull/9472#issuecomment-3717389314
@errose28 @sodonnel , I have removed short byte value and now using enum, pushed the code. Also current PR is having sampling present to provide only first 100 unhealthy containers, but in future PRs, we can remove sampling considering not much impact on memory as we saw with above computation analysis. so that in future , below CLI can provide any specific unhealthy state containers with some limit option, something like below:
`ozone admin container report --health-state UNDER_REPLICATED --limit 100`
Because with every RM run, we are updating the state in ContainerInfo also in this PR, so not sure if need both.
Based on your input if we need to keep sampling , I need to do code changes in this PR : #9258 to keep consistency of container health data in sync with SCM report.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14119. Capture all container replication status in SCM container info [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on PR #9472:
URL: https://github.com/apache/ozone/pull/9472#issuecomment-6433217038
@devmadhuu , thanks for the explaination!
> I propose a small follow-up PR to remove the numeric field, ...
It sounds great. Please do the follow-up.
Since it is for CLI report and metrics, let's also rename the ContainerHealthState enum to something like HealthStateMetricName.
> **QUASI_CLOSED_STUCK_UNDER_REPLICATED** - A container cannot safely finish closing, and the check for stuck quasi-closed containers also finds too few copies of the remaining original replicas.
But why there are no QUASI_CLOSED without STUCK?
The current enum definitions are non-precise and incomplete. We need to come up with some precise definitions first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14119. Capture all container replication status in SCM container info [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on PR #9472:
URL: https://github.com/apache/ozone/pull/9472#issuecomment-3837119034
> @errose28 can you pls re-review ?
Merged now as long time for no reivew, any further comments can be takenup as new JIRA
cc: @devmadhuu
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14119. Capture all container replication status in SCM container info [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl merged PR #9472:
URL: https://github.com/apache/ozone/pull/9472
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14119. Capture all container replication status in SCM container info [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on PR #9472:
URL: https://github.com/apache/ozone/pull/9472#issuecomment-3695402455
> > private short healthStateValue; // 2 bytes - memory efficient!
>
> Looks like an AI optimization, which means it has no context about real world clusters. I recently worked on a cluster with 32 million containers and 200PB+ of data running with 127GB of heap. Storing this as a short will consume 32Mil * 2B = **64MB** of space. Storing it as an enum, which is much easier for devs to work with, will consume 32Mil * 8B = **256MB** of space. (256mb - 64mb)/127gb = 0.001
>
> So all this inconvenience is for a 0.1% reduction in heap usage at scale.
>
> I do think we should track the container health state in the container info object in memory for easier queries, but based on real world numbers we can just use an enum for this.
Thanks @errose28 for providing your real time large sized cluster based computation on memory foot print. I agree with your explanation. We can work with enum itself rather short value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14119. Capture all container replication status in SCM container info [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on PR #9472:
URL: https://github.com/apache/ozone/pull/9472#issuecomment-3677721624
> In the RM report, we tried to only have a container in a single state, but it can be unhealthy and under / over replicated
I remembered why a container has 2 states. Its got a "lifecycle state" - open, closed, closing, qc, qc_stuck, unhealthy
And it has "replication states" - under, over and mis-replicated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14119. Capture all container replication status in SCM container info [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on PR #9472:
URL: https://github.com/apache/ozone/pull/9472#issuecomment-6398801808
> @devmadhuu , some questions:
>
> * It seems that the long description in the PR are mostly invalid. It particular "storing a 2-byte health state" and all the "memory efficient" calculations are invalid since the ContainerInfo is using the enum directly but not a short. If yes, could you revise the description?
>
>
> ```java
> // ContainerInfo
> private ContainerHealthState healthState;
> ```
>
> * Also, the ContainerInfo.value actually is unused -- it is only used in tests. If yes, could we remove it?
>
> * BTW, how the value is assigned?
>
> * Define `ContainerHealthState.HEALTHY`. The description "Container is healthy" does not explain what does it mean.
>
> * Honestly, I also don't know the definitions for the other enums. The description field is unclear about the exact meaning. We need precise definitions. @sumitagrawl , do you know the precise definitions?
>
> * UNDER_REPLICATED as an example, what are the difference between the cases below?
>
> 1. UNDER_REPLICATED
> 2. UNHEALTHY_UNDER_REPLICATED
> 3. MISSING_UNDER_REPLICATED
> 4. QUASI_CLOSED_STUCK_UNDER_REPLICATED
>
> * Why there are QUASI_CLOSED_STUCK_... enums but not QUASI_CLOSED without STUCK?
>
> * What was the goal for this PR? For metric and UI reports? Anything else?
>
>
> Cc @umamaheswararao
@szetszwo
1. You are right. The description reflects an earlier version that stored the health state as a short. I changed that to an enum reference following the earlier review, but missed updating the description.
The merged code stores `ContainerHealthState` directly, so the two-byte-per-container claim and the memory calculations based on it are not in line per code. I have updated the description and remove those calculations.
2. Yes, you are right, the numeric value belongs to `ContainerHealthState` and was left over from the earlier short-based implementation. I found no public API or callers of its numeric getter or conversion method in the checked code; the callers are tests.
The numbers are assigned explicitly in the enum declarations: 0–9 for individual states and 100–105 for combinations. They are not calculated from the container’s condition. The checks select an enum constant directly.
I propose a small follow-up PR to remove the numeric field, lookup map, conversion methods, and tests that only check those numbers. The enum names and metric names would remain.
3. I agree that the descriptions for various container health and states need to be more precise.
In the merged implementation, each container check starts with `HEALTHY`. If a check records another condition, that replaces it. Otherwise, the saved value remains HEALTHY.
It is also the default when `ContainerInfo` is created, before `ReplicationManager` checks it. Some checks stop processing early—for example, the closing-container check—so this value does not guarantee that every replication and placement check has passed.
I propose a follow-up PR to document the actual conditions for each state. We should agree on any stronger meaning for `HEALTHY` before changing its behavior and I think this would have had long history which @sodonnel might know better.
As per my best understanding on meaning of below states for example:
**`UNDER_REPLICATED`** - a Ratis container requiring three copies has only two healthy copies.
**`UNHEALTHY_UNDER_REPLICATED`** - More copies are needed in a situation involving unhealthy replicas. One example is a Ratis container with no healthy copies and fewer unhealthy copies than required.
**`MISSING_UNDER_REPLICATED`** - An EC container lacks enough distinct indexes to recover. A particularly useful example: EC MISSING does not necessarily mean zero replicas. For EC 3+2, retaining only indexes 1 and 2 is insufficient to reconstruct the data. If index 2 is on a decommissioning node without an in-service copy, the combination can be `MISSING_UNDER_REPLICATED`.
`**QUASI_CLOSED_STUCK_UNDER_REPLICATED**` - A container cannot safely finish closing, and the check for stuck quasi-closed containers also finds too few copies of the remaining original replicas.
4. The main goal was to retain `ReplicationManager’s` health result on each in-memory ContainerInfo, so that same can be used by admin CLI for full container report, metrics and reports use the enum too, but the future goal also was to include supporting queries by health state. After this PR, a health-filtered container ID listing in SCM through [PR #10736 (https://github.com/apache/ozone/commit/575fe4f1cd7bdd9f6226051fcf7c260d4a24e1c8)](https://github.com/apache/ozone/commit/575fe4f1cd7bdd9f6226051fcf7c260d4a24e1c8), was done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14119. Capture all container replication status in SCM container info [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on PR #9472:
URL: https://github.com/apache/ozone/pull/9472#issuecomment-6528798209
> ReplicationManagerReport.HealthState
Could you point out the code? I cannot find the HealthState class.
> ContainerHealthResult.HealthState represented ...
We should redesign ContainerHealthResult. HEALTHY and UNHEALTHY are ill-defined states.
> QUASI_CLOSED already exists in HddsProtos.LifeCycleState
You are right that we should reuse LifeCycleState. Then, we don't need "Open/close: OPEN_WITH_PIPELINE, OPEN_WITHOUT_PIPELINE, CLOSE, QUASI_CLOSED". We could have a
- Pipeline: HAS_PIPELINE, NO_PIPELINE
Your suggestion sounds great! Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org