You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "priyeshkaratha (via GitHub)" <gi...@apache.org> on 2025/07/07 07:42:37 UTC
[PR] HDDS-13192. Adding pending delete bytes info in storage report of DN [ozone]
priyeshkaratha opened a new pull request, #8755:
URL: https://github.com/apache/ozone/pull/8755
## What changes were proposed in this pull request?
### Accurate accounting of pending-delete bytes per container
- Added `incrPendingDeletions(long)` and `decrPendingDeletions(long)` to `ContainerData` with a new `pendingDeletionBytes` field that is persisted in container metadata.
- Integrated the counter with `DeleteBlocksCommandHandler`:
- Introduced helper method `incrPendingDeletionBytes()` which sums the sizes of blocks in each `DeletedBlocksTransaction` and updates the container’s counter when blocks are logically marked for deletion.
- When blocks are physically removed in `deleteViaTransactionStore()`, the counter is decremented accordingly.
### Datanode-level aggregation & reporting
- `StorageReport` now includes an aggregated `pendingDeletionBytes` value (sum across all containers).
- Introduced a short-lived in-memory cache (default TTL: 10 minutes) for the aggregated value to avoid recomputing on every heartbeat, reducing CPU overhead.
### Metrics & API exposure
- Exposed `pendingDeletionBytes` as a Datanode metric.
- Extended:
- Datanode-to-SCM heartbeat protobuf to include the new metric.
- Recon REST API (`/datanode`) to surface the pending deletion bytes.
### Upgrade path for existing clusters
- For containers created before the upgrade:
- During open, if `pendingDeletionBytes` is missing, it is computed using `KeyValueContainerUtil.populateContainerMetadata()` by scanning the block table.
- A background task reprocesses existing delete transactions and block metadata to compute accurate pending delete sizes.
- Mixed-version compatibility:
- Older DNs/SCMs gracefully ignore unknown fields.
- Recon detects support and falls back to legacy logic if needed.
## What is the link to the Apache JIRA
[HDDS-13192](https://issues.apache.org/jira/browse/HDDS-13192)
## How was this patch tested?
### Unit Tests
- Verified counter updates via unit tests in `ContainerData`.
- Added unit tests for `DeleteBlocksCommandHandler` flows that mark and delete blocks.
### Integration Testing
- Pending
### Upgrade & Backward Compatibility
- No impacts
### Performance Testing
- Pending
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13192. Adding pending delete bytes info in storage report of DN [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on PR #8755:
URL: https://github.com/apache/ozone/pull/8755#issuecomment-3090840155
> Our main goal here is to build a dashboard in Recon to show storage usage distribution across the cluster.
We should leave all dashboarding to Grafana instead of reimplementing the wheel in Recon.
> Recon already uses StorageReport for other usage stats, so we thought it would be a good idea to extend it to include pending deletion info as well, keeping everything in one place
StorageReport is not a catch-all to implement metrics collection in Recon, it is for reporting information about the disks within datanodes. In general Recon displays current categorical information (keys, containers, pipelines, volumes/disks) and metrics + dashboards track numeric information (bytes, durations, or event counts) over time. Deletion progress falls cleanly in the second category.
> We considered using Prometheus metrics, but based on my understanding. In ozone services these values might reset from beginning and become inaccurate in case of service restarts. This can lead to wrong conclusions in the dashboard.
Metrics collection is decoupled from persistence. Yes we will need to persist the counters in datanodes so they do not need to be recomputed on every restart, but this is independent of how the metrics are published and consumed by other services.
> Also I think to avoid dependency on Prometheus service running in this case (most customers do not have it set up when they hit issues related to deletion or anything related to space reclamation).
If any Ozone users want dashboards they will need Prometheus and Grafana. We do not have the bandwidth to build and maintain our own dashboarding setup when quality ones already exist.
This should be a reasonably small change if it is done the way Ozone is designed to handle it:
- Add metrics, with persistence underneath if necessary (this PR)
- Add the metrics to a dashboard, like [this](https://github.com/apache/ozone/blob/master/hadoop-ozone/dist/src/main/compose/common/grafana/dashboards/Ozone%20-%20DeleteKey%20Metrics.json) (follow-up PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13192. Adding pending delete bytes info in storage report of DN [ozone]
Posted by "priyeshkaratha (via GitHub)" <gi...@apache.org>.
priyeshkaratha commented on PR #8755:
URL: https://github.com/apache/ozone/pull/8755#issuecomment-3095197896
I agree with your points.
But here our goal is to enhance Recon's storage observability by providing a consolidated view of used capacity, including pending deletions. Recon already consumes the StorageReport for capacity metrics—extending this structure allows us to enrich the existing model without introducing a parallel system. I have already added metrics that can be published to prometheus too.
> If any Ozone users want dashboards they will need Prometheus and Grafana. We do not have the bandwidth to build and maintain our own dashboarding setup when quality ones already exist.
Are you telling that we should not rely on Recon for capacity distribution?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13192. Adding pending delete bytes info in storage report of DN [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on PR #8755:
URL: https://github.com/apache/ozone/pull/8755#issuecomment-3099100461
> Are you telling that we should not rely on Recon for capacity distribution?
"Capacity Distribution" will need a tighter definition in order to make this call. We should not rely on Recon for time series data, and the information added in this PR is time series data.
> Alternatively, we could introduce a new endpoint at the DataNode level, which Recon would call to gather and aggregate this information.
You have just described the process of Prometheus's metrics scraping : ) This is why we should not use Recon for this since Prometheus already handles this.
> since the storage report already includes other capacity-related fields, we considered extending it to include pending deletion information as well — which would simplify the implementation.
I don't see any evidence supporting this simplification claim. Making Ozone do this is far more complex to implement than leaving it to existing systems designed to handle time series data.
> Given that this data ultimately needs to be available at the Recon level — especially to enhance the current high-level view with more granular insights (e.g., how much space can be reclaimed after deletions or where deletions are stuck)
I disagree that this needs to be available in Recon. Recon is not designed to store or display time-series data. The terminology "stuck" used here is an indication that we are dealing with times series data, because it translates to deletion progress over time.
> which approach do you think is better?
Publish this time series data as metrics from the datanodes and let existing systems handle it much better than we can.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13192. Adding pending delete bytes info in storage report of DN [ozone]
Posted by "priyeshkaratha (via GitHub)" <gi...@apache.org>.
priyeshkaratha commented on PR #8755:
URL: https://github.com/apache/ozone/pull/8755#issuecomment-3096991543
@errose28 Followup to my previous comment.
To retrieve pending deletion data, one option is to query Prometheus metrics from each DataNode individually. Alternatively, we could introduce a new endpoint at the DataNode level, which Recon would call to gather and aggregate this information. However, since the storage report already includes other capacity-related fields, we considered extending it to include pending deletion information as well — which would simplify the implementation.
Given that this data ultimately needs to be available at the Recon level — especially to enhance the current high-level view with more granular insights (e.g., how much space can be reclaimed after deletions or where deletions are stuck) — which approach do you think is better?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13192. Adding pending delete bytes info in storage report of DN [ozone]
Posted by "priyeshkaratha (via GitHub)" <gi...@apache.org>.
priyeshkaratha commented on PR #8755:
URL: https://github.com/apache/ozone/pull/8755#issuecomment-3244431985
Based on the analysis, the JMX Metrics approach appears to be more efficient for exposing pending deletion information, primarily due to the reduced overhead and direct access for monitoring. While it introduces a JMX dependency, the benefits in terms of performance and data relevance outweigh the disadvantages of the Storage Report approach.
So, it will be handled as part of https://github.com/apache/ozone/pull/8995
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13192. Adding pending delete bytes info in storage report of DN [ozone]
Posted by "priyeshkaratha (via GitHub)" <gi...@apache.org>.
priyeshkaratha closed pull request #8755: HDDS-13192. Adding pending delete bytes info in storage report of DN
URL: https://github.com/apache/ozone/pull/8755
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13192. Adding pending delete bytes info in storage report of DN [ozone]
Posted by "priyeshkaratha (via GitHub)" <gi...@apache.org>.
priyeshkaratha commented on PR #8755:
URL: https://github.com/apache/ozone/pull/8755#issuecomment-3087541198
@errose28
Thank you for raising this valid point regarding the use of SCM for exposing pending delete information.
Our main goal here is to build a dashboard in Recon to show storage usage distribution across the cluster. Recon already uses StorageReport for other usage stats, so we thought it would be a good idea to extend it to include pending deletion info as well, keeping everything in one place.
We considered using Prometheus metrics, but based on my understanding. In ozone services these values might reset from beginning and become inaccurate in case of service restarts. This can lead to wrong conclusions in the dashboard. By passing the data through SCM to Recon, we can ensure more reliable and consistent reporting, even after restarts.
Also I think to avoid dependency on Prometheus service running in this case (most customers do not have it set up when they hit issues related to deletion or anything related to space reclamation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13192. Adding pending delete bytes info in storage report of DN [ozone]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #8755:
URL: https://github.com/apache/ozone/pull/8755#discussion_r2190587386
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/HddsVolume.java:
##########
@@ -624,4 +659,22 @@ public void compactDb() {
LOG.warn("compact rocksdb error in {}", dbFilePath, e);
}
}
+
+ private static class CachedPendingDeletion {
+ private final long size;
+ private final long expiryTimeMillis;
+
+ CachedPendingDeletion(long size, long cacheDurationMillis) {
+ this.size = size;
+ this.expiryTimeMillis = System.currentTimeMillis() + TimeUnit.MILLISECONDS.toMillis(cacheDurationMillis);
Review Comment:
[nitpick] The `cacheDurationMillis` is already in milliseconds, so wrapping it in `TimeUnit.MILLISECONDS.toMillis` is redundant. Consider simplifying to `System.currentTimeMillis() + cacheDurationMillis`.
```suggestion
this.expiryTimeMillis = System.currentTimeMillis() + cacheDurationMillis;
```
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/StorageLocationReport.java:
##########
@@ -362,7 +376,7 @@ public Builder setCommitted(long committed) {
}
/**
- * Sets the free space available to spare.
+ * Sets the pending deletetion bytes space available to spare.
Review Comment:
[nitpick] There is a typo in the Javadoc (`deletetion`) and the description is misleading. Please correct it to something like `Sets the bytes pending deletion for this storage location.`
```suggestion
* Sets the bytes pending deletion for this storage location.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org