You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "prathmesh12-coder (via GitHub)" <gi...@apache.org> on 2026/06/22 11:23:35 UTC

[PR] HDDS-15333. Expose volume and bucket utilization metrics on OM /prom [ozone]

prathmesh12-coder opened a new pull request, #10576:
URL: https://github.com/apache/ozone/pull/10576

   
   ## What changes were proposed in this pull request?
   
   This PR includes OM Prometheus metrics on /prom with volume-level utilization gauges and adds a missing bucket namespace usage gauge.
   
   OM already exposes bucket utilization metrics (bucket_utilization_metrics_*) on /prom, but volume-level used/available byte metrics were not exposed.
   
   Requested improvement:
   -> Expose OM Prometheus metrics for below.
   * volume used bytes
   * volume quota bytes
   * volume available bytes, if applicable
   * volume namespace quota
   * volume used namespace
   * bucket used namespace

   
   -> This would allow operators to build Grafana dashboards and alerts for both volume and bucket quota usage without relying on custom exporters or CLI polling.
   
   Changes made:
   1. Added VolumeUtilizationMetrics — a new MetricsSource registered on OM that publishes, per volume:
       * VolumeQuotaBytes — byte quota (-1 if unlimited)
       * VolumeUsedBytes — sum of usedBytes across all buckets in the volume
       * VolumeAvailableBytes — quota - used when a byte quota is set, otherwise -1
       * VolumeQuotaNamespace — existing namespace quota
       * VolumeUsedNamespace —  existing namespace usage

       
   2. Added OMMetadataManager.getVolumeIterator() — mirrors the existing getBucketIterator() API so volume metrics can iterate OM metadata.

   
   3. Extended BucketUtilizationMetrics — exposed BucketUsedNamespace (number of keys/directories in the bucket), which was previously missing from /prom. 

   
   4. On /prom, these appear as volume_utilization_metrics_* and bucket_utilization_metrics_bucket_used_namespace, following the same normalization as existing bucket utilization metrics.
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-15333
   
   ## How was this patch tested?
   
   * Unit tests
       * TestVolumeUtilizationMetrics — verifies volume quota, used bytes, available bytes, and namespace gauges.
       * TestBucketUtilizationMetrics — updated to cover BucketUsedNamespace.

       
   * Manual testing (docker compose cluster)
       * Verified metrics on OM /prom:

       ```
       curl -s http://localhost:9874/prom | grep volume_utilization_metrics

       curl -s http://localhost:9874/prom | grep bucket_utilization_metrics_bucket_used_namespace
       ```
       * Confirmed volume_utilization_metrics_volume_used_bytes increases after writes and volume_utilization_metrics_volume_available_bytes reflects quota minus usage.

       
   * CI — GitHub Actions `build-branch` succeeded on fork (HDDS-15333): https://github.com/prathmesh12-coder/ozone/actions/runs/27940909061
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15333. Expose volume and bucket utilization metrics on OM /prom [ozone]

Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #10576:
URL: https://github.com/apache/ozone/pull/10576#discussion_r3454188316


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/VolumeUtilizationMetrics.java:
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om;
+
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Map.Entry;
+import org.apache.hadoop.hdds.annotation.InterfaceAudience;
+import org.apache.hadoop.hdds.utils.db.cache.CacheKey;
+import org.apache.hadoop.hdds.utils.db.cache.CacheValue;
+import org.apache.hadoop.metrics2.MetricsCollector;
+import org.apache.hadoop.metrics2.MetricsInfo;
+import org.apache.hadoop.metrics2.MetricsSource;
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metrics;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.ozone.OzoneConsts;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+
+/**
+ * A class for collecting and reporting volume utilization metrics.
+ * <p>
+ * Available metrics:
+ * <ul>
+ *   <li>Volume quota in bytes.
+ *   <li>Volume used bytes (aggregated from all buckets in the volume).
+ *   <li>Volume available bytes. Calculated from difference between volume quota and used bytes.
+ *   If volume quota is not set then this metric shows -1 as value.
+ *   <li>Volume quota in namespace (maximum number of buckets).
+ *   <li>Volume used namespace (current number of buckets).
+ * </ul>
+ */
+@InterfaceAudience.Private
+@Metrics(about = "Ozone Volume Utilization Metrics", context = OzoneConsts.OZONE)
+public class VolumeUtilizationMetrics implements MetricsSource {
+
+  private static final String SOURCE = VolumeUtilizationMetrics.class.getSimpleName();
+
+  private final OMMetadataManager metadataManager;
+
+  public VolumeUtilizationMetrics(OMMetadataManager metadataManager) {
+    this.metadataManager = metadataManager;
+  }
+
+  public static VolumeUtilizationMetrics create(OMMetadataManager metadataManager) {
+    MetricsSystem ms = DefaultMetricsSystem.instance();
+    return ms.register(SOURCE, "Volume Utilization Metrics", new VolumeUtilizationMetrics(metadataManager));
+  }
+
+  @Override
+  public void getMetrics(MetricsCollector collector, boolean all) {
+    Map<String, OmVolumeArgs> volumes = new HashMap<>();
+    Iterator<Entry<CacheKey<String>, CacheValue<OmVolumeArgs>>> volumeIterator = metadataManager.getVolumeIterator();
+    while (volumeIterator.hasNext()) {
+      Entry<CacheKey<String>, CacheValue<OmVolumeArgs>> entry = volumeIterator.next();
+      OmVolumeArgs volumeArgs = entry.getValue().getCacheValue();
+      if (volumeArgs != null) {
+        volumes.put(volumeArgs.getVolume(), volumeArgs);
+      }
+    }
+
+    Map<String, Long> usedBytesPerVolume = new HashMap<>();
+    Iterator<Entry<CacheKey<String>, CacheValue<OmBucketInfo>>> bucketIterator = metadataManager.getBucketIterator();
+    while (bucketIterator.hasNext()) {
+      Entry<CacheKey<String>, CacheValue<OmBucketInfo>> entry = bucketIterator.next();
+      OmBucketInfo bucketInfo = entry.getValue().getCacheValue();
+      if (bucketInfo == null) {
+        continue;
+      }
+      usedBytesPerVolume.merge(bucketInfo.getVolumeName(), bucketInfo.getUsedBytes(), Long::sum);
+    }

Review Comment:
   Thanks for adding these metrics. Overall these look like good additions, I just have a concern with the volume used bytes. It is not directly tracked by Ozone because volumes don't have any connection to files, only buckets. Hence the used namespace tracked per volume is just a count of number of buckets and there is no used bytes for volumes.
   
   I think this could be aggregated with promQL outside of Ozone if needed.
   
   cc @sumitagrawl 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15333. Expose volume and bucket utilization metrics on OM /prom [ozone]

Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on PR #10576:
URL: https://github.com/apache/ozone/pull/10576#issuecomment-4770414923

   @rnblough You may be interested in this as well.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15333. Expose volume and bucket utilization metrics on OM /prom [ozone]

Posted by "sarvekshayr (via GitHub)" <gi...@apache.org>.
sarvekshayr closed pull request #10576: HDDS-15333. Expose volume and bucket utilization metrics on OM /prom
URL: https://github.com/apache/ozone/pull/10576


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15333. Expose volume and bucket utilization metrics on OM /prom [ozone]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10576:
URL: https://github.com/apache/ozone/pull/10576#issuecomment-5210734007

   This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15333. Expose volume and bucket utilization metrics on OM /prom [ozone]

Posted by "sarvekshayr (via GitHub)" <gi...@apache.org>.
sarvekshayr commented on PR #10576:
URL: https://github.com/apache/ozone/pull/10576#issuecomment-5276634082

   This PR is continued in https://github.com/apache/ozone/pull/11011.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15333. Expose volume and bucket utilization metrics on OM /prom [ozone]

Posted by "sarvekshayr (via GitHub)" <gi...@apache.org>.
sarvekshayr commented on PR #10576:
URL: https://github.com/apache/ozone/pull/10576#issuecomment-5262452212

   @prathmesh12-coder could you please resolve the conflicts in this patch so we can drive this to completion?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15333. Expose volume and bucket utilization metrics on OM /prom [ozone]

Posted by "prathmesh12-coder (via GitHub)" <gi...@apache.org>.
prathmesh12-coder commented on PR #10576:
URL: https://github.com/apache/ozone/pull/10576#issuecomment-4990195640

   Hi Team, 
   Please let me know if any changes/addition require here..
   cc : @errose28, @sarvekshayr, @Tejaskriya 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15333. Expose volume and bucket utilization metrics on OM /prom [ozone]

Posted by "prathmesh12-coder (via GitHub)" <gi...@apache.org>.
prathmesh12-coder commented on code in PR #10576:
URL: https://github.com/apache/ozone/pull/10576#discussion_r3459820047


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/VolumeUtilizationMetrics.java:
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om;
+
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Map.Entry;
+import org.apache.hadoop.hdds.annotation.InterfaceAudience;
+import org.apache.hadoop.hdds.utils.db.cache.CacheKey;
+import org.apache.hadoop.hdds.utils.db.cache.CacheValue;
+import org.apache.hadoop.metrics2.MetricsCollector;
+import org.apache.hadoop.metrics2.MetricsInfo;
+import org.apache.hadoop.metrics2.MetricsSource;
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metrics;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.ozone.OzoneConsts;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+
+/**
+ * A class for collecting and reporting volume utilization metrics.
+ * <p>
+ * Available metrics:
+ * <ul>
+ *   <li>Volume quota in bytes.
+ *   <li>Volume used bytes (aggregated from all buckets in the volume).
+ *   <li>Volume available bytes. Calculated from difference between volume quota and used bytes.
+ *   If volume quota is not set then this metric shows -1 as value.
+ *   <li>Volume quota in namespace (maximum number of buckets).
+ *   <li>Volume used namespace (current number of buckets).
+ * </ul>
+ */
+@InterfaceAudience.Private
+@Metrics(about = "Ozone Volume Utilization Metrics", context = OzoneConsts.OZONE)
+public class VolumeUtilizationMetrics implements MetricsSource {
+
+  private static final String SOURCE = VolumeUtilizationMetrics.class.getSimpleName();
+
+  private final OMMetadataManager metadataManager;
+
+  public VolumeUtilizationMetrics(OMMetadataManager metadataManager) {
+    this.metadataManager = metadataManager;
+  }
+
+  public static VolumeUtilizationMetrics create(OMMetadataManager metadataManager) {
+    MetricsSystem ms = DefaultMetricsSystem.instance();
+    return ms.register(SOURCE, "Volume Utilization Metrics", new VolumeUtilizationMetrics(metadataManager));
+  }
+
+  @Override
+  public void getMetrics(MetricsCollector collector, boolean all) {
+    Map<String, OmVolumeArgs> volumes = new HashMap<>();
+    Iterator<Entry<CacheKey<String>, CacheValue<OmVolumeArgs>>> volumeIterator = metadataManager.getVolumeIterator();
+    while (volumeIterator.hasNext()) {
+      Entry<CacheKey<String>, CacheValue<OmVolumeArgs>> entry = volumeIterator.next();
+      OmVolumeArgs volumeArgs = entry.getValue().getCacheValue();
+      if (volumeArgs != null) {
+        volumes.put(volumeArgs.getVolume(), volumeArgs);
+      }
+    }
+
+    Map<String, Long> usedBytesPerVolume = new HashMap<>();
+    Iterator<Entry<CacheKey<String>, CacheValue<OmBucketInfo>>> bucketIterator = metadataManager.getBucketIterator();
+    while (bucketIterator.hasNext()) {
+      Entry<CacheKey<String>, CacheValue<OmBucketInfo>> entry = bucketIterator.next();
+      OmBucketInfo bucketInfo = entry.getValue().getCacheValue();
+      if (bucketInfo == null) {
+        continue;
+      }
+      usedBytesPerVolume.merge(bucketInfo.getVolumeName(), bucketInfo.getUsedBytes(), Long::sum);
+    }

Review Comment:
   Hi @errose28, Thanks for the feedback! 
   Removed the VolumeUsedBytes and VolumeAvailableBytes metrics along with the aggregation logic.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org