You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Chi-Hsuan Huang (Jira)" <ji...@apache.org> on 2026/09/26 03:10:00 UTC
[jira] [Updated] (HDDS-16611) Use MutableGaugeLong for numVolumes, numBuckets and numKeys in OMMetrics
[ https://issues.apache.org/jira/browse/HDDS-16611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chi-Hsuan Huang updated HDDS-16611:
-----------------------------------
Description:
{{OMMetrics}} declares {{numVolumes}}, {{numBuckets}} and {{numKeys}} as {{MutableCounterLong}}, but they behave like gauges. They are decremented with {{incr(-1)}}, and {{setNum*()}} emulates a set via {{incr(val - oldVal)}}. This has been the case since HDDS-816.
The JMX values are correct, but {{/prom}} exports these metrics as {{# TYPE counter}}. Prometheus treats any decrease in a counter as a reset. So the *Key Creation Rate* panel in the Memory Consumption dashboard, which uses {{rate(om_metrics_num_keys[1m])}}, reports inflated values whenever keys are deleted.
HDDS-10597 fixed the same pattern in {{SafeModeMetrics}}.
Proposed changes:
* Switch the three fields to {{MutableGaugeLong}} and use {{set()}} in {{setNumVolumes}}, {{setNumBuckets}} and {{setNumKeys}}.
* Update {{TestOmMetrics}} to read these values with {{getLongGauge}} instead of {{getLongCounter}}.
* Update the Key Creation Rate panel to use {{deriv()}}, or {{rate(om_metrics_num_key_commits[1m])}}.
Metric names in {{/jmx}} and {{/prom}} stay the same.
was:
{{OMMetrics}} declares {{numVolumes}}, {{numBuckets}} and {{numKeys}} as {{MutableCounterLong}}, but they behave like gauges. They are decremented with {{incr\(\-1\)}}, and {{setNum\*\(\)}} emulates a set via {{incr\(val \- oldVal\)}}. This has been the case since HDDS\-816.
The JMX values are correct, but {{/prom}} exports these metrics as {{# TYPE counter}}. Prometheus treats any decrease in a counter as a reset. So the _Key Creation Rate_ panel in the Memory Consumption dashboard, which uses {{rate\(om\_metrics\_num\_keys\[1m\]\)}}, reports inflated values whenever keys are deleted.
HDDS\-10597 fixed the same pattern in {{SafeModeMetrics}}.
Proposed changes:
\- Switch the three fields to {{MutableGaugeLong}} and use {{set\(\)}} in {{setNumVolumes}}, {{setNumBuckets}} and {{setNumKeys}}.
\- Update {{TestOmMetrics}} to read these values with {{getLongGauge}} instead of {{getLongCounter}}.
\- Update the Key Creation Rate panel to use {{deriv\(\)}}, or {{rate\(om\_metrics\_num\_key\_commits\[1m\]\)}}.
Metric names in {{/jmx}} and {{/prom}} stay the same.
> Use MutableGaugeLong for numVolumes, numBuckets and numKeys in OMMetrics
> ------------------------------------------------------------------------
>
> Key: HDDS-16611
> URL: https://issues.apache.org/jira/browse/HDDS-16611
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Manager
> Reporter: Chi-Hsuan Huang
> Assignee: Chi-Hsuan Huang
> Priority: Major
>
> {{OMMetrics}} declares {{numVolumes}}, {{numBuckets}} and {{numKeys}} as {{MutableCounterLong}}, but they behave like gauges. They are decremented with {{incr(-1)}}, and {{setNum*()}} emulates a set via {{incr(val - oldVal)}}. This has been the case since HDDS-816.
> The JMX values are correct, but {{/prom}} exports these metrics as {{# TYPE counter}}. Prometheus treats any decrease in a counter as a reset. So the *Key Creation Rate* panel in the Memory Consumption dashboard, which uses {{rate(om_metrics_num_keys[1m])}}, reports inflated values whenever keys are deleted.
> HDDS-10597 fixed the same pattern in {{SafeModeMetrics}}.
> Proposed changes:
> * Switch the three fields to {{MutableGaugeLong}} and use {{set()}} in {{setNumVolumes}}, {{setNumBuckets}} and {{setNumKeys}}.
> * Update {{TestOmMetrics}} to read these values with {{getLongGauge}} instead of {{getLongCounter}}.
> * Update the Key Creation Rate panel to use {{deriv()}}, or {{rate(om_metrics_num_key_commits[1m])}}.
> Metric names in {{/jmx}} and {{/prom}} stay the same.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org