You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Andrey Yarovoy (Jira)" <ji...@apache.org> on 2026/08/25 21:41:00 UTC
[jira] [Assigned] (HDDS-16288) Convert OMPerformanceMetrics MutableRate latency counters to lock-free ConcurrentMutableStat
[ https://issues.apache.org/jira/browse/HDDS-16288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrey Yarovoy reassigned HDDS-16288:
-------------------------------------
Assignee: Andrey Yarovoy
> Convert OMPerformanceMetrics MutableRate latency counters to lock-free ConcurrentMutableStat
> --------------------------------------------------------------------------------------------
>
> Key: HDDS-16288
> URL: https://issues.apache.org/jira/browse/HDDS-16288
> Project: Apache Ozone
> Issue Type: Improvement
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
>
> h2. Problem
> OMPerformanceMetrics declares ~44 latency counters as @Metric MutableRate fields
> (OMPerformanceMetrics.java). MutableRate extends MutableStat, whose add(long) is
> synchronized on the counter instance:
> public synchronized void add(long) // hadoop-common MutableStat
> These counters are updated on the OM read/write RPC hot path, and several are hit
> per request. A single getKeyInfo, for example, records into six or seven of them in
> sequence (OmMetadataReader.getKeyInfo, ~lines 160-215):
> getKeyInfoResolveBucketLatencyNs, getKeyInfoAclCheckLatencyNs,
> getKeyInfoReadKeyInfoLatencyNs, getKeyInfoRefreshLocationLatencyNs,
> getKeyInfoGenerateBlockTokenLatencyNs, getKeyInfoSortDatanodesLatencyNs,
> getKeyInfoLatencyNs
> lookup, listKeys, createKey and the ACL/checkAccess paths have their own sets. Each
> counter is a distinct monitor, so contention is per-counter across threads: when many
> handlers execute the same read op concurrently, they serialize on the same synchronized
> add(). This is the same thundering-herd contention that HDDS-9377 (PR #11085) removed
> from OMLockMetrics and the PerformanceMetrics wrapper by replacing MutableStat with the lock-free ConcurrentMutableStat — but the raw MutableRate counters in OMPerformanceMetrics were not part of that change and still use the synchronized path.
> h2. Approach
> Reuse the ConcurrentMutableStat introduced in HDDS-9377 (hadoop-hdds/common) for the OMPerformanceMetrics latency counters instead of MutableRate, preserving the emitted metric names, num/avg samples, and min/max/stddev semantics so dashboards and JMX consumers are unaffected. Because these fields are currently auto-created by the @Metric annotation factory (which produces MutableRate), registration moves to explicit
> construction against the MetricsRegistry (mirroring how OMLockMetrics registers its
> ConcurrentMutableStat), and the capture helpers (captureLatencyNs / add* methods) are
> retargeted to the new type.
> h2. Scope
> In scope:
> - The MutableRate latency counters in OMPerformanceMetrics on OM request paths.
> Out of scope:
> - OMLockMetrics and PerformanceMetrics (already covered by HDDS-9377).
> - Changing hadoop-common MutableRate/MutableStat.
> - MutableGaugeLong service-latency gauges in OMPerformanceMetrics (single-writer,
> not contended).
> h2. Testing
> - Unit coverage that a converted counter records count/sum/min/max equivalently to
> MutableRate and that JMX/metrics-sink snapshot output is unchanged.
> - Re-run the mixed read/write OM workload used for HDDS-9377 to confirm the residual
> per-counter contention on OMPerformanceMetrics is removed and read p99 under
> concurrent load is unaffected or improved.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org