[jira] [Created] (HDDS-16693) Nested snapshot read locks on two stripes corrupt the OM lock hold time metrics

["Siyao Meng (Jira)" <ji...@apache.org>]

Siyao Meng created HDDS-16693:
---------------------------------

             Summary: Nested snapshot read locks on two stripes corrupt the OM lock hold time metrics
                 Key: HDDS-16693
                 URL: https://issues.apache.org/jira/browse/HDDS-16693
             Project: Apache Ozone
          Issue Type: Bug
            Reporter: Siyao Meng
         Attachments: HDDS-16693-CR-R13-nested-read-lock-metric.patch

h3. Mechanism
{{IOzoneManagerLock.ResourceManager}} keeps one read start time per resource per thread and removes the {{ThreadLocal}} on read ({{IOzoneManagerLock.java:157-205}}). With two nested read locks on different stripes of one resource, the second release computes elapsed time from {{LockUsageInfo}}'s -1 initial value ({{LockUsageInfo.java:25}}), and {{updateReadUnlockMetrics}} ({{OzoneManagerLock.java:526-540}}) feeds that into the statistic. The {{readHoldCount == 0}} guard covers reentrancy on one lock but not two stripes of one resource.

h3. Impact
Ordinary nested snapdiff read locks permanently poison {{readLockHeldTimeMsStat}} and {{getLongestReadLockHeldTimeMs}}. Operator visible metrics only, with no data or availability effect, but it degrades the very signal that would reveal the deadlocks reported separately for MC-4 and MC-5 in this umbrella.

h3. Reproduction
{{repro/test_bugCR-R13_nested_read_lock_metric.sh}}. Evidence in {{repro/logs/ev-cr2.log}}.

h3. Patch
{{HDDS-XXXXX-CR-R13-nested-read-lock-metric.patch}} is attached. It applies to 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535 and carries the fix plus a regression test that was verified to fail without the fix and pass with it.
{code:none}
mvn test -pl hadoop-ozone/ozone-manager -Dtest=TestOzoneManagerLock
{code}

Found by code review of the OM snapshot cache under HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. Generated with Specula (Claude Opus 5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org