[jira] [Created] (HDDS-16685) OM deadlock: nested snapshot get() takes SNAPSHOT_DB_LOCK stripes out of order against acquireAll

["Siyao Meng (Jira)" <ji...@apache.org>]

Siyao Meng created HDDS-16685:
---------------------------------

             Summary: OM deadlock: nested snapshot get() takes SNAPSHOT_DB_LOCK stripes out of order against acquireAll
                 Key: HDDS-16685
                 URL: https://issues.apache.org/jira/browse/HDDS-16685
             Project: Apache Ozone
          Issue Type: Bug
            Reporter: Siyao Meng


h3. Mechanism
{{OzoneManagerLock.acquireAll}} ({{:158-164}}) takes every stripe of a resource in ascending index order, blocking, with no backoff and no all or nothing semantics. {{SnapshotCache.lock()}} reaches it through {{acquireResourceWriteLock}} ({{:382}}).

{{SnapshotCache.get(UUID)}} instead takes exactly one stripe, {{acquireReadLock(SNAPSHOT_DB_LOCK, key.toString())}} at {{:217}}, chosen by the UUID hash, in whatever order the caller opens snapshots. A snapdiff holding the from snapshot on a high stripe and then requesting the to snapshot on a low stripe inverts against a concurrent {{acquireAll}} that has already taken the low stripe.

{{MultiSnapshotLocks}} exists and uses the ordered bulk path, so the ordered primitive was written on purpose and the snapdiff callers bypass it. The DAG tracker added by HDDS-13960 orders resources, not stripes within a resource, so it does not cover this case.

h3. Impact
Permanent deadlock between the snapdiff thread and the checkpoint or follower bootstrap thread. Every later snapshot operation that needs one of the already taken stripes blocks behind it. Untimed on both sides, restart only, and invisible to JVM deadlock detection because read holds have no owner.

h3. Reproduction
{{repro/test_bugMC-5_stripe_inversion_deadlock.sh}} with {{repro/java/SpeculaMc5StripeInversionTest.java}}. Evidence in {{repro/logs/ev-mc5.log}}: the Level 0 sequential case completes as expected, then the two thread case leaves both threads unfinished after 60s, with {{findDeadlockedThreads()=null}} and thread dumps showing the snapdiff thread parked in {{SnapshotCache.get}} at {{:217}} and the checkpoint thread parked in {{acquireAll}} under {{SnapshotCache.lock}}.

h3. No patch attached
Fixing this means deciding how nested snapshot acquisition is ordered. MultiSnapshotLocks already wraps the ordered bulk path, so routing the snapdiff callers through it is the natural fix, but that changes how those callers hold snapshots across a diff and the alternative is to order the stripes inside SnapshotCache.get itself. A regression test for the deadlock would be slow and timing dependent in CI. Filing the analysis so the approach can be agreed first.

Found by TLA+ model checking and code review of the OM snapshot cache under HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. Generated with Specula (Claude Opus 5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org