[jira] [Created] (HDDS-16686) Snapshot cache eviction erases a concurrent enqueue, leaking an open snapshot DB and breaking later checkpoints

["Siyao Meng (Jira)" <ji...@apache.org>]

Siyao Meng created HDDS-16686:
---------------------------------

             Summary: Snapshot cache eviction erases a concurrent enqueue, leaking an open snapshot DB and breaking later checkpoints
                 Key: HDDS-16686
                 URL: https://issues.apache.org/jira/browse/HDDS-16686
             Project: Apache Ozone
          Issue Type: Bug
            Reporter: Siyao Meng
         Attachments: HDDS-16686-MC-2-eviction-queue-erasure.patch

h3. Mechanism
{{SnapshotCache.cleanup(UUID, boolean)}} ({{SnapshotCache.java:379-417}}) reads {{v.getTotalRefCount()}} at {{:400}} inside the {{dbMap.compute}} lambda. On the keep branch it falls through to {{pendingEvictionQueue.remove(k)}} at {{:413}}, which executes on every branch.

A releasing thread that drops the last reference between those two points enqueues the key outside that bin lock, and the evictor then erases the enqueue. The entry stays in {{dbMap}} with refcount 0 and is no longer queued, so nothing will ever evict it.

h3. Impact
The snapshot DB stays open until OM restart, and {{SnapshotCache.lock()}}'s drain check at {{:334-336}} begins failing on every later attempt, which breaks admin checkpoint download and follower bootstrap. There is no automatic recovery.

h3. Reproduction
{{repro/test_bugMC-2_eviction_queue_erasure.sh}} with {{repro/java/SpeculaMc2EvictionQueueTest.java}}. Evidence in {{repro/logs/ev-mc2.log}} and {{ev-mc2b.log}}.

Note on probability: the race did not trigger in 4000 unassisted iterations, because the window between the refcount read at {{:400}} and the queue removal at {{:413}} is a single debug log statement. It was reproduced by widening that window with {{repro/level3-snapshotcache-keep-window.patch}}, which confirms the ordering and demonstrates the consequence: final state {{inDbMap=true refCount=0 inQueue=false}}, and the next {{snapshotCache.lock()}} throws {{IllegalStateException: Failed to acquire lock as cleanup did not drain the cache.}} The natural probability on an unpatched build is low; the fix is a one line move of the queue removal onto the evict branch.

h3. Patch
{{HDDS-XXXXX-MC-2-eviction-queue-erasure.patch}} is attached. It applies to 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535 and carries the fix plus a regression test that was verified to fail without the fix and pass with it.
{code:none}
mvn test -pl hadoop-ozone/ozone-manager -Dtest=TestSnapshotCache
{code}

Found by TLA+ model checking and code review of the OM snapshot cache under HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. Generated with Specula (Claude Opus 5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org