[jira] [Updated] (HDDS-16118) OM can deadlock at the unflushed transaction limit when KeyDeletingService retains snapshot DB handles

["Siyao Meng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-16118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siyao Meng updated HDDS-16118:
------------------------------
    Release Note: Fixed an OM deadlock that could occur when KeyDeletingService retained snapshot DB read locks during snapshot purge and the unflushed transaction limit was reached. The fix releases snapshot DB handles before synchronous requests. No configuration changes are required.

> OM can deadlock at the unflushed transaction limit when KeyDeletingService retains snapshot DB handles
> ------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-16118
>                 URL: https://issues.apache.org/jira/browse/HDDS-16118
>             Project: Apache Ozone
>          Issue Type: Bug
>          Components: Snapshot
>            Reporter: Siyao Meng
>            Assignee: Siyao Meng
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 2.3.0
>
>
> h2. Problem
> {{KeyDeletingService}} retains snapshot DB handles after scanning deleted-key metadata and while submitting synchronous OM requests. Each open handle holds a striped {{SNAPSHOT_DB_LOCK}} read lock.
> During snapshot purge, {{OMDoubleBufferFlushThread}} may need a write lock mapped to the same stripe. The reader and writer can reference different snapshot UUIDs because the lock is striped.
> A stripe collision normally causes only temporary blocking. It becomes a deadlock when the configured maximum number of unflushed transactions is reached before the KDS request is applied.
> h2. Deadlock sequence
> # {{KeyDeletingService}} opens the current and previous snapshot databases, acquiring {{SNAPSHOT_DB_LOCK}} read locks.
> # KDS scans the deleted-key and rename tables but retains the snapshot handles while calling {{processKeyDeletes()}} or {{submitSetSnapshotRequests()}}.
> # {{OMDoubleBufferFlushThread}} processes an {{OMSnapshotPurgeResponse}}.
> # The purge path calls {{deleteSnapshotCheckpointDirectories()}}, which requests a snapshot DB write lock.
> # The requested write lock maps to a stripe held by KDS, so the double-buffer flush blocks.
> # While flushing remains blocked, the number of unflushed transactions can increase.
> # When it reaches {{ozone.om.unflushed.transaction.max.count}}, the next transaction application blocks in {{acquireUnFlushedTransactions()}}.
> # The synchronous KDS request cannot be applied, so KDS cannot return and close its snapshot handles.
> # The snapshot purge cannot acquire its write lock, completing the cycle.
> The default unflushed transaction limit is {{10,000}}. Therefore, the full deadlock generally requires substantial transaction activity or an existing flush backlog. Without reaching the limit, the stripe collision should remain a temporary stall.
> h2. Observed evidence
> Thread and lock diagnostics showed two simultaneous stripe collisions:
> * Snapshot purge for {{snap-005051}} waited for the stripe 91 write lock. Its sole reader was {{KeyDeletingService#7}}, which held the handle for {{snap-018186}}.
> * Snapshot defrag for {{snap-011578}} waited for the stripe 102 write lock. Its sole reader was {{KeyDeletingService#3}}, which held the handle for {{snap-018180}}.
> * {{StateMachineUpdater}} was blocked in {{acquireUnFlushedTransactions()}}.
> * KDS threads were waiting in synchronous snapshot-property request submission.
> The purge collision formed the double-buffer transaction-application deadlock. The defrag collision independently demonstrates that KDS retained snapshot DB read locks longer than required.
> h2. Impact
> The OM leader can stop applying transactions and flushing the double buffer. Client operations may remain blocked until OM leadership changes or the affected OM is restarted.
> h2. Proposed fix
> Release snapshot DB handles after KDS finishes scanning snapshot tables and before it performs synchronous SCM or OM operations:
> * Close the current snapshot DB handle.
> * Close previous-snapshot handles opened by {{ReclaimableKeyFilter}} and {{ReclaimableRenameEntryFilter}}.
> * Continue holding the snapshot GC locks until the corresponding OM requests complete, preserving snapshot-chain and reclamation correctness.
> * Retain try-with-resources cleanup as the exception-safety fallback.
> No snapshot database is accessed after the early-close point; subsequent processing uses data already materialized in memory.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org