You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Siyao Meng (Jira)" <ji...@apache.org> on 2026/08/13 01:34:00 UTC
[jira] [Updated] (HDDS-16164) DirectoryDeletingService could deadlock snapshot purge while retaining snapshot DB handles
[ https://issues.apache.org/jira/browse/HDDS-16164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siyao Meng updated HDDS-16164:
------------------------------
Component/s: OM
> DirectoryDeletingService could deadlock snapshot purge while retaining snapshot DB handles
> ------------------------------------------------------------------------------------------
>
> Key: HDDS-16164
> URL: https://issues.apache.org/jira/browse/HDDS-16164
> Project: Apache Ozone
> Issue Type: Bug
> Components: OM
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
>
> h2. Problem
> {{DirectoryDeletingService}} (DDS) can retain snapshot DB handles while it submits synchronous OM requests. Each open handle holds a striped {{SNAPSHOT_DB_LOCK}} read lock.
> DDS has two relevant handle owners:
> * The task thread holds the current-snapshot handle while it waits for parallel directory workers and submits the final snapshot-property request.
> * Each directory worker can hold previous-snapshot handles through {{ReclaimableDirFilter}} and {{ReclaimableKeyFilter}} while it submits {{PurgeDirectories}} requests.
> Independently, {{OMDoubleBufferFlushThread}} can apply an earlier snapshot-purge response that needs a write lock on the same stripe. The DDS and purge snapshots can have different UUIDs that map to the same stripe.
> If the double-buffer flush blocks on a DDS read lock and the unflushed transaction count reaches {{ozone.om.unflushed.transaction.max.count}}, a synchronous DDS request cannot be applied. DDS cannot return and close the handle that blocks the flush, which completes the circular wait.
> This risk was identified by code inspection. It was not the path observed in the HDDS-16118 incident and has not yet been reproduced through DDS.
> h2. Impact
> The OM leader could stop applying transactions and flushing the double buffer. Client operations could remain blocked until OM leadership changes or the affected OM restarts.
> h2. Proposed fix
> Ensure that DDS closes all snapshot DB handles after it completes snapshot DB reads and before it submits synchronous OM requests:
> * Close the current-snapshot handle on its owner thread.
> * Close previous-snapshot handles opened by the reclaimable filters on their owner threads.
> * Retain the snapshot GC locks required for snapshot-chain and reclamation correctness until the corresponding OM requests complete.
> * Preserve DDS parallelism where practical, or process snapshot cleanup on one thread if this provides a simpler and safer ownership model.
> Add a deterministic regression test that uses one snapshot DB lock stripe and one allowed unflushed transaction to reproduce the complete DDS circular wait.
> h2. References
> * Follow-up to [HDDS-16118|https://issues.apache.org/jira/browse/HDDS-16118].
> * Identified during review of Apache Ozone PR 10987: [review comment by Weichiu Chuang|https://github.com/apache/ozone/pull/10987#pullrequestreview-4913044902].
> * Related but distinct from [HDDS-13219|https://issues.apache.org/jira/browse/HDDS-13219], which tracks snapshot read locking during deep cleaning.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org