You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "shuan1026 (via GitHub)" <gi...@apache.org> on 2026/10/12 06:27:20 UTC
[PR] HDDS-13880. Fix flaky TestKeyDeletingService#testSnapshotDeepClean [ozone]
shuan1026 opened a new pull request, #11485:
URL: https://github.com/apache/ozone/pull/11485
## What changes were proposed in this pull request?
`testSnapshotDeepClean` intermittently times out after 120s in `assertTableRowCount(snapshotInfoTable, ...)` right after deleting `snap2`. The cause is a deadlock between the test thread and the OM double-buffer flush thread, not a slow test.
The test held the `snap3` snapshot DB handle in one try-with-resources block while it deleted `snap2` and waited for the row to disappear from `snapshotInfoTable`. That handle keeps a `SNAPSHOT_DB_LOCK` read lock keyed by snapshot ID for its whole lifetime (`SnapshotCache.get`). The purge of `snap2` is written to RocksDB by `OMSnapshotPurgeResponse.addToDBBatch` on the single `OMDoubleBufferFlushThread`, which calls `deleteSnapshotCheckpointDirectories` and takes the `SNAPSHOT_DB_LOCK` write lock for `snap2` before deleting the `snapshotInfoTable` row. `OzoneManagerLock` maps keys onto 512 striped `ReentrantReadWriteLock`s by default, so whenever the UUIDs of `snap2` and `snap3` hash to the same stripe the write lock blocks behind the read lock the test holds. `countRowsInTable` only iterates RocksDB, so the wait can never succeed until the 120s timeout releases the handle. Both reproduced failures showed exactly this: the flush thread parked in `SnapshotCache.lock` and the two sn
apshot IDs on the same stripe.
This is the same invariant that HDDS-16118 fixed on the production side (`ReclaimableFilter.closeSnapshotDbHandles`: a synchronous request must not retain snapshot DB read locks because the double buffer may need the write lock to flush). That change did not touch this test.
The fix is test-only and follows the pattern already used in `TestSnapshotDeletingServiceIntegrationTest` (close the snapshot handle before waiting on `snapshotInfoTable`):
- Split the single try-with-resources into two narrow blocks, so `deleteSnapshot(snap2)` and the `snapshotInfoTable` wait run without any snapshot DB read lock held.
- Re-fetch the `snap3` deleted table in the second block, because `SnapshotCache` may close and reload the snapshot while no handle is held.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-13880
## How was this patch tested?
- Deterministic reproduction: with `ozone.om.lock.stripes.snapshot_db_lock=1` set temporarily in `Normal.setup` (same knob as `SnapshotDbHandlePermitDeadlock`), all `SNAPSHOT_DB_LOCK` keys map to the same stripe. The unpatched test timed out at the `snapshotInfoTable` wait after ~125s; after the fix it passed in ~19s. The knob is not part of this PR.
- ci: https://github.com/shuan1026/ozone/actions/runs/36136446945
- `flaky-test-check`, full `TestKeyDeletingService` (`test-name=ALL`, `submodule=ozone-manager`), 10 splits × 8 iterations:
| Ref | Run | Result |
| ----------------- | -------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- |
| master (baseline) | [33828743805](https://github.com/shuan1026/ozone/actions/runs/33828743805) | 2/80 iterations failed, both `testSnapshotDeepClean` at 124.3s with the deadlock |
| `fe6c5843ba` | [36144544072](https://github.com/shuan1026/ozone/actions/runs/36144544072) | 80/80 iterations passed, 0 failures |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13880. Fix flaky TestKeyDeletingService#testSnapshotDeepClean [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on PR #11485:
URL: https://github.com/apache/ozone/pull/11485#issuecomment-6127704681
Merged. Thanks @shuan1026 @smengcl
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13880. Fix flaky TestKeyDeletingService#testSnapshotDeepClean [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang merged PR #11485:
URL: https://github.com/apache/ozone/pull/11485
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13880. Fix flaky TestKeyDeletingService#testSnapshotDeepClean [ozone]
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on PR #11485:
URL: https://github.com/apache/ozone/pull/11485#issuecomment-6129837270
Thanks @shuan1026 for the patch. Thanks @jojochuang for merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org