[jira] [Created] (HDDS-17292) OMKeyRequest takes a second BUCKET lock while holding one, out of stripe order, which can deadlock the OM apply thread

["Siyao Meng (Jira)" <ji...@apache.org>]

Siyao Meng created HDDS-17292:
---------------------------------

             Summary: OMKeyRequest takes a second BUCKET lock while holding one, out of stripe order, which can deadlock the OM apply thread
                 Key: HDDS-17292
                 URL: https://issues.apache.org/jira/browse/HDDS-17292
             Project: Apache Ozone
          Issue Type: Bug
            Reporter: Siyao Meng
         Attachments: MC-5-release-bucket-lock-before-resolving-source-bucket.patch

h3. Mechanism
{{OMKeyRequest.generateRequiredEncryptionInfo}} takes the BUCKET read lock of the requested bucket and reads its {{OmBucketInfo}}. When that bucket points to a source bucket, it calls {{OzoneManager.resolveBucketLink}} while it still holds that lock, and the resolution takes the BUCKET read lock of the source bucket in {{BucketManagerImpl.getBucketInfo}}. {{OMKeyRequest.getFileEncryptionInfoForMpuKey}} does the same: it takes the read lock of the requested bucket first and calls {{resolveBucketLink}} inside it. Both run on the RPC handler thread, and both locks are stripes of the same {{BUCKET_LOCK}} of {{OzoneManagerLock}}. So the handler holds one stripe and waits for a second one, in whatever order the two stripes happen to be.

On the Ratis apply thread, {{OMKeyPurgeRequest.updateBucketSize}}, {{OMDirectoriesPurgeRequestWithFSO}} and {{S3ExpiredMultipartUploadsAbortRequest}} write lock all buckets of one request with {{IOzoneManagerLock.acquireWriteLocks}}, which takes the stripes in ascending order ({{Striped.bulkGet}}). That order prevents a cycle between two bulk holders, but not between a bulk holder and a thread that nests two single acquisitions in the opposite order.

h3. Trigger
Two lock holders in opposite stripe order, with stripe(B) below stripe(A):
# An RPC handler, in one of the two methods above, holds the BUCKET read lock stripe of bucket A and waits for the read lock stripe of B, the source of A.
# The apply thread, in a bulk acquire over a set of buckets with one bucket on stripe(B) and one on stripe(A), holds stripe(B) for write and waits for stripe(A) for write.

Neither wait has a timeout, and nothing else releases either lock.

h3. Impact
* The apply thread of the OM leader waits forever, so no later write is applied on that OM and every write waits for its reply.
* Each such write keeps its RPC handler while it waits in {{OzoneManagerRatisServer.submitRequest}} for the reply, which only comes after the apply thread has applied it. Reads on the leader do not go through the apply thread by default ({{ozone.om.ha.raft.server.read.option}} is {{DEFAULT}}), but they share the handler pool ({{ozone.om.handler.count.key}}), so once all handlers are held by waiting writes, reads wait in the call queue too. Reads that need a bucket on one of the two stripes block on the lock directly, since one stripe is write held and the other has a queued writer (code reading).
* The followers keep applying the log, because the nested acquisition only runs on the leader before submission. The leader stays leader, so the hang lasts until that OM is restarted (argued, restart not run). No data is lost.

By code reading at the release tags, not run on older code: the nested acquisition is present from 1.2.0 (HDDS-5530). Before 1.4.0 each bucket has its own BUCKET lock (no striping), and the cycle would need the apply thread to write lock the requested bucket itself, which does not appear in a purge. From 1.4.0 the BUCKET locks are striped (HDDS-8765) and {{OMDirectoriesPurgeRequestWithFSO}} holds several bucket write locks taken one at a time, and from 2.1.0 {{OMKeyPurgeRequest}} and {{OMDirectoriesPurgeRequestWithFSO}} use the sorted bulk acquire.

h3. Reproduction
No standalone reproduction is attached.

The regression test in the patch sets up the conflicting holder through internal calls. {{TestOMKeyCreateRequest.preExecuteReleasesBucketLockBeforeResolvingSourceBucket}} ({{hadoop-ozone/ozone-manager}}, also in the inherited {{...WithFSO}}, for both methods above) stubs {{resolveBucketLink}} so that, while the source bucket is resolved, another thread write locks the requested bucket and its source with {{acquireWriteLocks}}, as the apply thread does. That thread must complete within 5 seconds. Without the change all four cases fail with {{Expecting <CompletableFuture[Incomplete]> to be completed within 5S}}, because the resolving thread still holds the read lock of the requested bucket.

h3. Patch
[^MC-5-release-bucket-lock-before-resolving-source-bucket.patch], against e0de543cbe9d73f3bf5473a45e1c59cd6eae7173. It also applies to master at 1ff29ee6b86.

{{generateRequiredEncryptionInfo}} now releases the read lock of the requested bucket before it calls {{resolveBucketLink}}, and then reads the resolved bucket as before. {{getFileEncryptionInfoForMpuKey}} calls {{resolveBucketLink}} before it takes the read lock. {{resolveBucketLink}} takes and releases the lock of each bucket it reads on its own, so on these paths the handler now holds at most one BUCKET lock at a time and the cycle cannot form. The locks taken and the data read are otherwise the same: the resolved bucket was not read under its own lock on this path before either, and in both versions the lock is released before the request is submitted. Nothing changes for a bucket that does not point to another bucket, and there is no wire or protocol change.

With the patch the OM request unit suites that cover these two methods and the purge requests pass (593 tests in {{hadoop-ozone/ozone-manager}}, 1 skipped as before), as does {{TestOzoneAtRestEncryption}} on a mini cluster, and checkstyle on {{hadoop-ozone/ozone-manager}} is clean.

Found by TLA+ model checking and code review of the Ozone Manager lock ordering paths under HDDS-15926, on commit e0de543cbe9d73f3bf5473a45e1c59cd6eae7173. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. Related but not duplicates: HDDS-13729 (sorted bulk bucket locks in the directory purge), HDDS-7755 (an earlier OM hang from a leaked lock, HDDS-7925 is its duplicate). The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org