You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "spacemonkd (via GitHub)" <gi...@apache.org> on 2026/07/07 11:32:13 UTC

[PR] HDDS-11900. Introduce fine-grained locking for OBS [ozone]

spacemonkd opened a new pull request, #10683:
URL: https://github.com/apache/ozone/pull/10683

   ## What changes were proposed in this pull request?
   
   HDDS-11900. Introduce fine-grained locking for OBS key operations to enable leader-execution model.
   
   ## Description
   
   This pull request introduces fine-grained key-path-level locking for Ozone Object Store (OBS) key operations as a prerequisite for the leader-execution model (HDDS-11898). Currently, all OBS key operations within a bucket are serialized by a single bucket-level write lock (`BUCKET_LOCK`), preventing concurrent operations on different keys. This change enables parallel key operations by introducing a lock hierarchy that allows bucket-level read locks (preventing bucket deletion) paired with per-key write locks.
   
   ### Problem Statement
   
   The leader-execution design requires deterministic, repeated execution of requests: leaders execute request logic and produce a side-effect log, while followers replay the log to apply only the side effects (DB mutations). This model breaks if concurrent mutations on different keys within a bucket are possible—replaying in an arbitrary order risks different outcomes than the leader's execution.
   
   Today's coarse bucket-level write locks serialize all key operations, making determinism straightforward but limiting throughput. The solution is to introduce fine-grained locking: keys within a bucket can be locked independently using striped locks, enabling concurrent operations on different keys while maintaining determinism through ordered multi-key lock acquisition (`bulkGet()`).
   
   ### Changes Proposed
   
   **Lock Infrastructure:**
   - Extended `OzoneLockStrategy` interface with default methods for multi-key locks and bucket-read-only locks, enabling backward-compatible migration of handlers.
   - Implemented `OBSKeyPathLockStrategy`: acquires BUCKET_LOCK read + per-key KEY_PATH_LOCK writes using ordered `bulkGet()` for multi-key operations; bucket-read-only operations acquire only BUCKET_LOCK read.
   - Implemented `RegularBucketLockStrategy` fallback: preserves BUCKET_LOCK write behavior when key-path locking is disabled (config default: `ozone.om.key.path.lock.enabled=false`).
   
   **Request Handler Migrations:**
   - Single-key write operations: OMKeyCommitRequest, OMKeyDeleteRequest, OMAllocateBlockRequest, OMKeySetTimesRequest, S3MultipartUploadCompleteRequest, S3MultipartUploadAbortRequest, S3MultipartUploadCommitPartRequest, S3DeleteObjectTaggingRequest, S3PutObjectTaggingRequest, OMKeyAclRequest.
   - Bucket-read-only operations (no key lock): OMKeyCreateRequest, S3InitiateMultipartUploadRequest (per design doc, these write only to OpenKey table with unique clientIDs; no key conflict).
   - Multi-key ordered locks: OMKeysDeleteRequest (locks all deleted keys), OMKeyRenameRequest (locks source + destination), OMKeysRenameRequest (locks all from/to keys).
   
   ### Design & Approach
   The implementation follows the OBS locking strategy defined in :
   - **Lock Hierarchy**: S3_BUCKET_LOCK(0) → VOLUME_LOCK(1) → BUCKET_LOCK(2) → KEY_PATH_LOCK(5)
   - **Bucket-read + key-write model**: All key operations acquire BUCKET_LOCK read (prevents bucket deletion) + KEY_PATH_LOCK write (per-key serialization).
   - **Ordered multi-key locking**: Batch operations like DeleteKeys and Rename use `bulkGet()`-based ordered lock acquisition to guarantee deadlock-freedom.
   - **Backward compatibility**: Default config keeps key-path locking disabled; RegularBucketLockStrategy falls back to BUCKET_LOCK write, preserving existing behavior until the feature is enabled.
   
   The migration uses the Strategy pattern (`OzoneLockStrategy`) to avoid coupling handlers to specific lock schemes, allowing per-bucket-layout lock selection (OBS vs FSO) and per-config feature gating.
   
   ### References:
   @kerneltime's reference specification: https://github.com/kerneltime/ozone/tree/9c59ae0d66227ce10849b4f914f9efdce9318103/hadoop-hdds/docs/content/design/leader-execution
   Pre-Ratis Execution design document: https://github.com/apache/ozone/pull/10503
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-11900
   
   ## How was this patch tested?
   
   Unit Tests:
   - New `TestOBSKeyPathLockStrategy`: single-key, multi-key, bucket-read-only lock operations; concurrent lock acquisition on different keys; deadlock-free ordered locking.
   - Updated existing tests: fixed mock setup in `TestOMKeyRequest` and `TestS3MultipartRequest` base classes to mock `getOzoneLockProvider()`.
   
   Integration Tests (all passing)
   Compilation: `mvn clean install -U -DskipTests -DskipShade`
   
   **Feature Flag:**
   - Default config: `ozone.om.key.path.lock.enabled=false` (no behavior change; uses RegularBucketLockStrategy).
   - When enabled: OBSKeyPathLockStrategy activates per-key locking for OBS buckets, enabling concurrent key operations.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11900. Introduce fine-grained locking for OBS [ozone]

Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #10683:
URL: https://github.com/apache/ozone/pull/10683#discussion_r3573092934


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/lock/OBSKeyPathLockStrategy.java:
##########
@@ -89,4 +91,55 @@ public OMLockDetails releaseReadLock(OMMetadataManager omMetadataManager,
         .releaseReadLock(BUCKET_LOCK, volumeName, bucketName));
     return omLockDetails;
   }
+
+  @Override
+  public OMLockDetails acquireWriteLock(OMMetadataManager omMetadataManager,
+      String volumeName, String bucketName, Collection<String> keyNames)

Review Comment:
   @spacemonkd , just some quick questions: Why keyNames is a collection?  How many keys do you expected?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11900. Introduce fine-grained locking for OBS [ozone]

Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on PR #10683:
URL: https://github.com/apache/ozone/pull/10683#issuecomment-4960324662

   @yandrey321 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11900. Introduce fine-grained locking for OBS [ozone]

Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #10683:
URL: https://github.com/apache/ozone/pull/10683#discussion_r3608672592


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCreateRequest.java:
##########
@@ -242,8 +242,8 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, Execut
     try {
 
       mergeOmLockDetails(
-          ozoneLockStrategy.acquireWriteLock(omMetadataManager, volumeName,
-              bucketName, keyName));
+          ozoneLockStrategy.acquireBucketReadLock(omMetadataManager,
+              volumeName, bucketName));

Review Comment:
   Why replacing WriteLock with BucketReadLock?  It is changing the lock design.  This part should be done in a separated JIRA.



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/lock/OBSKeyPathLockStrategy.java:
##########
@@ -89,4 +91,55 @@ public OMLockDetails releaseReadLock(OMMetadataManager omMetadataManager,
         .releaseReadLock(BUCKET_LOCK, volumeName, bucketName));
     return omLockDetails;
   }
+
+  @Override
+  public OMLockDetails acquireWriteLock(OMMetadataManager omMetadataManager,
+      String volumeName, String bucketName, Collection<String> keyNames)

Review Comment:
   Thanks for the info.  As in [Striped.bulkGet](https://guava.dev/releases/19.0/api/docs/com/google/common/util/concurrent/Striped.html#bulkGet(java.lang.Iterable)), use Iterable is better and don't ever copy the underlying collection.  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11900. Introduce fine-grained locking for OBS [ozone]

Posted by "spacemonkd (via GitHub)" <gi...@apache.org>.
spacemonkd commented on PR #10683:
URL: https://github.com/apache/ozone/pull/10683#issuecomment-5011904613

   Sure, I'm just thinking we might have to go into a feature branch sometime down the line. Would you prefer a feature branch instead and we can merge smaller changes there without worrying about breaking things


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11900. Introduce fine-grained locking for OBS [ozone]

Posted by "spacemonkd (via GitHub)" <gi...@apache.org>.
spacemonkd commented on code in PR #10683:
URL: https://github.com/apache/ozone/pull/10683#discussion_r3573890725


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/lock/OBSKeyPathLockStrategy.java:
##########
@@ -89,4 +91,55 @@ public OMLockDetails releaseReadLock(OMMetadataManager omMetadataManager,
         .releaseReadLock(BUCKET_LOCK, volumeName, bucketName));
     return omLockDetails;
   }
+
+  @Override
+  public OMLockDetails acquireWriteLock(OMMetadataManager omMetadataManager,
+      String volumeName, String bucketName, Collection<String> keyNames)

Review Comment:
   Thanks for taking a look at this PR.
   Here this method can be called by various classes.
   For example:
   - OMKeyRenameRequest only passes 2 keys as [list](https://github.com/spacemonkd/ozone/blob/101b7aa2befbfefb5dba8e05a3c3ff7db03c232b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyRenameRequest.java#L155). 
   - OMKeyDeleteRequest can pass multiple keys [here](https://github.com/spacemonkd/ozone/blob/101b7aa2befbfefb5dba8e05a3c3ff7db03c232b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeysDeleteRequest.java#L146) as multiple keys can be deleted in a batch as we call [getKeysAsList()](https://github.com/spacemonkd/ozone/blob/101b7aa2befbfefb5dba8e05a3c3ff7db03c232b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeysDeleteRequest.java#L99)
   - OMKeysRenameRequest is similar to OMKeyRenameRequest but instead we pass a batch of keys so technically here it can be 2*N keys (2 keys for from/to and N total keys in the batch)
   
   So the collection was more suitable here as then we can pass list, set or whatever is suitable.
   But I am definitely open to suggestions on how this could be improved.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11900. Introduce fine-grained locking for OBS [ozone]

Posted by "spacemonkd (via GitHub)" <gi...@apache.org>.
spacemonkd closed pull request #10683: HDDS-11900. Introduce fine-grained locking for OBS
URL: https://github.com/apache/ozone/pull/10683


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11900. Introduce fine-grained locking for OBS [ozone]

Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on PR #10683:
URL: https://github.com/apache/ozone/pull/10683#issuecomment-5011920459

   For this JIRA, a feature branch may not be needed.  Let's start with below:
   
   > Why replacing WriteLock with BucketReadLock? It is changing the lock design. This part should be done in a separated JIRA.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11900. Introduce fine-grained locking for OBS [ozone]

Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on PR #10683:
URL: https://github.com/apache/ozone/pull/10683#issuecomment-5011950603

   > As in [Striped.bulkGet](https://guava.dev/releases/19.0/api/docs/com/google/common/util/concurrent/Striped.html#bulkGet(java.lang.Iterable)), use Iterable is better ...
   Filed HDDS-15907.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org