You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "yandrey321 (via GitHub)" <gi...@apache.org> on 2026/10/22 02:29:54 UTC
[PR] HDDS-16951. Remove the per-call InfoBucket from rooted-ofs rename on FSO buckets [ozone]
yandrey321 opened a new pull request, #11674:
URL: https://github.com/apache/ozone/pull/11674
## What changes were proposed in this pull request?
**Problem.** Rooted-ofs `rename` fetches the bucket (`InfoBucket`, plus `InfoVolume` + `InfoBucket` per link hop)
only to choose the algorithm: FSO sends one `RenameKey`, LEGACY iterates over the keys, OBJECT_STORE is rejected on
the client. So every FSO rename costs 2 OM requests. OM cannot simply reject non-FSO buckets in `RenameKey`: the
shell, trash, key lifecycle and `OzoneBucket.renameKey` send it on any layout, and on LEGACY it renames one key.
**Implementation** (same shape as HDDS-16760 / HDDS-16948):
- `RenameKeyRequest` gets `optional bool requireFileSystemOptimized = 4`. When it is set,
`OMKeyRenameRequest.preExecute` (inherited by the FSO request) rejects any bucket whose link-resolved layout is
not FILE_SYSTEM_OPTIMIZED with `NOT_SUPPORTED_OPERATION`, before Ratis. It tests the layout, not the request class,
because the OBJECT_STORE request class also serves LEGACY. Unlike `requireFileSystemLayout`, it rejects LEGACY too.
- New `ClientProtocol.renameKey(..., requireFileSystemOptimized)`; the existing overload is unchanged.
- New `OzoneManagerVersion.RENAME_KEY_REJECTS_NON_FSO(104)`. From that version on, `BasicRootedOzoneFileSystem.rename`
first sends the flagged `RenameKey` with no `getBucket()`. Success and `KEY_ALREADY_EXISTS` / `KEY_RENAME_ERROR` /
`KEY_NOT_FOUND` are handled exactly as in today's FSO rename. `NOT_SUPPORTED_OPERATION`, `BUCKET_NOT_FOUND` and
`VOLUME_NOT_FOUND` fall back to today's path from `getBucket()` on, so LEGACY renames, OBJECT_STORE rejections and
missing-bucket errors are unchanged. Older OMs, including `ZDU`, keep today's path.
**Side effects.**
- FSO: 2 -> 1 OM request per rename. LEGACY and OBJECT_STORE: one extra rejected `RenameKey` before today's path
(the audit log shows a failed `RENAME_KEY` before `INFO_BUCKET`).
- Callers that do not set the field (old clients, shell, trash, lifecycle, `OzoneBucket.renameKey`) are unaffected.
Generated-by: Claude Code (Claude Opus 5.5)
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16951
## How was this patch tested?
**Unit and integration tests**
**Benchmark**
**CPU savings.** CPU profiles of the 10-thread run show the patch saves about 110 us of CPU per FSO rename: about
67 us on the client (46% of its CPU per rename) and about 44 us on the OM (about 20% of its request-handling CPU; the
write pipeline is unchanged). Almost all of it is per-RPC overhead, not OM work: the `InfoBucket` request itself costs
about 2 us. It does not show up as throughput because the benchmark is bound by the OM's Ratis log fsync, not CPU.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16951. Remove the per-call InfoBucket from rooted-ofs rename on FSO buckets [ozone]
Posted by "chungen0126 (via GitHub)" <gi...@apache.org>.
chungen0126 commented on code in PR #11674:
URL: https://github.com/apache/ozone/pull/11674#discussion_r4366548611
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyRenameRequest.java:
##########
@@ -76,6 +76,14 @@ public OMRequest preExecute(OzoneManager ozoneManager) throws IOException {
.getRenameKeyRequest();
Objects.requireNonNull(renameKeyRequest, "renameKeyRequest == null");
+ // Checked before Ratis only: preExecute does not run when the request is applied. The request was created for the
+ // link-resolved layout, and the OBJECT_STORE request class also serves LEGACY, so test the layout itself.
+ if (renameKeyRequest.getRequireFileSystemOptimized() && !getBucketLayout().isFileSystemOptimized()) {
+ throw new OMException("Bucket: " + renameKeyRequest.getKeyArgs().getBucketName()
+ + " has layout: " + getBucketLayout() + ", but the rename requires "
+ + BucketLayout.FILE_SYSTEM_OPTIMIZED + ".", OMException.ResultCodes.NOT_SUPPORTED_OPERATION);
+ }
+
Review Comment:
I have created [HDDS-17382](https://issues.apache.org/jira/browse/HDDS-17382) to track this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16951. Remove the per-call InfoBucket from rooted-ofs rename on FSO buckets [ozone]
Posted by "chungen0126 (via GitHub)" <gi...@apache.org>.
chungen0126 commented on code in PR #11674:
URL: https://github.com/apache/ozone/pull/11674#discussion_r4358361190
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyRenameRequest.java:
##########
@@ -76,6 +76,14 @@ public OMRequest preExecute(OzoneManager ozoneManager) throws IOException {
.getRenameKeyRequest();
Objects.requireNonNull(renameKeyRequest, "renameKeyRequest == null");
+ // Checked before Ratis only: preExecute does not run when the request is applied. The request was created for the
+ // link-resolved layout, and the OBJECT_STORE request class also serves LEGACY, so test the layout itself.
+ if (renameKeyRequest.getRequireFileSystemOptimized() && !getBucketLayout().isFileSystemOptimized()) {
+ throw new OMException("Bucket: " + renameKeyRequest.getKeyArgs().getBucketName()
+ + " has layout: " + getBucketLayout() + ", but the rename requires "
+ + BucketLayout.FILE_SYSTEM_OPTIMIZED + ".", OMException.ResultCodes.NOT_SUPPORTED_OPERATION);
+ }
+
Review Comment:
Not directly related to this PR, just noting down for the record: if the operation fails in preExecute, it won't be recorded in the audit log.
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicRootedOzoneClientAdapterImpl.java:
##########
@@ -537,6 +537,27 @@ void rename(OzoneBucket bucket, String path, String newPath)
bucket.renameKey(key, newKey);
}
+ /**
+ * Whether OM rejects a rename that requires a FILE_SYSTEM_OPTIMIZED bucket on any other layout (HDDS-16951), so a
+ * file system rename can be sent without fetching the bucket to learn its layout.
+ */
+ boolean isRenameRequiringFileSystemOptimizedSupported() {
+ return proxy.getOmVersion().compareTo(OzoneManagerVersion.RENAME_KEY_REJECTS_NON_FSO) >= 0;
Review Comment:
Please use `ComponentVersion#isSupportedBy`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16951. Remove the per-call InfoBucket from rooted-ofs rename on FSO buckets [ozone]
Posted by "chungen0126 (via GitHub)" <gi...@apache.org>.
chungen0126 commented on code in PR #11674:
URL: https://github.com/apache/ozone/pull/11674#discussion_r4366548611
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyRenameRequest.java:
##########
@@ -76,6 +76,14 @@ public OMRequest preExecute(OzoneManager ozoneManager) throws IOException {
.getRenameKeyRequest();
Objects.requireNonNull(renameKeyRequest, "renameKeyRequest == null");
+ // Checked before Ratis only: preExecute does not run when the request is applied. The request was created for the
+ // link-resolved layout, and the OBJECT_STORE request class also serves LEGACY, so test the layout itself.
+ if (renameKeyRequest.getRequireFileSystemOptimized() && !getBucketLayout().isFileSystemOptimized()) {
+ throw new OMException("Bucket: " + renameKeyRequest.getKeyArgs().getBucketName()
+ + " has layout: " + getBucketLayout() + ", but the rename requires "
+ + BucketLayout.FILE_SYSTEM_OPTIMIZED + ".", OMException.ResultCodes.NOT_SUPPORTED_OPERATION);
+ }
+
Review Comment:
Created [HDDS-17382](https://issues.apache.org/jira/browse/HDDS-17382) for follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16951. Remove the per-call InfoBucket from rooted-ofs rename on FSO buckets [ozone]
Posted by "chungen0126 (via GitHub)" <gi...@apache.org>.
chungen0126 commented on code in PR #11674:
URL: https://github.com/apache/ozone/pull/11674#discussion_r4366548611
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyRenameRequest.java:
##########
@@ -76,6 +76,14 @@ public OMRequest preExecute(OzoneManager ozoneManager) throws IOException {
.getRenameKeyRequest();
Objects.requireNonNull(renameKeyRequest, "renameKeyRequest == null");
+ // Checked before Ratis only: preExecute does not run when the request is applied. The request was created for the
+ // link-resolved layout, and the OBJECT_STORE request class also serves LEGACY, so test the layout itself.
+ if (renameKeyRequest.getRequireFileSystemOptimized() && !getBucketLayout().isFileSystemOptimized()) {
+ throw new OMException("Bucket: " + renameKeyRequest.getKeyArgs().getBucketName()
+ + " has layout: " + getBucketLayout() + ", but the rename requires "
+ + BucketLayout.FILE_SYSTEM_OPTIMIZED + ".", OMException.ResultCodes.NOT_SUPPORTED_OPERATION);
+ }
+
Review Comment:
Created https://issues.apache.org/jira/browse/HDDS-17382 for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16951. Remove the per-call InfoBucket from rooted-ofs rename on FSO buckets [ozone]
Posted by "chungen0126 (via GitHub)" <gi...@apache.org>.
chungen0126 commented on PR #11674:
URL: https://github.com/apache/ozone/pull/11674#issuecomment-6382030960
@yandrey321 Please rebase to fix the failing CI checks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org