You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "jojochuang (via GitHub)" <gi...@apache.org> on 2026/09/09 01:21:43 UTC
[PR] HDDS-15925. Avoid redundant getBucketInfo RPC on OFS getFileStatus [ozone]
jojochuang opened a new pull request, #10792:
URL: https://github.com/apache/ozone/pull/10792
## What changes were proposed in this pull request?
Rooted OFS `getFileStatus()` on normal bucket/key paths (non-snapshot) previously called `getBucket()` → InfoBucket, then `bucket.getFileStatus()`, which is already `proxy.getOzoneFileStatus()`. This PR skips the redundant **InfoBucket** RPC and calls **`GetFileStatus` only** via `proxy.getOzoneFileStatus(..., headOp)`.
Snapshot indicator paths are unchanged: they still use `getBucket()` and synthetic status (no `GetFileStatus`).
**Behavior change (accepted):** Client-side `validateBucketLayout()` no longer runs on the stat path. OM accepts `GetFileStatus` for **OBJECT_STORE** buckets, so rooted `getFileStatus` may succeed or return `FILE_NOT_FOUND` instead of throwing `IllegalArgumentException`. Mutating OFS operations (`mkdir`, `create`, `delete`, etc.) still call `getBucket()` and reject OBJECT_STORE as before.
This is a **client-side** optimization for [HDDS-15925](https://issues.apache.org/jira/browse/HDDS-15925). It is **not** the [HDDS-11232](https://issues.apache.org/jira/browse/HDDS-11232) approach (no combined OM response / server-side merge of bucket info with file status). HDDS-15925 references HDDS-11232 for the same RPC-reduction theme.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-15925
## How was this patch tested?
- [x] Unit: `TestBasicRootedOzoneClientAdapterHeadOp` (direct `getOzoneFileStatus` / `headOp`)
- [x] Integration: `TestOFS#testGetFileStatusUsesSingleOmRpc` (OM metrics: no InfoBucket increment, `GetFileStatus` +1 per stat)
- [x] Integration: missing-bucket stat expects `FileNotFoundException` (client mapping)
- [x] Smoketest: `ozonefs-obs.robot` updated for OBS stat vs mutating ops
- [x] CI on https://github.com/apache/ozone/pull/10792
Generated-by: Cursor (Composer)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15801. Avoid redundant getBucketInfo RPC on OFS getFileStatus [ozone]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #10792:
URL: https://github.com/apache/ozone/pull/10792#discussion_r3606133579
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicRootedOzoneClientAdapterImpl.java:
##########
@@ -274,6 +280,47 @@ OzoneBucket getBucket(OFSPath ofsPath, boolean createIfNotExist)
createIfNotExist);
}
+ private static String bucketLayoutCacheKey(String volumeStr, String bucketStr) {
+ return volumeStr + OZONE_URI_DELIMITER + bucketStr;
+ }
+
+ private BucketLayout validateBucketLayoutForBucket(OzoneBucket bucket)
+ throws IOException {
+ BucketLayout resolvedBucketLayout =
+ OzoneClientUtils.resolveLinkBucketLayout(bucket, objectStore,
+ new HashSet<>());
+ OzoneFSUtils.validateBucketLayout(bucket.getName(), resolvedBucketLayout);
+ return resolvedBucketLayout;
+ }
+
+ private void cacheValidatedBucketLayout(String volumeStr, String bucketStr,
+ BucketLayout layout) {
+ validatedBucketLayouts.putIfAbsent(
+ bucketLayoutCacheKey(volumeStr, bucketStr), layout);
+ }
+
+ private void validateAndCacheBucketLayout(String volumeStr, String bucketStr,
+ OzoneBucket bucket) throws IOException {
+ BucketLayout layout = validateBucketLayoutForBucket(bucket);
+ cacheValidatedBucketLayout(volumeStr, bucketStr, layout);
+ }
+
+ private void resolveAndValidateBucketLayout(String volumeStr, String bucketStr)
+ throws IOException {
+ OzoneBucket bucket = proxy.getBucketDetails(volumeStr, bucketStr);
+ validateAndCacheBucketLayout(volumeStr, bucketStr, bucket);
+ }
+
+ private void ensureBucketLayoutValid(OFSPath ofsPath) throws IOException {
+ String volumeStr = ofsPath.getVolumeName();
+ String bucketStr = ofsPath.getBucketName();
+ if (validatedBucketLayouts.containsKey(
+ bucketLayoutCacheKey(volumeStr, bucketStr))) {
+ return;
+ }
+ resolveAndValidateBucketLayout(volumeStr, bucketStr);
+ }
Review Comment:
ensureBucketLayoutValid() uses containsKey() followed by a separate getBucketDetails()+putIfAbsent(). Under concurrent access on a cold bucket, multiple threads can observe the cache miss and all issue redundant getBucketDetails (GetBucketInfo) RPCs, which undermines the goal of avoiding extra OM calls. Using computeIfAbsent() makes the cache population atomic and ensures at most one RPC per (volume,bucket) when warming the cache.
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicRootedOzoneClientAdapterImpl.java:
##########
@@ -132,6 +133,11 @@ public class BasicRootedOzoneClientAdapterImpl
private BucketLayout defaultOFSBucketLayout;
private final OzoneConfiguration config;
private final OzoneClientConfig clientConfig;
+ /**
+ * Cache of successfully validated bucket layouts for read paths.
+ */
+ private final ConcurrentHashMap<String, BucketLayout> validatedBucketLayouts =
+ new ConcurrentHashMap<>();
Review Comment:
validatedBucketLayouts is an unbounded cache that grows monotonically with every distinct (volume,bucket) accessed by this client process. In long-lived OzoneFS clients that touch many buckets (e.g., gateway services or multi-tenant jobs), this can become an unbounded memory growth vector. Consider using a bounded/expiring cache (e.g., Guava Cache with maximumSize/expireAfterAccess) or otherwise limiting growth/invalidation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15925. Avoid redundant getBucketInfo RPC on OFS getFileStatus [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang closed pull request #10792: HDDS-15925. Avoid redundant getBucketInfo RPC on OFS getFileStatus
URL: https://github.com/apache/ozone/pull/10792
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15925. Avoid redundant getBucketInfo RPC on OFS getFileStatus [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on PR #10792:
URL: https://github.com/apache/ozone/pull/10792#issuecomment-5686446474
Superseded by #11226
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15925. Avoid redundant getBucketInfo RPC on OFS getFileStatus [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on code in PR #10792:
URL: https://github.com/apache/ozone/pull/10792#discussion_r3646822154
##########
hadoop-ozone/dist/src/main/smoketest/ozonefs/ozonefs-obs.robot:
##########
@@ -51,10 +51,10 @@ Verify ls fails on OBS bucket
Create key in OBS bucket
Execute ozone sh key put /${volume}/${bucket}/testfile NOTICE.txt
-Verify ls fails on OBS bucket key
+Verify ls succeeds on OBS bucket key
${url} = Format FS URL ${SCHEME} ${volume} ${bucket} testfile
- ${result} = Execute and checkrc ozone fs -ls ${url} 255
- Should contain ${result} Bucket: ${bucket} has layout: OBJECT_STORE, which does not support file system semantics. Bucket Layout must be FILE_SYSTEM_OPTIMIZED or LEGACY.
Review Comment:
this is a breaking change. Interestingly, OM does not reject getFileStatus() call.
Without breaking this behavior, it would require a client side cache to store bucket info. But that's more complex.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15925. Avoid redundant getBucketInfo RPC on OFS getFileStatus [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10792:
URL: https://github.com/apache/ozone/pull/10792#issuecomment-5383275911
Thank you for your contribution. This PR is being closed due to inactivity. Please contact a maintainer if you would like to reopen it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11232. Avoid redundant getBucketInfo RPC on OFS getFileStatus [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on PR #10792:
URL: https://github.com/apache/ozone/pull/10792#issuecomment-5008844072
@yandrey321 this PR aims to remove the extra RPC call in most cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15925. Avoid redundant getBucketInfo RPC on OFS getFileStatus [ozone]
Posted by "yandrey321 (via GitHub)" <gi...@apache.org>.
yandrey321 commented on code in PR #10792:
URL: https://github.com/apache/ozone/pull/10792#discussion_r3633107722
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicRootedOzoneClientAdapterImpl.java:
##########
@@ -132,6 +133,11 @@ public class BasicRootedOzoneClientAdapterImpl
private BucketLayout defaultOFSBucketLayout;
private final OzoneConfiguration config;
private final OzoneClientConfig clientConfig;
+ /**
+ * Cache of successfully validated bucket layouts for read paths.
+ */
+ private final ConcurrentHashMap<String, BucketLayout> validatedBucketLayouts =
+ new ConcurrentHashMap<>();
Review Comment:
it might be the case for HBase, so I'd agree with Copilot that we need TTL/eviction policy and size limit for the cache.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15925. Avoid redundant getBucketInfo RPC on OFS getFileStatus [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10792:
URL: https://github.com/apache/ozone/pull/10792#issuecomment-5299358191
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15925. Avoid redundant getBucketInfo RPC on OFS getFileStatus [ozone]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #10792:
URL: https://github.com/apache/ozone/pull/10792#discussion_r3641905912
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicRootedOzoneClientAdapterImpl.java:
##########
@@ -690,19 +696,24 @@ public FileStatusAdapter getFileStatus(String path, URI uri,
* Return FileStatusAdapter based on OFSPath being a
* valid bucket path or valid snapshot path.
* Throws exception in case of failure.
+ *
+ * <p>Non-snapshot paths call OM GetFileStatus directly (HDDS-15925) without a
+ * prior InfoBucket RPC. OBJECT_STORE buckets are not rejected on this path;
+ * mutating OFS operations still validate layout via {@link #getBucket(OFSPath, boolean)}.
Review Comment:
The updated Javadoc around getFileStatusForKeyOrSnapshot() is potentially misleading: it states that "Non-snapshot paths" avoid InfoBucket and implies layout validation is only done for mutating operations, but listStatus() (a read operation) still calls getBucket() and enforces layout validation. Also, the PR description says OBJECT_STORE rejection is unchanged, while this method comment suggests otherwise. Consider tightening the wording to explicitly scope the optimization to key paths and to describe how OBJECT_STORE/layout validation is handled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15801. Avoid redundant getBucketInfo RPC on OFS getFileStatus [ozone]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #10792:
URL: https://github.com/apache/ozone/pull/10792#discussion_r3605905640
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/AbstractRootedOzoneFileSystemTest.java:
##########
@@ -1847,6 +1847,56 @@ void testGetFileStatus() throws Exception {
fs.delete(volume, false);
}
+ @Test
+ void testGetFileStatusUsesSingleOmRpcAfterCacheWarm() throws Exception {
+ Path path = new Path(bucketPath, "single-rpc-stat-test");
+ fs.mkdirs(path);
+ // Warm the bucket layout cache; ignore metrics from the first stat.
+ fs.getFileStatus(path);
+
+ long getFileStatusBefore = getOMMetrics().getNumGetFileStatus();
+ long bucketInfoBefore = getOMMetrics().getNumBucketInfos();
+
+ FileStatus status = fs.getFileStatus(path);
+ assertTrue(status.isDirectory());
+
+ assertEquals(getFileStatusBefore + 1, getOMMetrics().getNumGetFileStatus());
+ assertEquals(bucketInfoBefore, getOMMetrics().getNumBucketInfos());
+ }
+
+ @Test
+ void testGetFileStatusOnBucketRoot() throws Exception {
+ FileStatus status = fs.getFileStatus(bucketPath);
+ assertTrue(status.isDirectory());
+ }
+
+ @Test
+ void testGetFileStatusOnObjectStoreBucketRejectsInvalidLayout()
+ throws Exception {
+ OzoneBucket obsBucket =
+ TestDataUtil.createVolumeAndBucket(client, BucketLayout.OBJECT_STORE);
+ Path obsBucketPath = new Path(
+ new Path(OZONE_URI_DELIMITER + obsBucket.getVolumeName()),
+ obsBucket.getName());
+ try {
+ IllegalArgumentException exception = assertThrows(
+ IllegalArgumentException.class, () -> fs.getFileStatus(obsBucketPath));
+ assertThat(exception.getMessage())
+ .contains(BucketLayout.OBJECT_STORE.name());
+ } finally {
+ objectStore.deleteVolume(obsBucket.getVolumeName());
+ }
Review Comment:
The cleanup in `testGetFileStatusOnObjectStoreBucketRejectsInvalidLayout` deletes the volume but never deletes the bucket created by `TestDataUtil.createVolumeAndBucket(...)`. `ObjectStore#deleteVolume` delegates to OM and typically fails when the volume is non-empty, making the test cleanup flaky/leaky. Delete the bucket first, then the volume (consistent with other tests in this class).
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicRootedOzoneClientAdapterImpl.java:
##########
@@ -274,6 +280,47 @@ OzoneBucket getBucket(OFSPath ofsPath, boolean createIfNotExist)
createIfNotExist);
}
+ private static String bucketLayoutCacheKey(String volumeStr, String bucketStr) {
+ return volumeStr + OZONE_URI_DELIMITER + bucketStr;
+ }
+
+ private BucketLayout validateBucketLayoutForBucket(OzoneBucket bucket)
+ throws IOException {
+ BucketLayout resolvedBucketLayout =
+ OzoneClientUtils.resolveLinkBucketLayout(bucket, objectStore,
+ new HashSet<>());
+ OzoneFSUtils.validateBucketLayout(bucket.getName(), resolvedBucketLayout);
+ return resolvedBucketLayout;
+ }
+
+ private void cacheValidatedBucketLayout(String volumeStr, String bucketStr,
+ BucketLayout layout) {
+ validatedBucketLayouts.putIfAbsent(
+ bucketLayoutCacheKey(volumeStr, bucketStr), layout);
+ }
+
+ private void validateAndCacheBucketLayout(String volumeStr, String bucketStr,
+ OzoneBucket bucket) throws IOException {
+ BucketLayout layout = validateBucketLayoutForBucket(bucket);
+ cacheValidatedBucketLayout(volumeStr, bucketStr, layout);
+ }
+
+ private void resolveAndValidateBucketLayout(String volumeStr, String bucketStr)
+ throws IOException {
+ OzoneBucket bucket = proxy.getBucketDetails(volumeStr, bucketStr);
+ validateAndCacheBucketLayout(volumeStr, bucketStr, bucket);
+ }
+
+ private void ensureBucketLayoutValid(OFSPath ofsPath) throws IOException {
+ String volumeStr = ofsPath.getVolumeName();
+ String bucketStr = ofsPath.getBucketName();
+ if (validatedBucketLayouts.containsKey(
+ bucketLayoutCacheKey(volumeStr, bucketStr))) {
+ return;
+ }
+ resolveAndValidateBucketLayout(volumeStr, bucketStr);
+ }
Review Comment:
`ensureBucketLayoutValid` uses a non-atomic `containsKey` + RPC pattern. Under concurrent access, multiple threads can race and each issue `getBucketDetails`, reintroducing redundant OM calls and partially defeating the intended optimization. Using `computeIfAbsent` (with checked-exception wrapping) makes the cache population atomic per bucket.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15925. Avoid redundant getBucketInfo RPC on OFS getFileStatus [ozone]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #10792:
URL: https://github.com/apache/ozone/pull/10792#discussion_r3641371524
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicRootedOzoneClientAdapterImpl.java:
##########
@@ -690,19 +696,24 @@ public FileStatusAdapter getFileStatus(String path, URI uri,
* Return FileStatusAdapter based on OFSPath being a
* valid bucket path or valid snapshot path.
* Throws exception in case of failure.
+ *
+ * <p>Non-snapshot paths call OM GetFileStatus directly (HDDS-15925) without a
+ * prior InfoBucket RPC. OBJECT_STORE buckets are not rejected on this path;
+ * mutating OFS operations still validate layout via {@link #getBucket(OFSPath, boolean)}.
*/
private FileStatusAdapter getFileStatusForKeyOrSnapshot(
OFSPath ofsPath, URI uri, Path qualifiedPath, String userName,
boolean headOp) throws IOException {
String key = ofsPath.getKeyName();
try {
- OzoneBucket bucket = getBucket(ofsPath, false);
if (ofsPath.isSnapshotPath()) {
+ OzoneBucket bucket = getBucket(ofsPath, false);
OzoneVolume volume = objectStore.getVolume(ofsPath.getVolumeName());
return getFileStatusAdapterWithSnapshotIndicator(
volume, bucket, uri);
} else {
- OzoneFileStatus status = bucket.getFileStatus(key, headOp);
+ OzoneFileStatus status = proxy.getOzoneFileStatus(ofsPath.getVolumeName(),
+ ofsPath.getBucketName(), key, headOp);
return toFileStatusAdapter(status, userName, uri, qualifiedPath,
Review Comment:
The new non-snapshot path in getFileStatusForKeyOrSnapshot bypasses getBucket(...), so it no longer runs link-bucket layout resolution + OzoneFSUtils.validateBucketLayout(...). That means getFileStatus can now succeed on OBJECT_STORE (and link-to-OBJECT_STORE) buckets even though other OFS operations still reject them via getBucket(). This appears to contradict the PR description (“OBJECT_STORE rejection” and “link-bucket layout validation” unchanged) and also makes OFS behavior inconsistent across operations.
Recommendation: please clarify the intended semantics. If getFileStatus should continue to reject OBJECT_STORE / invalid link layouts, keep the optimization by validating once (eg, validate+cache per (volume,bucket) with size/TTL) and only falling back to an InfoBucket lookup on first encounter; otherwise update the PR description and add/adjust tests to lock in the new relaxed behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15801. Avoid redundant getBucketInfo RPC on OFS getFileStatus [ozone]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #10792:
URL: https://github.com/apache/ozone/pull/10792#discussion_r3606474660
##########
hadoop-ozone/ozonefs-common/src/test/java/org/apache/hadoop/fs/ozone/TestBasicRootedOzoneClientAdapterHeadOp.java:
##########
@@ -77,10 +82,29 @@ public void setUp() throws Exception {
when(volume.getCreationTime()).thenReturn(Instant.EPOCH);
ObjectStore objectStore = mock(ObjectStore.class);
when(objectStore.getVolume(anyString())).thenReturn(volume);
- Field field =
- BasicRootedOzoneClientAdapterImpl.class.getDeclaredField("objectStore");
+ setField(adapter, "objectStore", objectStore);
+ setField(adapter, "proxy", proxy);
+ warmLayoutCache("vol", "bucket");
+ }
+
+ private static void setField(Object target, String name, Object value)
+ throws Exception {
+ Field field = BasicRootedOzoneClientAdapterImpl.class.getDeclaredField(name);
field.setAccessible(true);
- field.set(adapter, objectStore);
+ field.set(target, value);
+ }
+
+ @SuppressWarnings("unchecked")
+ private void warmLayoutCache(String volumeName, String bucketName)
+ throws Exception {
+ Field cacheField =
+ BasicRootedOzoneClientAdapterImpl.class.getDeclaredField(
+ "validatedBucketLayouts");
+ cacheField.setAccessible(true);
+ ConcurrentHashMap<String, BucketLayout> cache =
+ new ConcurrentHashMap<>();
+ cache.put(volumeName + "/" + bucketName, BucketLayout.FILE_SYSTEM_OPTIMIZED);
+ cacheField.set(adapter, cache);
Review Comment:
warmLayoutCache() replaces the validatedBucketLayouts field via reflection. In production this field is final, and swapping the reference can be blocked or behave inconsistently under newer JDKs. It’s safer to read the existing map instance from the field and mutate it (put the warm entry) rather than setting a new ConcurrentHashMap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15925. Avoid redundant getBucketInfo RPC on OFS getFileStatus [ozone]
Posted by "yandrey321 (via GitHub)" <gi...@apache.org>.
yandrey321 commented on PR #10792:
URL: https://github.com/apache/ozone/pull/10792#issuecomment-5050258262
this is a good fix for reducing number of RPC calls for metadata, we can extend this fix or file a separate JIRA to route getFileChecksum, listStatus, getVolumeInfo and getBucket through the same cache.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15925. Avoid redundant getBucketInfo RPC on OFS getFileStatus [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #10792: HDDS-15925. Avoid redundant getBucketInfo RPC on OFS getFileStatus
URL: https://github.com/apache/ozone/pull/10792
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15925. Avoid redundant getBucketInfo RPC on OFS getFileStatus [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on code in PR #10792:
URL: https://github.com/apache/ozone/pull/10792#discussion_r3963763742
##########
hadoop-ozone/dist/src/main/smoketest/ozonefs/ozonefs-obs.robot:
##########
@@ -51,10 +51,10 @@ Verify ls fails on OBS bucket
Create key in OBS bucket
Execute ozone sh key put /${volume}/${bucket}/testfile NOTICE.txt
-Verify ls fails on OBS bucket key
+Verify ls succeeds on OBS bucket key
${url} = Format FS URL ${SCHEME} ${volume} ${bucket} testfile
- ${result} = Execute and checkrc ozone fs -ls ${url} 255
- Should contain ${result} Bucket: ${bucket} has layout: OBJECT_STORE, which does not support file system semantics. Bucket Layout must be FILE_SYSTEM_OPTIMIZED or LEGACY.
Review Comment:
fixed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org