You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Andrey Yarovoy (Jira)" <ji...@apache.org> on 2026/10/18 02:07:00 UTC
[jira] [Updated] (HDDS-16948) Remove the per-call InfoBucket from rooted-ofs open
[ https://issues.apache.org/jira/browse/HDDS-16948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrey Yarovoy updated HDDS-16948:
----------------------------------
Description:
`BasicRootedOzoneClientAdapterImpl.readFile` (behind `FileSystem.open`) calls `getBucket()`, which is an
`InfoBucket` RPC. For a link bucket it adds an `InfoVolume` and an `InfoBucket` per hop. It then calls
`OzoneBucket.readFile`. Every `open` therefore costs at least 2 OM requests. On read-heavy file system
workloads (Hive, Impala, Spark scans) `open` is one of the most frequent metadata operations, so this is
the largest single `InfoBucket` source left after HDDS-15925 and HDDS-16760.
The fetch does one job: `OzoneFSUtils.validateBucketLayout`, which rejects OBJECT_STORE. `readFile`
reads nothing else from the bucket.
Why HDDS-15951's lookupFile check does not cover this
`RpcClient.readFile` uses `lookupFile` only for an OM older than `OPTIMIZED_GET_KEY_INFO` (3). For every
current OM it sends *{*}`GetKeyInfo`{*}*, then checks `isFile()` on the client. Its read-retry path
(`RpcClient.getKeyInfo`) also sends `GetKeyInfo`. So the OBJECT_STORE rejection that HDDS-15951 added to
`lookupFile` is never reached from `open`.
*Why the rejection is opt-in*
`GetKeyInfo` is the object store read RPC. It sits behind `OzoneBucket.readKey`, `getKeyDetails`,
`headObject` and the S3 key lookups (`assumeS3Context=true`), which the S3 Gateway uses for GetObject and
HeadObject. `cli-shell`, `freon` and `cli-debug` call it as well. An unconditional OBJECT_STORE rejection
would break all of them. As in HDDS-16760, the client asks for the check:
{code:java}
message GetKeyInfoRequest {
required KeyArgs keyArgs = 1;
optional bool assumeS3Context = 2;
// Set by file system clients: OM rejects the request with
// NOT_SUPPORTED_OPERATION if the resolved bucket layout is OBJECT_STORE.
optional bool requireFileSystemLayout = 3;
}
{code}
*Proposed change*
*OM.*
* `OzoneManagerRequestHandler.getKeyInfo` copies the field into `OmKeyArgs`. The `requireFileSystemLayout`
field added there by HDDS-16760 is reused, and the builder already survives `ResolvedBucket.update`.
* `OmMetadataReader.getKeyInfo` already holds the link-resolved `ResolvedBucket`. When the field is set,
it calls the existing `validateFileSystemLayout(bucket)` before the ACL check. This is the same
placement as `listStatus` in HDDS-16760, so it adds no lookup.
*Client.*
* `RpcClient.readFile` sets the field on the `OmKeyArgs` it builds for `GetKeyInfo`. The read-retry
`getKeyInfo` call also sets it, so a retried read is checked too.
* `OzoneManagerProtocolClientSideTranslatorPB.getKeyInfo` builds `KeyArgs` through
`OmKeyArgs.toProtobuf()` but sets the request-level field explicitly. Do not set it unconditionally:
`readKey` / `getKeyDetails` share this translator method.
* The adapter needs a way to ask for it, mirroring HDDS-16760's `ListStatusLightOptions`. Options are a
`readFile` overload or an options object on `ClientProtocol`, or `OzoneBucket.readFile` setting it
itself. `OzoneBucket.readFile` is a file-oriented API, so it could always set it; decide on review.
* The adapter skips `getBucket()` when `proxy.getOmVersion() >=` a new `OzoneManagerVersion` constant.
Place it after `LIST_STATUS_REJECTS_OBS(101)` and never below `ZDU(100)`; the number is assigned at
merge. It maps `NOT_SUPPORTED_OPERATION` to `IllegalArgumentException`. Against any older OM, including
one that reports ZDU, it keeps the current path.
* The `lookupFile` branch for OMs older than version 3 is unchanged, and such OMs take the fallback anyway.
*Rejected alternative*
`GetKeyInfoResponse` already returns `bucketLayout` (HDDS-16362), so the client could check that instead
and need no new request field. Two problems:
* It only works when the key exists. A missing key on an OBJECT_STORE bucket would then report
`FileNotFoundException` instead of today's `IllegalArgumentException`.
* The field was added without a version gate, so an older OM leaves it unset, and the client cannot
tell "not OBJECT_STORE" from "not reported".
#
##
### Compatibility
| |Behavior|
|—|—|
|New client, new OM|1 RPC (`GetKeyInfo`); OM rejects OBJECT_STORE|
|New client, older OM (incl. ZDU)|`getBucket()` + `GetKeyInfo`, as today|
|Old client / S3 Gateway / `readKey` / `getKeyDetails`, new OM|field absent ⇒ unchanged|
There is one new optional proto field, no new result code and no new configuration.
*Testing*
* Adapter (Mockito):
*`open` makes exactly one OM call, carrying the field.
*`NOT_SUPPORTED_OPERATION` maps to `IllegalArgumentException`; `KEY_NOT_FOUND` / `FILE_NOT_FOUND` /
`NOT_A_FILE` map to `FileNotFoundException`, as today.
* An older OM, including `ZDU`, falls back to `getBucket()`.
* `TestOMMetadataReader`:
* With the field set, `getKeyInfo` rejects OBJECT_STORE without reaching `KeyManager` and serves FSO
and *{*}LEGACY{*}*.
* Without the field, it still serves OBJECT_STORE: the S3 Gateway case, and the guard against anyone
making the check unconditional.
* The same holds with `assumeS3Context=true`.
* `TestOzoneManagerRequestHandler`: the field is forwarded into `OmKeyArgs`.
* Existing S3 and object-store read suites pass unchanged.
* `AbstractRootedOzoneFileSystemTest`: `open` on an OBJECT_STORE bucket fails with
`IllegalArgumentException`. Do RPC counting in the adapter test, not with exact OMMetrics deltas
(HDDS-16513).
* Benchmark `TestOfsOpenRpcBenchmark`, same shape as `TestOfsListStatusRpcBenchmark`:
* 200 FSO buckets, open + close of a small file, single thread and 10 threads.
* n=3 per arm, built and run strictly one after the other.
* Invariant: exactly 1 OM RPC per `open` instead of 2. `open` alone, with no data read, so DN reads do
not mask the OM saving.
was:
`BasicRootedOzoneClientAdapterImpl.readFile` (behind `FileSystem.open`) calls `getBucket()`, which is an
`InfoBucket` RPC. For a link bucket it adds an `InfoVolume` and an `InfoBucket` per hop. It then calls
`OzoneBucket.readFile`. Every `open` therefore costs at least 2 OM requests. On read-heavy file system
workloads (Hive, Impala, Spark scans) `open` is one of the most frequent metadata operations, so this is
the largest single `InfoBucket` source left after HDDS-15925 and HDDS-16760.
The fetch does one job: `OzoneFSUtils.validateBucketLayout`, which rejects OBJECT_STORE. `readFile`
reads nothing else from the bucket.
### Why HDDS-15951's lookupFile check does not cover this
`RpcClient.readFile` uses `lookupFile` only for an OM older than `OPTIMIZED_GET_KEY_INFO` (3). For every
current OM it sends **`GetKeyInfo`**, then checks `isFile()` on the client. Its read-retry path
(`RpcClient.getKeyInfo`) also sends `GetKeyInfo`. So the OBJECT_STORE rejection that HDDS-15951 added to
`lookupFile` is never reached from `open`.
### Why the rejection is opt-in
`GetKeyInfo` is the object store read RPC. It sits behind `OzoneBucket.readKey`, `getKeyDetails`,
`headObject` and the S3 key lookups (`assumeS3Context=true`), which the S3 Gateway uses for GetObject and
HeadObject. `cli-shell`, `freon` and `cli-debug` call it as well. An unconditional OBJECT_STORE rejection
would break all of them. As in HDDS-16760, the client asks for the check:
{code}
message GetKeyInfoRequest {
required KeyArgs keyArgs = 1;
optional bool assumeS3Context = 2;
// Set by file system clients: OM rejects the request with
// NOT_SUPPORTED_OPERATION if the resolved bucket layout is OBJECT_STORE.
optional bool requireFileSystemLayout = 3;
}
{code}
### Proposed change
*OM.*
* `OzoneManagerRequestHandler.getKeyInfo` copies the field into `OmKeyArgs`. The `requireFileSystemLayout`
field added there by HDDS-16760 is reused, and the builder already survives `ResolvedBucket.update`.
* `OmMetadataReader.getKeyInfo` already holds the link-resolved `ResolvedBucket`. When the field is set,
it calls the existing `validateFileSystemLayout(bucket)` before the ACL check. This is the same
placement as `listStatus` in HDDS-16760, so it adds no lookup.
*Client.*
* `RpcClient.readFile` sets the field on the `OmKeyArgs` it builds for `GetKeyInfo`. The read-retry
`getKeyInfo` call also sets it, so a retried read is checked too.
* `OzoneManagerProtocolClientSideTranslatorPB.getKeyInfo` builds `KeyArgs` through
`OmKeyArgs.toProtobuf()` but sets the request-level field explicitly. Do not set it unconditionally:
`readKey` / `getKeyDetails` share this translator method.
* The adapter needs a way to ask for it, mirroring HDDS-16760's `ListStatusLightOptions`. Options are a
`readFile` overload or an options object on `ClientProtocol`, or `OzoneBucket.readFile` setting it
itself. `OzoneBucket.readFile` is a file-oriented API, so it could always set it; decide on review.
* The adapter skips `getBucket()` when `proxy.getOmVersion() >=` a new `OzoneManagerVersion` constant.
Place it after `LIST_STATUS_REJECTS_OBS(101)` and never below `ZDU(100)`; the number is assigned at
merge. It maps `NOT_SUPPORTED_OPERATION` to `IllegalArgumentException`. Against any older OM, including
one that reports ZDU, it keeps the current path.
* The `lookupFile` branch for OMs older than version 3 is unchanged, and such OMs take the fallback anyway.
### Rejected alternative
`GetKeyInfoResponse` already returns `bucketLayout` (HDDS-16362), so the client could check that instead
and need no new request field. Two problems:
* It only works when the key exists. A missing key on an OBJECT_STORE bucket would then report
`FileNotFoundException` instead of today's `IllegalArgumentException`.
* The field was added without a version gate, so an older OM leaves it unset, and the client cannot
tell "not OBJECT_STORE" from "not reported".
### Compatibility
| | Behavior |
|---|---|
| New client, new OM | 1 RPC (`GetKeyInfo`); OM rejects OBJECT_STORE |
| New client, older OM (incl. ZDU) | `getBucket()` + `GetKeyInfo`, as today |
| Old client / S3 Gateway / `readKey` / `getKeyDetails`, new OM | field absent ⇒ unchanged |
There is one new optional proto field, no new result code and no new configuration.
## Testing
* Adapter (Mockito):
*`open` makes exactly one OM call, carrying the field.
*`NOT_SUPPORTED_OPERATION` maps to `IllegalArgumentException`; `KEY_NOT_FOUND` / `FILE_NOT_FOUND` /
`NOT_A_FILE` map to `FileNotFoundException`, as today.
* An older OM, including `ZDU`, falls back to `getBucket()`.
* `TestOMMetadataReader`:
* With the field set, `getKeyInfo` rejects OBJECT_STORE without reaching `KeyManager` and serves FSO
and **LEGACY**.
* Without the field, it still serves OBJECT_STORE: the S3 Gateway case, and the guard against anyone
making the check unconditional.
* The same holds with `assumeS3Context=true`.
* `TestOzoneManagerRequestHandler`: the field is forwarded into `OmKeyArgs`.
* Existing S3 and object-store read suites pass unchanged.
* `AbstractRootedOzoneFileSystemTest`: `open` on an OBJECT_STORE bucket fails with
`IllegalArgumentException`. Do RPC counting in the adapter test, not with exact OMMetrics deltas
(HDDS-16513).
* Benchmark `TestOfsOpenRpcBenchmark`, same shape as `TestOfsListStatusRpcBenchmark`:
* 200 FSO buckets, open + close of a small file, single thread and 10 threads.
* n=3 per arm, built and run strictly one after the other.
* Invariant: exactly 1 OM RPC per `open` instead of 2. `open` alone, with no data read, so DN reads do
not mask the OM saving.
> Remove the per-call InfoBucket from rooted-ofs open
> ---------------------------------------------------
>
> Key: HDDS-16948
> URL: https://issues.apache.org/jira/browse/HDDS-16948
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Ozone Client
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
>
> `BasicRootedOzoneClientAdapterImpl.readFile` (behind `FileSystem.open`) calls `getBucket()`, which is an
> `InfoBucket` RPC. For a link bucket it adds an `InfoVolume` and an `InfoBucket` per hop. It then calls
> `OzoneBucket.readFile`. Every `open` therefore costs at least 2 OM requests. On read-heavy file system
> workloads (Hive, Impala, Spark scans) `open` is one of the most frequent metadata operations, so this is
> the largest single `InfoBucket` source left after HDDS-15925 and HDDS-16760.
> The fetch does one job: `OzoneFSUtils.validateBucketLayout`, which rejects OBJECT_STORE. `readFile`
> reads nothing else from the bucket.
> Why HDDS-15951's lookupFile check does not cover this
> `RpcClient.readFile` uses `lookupFile` only for an OM older than `OPTIMIZED_GET_KEY_INFO` (3). For every
> current OM it sends *{*}`GetKeyInfo`{*}*, then checks `isFile()` on the client. Its read-retry path
> (`RpcClient.getKeyInfo`) also sends `GetKeyInfo`. So the OBJECT_STORE rejection that HDDS-15951 added to
> `lookupFile` is never reached from `open`.
> *Why the rejection is opt-in*
> `GetKeyInfo` is the object store read RPC. It sits behind `OzoneBucket.readKey`, `getKeyDetails`,
> `headObject` and the S3 key lookups (`assumeS3Context=true`), which the S3 Gateway uses for GetObject and
> HeadObject. `cli-shell`, `freon` and `cli-debug` call it as well. An unconditional OBJECT_STORE rejection
> would break all of them. As in HDDS-16760, the client asks for the check:
> {code:java}
> message GetKeyInfoRequest {
> required KeyArgs keyArgs = 1;
> optional bool assumeS3Context = 2;
> // Set by file system clients: OM rejects the request with
> // NOT_SUPPORTED_OPERATION if the resolved bucket layout is OBJECT_STORE.
> optional bool requireFileSystemLayout = 3;
> }
> {code}
> *Proposed change*
> *OM.*
> * `OzoneManagerRequestHandler.getKeyInfo` copies the field into `OmKeyArgs`. The `requireFileSystemLayout`
> field added there by HDDS-16760 is reused, and the builder already survives `ResolvedBucket.update`.
> * `OmMetadataReader.getKeyInfo` already holds the link-resolved `ResolvedBucket`. When the field is set,
> it calls the existing `validateFileSystemLayout(bucket)` before the ACL check. This is the same
> placement as `listStatus` in HDDS-16760, so it adds no lookup.
> *Client.*
> * `RpcClient.readFile` sets the field on the `OmKeyArgs` it builds for `GetKeyInfo`. The read-retry
> `getKeyInfo` call also sets it, so a retried read is checked too.
> * `OzoneManagerProtocolClientSideTranslatorPB.getKeyInfo` builds `KeyArgs` through
> `OmKeyArgs.toProtobuf()` but sets the request-level field explicitly. Do not set it unconditionally:
> `readKey` / `getKeyDetails` share this translator method.
> * The adapter needs a way to ask for it, mirroring HDDS-16760's `ListStatusLightOptions`. Options are a
> `readFile` overload or an options object on `ClientProtocol`, or `OzoneBucket.readFile` setting it
> itself. `OzoneBucket.readFile` is a file-oriented API, so it could always set it; decide on review.
> * The adapter skips `getBucket()` when `proxy.getOmVersion() >=` a new `OzoneManagerVersion` constant.
> Place it after `LIST_STATUS_REJECTS_OBS(101)` and never below `ZDU(100)`; the number is assigned at
> merge. It maps `NOT_SUPPORTED_OPERATION` to `IllegalArgumentException`. Against any older OM, including
> one that reports ZDU, it keeps the current path.
> * The `lookupFile` branch for OMs older than version 3 is unchanged, and such OMs take the fallback anyway.
> *Rejected alternative*
> `GetKeyInfoResponse` already returns `bucketLayout` (HDDS-16362), so the client could check that instead
> and need no new request field. Two problems:
> * It only works when the key exists. A missing key on an OBJECT_STORE bucket would then report
> `FileNotFoundException` instead of today's `IllegalArgumentException`.
> * The field was added without a version gate, so an older OM leaves it unset, and the client cannot
> tell "not OBJECT_STORE" from "not reported".
> #
> ##
> ### Compatibility
> | |Behavior|
> |—|—|
> |New client, new OM|1 RPC (`GetKeyInfo`); OM rejects OBJECT_STORE|
> |New client, older OM (incl. ZDU)|`getBucket()` + `GetKeyInfo`, as today|
> |Old client / S3 Gateway / `readKey` / `getKeyDetails`, new OM|field absent ⇒ unchanged|
> There is one new optional proto field, no new result code and no new configuration.
> *Testing*
> * Adapter (Mockito):
> *`open` makes exactly one OM call, carrying the field.
> *`NOT_SUPPORTED_OPERATION` maps to `IllegalArgumentException`; `KEY_NOT_FOUND` / `FILE_NOT_FOUND` /
> `NOT_A_FILE` map to `FileNotFoundException`, as today.
> * An older OM, including `ZDU`, falls back to `getBucket()`.
> * `TestOMMetadataReader`:
> * With the field set, `getKeyInfo` rejects OBJECT_STORE without reaching `KeyManager` and serves FSO
> and *{*}LEGACY{*}*.
> * Without the field, it still serves OBJECT_STORE: the S3 Gateway case, and the guard against anyone
> making the check unconditional.
> * The same holds with `assumeS3Context=true`.
> * `TestOzoneManagerRequestHandler`: the field is forwarded into `OmKeyArgs`.
> * Existing S3 and object-store read suites pass unchanged.
> * `AbstractRootedOzoneFileSystemTest`: `open` on an OBJECT_STORE bucket fails with
> `IllegalArgumentException`. Do RPC counting in the adapter test, not with exact OMMetrics deltas
> (HDDS-16513).
> * Benchmark `TestOfsOpenRpcBenchmark`, same shape as `TestOfsListStatusRpcBenchmark`:
> * 200 FSO buckets, open + close of a small file, single thread and 10 threads.
> * n=3 per arm, built and run strictly one after the other.
> * Invariant: exactly 1 OM RPC per `open` instead of 2. `open` alone, with no data read, so DN reads do
> not mask the OM saving.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org