[jira] [Commented] (HDDS-7715) keyManagerImpl#listStatus exceeds the maximum RPC length

["Tanvi Penumudy (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDDS-7715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17771330#comment-17771330 ] 

Tanvi Penumudy commented on HDDS-7715:
--------------------------------------

Solution - PR [#5275|https://github.com/apache/ozone/pull/5275]: To introduce a newer listStatus API (we have chosen to introduce a newer API over modifying the existing API as it would break compatibility for older ozone clients) which uses a trimmed {{KeyInfo}} {{BasicKeyInfo}} proto message for excluding fields such as {{KeyLocations}} which are not very useful on the client side:

The new lightweight listStatus API would be used for:
 * {{o3}} CLI: ({{ozone sh key list}})
 * S3 ListObject requests

The existing listStatus API will continue being used for:
 * {{ofs}} commands

{code:java}
message BasicKeyInfo {
    optional string keyName = 1;
    optional uint64 dataSize = 2;
    optional uint64 creationTime = 3;
    optional uint64 modificationTime = 4;
    optional hadoop.hdds.ReplicationType type = 5;
    optional hadoop.hdds.ReplicationFactor factor = 6;
    optional hadoop.hdds.ECReplicationConfig ecReplicationConfig = 7;
}
{code}
 
This patch shall utilize the server-side Ozone key encapsulations and proto messages introduced as part of HDDS-9079. Implement a lightweight listKeys API (PR [#5115|https://github.com/apache/ozone/pull/5115]).

> keyManagerImpl#listStatus exceeds the maximum RPC length
> --------------------------------------------------------
>
>                 Key: HDDS-7715
>                 URL: https://issues.apache.org/jira/browse/HDDS-7715
>             Project: Apache Ozone
>          Issue Type: Bug
>          Components: OM, Ozone Manager
>    Affects Versions: 1.3.0
>         Environment: Version: Our forked version based ozone-1.3.0 branch. commit hash is [https://github.com/apache/ozone/commit/9c61a8aa497ab96c014ad3bb7b1ee4f731ebfaf8]
> FSO-enabled bucket
>            Reporter: Kohei Sugihara
>            Assignee: Tanvi Penumudy
>            Priority: Major
>              Labels: ozone-performance, pull-request-available
>         Attachments: large-protos.png
>
>
> keyManagerImpl#listStatus for FSO buckets serves a KeyLocationList but might be large size of response (> 100 MB) and exceed the limit. The problem is KeyLocation field is large enough (> 1MB) so just listing some multi-part uploaded keys exceeds the limit. A possible solution is to reduce KeyLocation size by omitting pipeline fields.
> {quote}% ozone sh key ls /s3v/bucket --prefix=path/to/prefix
> RPC response exceeds maximum data length
> {quote}
>  
> We checked a List<OzoneFileStatus> in our environment and it turned out that protobufs corresponding to some multi-part keys are larger than 1MB. AFAIK, the maximum IPC length is -100MB- 128MB by default so it easily exceeds the limit.
> !large-protos.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org