You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Kohei Sugihara (Jira)" <ji...@apache.org> on 2022/10/07 03:09:00 UTC

[jira] [Updated] (HDDS-7295) listKeys for large FSO-enabled bucket is timeout

     [ https://issues.apache.org/jira/browse/HDDS-7295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kohei Sugihara updated HDDS-7295:
---------------------------------
    Description: 
When we have a large bucket (e.g. # of keys > 1M; shows below) with FSO-enabled, listKeys for top of bucket (ex. s3://bucket/) via S3G is timeout even non-recursive request (also recursive request).

{{+ s3v/}}
{{  + bucket/}}
{{    + prefix1/}}
{{      - <1M keys>}}
{{    + prefix2/}}
{{      - <1M keys>}}
{{    + prefix3/}}
{{      - <1M keys>}}
{{    + ...}}

For S3G, I think BucketEndpoint.java [1] always queries all keys recursively (both {{KeyIterator}} and {{{}KeyIteratorWithFSO{}}}).

[1]https://github.com/apache/ozone/blob/a8808d1c3781627c40e0ed25d0bb4ec1e74e3de2/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java#L134-L144

  was:
When we have a large bucket (e.g. # of keys > 1M; shows below) with FSO-enabled, listKeys for top of bucket (ex. s3://bucket/) via S3G is timeout even non-recursive request (also recursive request).

{{+ s3v/}}
{{  + bucket/}}
{{    + prefix1/}}
{{      - <1M keys>}}
{{    + prefix2/}}
{{{}      - <1M keys>{}}}{{{}{}}}
{{    + prefix3/}}
{{      - <1M keys>}}
{{{}    + ...{}}}{{{}{}}}{{{}{}}}{{{}{}}}{{{}{}}}

For S3G, I think BucketEndpoint.java [1] always queries all keys recursively (both {{KeyIterator}} and {{{}KeyIteratorWithFSO{}}}).


[1]https://github.com/apache/ozone/blob/a8808d1c3781627c40e0ed25d0bb4ec1e74e3de2/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java#L134-L144


> listKeys for large FSO-enabled bucket is timeout
> ------------------------------------------------
>
>                 Key: HDDS-7295
>                 URL: https://issues.apache.org/jira/browse/HDDS-7295
>             Project: Apache Ozone
>          Issue Type: Bug
>          Components: S3
>    Affects Versions: 1.2.1
>         Environment: 1.3.0-SNAPSHOT (https://github.com/apache/ozone/tree/843fac2fb646eecfc33103fdb16eaf77a66ca062)
>            Reporter: Kohei Sugihara
>            Priority: Major
>
> When we have a large bucket (e.g. # of keys > 1M; shows below) with FSO-enabled, listKeys for top of bucket (ex. s3://bucket/) via S3G is timeout even non-recursive request (also recursive request).
> {{+ s3v/}}
> {{  + bucket/}}
> {{    + prefix1/}}
> {{      - <1M keys>}}
> {{    + prefix2/}}
> {{      - <1M keys>}}
> {{    + prefix3/}}
> {{      - <1M keys>}}
> {{    + ...}}
> For S3G, I think BucketEndpoint.java [1] always queries all keys recursively (both {{KeyIterator}} and {{{}KeyIteratorWithFSO{}}}).
> [1]https://github.com/apache/ozone/blob/a8808d1c3781627c40e0ed25d0bb4ec1e74e3de2/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java#L134-L144



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org