You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Kohei Sugihara (Jira)" <ji...@apache.org> on 2022/10/07 03:09:00 UTC
[jira] [Updated] (HDDS-7295) listKeys for large FSO-enabled bucket is timeout
[ https://issues.apache.org/jira/browse/HDDS-7295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kohei Sugihara updated HDDS-7295:
---------------------------------
Description:
When we have a large bucket (e.g. # of keys > 1M; shows below) with FSO-enabled, listKeys for top of bucket (ex. s3://bucket/) via S3G is timeout even non-recursive request (also recursive request).
{{+ s3v/}}
{{ + bucket/}}
{{ + prefix1/}}
{{ - <1M keys>}}
{{ + prefix2/}}
{{ - <1M keys>}}
{{ + prefix3/}}
{{ - <1M keys>}}
{{ + ...}}
For S3G, I think BucketEndpoint.java [1] always queries all keys recursively (both {{KeyIterator}} and {{{}KeyIteratorWithFSO{}}}).
[1]https://github.com/apache/ozone/blob/a8808d1c3781627c40e0ed25d0bb4ec1e74e3de2/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java#L134-L144
was:
When we have a large bucket (e.g. # of keys > 1M; shows below) with FSO-enabled, listKeys for top of bucket (ex. s3://bucket/) via S3G is timeout even non-recursive request (also recursive request).
{{+ s3v/}}
{{ + bucket/}}
{{ + prefix1/}}
{{ - <1M keys>}}
{{ + prefix2/}}
{{{} - <1M keys>{}}}{{{}{}}}
{{ + prefix3/}}
{{ - <1M keys>}}
{{{} + ...{}}}{{{}{}}}{{{}{}}}{{{}{}}}{{{}{}}}
For S3G, I think BucketEndpoint.java [1] always queries all keys recursively (both {{KeyIterator}} and {{{}KeyIteratorWithFSO{}}}).
[1]https://github.com/apache/ozone/blob/a8808d1c3781627c40e0ed25d0bb4ec1e74e3de2/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java#L134-L144
> listKeys for large FSO-enabled bucket is timeout
> ------------------------------------------------
>
> Key: HDDS-7295
> URL: https://issues.apache.org/jira/browse/HDDS-7295
> Project: Apache Ozone
> Issue Type: Bug
> Components: S3
> Affects Versions: 1.2.1
> Environment: 1.3.0-SNAPSHOT (https://github.com/apache/ozone/tree/843fac2fb646eecfc33103fdb16eaf77a66ca062)
> Reporter: Kohei Sugihara
> Priority: Major
>
> When we have a large bucket (e.g. # of keys > 1M; shows below) with FSO-enabled, listKeys for top of bucket (ex. s3://bucket/) via S3G is timeout even non-recursive request (also recursive request).
> {{+ s3v/}}
> {{ + bucket/}}
> {{ + prefix1/}}
> {{ - <1M keys>}}
> {{ + prefix2/}}
> {{ - <1M keys>}}
> {{ + prefix3/}}
> {{ - <1M keys>}}
> {{ + ...}}
> For S3G, I think BucketEndpoint.java [1] always queries all keys recursively (both {{KeyIterator}} and {{{}KeyIteratorWithFSO{}}}).
> [1]https://github.com/apache/ozone/blob/a8808d1c3781627c40e0ed25d0bb4ec1e74e3de2/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java#L134-L144
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org