You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Kohei Sugihara (Jira)" <ji...@apache.org> on 2022/10/07 03:09:00 UTC
[jira] [Created] (HDDS-7295) listKeys for large FSO-enabled bucket is timeout
Kohei Sugihara created HDDS-7295:
------------------------------------
Summary: listKeys for large FSO-enabled bucket is timeout
Key: HDDS-7295
URL: https://issues.apache.org/jira/browse/HDDS-7295
Project: Apache Ozone
Issue Type: Bug
Components: S3
Affects Versions: 1.2.1
Environment: 1.3.0-SNAPSHOT (https://github.com/apache/ozone/tree/843fac2fb646eecfc33103fdb16eaf77a66ca062)
Reporter: Kohei Sugihara
When we have a large bucket (e.g. # of keys > 1M; shows below) with FSO-enabled, listKeys for top of bucket (ex. s3://bucket/) via S3G is timeout even non-recursive request (also recursive request).
{{+ s3v/}}
{{ + bucket/}}
{{ + prefix1/}}
{{ - <1M keys>}}
{{ + prefix2/}}
{{{} - <1M keys>{}}}{{{}{}}}
{{ + prefix3/}}
{{ - <1M keys>}}
{{{} + ...{}}}{{{}{}}}{{{}{}}}{{{}{}}}{{{}{}}}
For S3G, I think BucketEndpoint.java [1] always queries all keys recursively (both {{KeyIterator}} and {{{}KeyIteratorWithFSO{}}}).
[1]https://github.com/apache/ozone/blob/a8808d1c3781627c40e0ed25d0bb4ec1e74e3de2/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java#L134-L144
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org