You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Huang Kuan Hao (Jira)" <ji...@apache.org> on 2026/10/02 09:14:00 UTC
[jira] [Updated] (HDDS-16673) Avoid redundant ListBuckets RPCs when S3 bucket listing reaches the end
[ https://issues.apache.org/jira/browse/HDDS-16673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Huang Kuan Hao updated HDDS-16673:
----------------------------------
Description:
RootEndpoint calls hasNext() again to determine whether a continuation token is needed after the listing loop has already reached the end. The bucket iterator fetches the same empty page again, adding a redundant ListBuckets RPC to non-empty final pages in paginated ListBuckets and ListDirectoryBuckets requests.
Avoid probing the exhausted iterator again, while retaining the lookahead needed for full pages and the existing listing behavior.
AWS API references: [ListBuckets|https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListBuckets.html], [ListDirectoryBuckets|https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListDirectoryBuckets.html].
was:
RootEndpoint checks the bucket iterator again for a continuation token after the listing loop has already observed an empty page. OzoneVolume's bucket iterator does not retain the exhausted state, so this sends a duplicate ListBuckets RPC for non-empty final pages in paginated ListBuckets and ListDirectoryBuckets requests.
Check the response limit before calling hasNext() in the loop, and perform the post-loop lookahead only when the limit is reached. Keep that lookahead for full pages, since reaching the requested limit alone does not establish whether more buckets remain. This saves one ListBuckets RPC for affected requests while retaining the initial listing request, existing authorization, bucket filtering, and continuation-token construction.
Regression tests should use the real OzoneVolume iterator and verify:
* The terminal empty page is fetched only once for both underfilled and exact-limit final pages.
* Full pages with more results retain a continuation token, and continuation requests return the remaining results correctly.
* Empty lists, bucket results, and owner information retain their existing behavior.
* ListDirectoryBuckets retains its FSO filtering with mixed bucket layouts and its existing max-directory-buckets=0 behavior.
AWS API references:
* [ListBuckets|https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListBuckets.html]: max-buckets ranges from 1 to 10000; ContinuationToken indicates more buckets can be listed.
* [ListDirectoryBuckets|https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListDirectoryBuckets.html]: max-directory-buckets ranges from 0 to 1000; ContinuationToken is used for pagination.
> Avoid redundant ListBuckets RPCs when S3 bucket listing reaches the end
> -----------------------------------------------------------------------
>
> Key: HDDS-16673
> URL: https://issues.apache.org/jira/browse/HDDS-16673
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Huang Kuan Hao
> Assignee: Huang Kuan Hao
> Priority: Major
>
> RootEndpoint calls hasNext() again to determine whether a continuation token is needed after the listing loop has already reached the end. The bucket iterator fetches the same empty page again, adding a redundant ListBuckets RPC to non-empty final pages in paginated ListBuckets and ListDirectoryBuckets requests.
> Avoid probing the exhausted iterator again, while retaining the lookahead needed for full pages and the existing listing behavior.
> AWS API references: [ListBuckets|https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListBuckets.html], [ListDirectoryBuckets|https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListDirectoryBuckets.html].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org