You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-dev@hadoop.apache.org by "Steve Loughran (Jira)" <ji...@apache.org> on 2021/10/11 15:28:00 UTC

[jira] [Created] (HADOOP-17961) s3 and abfs incremental listing: ask for a smaller first batch

Steve Loughran created HADOOP-17961:
---------------------------------------

             Summary: s3 and abfs incremental listing: ask for a smaller first batch
                 Key: HADOOP-17961
                 URL: https://issues.apache.org/jira/browse/HADOOP-17961
             Project: Hadoop Common
          Issue Type: Sub-task
          Components: fs/azure, fs/s3
    Affects Versions: 3.3.2
            Reporter: Steve Loughran


With code gradually adopting listStatusIncremental(), asking for a smaller initial batch could permit faster ramp of result processing.

probably most significant on an s3 versioned bucket, as there the need to skip tombstones can result in significantly slower listings -but could benefit ABFS too



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org