[GitHub] [arrow-rs] wjones127 commented on issue #3970: [object_store] Add option to start listing at a particular key

["wjones127 (via GitHub)" <gi...@apache.org>]

wjones127 commented on issue #3970:
URL: https://github.com/apache/arrow-rs/issues/3970#issuecomment-1487380859

   It looks like support isn't *that* wide:
   
   * [S3](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html#API_ListObjectsV2_RequestParameters): has `start-after` (exclusive?)
   * [GCS](https://cloud.google.com/storage/docs/json_api/v1/objects/list): has a `startOffset` (inclusive) 
   * [Azure Blob Store](https://learn.microsoft.com/en-us/rest/api/storageservices/list-blobs?tabs=azure-ad) not supported.
   * I don't see any obvious API for local filesystems.
   
   So this is mostly providing a useful optimization for S3 and GCS. There can be a default implementation that just throws out earlier entries. Also, for consistency between S3 and GCS, we would have to make the lower bound exclusive, since that seems to be the S3 behavior.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org