You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "clbarnes (via GitHub)" <gi...@apache.org> on 2023/08/02 09:22:40 UTC
[GitHub] [arrow-rs] clbarnes commented on issue #4612: object_store: multipart ranges for HTTP
clbarnes commented on issue #4612:
URL: https://github.com/apache/arrow-rs/issues/4612#issuecomment-1661852642
As part of the [zarr](https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html) project, we plan to store large tensors on a variety of backends (local/ HTTP/ object store), which are chunked into many separate files/ objects. As part of the [sharding](https://zarr-specs.readthedocs.io/en/latest/v3/codecs/sharding-indexed/v1.0.html) specification, each chunk (=shard) could contain many sub-chunks which are independently encoded and then concatenated. We'd want to read a footer to find the byte addresses of sub-chunks (see #4611 ), and then read (possibly multiple) byte ranges from the shard.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org