You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "jdrouet (via GitHub)" <gi...@apache.org> on 2025/04/30 11:44:58 UTC
[PR] [WIP] implement multi range query in single request [arrow-rs-object-store]
jdrouet opened a new pull request, #345:
URL: https://github.com/apache/arrow-rs-object-store/pull/345
# Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
Closes #.
# Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
Right now, when requesting multiple ranges for a file, multiple queries are fired (if the gap between the ranges > `OBJECT_STORE_COALESCE_DEFAULT`). This could be avoided by using the [multi ranges capability](https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Range_requests#multipart_ranges) when doing fetching ranges through HTTP.
# What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
This PR introduces a new `get_manyranges` method to extract the multipart version of the response.
It also updates the `GetOptions` to add an type parameter so that `Range` can be `GetRange` or `GetManyRanges`.
# Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
`GetOptions` now has a parameter `Range` but has `GetRange` as a default type which should be transparent for the users.
<!---
If there are any breaking changes to public APIs, please call them out.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] implement multi range query in single request [arrow-rs-object-store]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #345:
URL: https://github.com/apache/arrow-rs-object-store/pull/345#issuecomment-2842916261
> If there are any providers that do support this, I'd expect for this to be an implementation detail of get_ranges instead of a separate method.
I know of azure supports this, but other major services are not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] implement multi range query in single request [arrow-rs-object-store]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #345:
URL: https://github.com/apache/arrow-rs-object-store/pull/345#issuecomment-2844151355
> You're saying that Azure _does_ support multiple byte ranges? This [SO answer](https://stackoverflow.com/a/57882772) says it doesn't, and this [relevant blob storage doc](https://learn.microsoft.com/en-us/rest/api/storageservices/specifying-the-range-header-for-blob-service-operations) doesn't mention it.
You are right. I double-checked the docs and confirmed that none of the major cloud storage services support this. I'm not sure why I had such a mistaken impression about this. Sorry for providing incorrect information.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] implement multi range query in single request [arrow-rs-object-store]
Posted by "kylebarron (via GitHub)" <gi...@apache.org>.
kylebarron commented on PR #345:
URL: https://github.com/apache/arrow-rs-object-store/pull/345#issuecomment-2843077560
You're saying that Azure _does_ support multiple byte ranges? This [SO answer](https://stackoverflow.com/a/57882772) says it doesn't, and this [relevant blob storage doc](https://learn.microsoft.com/en-us/rest/api/storageservices/specifying-the-range-header-for-blob-service-operations) doesn't mention it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] implement multi range query in single request [arrow-rs-object-store]
Posted by "kylebarron (via GitHub)" <gi...@apache.org>.
kylebarron commented on PR #345:
URL: https://github.com/apache/arrow-rs-object-store/pull/345#issuecomment-2841911366
Do any object storage providers support this? I know S3 explicitly does not:
<img width="797" alt="image" src="https://github.com/user-attachments/assets/b0688fec-0a2a-4e24-ae24-14d3f510d3ba" />
https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html#API_GetObject_RequestSyntax
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] implement multi range query in single request [arrow-rs-object-store]
Posted by "jdrouet (via GitHub)" <gi...@apache.org>.
jdrouet commented on PR #345:
URL: https://github.com/apache/arrow-rs-object-store/pull/345#issuecomment-2842026376
@kylebarron that's actually a good question, I've been around most of them and they only mention single ranges.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] implement multi range query in single request [arrow-rs-object-store]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on PR #345:
URL: https://github.com/apache/arrow-rs-object-store/pull/345#issuecomment-2846996244
Perhaps we could file an issue where we could discuss your use-case and then decide what the best course of action is?
IMO if there is a unobtrusive way to support this as an implementation detail of HttpStore::get_ranges I don't see a major issue with adding it, but I do agree with @Xuanwo that as currently written this PR is rather intrusive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] implement multi range query in single request [arrow-rs-object-store]
Posted by "jdrouet (via GitHub)" <gi...@apache.org>.
jdrouet closed pull request #345: [WIP] implement multi range query in single request
URL: https://github.com/apache/arrow-rs-object-store/pull/345
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] implement multi range query in single request [arrow-rs-object-store]
Posted by "kylebarron (via GitHub)" <gi...@apache.org>.
kylebarron commented on PR #345:
URL: https://github.com/apache/arrow-rs-object-store/pull/345#issuecomment-2842895278
If there are any providers that _do_ support this, I'd expect for this to be an implementation detail of `get_ranges` instead of a separate method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org