You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "drwelby (via GitHub)" <gi...@apache.org> on 2025/01/16 20:40:04 UTC
[I] Support query parameters in HTTP objects [arrow-rs]
drwelby opened a new issue, #6989:
URL: https://github.com/apache/arrow-rs/issues/6989
### Is your feature request related to a problem or challenge? Please describe what you are trying to do.
We are trying to access objects with query parameters, which can include things like formatting options for the object, or authentication keys.
An imaginary example would look like:
`https://cats.server/kitten.jpg?color=black&api_key=1234` to get a picture of a black kitten.
At present, it appears the URI is parsed and the parameters are stripped.
-----
### Describe the solution you'd like
A basic solution would be to allow the URI parameters to pass through.
A more flexible implementation would allow for passing parameters as an argument.
-----
### Describe alternatives you've considered
I could not find any alternative, but have considered that objects described by parameters may be anti-pattern or out of scope.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support query parameters in HTTP objects [arrow-rs]
Posted by "drwelby (via GitHub)" <gi...@apache.org>.
drwelby commented on issue #6989:
URL: https://github.com/apache/arrow-rs/issues/6989#issuecomment-2599798779
The use case stems from @kylebarron's development of using `object_store` to achieve high-throughput, high-concurrency reading of geospatial images using a Rust-based core: https://github.com/developmentseed/aiocogeo-rs
The structure of these files is especially suited for http access from cloud storage from S3 so `object_store` fits the base use case very well.
However, in my case we have an API that _emulates_ a file in cloud storage. It behaves like an image file stored in S3, but the image is processed on the fly as the file is accessed.
At it's core, there is a single "object" - the raw sensor data of the image, but the processing options to create a final image are passed as parameters.
So conceptually it's not a great fit for an object store abstraction, but if it's not problematic to allow parameters to slip through, we could use this library, understanding it's "out of spec". Allowing parameters for the state goal of authorization feels like a harmless ask.
I have also considered that we instead pack the parameters into a URL structure to make the API behave more RESTfully. Perhaps that's a better direction in general for these times but right now we have parameters.
If it's simply not a good fit, that's OK, but I thought I'd ask anyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support query parameters in HTTP objects [arrow-rs]
Posted by "kylebarron (via GitHub)" <gi...@apache.org>.
kylebarron commented on issue #6989:
URL: https://github.com/apache/arrow-rs/issues/6989#issuecomment-2596990084
I think there's a difference between query parameters for authentication can potentially be in line with the existing ObjectStore APIs, but using query parameters to find specific objects would be a departure from what the S3 API provides.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support query parameters in HTTP objects [arrow-rs]
Posted by "drwelby (via GitHub)" <gi...@apache.org>.
drwelby commented on issue #6989:
URL: https://github.com/apache/arrow-rs/issues/6989#issuecomment-2596847712
cc @kylebarron
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support query parameters in HTTP objects [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #6989:
URL: https://github.com/apache/arrow-rs/issues/6989#issuecomment-2599641525
Could you perhaps expand upon the use-case for this. Why do you want to access an HTTP API as an ObjectStore? Are the query parameters static? Does the server support listing or any APIs other than basic retrieval?
On the face of it this seems like a bit of an unusual ask, and might suggest a missing abstraction somewhere - e.g. using ObjectStore as opposed to a more targeted trait
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org