You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/07/26 21:58:45 UTC
[GitHub] [arrow-rs] alamb opened a new issue, #2176: object_store: Move Away From SDKs
alamb opened a new issue, #2176:
URL: https://github.com/apache/arrow-rs/issues/2176
ported from https://github.com/influxdata/object_store_rs/issues/18 by @tustvold
Note there are several good comments on that ticket by @tustvold and @wjones127 that I did not copy over
**Problem**
Currently this crate uses upstream crates to provide interaction with object storage. This comes with a few downsides:
* Missing features, e.g. range support, conditionals, etc...
* Inconsistent error handling, e.g. it can be next to impossible to get the HTTP status code
* Extreme dependency bloat
**Proposal**
This crate does not intend to cover more than the basic APIs of each store, which boils down to just a couple of different request types. I would like therefore like to propose:
* Move to using a reqwest client directly
* Use serde to serialize payloads
* Use ring to handle signatures/etc...
* Use rustls to handle TLS
This will allow:
* Consistent error handling, retries, etc...
* Smaller dependency footprint
* New features without waiting on upstreams
* Simpler codebase
It's up to whoever picks this up where to start, but I would suggest starting with GCS as:
* It will unlock range requests
* The authentication logic is simpler than say AWS
* The JSON API is relatively straightforward
If that goes well, we can then look to move onto the others
**Additional Context**
https://github.com/influxdata/object_store_rs/pull/15#issuecomment-1134893330
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold closed issue #2176: object_store: Move Away From SDKs
Posted by GitBox <gi...@apache.org>.
tustvold closed issue #2176: object_store: Move Away From SDKs
URL: https://github.com/apache/arrow-rs/issues/2176
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org