You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "nathschmidt (via GitHub)" <gi...@apache.org> on 2025/03/23 19:25:17 UTC
[PR] Allow setting ClientOptions for all `datafusion.object_store` contexts [datafusion-python]
nathschmidt opened a new pull request, #1083:
URL: https://github.com/apache/datafusion-python/pull/1083
# Which issue does this PR close?
Closes #1082
# Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
This change adds a `PyClientOptions` wrapper around `object_store.ClientOptions` so that it can be instantiated and then used from Python.
# What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
Introduces a new `PyClientOptions` struct and necessary wiring to use it.
# Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
New optional user-facing arguments for setting `ClientOptions`.
Given this is my first change - and I don't usually work in Rust - looking to get some confirmation on approach and acceptability of this PR.
Outstanding TODO's:
- [ ] Get some confirmation on overall approach
- [ ] Add necessary wiring for all contexts
- [ ] Add some simple tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [Draft] Allow setting ClientOptions for all `datafusion.object_store` contexts [datafusion-python]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on PR #1083:
URL: https://github.com/apache/datafusion-python/pull/1083#issuecomment-2970831051
https://github.com/apache/datafusion-python/pull/1143/ updates us to DF48 which uses object store 0.12 so I think we may be unblocked on moving to pyo3-object_store!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow setting ClientOptions for all `datafusion.object_store` contexts [datafusion-python]
Posted by "nathschmidt (via GitHub)" <gi...@apache.org>.
nathschmidt commented on code in PR #1083:
URL: https://github.com/apache/datafusion-python/pull/1083#discussion_r2009206331
##########
src/store.rs:
##########
@@ -164,6 +166,41 @@ impl PyGoogleCloudContext {
}
}
+#[pyclass(name = "ClientOptions", module = "datafusion.store", subclass)]
+#[derive(Debug, Clone)]
+pub struct PyClientOptions {
+ pub inner: ClientOptions,
Review Comment:
Main question I have is does this need to be `Arc<ClientOptions>` rather than just `ClientOptions`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [Draft] Allow setting ClientOptions for all `datafusion.object_store` contexts [datafusion-python]
Posted by "kylebarron (via GitHub)" <gi...@apache.org>.
kylebarron commented on PR #1083:
URL: https://github.com/apache/datafusion-python/pull/1083#issuecomment-2746392406
This is somewhat related to https://github.com/apache/datafusion-python/issues/899.
Up for discussion with maintainers but I would personally argue for adopting my project https://docs.rs/pyo3-object_store, which exposes all possible client options automatically, and would give less of a maintenance burden here. You can see the exported client API, which matches [these python docs ](https://developmentseed.org/obstore/latest/api/store/).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [Draft] Allow setting ClientOptions for all `datafusion.object_store` contexts [datafusion-python]
Posted by "kylebarron (via GitHub)" <gi...@apache.org>.
kylebarron commented on PR #1083:
URL: https://github.com/apache/datafusion-python/pull/1083#issuecomment-2749582101
I think the current exports are `datafusion.object_store`? So we could deprecate that and use `datafusion.store` instead?
I've been thinking perhaps it makes sense to cut an 0.3 release of `pyo3-object_store` that uses `object_store` **0.11**, so that projects (including my own) can start integrating `pyo3-object_store`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [Draft] Allow setting ClientOptions for all `datafusion.object_store` contexts [datafusion-python]
Posted by "nathschmidt (via GitHub)" <gi...@apache.org>.
nathschmidt commented on PR #1083:
URL: https://github.com/apache/datafusion-python/pull/1083#issuecomment-2749577539
@kylebarron - this seems ideal to me. I'm happy to have a go at it - potentially as a separate API to ease any compatibility woes (i.e. deprecate `datafusion.store`, replace with `datafusion.obstore` or something - though tbh given datafusion-python is mostly major releases it should be an acceptable API change so long as there are updated examples).
While we're still on version 0.0.11 of object_store I can try and put together a demo branch with some cargo hackery. Unsure if there's a timeline on getting an official datafusion release depending on 0.0.12 out yet that'd obviously be a blocker to actually merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [Draft] Allow setting ClientOptions for all `datafusion.object_store` contexts [datafusion-python]
Posted by "kylebarron (via GitHub)" <gi...@apache.org>.
kylebarron commented on PR #1083:
URL: https://github.com/apache/datafusion-python/pull/1083#issuecomment-2970835101
Oh, I published versions of `pyo3-object_store` for compatibility with `object_store` 0.11, so that wasn't actually a blocker 😅
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [Draft] Allow setting ClientOptions for all `datafusion.object_store` contexts [datafusion-python]
Posted by "kylebarron (via GitHub)" <gi...@apache.org>.
kylebarron commented on PR #1083:
URL: https://github.com/apache/datafusion-python/pull/1083#issuecomment-2749601811
I published new versions of pyo3-object_store for pyo3 0.23 and object_store 0.11: https://github.com/developmentseed/obstore/tree/main/pyo3-object_store#version-compatibility
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org