You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "timsaucer (via GitHub)" <gi...@apache.org> on 2026/09/15 13:29:23 UTC
[I] Allow extension libraries to contribute object stores over FFI [datafusion-python]
timsaucer opened a new issue, #1737:
URL: https://github.com/apache/datafusion-python/issues/1737
# Is your feature request related to a problem or challenge?
Split out of #1676, which proposed adding an `object_stores` field to `SessionExtensionComponents` so an extension library could declare its object stores alongside its codecs, functions, and providers and have `with_extensions` install everything in one call. The other fields in that issue are actionable and are being implemented; this one is blocked, so it is being tracked separately rather than holding up the rest.
An extension library that reads from a storage system datafusion-python does not know about has no way to contribute an object store. Every other extension point in the FFI surface — table providers, catalog providers, functions, codecs, query planners, physical optimizer rules — has a `__datafusion_*__` capsule getter that lets a separately compiled library hand over an implementation. Object stores have none.
# Describe the solution you'd like
Ultimately, a `__datafusion_object_store__` capsule getter following the same convention as the rest of the protocol, so a library can export an `ObjectStore` implementation across the FFI boundary, plus an `object_stores` field on `SessionExtensionComponents` keyed by scheme.
**This is blocked upstream and cannot be built here first.** There is no `FFI_ObjectStore` in `datafusion-ffi` — no object-store module exists in the crate at all. Without one there is nothing for a capsule to carry.
The Python-side surface is also closed today. `SessionContext.register_object_store` takes `StorageContexts`, a closed enum over five built-in pyclasses (`AmazonS3`, `GoogleCloudStorage`, `MicrosoftAzure`, `LocalFileSystem`, `HTTP`), and `crates/core/src/context.rs` matches all five exhaustively to pull out an `Arc<dyn ObjectStore>`. There is no `RustWrappedPyObjectStore` equivalent to the wrappers the other extension points have. So even a Python-native path would need new work, and a third-party Rust cdylib could not participate at all — it would have to import `datafusion.object_store` and call back into the host to construct one of our own objects.
The dependency order is therefore: an `FFI_ObjectStore` upstream in `datafusion-ffi`, then an importer and `__datafusion_object_store__` hook here, then optionally the `object_stores` bundle field.
# Describe alternatives you've considered
**What works today.** A library that wants to ship a configured store returns one of datafusion-python's own objects and lets the caller register it:
```python
ctx.register_object_store("s3://my-bucket", my_library.configured_s3_store())
```
That is one line and needs no protocol. It covers the case where the library is packaging credentials or endpoint configuration for a store datafusion-python already supports, which is probably the common case.
What it does not cover is a library implementing a genuinely new `ObjectStore` — an internal blob service, a content-addressed store, a caching layer in front of another store. That case needs the FFI type and has no workaround short of the library vendoring its own DataFusion.
**Adding `object_stores` to `SessionExtensionComponents` anyway**, carrying the five existing `StorageContexts` variants. Rejected in #1676: it would be the only field in that dataclass carrying nothing foreign, there would be no capsule to validate, and it would bake the `format!("{scheme}{derived_host}")` key construction into a tuple shape for no benefit over the one-line call above.
# Additional context
Follow-up to #1676; see https://github.com/apache/datafusion-python/issues/1676#issuecomment-5680959562 for the analysis this was split out of. Blocked on an upstream `datafusion-ffi` change — worth raising in apache/datafusion before any work starts here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org