You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "lisasgoh (via GitHub)" <gi...@apache.org> on 2025/02/04 17:54:34 UTC
[I] Supporting object store FFI [arrow-rs]
lisasgoh opened a new issue, #7075:
URL: https://github.com/apache/arrow-rs/issues/7075
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
The use case here would be to be able to dynamically load libraries with custom object store implementations outside of s3/azure/gcp etc in libraries like polars in which there’s no way currently do register a new object store.
**Describe the solution you'd like**
An object store FFI would be necessary due to Rust’s unstable ABI.
**Describe alternatives you've considered**
<!--
A clear and concise description of any alternative solutions or features you've considered.
-->
**Additional context**
<!--
Add any other context or screenshots about the feature request here.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Supporting object store FFI [arrow-rs]
Posted by "lisasgoh (via GitHub)" <gi...@apache.org>.
lisasgoh commented on issue #7075:
URL: https://github.com/apache/arrow-rs/issues/7075#issuecomment-2637994901
Are you referring to something like SessionContext that Datafusion has? It would be nice but AFAIK, I don't think such a concept exists in Polars unfortunately. I did float the idea of using libloading to the Polars devs but the main concern/blocker that was raised was the lack of a stable C FFI for ObjectStore. I agree from a UX perspective it might be non-ideal, but given that a similar concept already exists in Polars (registering a Rust plugin via python), this wouldn't be unprecedented. Is there any chance for an ObjectStore C FFI to be on the roadmap?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Supporting object store FFI [arrow-rs]
Posted by "lisasgoh (via GitHub)" <gi...@apache.org>.
lisasgoh commented on issue #7075:
URL: https://github.com/apache/arrow-rs/issues/7075#issuecomment-2646962621
A potential temporary solution could be to create a fork of Polars to integrate my new object store, though it creates some maintenance overhead which isn't ideal. There seems to be some demand for ObjectStore FFI, e.g. [here](https://github.com/delta-io/delta-rs/issues/2818), so it would be immensely helpful if the Arrow team could consider adding this capability to their roadmap!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Supporting object store FFI [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #7075:
URL: https://github.com/apache/arrow-rs/issues/7075#issuecomment-2634952585
Tagging @kylebarron.
I think there are a couple of questions here:
* Is the major use-case python interop
* Does a python focused abstraction have more utliity than a generic FFI abstraction
* Is there maintainer bandwidth for this within arrow-rs, which is primarily rust focused
* Is the project better served incubating outside by a group of motivated individuals (who may not be arrow committers)
* Do the various python projects have sufficient commonality that sharing this is even useful (e.g. have they already built their own bespoke setup)
I think there are many people interested in such functionality, however, I suspect there is a significant non-technical aspect to any such initiative.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Supporting object store FFI [arrow-rs]
Posted by "kylebarron (via GitHub)" <gi...@apache.org>.
kylebarron commented on issue #7075:
URL: https://github.com/apache/arrow-rs/issues/7075#issuecomment-2634984317
It sounds like the OP here is interested in a custom implementation of `ObjectStore` to serve something other than AWS/GCP/Azure.
That's a bit different than what I've been focused on. The solution I've been working towards is to have [reusable Python bindings](https://github.com/developmentseed/obstore/tree/main/pyo3-object_store) that other Rust-Python developers can use in their own Python bindings. This doesn't require FFI (though by not using FFI it means you can't share `ObjectStore` instances across Python libraries). But it's also not a solution to a third-party extensible implementation of `ObjectStore`.
I don't think I have the bandwidth to try and implement stable object store FFI. I'll tag @timsaucer who wrote the DataFusion FFI support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Supporting object store FFI [arrow-rs]
Posted by "lisasgoh (via GitHub)" <gi...@apache.org>.
lisasgoh commented on issue #7075:
URL: https://github.com/apache/arrow-rs/issues/7075#issuecomment-2635059503
The use case here would be that unlike Datafusion that has a `register_object_store` method that allows for custom implementations for object stores, Polars doesn't (https://github.com/pola-rs/polars/issues/20568). I'm looking to contribute to polars to add support for it, but the main blocker is the lack of an existing ObjectStore FFI.
The idea I have is to have a Python API `register_object_store(path: str, method: str, scheme: str)` in Polars (similar to DF) that accepts the path to the library with the custom object store, a method name and a url scheme. This will call into Rust via PyO3 which would dynamically load that library via libloading and call the method name which would return an instance of the object store.
The main issue here is that there's no ObjectStore FFI at the moment AFAIK, which makes it tricky to implement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Supporting object store FFI [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #7075:
URL: https://github.com/apache/arrow-rs/issues/7075#issuecomment-2638073743
> Is there any chance for an ObjectStore C FFI to be on the roadmap?
I will be frank, ObjectStore has a fairly large, async API, and so defining and maintaining an FFI interface for it would be a fairly substantial undertaking. Given what I know of the interests of the various maintainers, I think such an initiative would likely stand the best chance of success incubating as a third-party project.
> I did float the idea of using libloading to the Polars devs but the main concern/blocker that was raised was the lack of a stable C FFI for ObjectStore
IMO I would suggest adding support for custom ObjectStore within polars from the Rust API first, i.e. introducing something similar to [ObjectStoreRegistry](https://docs.rs/datafusion/latest/datafusion/datasource/object_store/trait.ObjectStoreRegistry.html). Once such an abstraction exists, then it will be possible to devise ways to potentially orchestrate that from python code.
This wouldn't necessarily require a C FFI, for example, you could potentially build a shared library bundling both polars and your extension code, and then use that from python or something along those lines.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Supporting object store FFI [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #7075:
URL: https://github.com/apache/arrow-rs/issues/7075#issuecomment-2645188605
If this is indeed the case, then polars will indeed require a stable C FFI for all such extensions. My point was that this is a very limiting methodology, and it seems unfortunate that polars would not support build-time extension in addition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Supporting object store FFI [arrow-rs]
Posted by "lisasgoh (via GitHub)" <gi...@apache.org>.
lisasgoh commented on issue #7075:
URL: https://github.com/apache/arrow-rs/issues/7075#issuecomment-2637921175
@tustvold The Rust API to directly provide an object store makes sense, but I'll be interacting with polars via python so I was wondering what's the best way to register a new object store in python. I might be misunderstanding, but regarding your idea for python, does it involve creating some sort of generic ObjectStore python impl that wraps the user provided python impl and invokes its methods? Something like:
`Python::with_gil(|py| {
let _: PyResult<()> = obj.call_method1("put", (location, bytes));
})`
I am worried that for this approach, doesn't this mean that the interactions with the object store will be -> rust -> python -> rust? This would have some overheard and I'm not sure if there would be performance implications due to the GIL. It seems like libloading is already used in polars for plugins https://docs.pola.rs/api/python/stable/reference/plugins.html, which is where I got the idea from.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Supporting object store FFI [arrow-rs]
Posted by "lisasgoh (via GitHub)" <gi...@apache.org>.
lisasgoh commented on issue #7075:
URL: https://github.com/apache/arrow-rs/issues/7075#issuecomment-2638781547
Just to make sure I understand correctly, is the idea here along the lines of:
1. Create a register API in Rust to store the custom object store in memory.
2. Maybe create a Rust function in the external library with the custom impl that would invoke that register API
3. Expose that function to python and call it at runtime before executing any query via python polars?
Or am I getting it wrong?
I’m not sure this would work, since wouldn’t the Rust binary for the Rust API be different from the underlying Rust binary for python polars which is separately compiled and published? So the state in Rust polars with the registered object store won’t be shared with python polars.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Supporting object store FFI [arrow-rs]
Posted by "kylebarron (via GitHub)" <gi...@apache.org>.
kylebarron commented on issue #7075:
URL: https://github.com/apache/arrow-rs/issues/7075#issuecomment-2640152758
I think the confusion here might be that @tustvold is expecting Polars to act like DataFusion, where it's intended to be fully embedded into your own project. Whereas my understanding is that the Polars Python API isn't designed to be embedded. The intended extension point is via runtime-linked extensions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Supporting object store FFI [arrow-rs]
Posted by "lisasgoh (via GitHub)" <gi...@apache.org>.
lisasgoh commented on issue #7075:
URL: https://github.com/apache/arrow-rs/issues/7075#issuecomment-2638243977
> I will be frank, ObjectStore has a fairly large, async API, and so defining and maintaining an FFI interface for it would be a fairly substantial undertaking. Given what I know of the interests of the various maintainers, I think such an initiative would likely stand the best chance of success incubating as a third-party project.
Ah that's unfortunate :( There's only https://github.com/RelationalAI/object_store_ffi I think.
> This wouldn't necessarily require a C FFI, for example, you could potentially build a shared library bundling both polars and your extension code, and then use that from python or something along those lines.
Do you mind elaborating on this approach? I was thinking that a Rust API for this might not really be helpful since after registering a custom object store into the registry (which maybe can be represented as some sort of global map), how will python polars be aware of this state?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Supporting object store FFI [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #7075:
URL: https://github.com/apache/arrow-rs/issues/7075#issuecomment-2651558167
Having thought about this a bit more, I think the request is not actually for a stable C FFI for ObjectStore itself, i.e. a C API that can be used to call any Rust implementation of the ObjectStore trait, but rather a Rust ObjectStore implementation that calls through to a C FFI interface to perform its operations.
This has some key implications:
* The C FFI can be partial / incomplete / lag the full ObjectStore trait - e.g. it could just support basic range requests
* The C FFI could be a simple callback based API, as opposed to needing to shim streams/futures/wakers over the FFI boundary
* Can support multiple ABI-incompatible interfaces simultaneously allowing for evolution (provided the ObjectStoreShim is told which one)
I think this is probably tractable, although definitely not easy, if someone wanted to do work on this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Supporting object store FFI [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #7075:
URL: https://github.com/apache/arrow-rs/issues/7075#issuecomment-2638277394
> how will python polars be aware of this state?
My understanding is that the polars python API really just acts as glue to orchestrate the underlying Rust execution engine. As such it should be possible to do the initial setup in Rust and then do further orchestration from Python.
Ultimately I'd be very surprised if adding support for this from Rust wasn't a necessary precondition to python support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Supporting object store FFI [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #7075:
URL: https://github.com/apache/arrow-rs/issues/7075#issuecomment-2639234236
> separately compiled and published
You wouldn't use the standard polars distribution, just the custom one with your extension built into it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Supporting object store FFI [arrow-rs]
Posted by "santosh-d3vpl3x (via GitHub)" <gi...@apache.org>.
santosh-d3vpl3x commented on issue #7075:
URL: https://github.com/apache/arrow-rs/issues/7075#issuecomment-2650823868
> A potential temporary solution could be to create a fork of Polars to integrate my new object store, though it creates some maintenance overhead which isn't ideal.
Have a look at: https://github.com/pola-rs/polars/pull/17506. This was my attempt to add HDFS support for polars. I thought of maintaining a fork and decided against it. It is a clunky solution to a systemic problem.
1. It is fair that polars doesn't want to add dependency on all the object stores out there.
2. Python is still the go to language for most of the data processing frameworks.
How do other frameworks add support for more object stores? Any ideas? Anything reusable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Supporting object store FFI [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #7075:
URL: https://github.com/apache/arrow-rs/issues/7075#issuecomment-2637950783
> I am worried that for this approach, doesn't this mean that the interactions with the object store will be -> rust -> python -> rust?
I was somewhat presuming that if you're using the python API you want to author your extension in python. I agree if the implementation is in Rust, proxying via python seems unnecessary, even if the performance impact is likely irrelevant when compared to network overheads. I'm not familiar with how polars has hooked up its python bindings, but I wonder if you can setup a polars "context" in Rust and then invoke it from python?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Supporting object store FFI [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #7075:
URL: https://github.com/apache/arrow-rs/issues/7075#issuecomment-2637845734
IMO polars has both a python and Rust API, and IMO trying to support both with one system using mechanisms like libloading is a bad UX for both.
Instead I'd suggest Rust code should be able to provide an ObjectStore directly.
Similarly for python, polars could provide an object store shim that delegates to a user provided python impl (this may even already exist). This would allow people to plugin python based object stores.
If libloading is important for other reasons, e.g. some GPL dance, then these Rust/python impls can orchestrate that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org