You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@opendal.apache.org by "chitralverma (via GitHub)" <gi...@apache.org> on 2025/04/06 16:28:50 UTC
(rust) How to use opendal to load data into DuckDB or Polars LazyFrame?
GitHub user chitralverma created a discussion: (rust) How to use opendal to load data into DuckDB or Polars LazyFrame?
I’m trying to load data from a remote storage backend (e.g., S3, local FS) using opendal, and I want to pass that data into either:
- a DuckDB table, or
- a Polars LazyFrame for further processing.
My current idea is to read the data into memory using op.read() and then wrap it with Cursor, but I’m unsure if this is the best or most idiomatic way to integrate with these libraries, especially for streaming large datasets.
Has anyone successfully done this? Are there preferred patterns or best practices for interoperability between opendal and DuckDB or Polars?
Thanks!
GitHub link: https://github.com/apache/opendal/discussions/5972
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: (rust) How to use opendal to load data into DuckDB or Polars LazyFrame?
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
GitHub user Xuanwo edited a comment on the discussion: (rust) How to use opendal to load data into DuckDB or Polars LazyFrame?
For Polars, one possible approach is to use `object_store_opendal`, which allows using OpenDAL as an implementation of `ObjectStore`. However, the Polars Rust API is relatively closed and doesn't offer a way to inject custom implementations. For instance, their [`ParquetAsyncReader`](https://docs.rs/polars-io/0.46.0/polars_io/parquet/read/struct.ParquetAsyncReader.html) only supports the [`from_uri`](https://docs.rs/polars-io/0.46.0/polars_io/parquet/read/struct.ParquetAsyncReader.html#method.from_uri) method.
cc @reswqa for more help, can we use opendal in polars directly?
---
As for DuckDB, I haven't examined it closely, but from what I remember, there isn't a straightforward way to interact with the DuckDB table class from Rust. Please correct me if that has changed.
GitHub link: https://github.com/apache/opendal/discussions/5972#discussioncomment-12745897
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: (rust) How to use opendal to load data into DuckDB or Polars LazyFrame?
Posted by "chitralverma (via GitHub)" <gi...@apache.org>.
GitHub user chitralverma edited a comment on the discussion: (rust) How to use opendal to load data into DuckDB or Polars LazyFrame?
@Xuanwo I had one idea to make this work for all 3 - pyarrow, polars and duckdb.
Instead of doing things on rust/ c side in each project, I saw that the python bindings all all 3 projects support fsspec based file systems - [pyarrow](https://arrow.apache.org/docs/python/filesystems.html#filesystem-fsspec), [duckdb](https://duckdb.org/docs/stable/guides/python/filesystems) and [polars](https://docs.pola.rs/user-guide/io/cloud-storage/#scanning-with-pyarrow)
So if the opendal team sees it as a good fit, we can provide an opendal backed implementation of the [fsspec.AsyncFileSystem](https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/asyn.html#AsyncFileSystem) + maybe [pyarrow.fs.FSSpecHandler](https://arrow.apache.org/docs/python/generated/pyarrow.fs.FSSpecHandler.html#pyarrow.fs.FSSpecHandler), directly in the python bindings for end users to use (maybe behind an extra/ group??).
Beyond polars and duckdb, also this will help users to use opendal backends through these interfaces.
Someone is also partially maintaining this in a separate [repo](https://github.com/fsspec/opendalfs), might as well do it officially. wdyt?
GitHub link: https://github.com/apache/opendal/discussions/5972#discussioncomment-13062336
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: (rust) How to use opendal to load data into DuckDB or Polars LazyFrame?
Posted by "chitralverma (via GitHub)" <gi...@apache.org>.
GitHub user chitralverma added a comment to the discussion: (rust) How to use opendal to load data into DuckDB or Polars LazyFrame?
these are examples in python, i was wondering if theres a solution in rust directly
GitHub link: https://github.com/apache/opendal/discussions/5972#discussioncomment-12745702
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: (rust) How to use opendal to load data into DuckDB or Polars LazyFrame?
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
GitHub user Xuanwo added a comment to the discussion: (rust) How to use opendal to load data into DuckDB or Polars LazyFrame?
Hi, we have https://github.com/apache/opendal/blob/main/bindings/python/examples/polars.ipynb and https://github.com/apache/opendal/blob/main/bindings/python/examples/pandas.ipynb as basic examples. Would these be helpful to you?
GitHub link: https://github.com/apache/opendal/discussions/5972#discussioncomment-12744760
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: (rust) How to use opendal to load data into DuckDB or Polars LazyFrame?
Posted by "chitralverma (via GitHub)" <gi...@apache.org>.
GitHub user chitralverma added a comment to the discussion: (rust) How to use opendal to load data into DuckDB or Polars LazyFrame?
awesome, that's what i thought as well.
but my suggestion was to ship this as a part of the python bindings themselves, i assume this will reduce a lot of package management, version compatibility, backend service support and release stuff as well if its in the same repo
GitHub link: https://github.com/apache/opendal/discussions/5972#discussioncomment-13063127
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: (rust) How to use opendal to load data into DuckDB or Polars LazyFrame?
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
GitHub user Xuanwo added a comment to the discussion: (rust) How to use opendal to load data into DuckDB or Polars LazyFrame?
Hi @chitralverma, I started https://github.com/fsspec/opendalfs to explore whether it would be better to build an fsspec implementation directly in Rust. However, I've realized that it duplicates much of the work we've already done on the Python side. It might be better to use the OpenDAL Python binding directly in opendalfs. I'll start a discussion about this in opendalfs instead.
For `pyarrow.fs.FSSpecHandler`, maybe we can build a new integration like we do for other crates.
GitHub link: https://github.com/apache/opendal/discussions/5972#discussioncomment-13063020
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: (rust) How to use opendal to load data into DuckDB or Polars LazyFrame?
Posted by "chitralverma (via GitHub)" <gi...@apache.org>.
GitHub user chitralverma edited a comment on the discussion: (rust) How to use opendal to load data into DuckDB or Polars LazyFrame?
@Xuanwo I had one idea to make this work for all 3 - pyarrow, polars and duckdb.
Instead of doing things on rust/ c side in each project, I saw that the python bindings all all 3 projects support fsspec based file systems - [pyarrow](https://arrow.apache.org/docs/python/filesystems.html#filesystem-fsspec), [duckdb](https://duckdb.org/docs/stable/guides/python/filesystems) and [polars](https://docs.pola.rs/user-guide/io/cloud-storage/#scanning-with-pyarrow)
So if the opendal team sees it as a good fit, we can provide an opendal backed implementation of the [fsspec.AsyncFileSystem](https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/asyn.html#AsyncFileSystem) + maybe [pyarrow.fs.FSSpecHandler](https://arrow.apache.org/docs/python/generated/pyarrow.fs.FSSpecHandler.html#pyarrow.fs.FSSpecHandler), directly in the python bindings for end users to use (maybe behind an extra/ group??).
Beyond polars and duckdb, also this will help users to use opendal backends through these interfaces.
Someone is also partially maintaining this in a separate [repo](https://github.com/fsspec/opendalfs), might as well do it officially wdyt?
GitHub link: https://github.com/apache/opendal/discussions/5972#discussioncomment-13062336
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: (rust) How to use opendal to load data into DuckDB or Polars LazyFrame?
Posted by "chitralverma (via GitHub)" <gi...@apache.org>.
GitHub user chitralverma added a comment to the discussion: (rust) How to use opendal to load data into DuckDB or Polars LazyFrame?
@Xuanwo @reswqa please see one of the comments from a collaborator on the issue above, might need your support as im not very familiar with the internals of opendal
> That is unfortunately not possible. Our internal API is based on memory maps, this does allow non-eager reading. AsyncRead and Streams are very hard for us to support efficiently, since many file formats rely on seeking across the file or would require significant buffering anyway.
GitHub link: https://github.com/apache/opendal/discussions/5972#discussioncomment-12764885
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: (rust) How to use opendal to load data into DuckDB or Polars LazyFrame?
Posted by "chitralverma (via GitHub)" <gi...@apache.org>.
GitHub user chitralverma added a comment to the discussion: (rust) How to use opendal to load data into DuckDB or Polars LazyFrame?
Yes for Polars I've raised https://github.com/pola-rs/polars/issues/22154, hopefully this can be seen as an enhancement.
GitHub link: https://github.com/apache/opendal/discussions/5972#discussioncomment-12745998
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: (rust) How to use opendal to load data into DuckDB or Polars LazyFrame?
Posted by "chitralverma (via GitHub)" <gi...@apache.org>.
GitHub user chitralverma added a comment to the discussion: (rust) How to use opendal to load data into DuckDB or Polars LazyFrame?
@Xuanwo I had one idea to make this work for all 3 - pyarrow, polars and duckdb.
Instead of doing things on rust/ c side in each project, I saw that the python bindings all all 3 projects support fsspec based file systems - [pyarrow](https://arrow.apache.org/docs/python/filesystems.html#filesystem-fsspec), [duckdb](https://duckdb.org/docs/stable/guides/python/filesystems) and [polars](https://docs.pola.rs/user-guide/io/cloud-storage/#scanning-with-pyarrow)
So if the opendal team sees it as a good fit, we can provide an opendal backed implementation of the [fsspec.AsyncFileSystem](https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/asyn.html#AsyncFileSystem) + maybe [pyarrow.fs.FSSpecHandler](https://arrow.apache.org/docs/python/generated/pyarrow.fs.FSSpecHandler.html#pyarrow.fs.FSSpecHandler), directly in the python bindings for end users to use (maybe behind an extra/ group??).
Beyond polars and duckdb, also this will help users to use opendal backends through these interfaces.
wdyt?
GitHub link: https://github.com/apache/opendal/discussions/5972#discussioncomment-13062336
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: (rust) How to use opendal to load data into DuckDB or Polars LazyFrame?
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
GitHub user Xuanwo added a comment to the discussion: (rust) How to use opendal to load data into DuckDB or Polars LazyFrame?
For Polars, one possible approach is to use `object_store_opendal`, which allows using OpenDAL as an implementation of `ObjectStore`. However, the Polars Rust API is relatively closed and doesn't offer a way to inject custom implementations. For instance, their [`ParquetAsyncReader`](https://docs.rs/polars-io/0.46.0/polars_io/parquet/read/struct.ParquetAsyncReader.html) only supports the [`from_uri`](https://docs.rs/polars-io/0.46.0/polars_io/parquet/read/struct.ParquetAsyncReader.html#method.from_uri) method.
cc @reswqa for more help, can we use opendal in polars direcrly?
---
As for DuckDB, I haven't examined it closely, but from what I remember, there isn't a straightforward way to interact with the DuckDB table class from Rust. Please correct me if that has changed.
GitHub link: https://github.com/apache/opendal/discussions/5972#discussioncomment-12745897
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org