You are viewing a plain text version of this content. The canonical link for it is here.

Posted to github@datafusion.apache.org by "kylebarron (via GitHub)" <gi...@apache.org> on 2025/09/03 17:16:14 UTC

[I] Remove pyarrow as required dependency, relying on Arrow PyCapsule Interface [datafusion-python]

kylebarron opened a new issue, #1227:
URL: https://github.com/apache/datafusion-python/issues/1227

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

PyArrow is a massive dependency. Unpacked, it tends to be >100MB in size, and, until the latest versions (I think?) also required numpy as its own non-optional dependency.

It's also, in effect the only current dependency
https://github.com/apache/datafusion-python/blob/f0bbad7543717c5f08ba2acb92d42c9d30fd2355/pyproject.toml#L46

It would be great if we could remove it, and that would greatly lessen the minimal environment size for datafusion python.

[Many other Python Arrow libraries](https://github.com/apache/arrow/issues/39195#issuecomment-2245718008) implement the PyCapsule Interface, so the user can use nanoarrow, arro3, Polars, DuckDB, etc, or pyarrow. Whatever is best for them.

**Describe the solution you'd like**

The Arrow PyCapsule Interface is a lightweight, decentralized protocol for sharing Arrow data between Python libraries. We already implement the PyCapsule Interface, so it's just a matter of removing places where we hard-code use of pyarrow.

**Describe alternatives you've considered**

Keep pyarrow dependency.

**Additional context**

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] Remove pyarrow as required dependency, relying on Arrow PyCapsule Interface [datafusion-python]

Posted by "kylebarron (via GitHub)" <gi...@apache.org>.

kylebarron commented on issue #1227:
URL: https://github.com/apache/datafusion-python/issues/1227#issuecomment-3339863263

   You can turn off the default features of pyo3-arrow


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] Remove pyarrow as required dependency, relying on Arrow PyCapsule Interface [datafusion-python]

Posted by "ion-elgreco (via GitHub)" <gi...@apache.org>.

ion-elgreco commented on issue #1227:
URL: https://github.com/apache/datafusion-python/issues/1227#issuecomment-3303320257

   I agree with removing pyarrow as a dependency. We have to consider that there are users out there that want to create small containers but do data operations in them, if you pull in pyarrow which is so heavyweight that complicates it a lot. 
   
   In delta-rs I've already done this, https://github.com/delta-io/delta-rs/pull/3459 some time ago, which was very easy due to Arro3. (Thanks @kylebarron  :D)
   
   As I am thinking about it, @kylebarron a pointer might make more sense since it's more to be interopable between other libs, if you want to analyze the data it can be done directly in datafusion. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] Remove pyarrow as required dependency, relying on Arrow PyCapsule Interface [datafusion-python]

Posted by "kylebarron (via GitHub)" <gi...@apache.org>.

kylebarron commented on issue #1227:
URL: https://github.com/apache/datafusion-python/issues/1227#issuecomment-3303389690

   We already have implementations of the PyCapsule interface, and we're improving it in https://github.com/apache/datafusion-python/pull/1222 (thanks to @kosiew ). It's just methods like `to_batches` that are just syntactic sugar for calling `pa.table(datafusion_df)`. What should we do about them?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] Remove pyarrow as required dependency, relying on Arrow PyCapsule Interface [datafusion-python]

Posted by "kylebarron (via GitHub)" <gi...@apache.org>.

kylebarron commented on issue #1227:
URL: https://github.com/apache/datafusion-python/issues/1227#issuecomment-3332072263

   I think it'll be easiest to focus on input parameters first. You can either use the `arrow-rs` `pyarrow` feature flag or use my crate [`pyo3-arrow`](https://crates.io/crates/pyo3-arrow). I'm partial to `pyo3-arrow` because of [these issues](https://docs.rs/pyo3-arrow/latest/pyo3_arrow/#why-not-use-arrow-rss-python-integration). So, for example, if we ever have a place where we need a _column input_ then we need to use `pyo3-arrow` (`arrow` doesn't have a way to import a column).
   
   > ### Input Parameters
   > **SessionContext methods:**
   > 
   > * `from_arrow_table(data: pa.Table)`
   
   This can be updated to import a table via the PyCapsule Interface, without any breaking change. On the Python side the type hint can be updated to 	
   ```py
   from typing import Tuple, Protocol
   
   class ArrowStreamExportable(Protocol):
       def __arrow_c_stream__(
           self,
           requested_schema: object | None = None
       ) -> object:
           ...
   ```
   
   > * `create_dataframe(partitions: list[list[pa.RecordBatch]])`
   
   Similarly, each `RecordBatch` can be imported via pycapsules. 
   
   > * `register_csv(..., schema: pa.Schema)`
   > * `register_parquet(..., schema: pa.Schema)`
   > * `register_json(..., schema: pa.Schema)`
   
   These can import a schema via the pycapsule interface, without any breaking change. On the python side the type hint can be updated to [this protocol](https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html#protocol-typehints)
   
   ```py
   from typing import Tuple, Protocol
   
   class ArrowSchemaExportable(Protocol):
       def __arrow_c_schema__(self) -> object: ...
   ```
   
   
   > * `register_dataset(dataset: pa.dataset.Dataset)`
   
   This is specific to a pyarrow API, and so this can stay as-is with pyarrow as an optional dependency (I wish this was named `register_pyarrow_dataset` to avoid confusion)
   
   > 
   > **DataFrame methods:**
   > 
   > * `cast(mapping: dict[str, pa.DataType])`
   
   Why isn't this a schema? Or is intentional that you only want to cast a couple specified columns, leaving the others alone, not "projecting" to the specific schema?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] Remove pyarrow as required dependency, relying on Arrow PyCapsule Interface [datafusion-python]

Posted by "ion-elgreco (via GitHub)" <gi...@apache.org>.

ion-elgreco commented on issue #1227:
URL: https://github.com/apache/datafusion-python/issues/1227#issuecomment-3313444410

   > First, let's establish a list of all the places where pyarrow is used. Where is it used as an input parameter and where is it used as a return type?
   > 
   > Wherever pyarrow is used as an input parameter is pretty trivially replaced by the PyCapsule Interface, because the interface can accept pyarrow objects without us needing to depend on pyarrow as a dependency.
   > 
   > Wherever it's used as a _return type_ is a little more tricky, because we need to return something.
   
   Where we return a pyarrow type we either need to remove it and implement on the Datafusion type directly, for example for the schema we just need to have it be a Datafusion.schema that's how I mentioned it in the initial issue when I had time :p


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org