You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "kevinjqliu (via GitHub)" <gi...@apache.org> on 2026/08/20 17:33:16 UTC
[I] [pyiceberg-core] Remove the DataFusion `TableProvider` export from the Python bindings [iceberg-rust]
kevinjqliu opened a new issue, #3036:
URL: https://github.com/apache/iceberg-rust/issues/3036
## Summary
Remove `pyiceberg_core.datafusion.IcebergDataFusionTable` from the `pyiceberg-core` Python bindings.
This proposal only removes the Python export. It does **not** remove any DataFusion table provider from iceberg-rust.
## Rationale
This experimental feature was intended to integrate PyIceberg with DataFusion Python. **However, the resulting query does not use PyIceberg's scan implementation.** PyIceberg passes the table identifier, metadata location, and file I/O properties to `pyiceberg-core`, which loads a separate Rust `StaticTable`. iceberg-rust then handles metadata and manifest parsing, pruning, scan planning, delete handling, and data reads.
**This creates two independent Iceberg implementations behind a PyIceberg `Table`: Python for `Table.scan()` and Rust when the table is registered with DataFusion.** Their features and behavior can differ, and PyIceberg fixes do not apply to the Rust path. The API therefore appears to query through PyIceberg while actually relying on iceberg-rust's feature set and semantics.
The binding also tightly couples `pyiceberg-core` to DataFusion's Rust FFI and its major-version compatibility requirements.
## Proposal
Remove the following from the `pyiceberg-core` Python bindings:
- The `pyiceberg_core.datafusion` module
- The `IcebergDataFusionTable` Python class
- Related module registration, dependencies, and tests
The corresponding PyIceberg change should remove `Table.__datafusion_table_provider__` and update its documentation and optional dependencies.
## Non-goals
This proposal does **not** remove or deprecate the Rust `iceberg-datafusion` crate, `IcebergTableProvider`, or `IcebergStaticTableProvider`. The Rust integration remains available to applications that choose iceberg-rust as their Iceberg implementation.
Because the Python API has been released, the removal should be coordinated with PyIceberg and included in an appropriate breaking release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [pyiceberg-core] Remove the DataFusion `TableProvider` export from the Python bindings [iceberg-rust]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on issue #3036:
URL: https://github.com/apache/iceberg-rust/issues/3036#issuecomment-5375839483
I think it's fine to remove this.
Users who wish to use datafusion integration can use iceberg-rust directly to scan the table. Using the `IcebergDataFusionTable` is also becoming more and more limiting as iceberg-rust tries to allow new components to be injected into the catalog(Storage, Runtime, AuthManager, etc.). these components cannot work with Pyiceberg without having extra python wrappers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [pyiceberg-core] Remove the DataFusion `TableProvider` export from the Python bindings [iceberg-rust]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #3036:
URL: https://github.com/apache/iceberg-rust/issues/3036#issuecomment-5530481376
I opened #3143 to remove DataFusion TableProvider from pyiceberg-core
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [pyiceberg-core] Remove the DataFusion `TableProvider` export from the Python bindings [iceberg-rust]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #3036:
URL: https://github.com/apache/iceberg-rust/issues/3036#issuecomment-5530471377
I think DataFusion can still be a good candidate for data-intensive compute in PyIceberg as an alternative to PyArrow. It’s just that the current integration through `pyiceberg-core` conflicts with PyIceberg’s internal implementation. The Rust `iceberg-datafusion` crate, `IcebergTableProvider`, and `IcebergStaticTableProvider` remain available on the Rust side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [pyiceberg-core] Remove the DataFusion `TableProvider` export from the Python bindings [iceberg-rust]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #3036:
URL: https://github.com/apache/iceberg-rust/issues/3036#issuecomment-5359526193
To summarize from the `pyiceberg` angle, it is weird to use `iceberg-rust` (datafusion table provider) for table scan and read when we have already implemented that logic in python. Its like using `duckdb` with its iceberg integration in pyiceberg, totally orthogonal!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [pyiceberg-core] Remove the DataFusion `TableProvider` export from the Python bindings [iceberg-rust]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #3036:
URL: https://github.com/apache/iceberg-rust/issues/3036#issuecomment-5359469961
AFAIK pyiceberg is the only one relying on `pyiceberg-core`'s datafusion functionalities
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [pyiceberg-core] Remove the DataFusion `TableProvider` export from the Python bindings [iceberg-rust]
Posted by "qzyu999 (via GitHub)" <gi...@apache.org>.
qzyu999 commented on issue #3036:
URL: https://github.com/apache/iceberg-rust/issues/3036#issuecomment-5530385363
Hi @kevinjqliu, does this mean that DF is no longer positioned to act as a scalable compute backend for PyIceberg, or is it cleaning up a previous integration that wasn't executed cleanly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [pyiceberg-core] Remove the DataFusion `TableProvider` export from the Python bindings [iceberg-rust]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #3036:
URL: https://github.com/apache/iceberg-rust/issues/3036#issuecomment-5482251372
I think it makes sense to remove it. This will also allow us to drop the dependency, which simplifies things.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [pyiceberg-core] Remove the DataFusion `TableProvider` export from the Python bindings [iceberg-rust]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY closed issue #3036: [pyiceberg-core] Remove the DataFusion `TableProvider` export from the Python bindings
URL: https://github.com/apache/iceberg-rust/issues/3036
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org