(iceberg-rust) branch main updated: test: pyiceberg-core, pin datafusion and unpin pyarrow (#1648)

[li...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

liurenjie1024 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/iceberg-rust.git


The following commit(s) were added to refs/heads/main by this push:
     new 7362bbdcf test: pyiceberg-core, pin datafusion and unpin pyarrow (#1648)
7362bbdcf is described below

commit 7362bbdcf72910d95ee88940f229d2eea40b3fa1
Author: Kevin Liu <ke...@users.noreply.github.com>
AuthorDate: Fri Sep 5 02:23:39 2025 -0700

    test: pyiceberg-core, pin datafusion and unpin pyarrow (#1648)
    
    ## Which issue does this PR close?
    
    - Closes #.
    
    ## What changes are included in this PR?
    Follow up to #1635
    We dont need to pin pyarrow specifically, just grab it from pyiceberg.
    Revert datafusion version threshold to 45. Version 45 is the lowest
    version which will still work.
    The underlying issue for #1635 is with the **newer** versions of
    datafusion, mainly datafusion 49 due to a breaking change described in
    #1647
    So pyiceberg-core works with datafusion (python) library 45, 46, 47, and
    48. just not 49
    
    I verified locally with all the versions above:
    ```
    hatch env prune && hatch run dev:test
    ```
    
    i double checked. `insert_into` is available in datafusion 45.
    https://github.com/apache/datafusion/commit/7f9a8c024c2993732d600bf32d3ee64ef7afa96d
    
    
    
    ## Are these changes tested?
---
 bindings/python/pyproject.toml                          | 2 +-
 bindings/python/tests/test_datafusion_table_provider.py | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/bindings/python/pyproject.toml b/bindings/python/pyproject.toml
index 346ee168d..85af1e000 100644
--- a/bindings/python/pyproject.toml
+++ b/bindings/python/pyproject.toml
@@ -44,7 +44,7 @@ python-source = "python"
 ignore = ["F403", "F405"]
 
 [tool.hatch.envs.dev]
-dependencies = ["maturin>=1.0,<2.0", "pytest>=8.3.2", "pyarrow==16.*", "datafusion==47.*", "pyiceberg[sql-sqlite]>=0.9.1"]
+dependencies = ["maturin>=1.0,<2.0", "pytest>=8.3.2", "datafusion==45.*", "pyiceberg[sql-sqlite,pyarrow]>=0.9.1"]
 
 [tool.hatch.envs.dev.scripts]
 build = "maturin build --out dist --sdist"
diff --git a/bindings/python/tests/test_datafusion_table_provider.py b/bindings/python/tests/test_datafusion_table_provider.py
index 5c6a15eb7..915b78769 100644
--- a/bindings/python/tests/test_datafusion_table_provider.py
+++ b/bindings/python/tests/test_datafusion_table_provider.py
@@ -27,8 +27,8 @@ from pathlib import Path
 import datafusion
 
 assert (
-    datafusion.__version__ >= "47"
-)  # iceberg table provider only works for datafusion >= 47
+    datafusion.__version__ >= "45"
+)  # iceberg table provider only works for datafusion >= 45
 
 
 @pytest.fixture(scope="session")