You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@iceberg.apache.org by "qzyu999 (via GitHub)" <gi...@apache.org> on 2026/06/24 05:55:46 UTC

[I] Integrate DataFusion as execution engine for compute-heavy operations [iceberg-python]

qzyu999 opened a new issue, #3554:
URL: https://github.com/apache/iceberg-python/issues/3554

   ### Feature Request / Improvement
   
   # Problem
   PyIceberg cannot perform several operations at production scale due to unbounded memory requirements in the PyArrow execution path:
   - Tables with equality deletes are unreadable (hard `ValueError`)
   - CoW deletes OOM on large Parquet files (~1GB)
   - CoW overwrite OOMs (same pattern as delete)
   - Upsert uses O(n²) row-by-row comparison
   - Compaction not implemented, requires external sort (infeasible in-memory for large tables)
   - Orphan file deletion OOMs (LEFT ANTI JOIN of millions of paths)
   
   The list goes on (documented below) with operations that don't scale in the typical single-node environment of PyIceberg. These block PyIceberg from achieving feature parity with Java Iceberg for V2/V3.
   
   # Proposed Solution
   Integrate Apache DataFusion as an optional execution engine (pip install 'pyiceberg[pyiceberg-core]') behind an automatic engine-resolution layer. When installed, compute-heavy operations use DataFusion's spill-to-disk execution (bounded memory). When not installed, the existing PyArrow path remains unchanged (works for small data, OOMs gracefully on large).
   
   No existing behavior changes. No forced dependency. DuckDB-style UX where only a developer only needs to configure a memory budget if they so choose.
   
   # Design Doc
   ...
   
   # Operations Unblocked
   - [ ] Equality delete read resolution
   - [ ] Streaming CoW delete/overwrite
   - [ ] Table compaction (sort + rewrite)
   - [ ] Orphan file deletion
   - [ ] Upsert via hash join
   - [ ] Equality-to-positional conversion
   - [ ] Position delete compaction
   - [ ] Full MoR compaction
   - [ ] Z-Order / Hilbert sorting
   - [ ] DV compaction
   - [ ] Incremental compaction
   - [ ] Sort-order enforcement on write
   - [ ] Dynamic partition overwrite (bounded memory)
   
   # Related Issues
   
   ## PyIceberg
   - #1078 (MoR support epic)
   - #1210 / #3270 (equality delete reads)
   - #3356 (execution path isolation)
   - #1092 (data compaction)
   - #1200 (orphan file deletion)
   - #3285 (`DeleteFileIndex` for equality deletes)
   - #3319 / #3320 (commit retry, prerequisite for safe compaction commits)
   - #3130 / #3131 (`REPLACE` API, prerequisite for compaction)
   - #1818 (V3 tracking, DV compaction)
   - #1808 (positional delete write support)
   - #2918 (`DeleteFileIndex` for positional deletes, merged foundation)
   
   ## iceberg-rust
   - [iceberg-rust#2186](https://github.com/apache/iceberg-rust/issues/2186) (MoR scan-side delete reconciliation)
   - [iceberg-rust#2205](https://github.com/apache/iceberg-rust/issues/2205) (equality delete reader)
   - [iceberg-rust#1530](https://github.com/apache/iceberg-rust/issues/1530) (delete file support in scan)
   - [iceberg-rust#2269](https://github.com/apache/iceberg-rust/issues/2269) (DataFusion write actions)
   
   ## datafusion-python
   - [datafusion-python#1217](https://github.com/apache/datafusion-python/issues/1217) (FFI boundary stability)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] Integrate DataFusion as execution engine for compute-heavy operations [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.

kevinjqliu commented on issue #3554:
URL: https://github.com/apache/iceberg-python/issues/3554#issuecomment-4827221121

   Thanks! I think a good next step is to understand how we use pyarrow today and how to decouple it from the rest of the code. 
   
   Would be great to be able to use different backend for reading and writing -- datafusion, duckdb, pyarrow, etc 😄 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] Integrate DataFusion as execution engine for compute-heavy operations [iceberg-python]

Posted by "timsaucer (via GitHub)" <gi...@apache.org>.

timsaucer commented on issue #3554:
URL: https://github.com/apache/iceberg-python/issues/3554#issuecomment-4831966686

   Interesting proposal. I'm definitely interested in following the discussion here and available for consult about DataFusion / df-python related topics. This is also closely related to some work I've done at [Rerun](https://rerun.io/) where we have a mixture of compute heavy tasks and IO tasks using DataFusion for the execution engine. I'm pretty swamped at the moment but I'll try to review this proposal in detail sometime this week.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] Integrate DataFusion as execution engine for compute-heavy operations [iceberg-python]

Posted by "abnobdoss (via GitHub)" <gi...@apache.org>.

abnobdoss commented on issue #3554:
URL: https://github.com/apache/iceberg-python/issues/3554#issuecomment-4861345525

Thank you for putting this together, this looks great! Personally, and from what I've seen across the community, the items on this list are things a lot of people are looking for.

My 2 cents, for what it's worth: I think this might benefit from being split into three separate tracks, each of which is easier to land on its own.

1. **The v2/v3 semantics that are engine-independent.** Most items on the "Operations Unblocked" checklist are gated on table-format work that has to be built in pyiceberg under any engine: equality delete reads (a missing v2 feature tracked in #1210; #3285 already builds the DeleteFileIndex foundation, which this proposal cites), delete-file writes (v2 positional deletes, v3 deletion vectors), and the REPLACE API and commit retry work the proposal already lists as prerequisites for compaction. Whichever executor runs the data step, these are what actually unlock the operations.

2. **Streaming over materialization in the existing pyarrow path.** We are close to having workable end-to-end streaming paths in pyarrow that would mitigate the OOM concerns for delete and overwrite: #3335 (RecordBatchReader in append/overwrite) merged recently, and there's prior PR work on the eager file materialization read path (#3036) to build on. Meanwhile CoW delete still loads each affected file fully into memory (`Transaction.delete` calls `to_table` per file) even though the complementary filter is per-batch. Reviving and finishing that conversion gets us there with no new dependency, and it helps every pyiceberg user by default, no new engine to opt into.

3. **DataFusion for the operations that are genuinely blocking.** What's left after 1 and 2 is the ops where an operator has to hold more than memory: global sort for compaction, large join builds (huge equality delete files, upserts with a huge source). That's where spill-to-disk earns its place, and where I think this proposal is strongest. One thing worth designing in early: delegating these ops crosses a semantics boundary (for example, the spec requires a null delete value to match null rows, which needs `IS NOT DISTINCT FROM` in datafusion rather than `=`, and pyarrow's join can't express it at all), so spec fixtures that run identically against every execution path would keep divergences visible as test failures.

Part of why I'd sequence it this way is that review bandwidth is the scarcest resource in any of these projects. Tracks 1 and 2 sit in code paths that are well established in pyiceberg, while a new engine is a long-term commitment (docs, CI, version compatibility, triage), so the narrower its scope when it arrives, the better its odds of landing and staying healthy.

More broadly, this issue has surfaced how blurry the line is between the engine pyiceberg uses internally and the engines it serves externally. On that note, @kevinjqliu @Fokko I'd be keen to understand where the existing iceberg-core expansion tickets stand (#2396, #2303, apache/iceberg-rust#1144, apache/iceberg-rust#1694): are those still the desired end state, or deprioritized in favor of this direction? The community has been very active lately, so I think it would be awesome to have a fresh roadmap discussion along the lines of #1856 to recap the longer-term plans and 2026 goals.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org