You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "NoahKusaba (via GitHub)" <gi...@apache.org> on 2026/09/29 22:24:41 UTC

[I] [EPIC] Distributed execution with Ballista [datafusion-iceberg]

NoahKusaba opened a new issue, #23:
URL: https://github.com/apache/datafusion-iceberg/issues/23

   ## Goal
   
   Run Iceberg reads and writes in parallel on
   [Ballista](https://github.com/apache/datafusion-ballista) executors: apache/datafusion-ballista#2217 adds an `iceberg-ballista` crate whose
   logical and physical extension codecs send Iceberg plans to the scheduler and
   executors.
   
   A codec in another process has to name each node's type, read what it was built
   from, and rebuild an equivalent node from those parts alone. This epic tracks the
   changes that make that possible, split into what blocks
   apache/datafusion-ballista#2217 and what improves the integration once it has
   landed.
   
   ## Required (blocking)
   
   apache/datafusion-ballista#2217 can't merge until these are on
   datafusion-iceberg `main`. Today it depends on a fork branch that carries all of
   them except partitioned scans.
   
   - [x] #14 Make `PartitionExpr` reconstructible (closes #13)
   - [ ] #20 Make the plan nodes inspectable and rebuildable: public
         `IcebergCommitExec`, `IcebergWriteExec`, `IcebergMetadataScan`, accessors,
         and constructors that take only what the accessors return
   - [ ] `IcebergTableProvider::new_with_schema` (PR to be opened; draft, stacked on
         #20). Lets the scheduler rebuild a catalog-backed provider with the schema the
         client planned against, without loading the table.
   - [ ] Opt-in idempotent commits (PR to be opened; draft, stacked on #20).
         `IcebergCommitExec` records a commit id in the snapshot summary and skips
         committing when an earlier execution already did, since Ballista reruns tasks
         after losing an executor. Two attempts running at the same time can still
         both commit; see the idempotency key under iceberg-rust below.
   - [ ] Partitioned scans. `IcebergTableScan` reports `UnknownPartitioning(1)`, so
         every scan runs as one task on one executor, which lists and reads every
         file. It also limits writes: Ballista drops round-robin repartitions, so an
         INSERT into an unpartitioned table, or one with no identity or bucket
         partition field, runs one writer task per input partition. Into such a
         table, `INSERT INTO t SELECT … FROM <iceberg table>` writes from a single
         task.
         apache/iceberg-rust#2671 (@toutane, closes apache/iceberg-rust#2220)
         implements this: `scan()` plans the file tasks eagerly, groups them across
         `target_partitions`, and `execute(i)` reads group i. It went through several
         review rounds there but was stranded by the move to this repo, and
         apache/iceberg-rust#3126 (sort-order reporting) builds on it. Porting it
         here also needs, for Ballista:
     - [ ] The node exposes its task groups and has a constructor that takes them,
           like #20's nodes. The files must be planned once, on the scheduler, and
           shipped to executors: an executor that planned again would read a
           different grouping, since file planning order isn't deterministic.
           `FileScanTask` is public and serializable.
     - [ ] No dependency on `TableScan::arrow_reader_builder()`, which #2671 adds to
           iceberg-rust but neither iceberg-rust `main` nor the revision #21 moves
           to has.
           Build the `ArrowReaderBuilder` here instead (its setters are public, and
           `Runtime::try_current()` works at execute time), or upstream the
           accessor first (see iceberg-rust below).
     - [ ] If eager planning stays opt-in (#2671 uses
           `iceberg.enable_eager_scan_planning`), the setting reaches the Ballista
           scheduler, or iceberg-ballista turns it on.
     - [ ] A scan with no files reports one partition, not zero.
   
   ## Nice to have
   
   Fixes and improvements that don't block apache/datafusion-ballista#2217. Most
   of the iceberg-rust items remove a workaround in iceberg-ballista or close a
   known gap.
   
   ### datafusion-iceberg
   
   - [ ] #19 Accept NOT NULL input columns in partitioned INSERT. Without it, a
         partitioned INSERT into optional columns from a NOT NULL source (such as a
         `VALUES` list with no NULLs, whose columns DataFusion infers as
         non-nullable) fails at plan time.
     - [ ] #22 follow-up: unpartitioned tables write NULLs into required columns;
           partitioned tables reject nullable sources at plan time
   - [ ] Hash-partition INSERT input for every partition transform. `repartition.rs`
         hashes on `_partition` only when the spec has an identity or bucket field,
         and otherwise round-robins, which Ballista drops. Each writer task can then
         write to every table partition, and the INSERT produces many small files.
         `_partition` already holds the transformed value, so hashing on it works
         for any transform.
   - [ ] Scan statistics. With eager planning, file record counts give row counts
         nearly for free, which helps join planning. An earlier attempt,
         apache/iceberg-rust#880, went stale.
   - [ ] Pass the session's `batch_size` to the scan's reader, and bound per-partition
         file-read concurrency once a scan has several partitions per executor.
   
   ### iceberg-rust
   
   In review:
   - [ ] apache/iceberg-rust#2904 A scan projects a stale schema after a schema change
         with no write since
   - [ ] apache/iceberg-rust#3286 `with_runtime` on the catalog loader's
         `BoxedCatalogBuilder`, so a loaded catalog can be bound to a chosen runtime
   
   Fix ready, PR not opened yet:
   - [ ] apache/iceberg-rust#3297 `strip_metadata_from_schema` fails on list and map
         columns. Blocks the list and map nullability tests staged in #19.
   
   Not opened yet, most important first:
   - [ ] Idempotency key on `fast_append`, checked inside the commit retry loop.
         Closes the concurrent-commit gap above: today a conflicting commit is retried
         and re-applied on the new head without re-checking the commit id.
   - [ ] A scan on a runtime that has shut down returns no rows instead of failing
   - [ ] The REST catalog never refreshes its OAuth token, so a long-lived scheduler
         fails to plan once its token expires
   - [ ] `Debug` output of `FileIO`/`StorageConfig` prints storage credentials
   - [ ] `Table::with_runtime` that keeps the table's manifest cache
   - [ ] `TableScan::arrow_reader_builder()`, from apache/iceberg-rust#2671, so a
         partitioned scan reads with exactly the reader settings its `TableScan`
         was built with
   
   To investigate: whether `FileIO` can refresh catalog-vended storage credentials
   during a long job.
   
   ## Merge order
   
   1. #20
   2. The rest of the required items, which build on #20, in any order:
      - `new_with_schema` uses #20's public `try_new` and `table_ident()`
      - idempotent commits add to `IcebergCommitExec`, which #20 makes public
      - partitioned scans rework the `IcebergTableScan` code #20 changes
   
   The nice-to-haves can land in any order.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] [EPIC] Distributed Execution for Reads/Writes [datafusion-iceberg]

Posted by "NoahKusaba (via GitHub)" <gi...@apache.org>.
NoahKusaba commented on issue #23:
URL: https://github.com/apache/datafusion-iceberg/issues/23#issuecomment-5900511153

   cc @alexanderbianchi --> I know you represent the datafusion-distributed interested party.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] [EPIC] Distributed Execution for Reads/Writes [datafusion-iceberg]

Posted by "NoahKusaba (via GitHub)" <gi...@apache.org>.
NoahKusaba commented on issue #23:
URL: https://github.com/apache/datafusion-iceberg/issues/23#issuecomment-5900412737

   cc @mbutrovich 
   
   At your suggestion I put together an EPIC to track unfinished work for distributed datafusion execution, which is relevant for both ballista and datafusion-distributed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] [EPIC] Distributed Execution for Reads/Writes [datafusion-iceberg]

Posted by "NoahKusaba (via GitHub)" <gi...@apache.org>.
NoahKusaba commented on issue #23:
URL: https://github.com/apache/datafusion-iceberg/issues/23#issuecomment-5900721488

   As mbutrovich has stated their is more review bandwidth for iceberg-rust PR's, I'm going to first focus on the "idemopotent key change on fast_appends", as it closes the only gap in correct writes and the https://github.com/apache/iceberg-rust/issues/3297 we reviewed, as it's mostly done.
   
   On the datafusion-iceberg side, I'm going to wait on PR #20 to merge while I'm doing the iceberg-rust stuff, and then start opening PR's the remaining stack on top of it. 
   
   I've also messaged @toutane on slack about re-opening his partitioned reads PR on this repo, and I'll look to re-open it myself by end-of-october as it's a major blocker.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] [EPIC] Distributed Execution for Reads/Writes [datafusion-iceberg]

Posted by "NoahKusaba (via GitHub)" <gi...@apache.org>.
NoahKusaba commented on issue #23:
URL: https://github.com/apache/datafusion-iceberg/issues/23#issuecomment-5900436479

   cc @andygrove and @milenkovicm 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] [EPIC] Distributed Execution for Reads/Writes [datafusion-iceberg]

Posted by "alexanderbianchi (via GitHub)" <gi...@apache.org>.
alexanderbianchi commented on issue #23:
URL: https://github.com/apache/datafusion-iceberg/issues/23#issuecomment-5900977145

   Thanks for the write-up! A lot of these primitives overlap with what DFD needs, which is great. In DFD, we’ll maintain our own `DataSourceExec` integration because it’s built on the `WorkUnitFeeds` abstraction. Ideally, we can reuse upstream for everything else.
   
   As these APIs become public, I’d like to make sure they also accommodate DFD’s execution model—particularly running writers on workers and collecting their results for a coordinator-side commit.
   
   Breaking this into sub-issues would help; I know a few engineers looking to pick up tickets, including myself.
   
   I can ask @toutane whether he wants to move [[iceberg-rust#2671](https://github.com/apache/iceberg-rust/pull/2671)](https://github.com/apache/iceberg-rust/pull/2671) here or whether I should carry it forward—we work together.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org