You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "unikdahal (via GitHub)" <gi...@apache.org> on 2026/10/06 20:11:33 UTC
[I] Design question: long-term, should native Iceberg scans run through DataFusion's ParquetSource instead of iceberg-rust's ArrowReader? [datafusion-comet]
unikdahal opened a new issue, #6737:
URL: https://github.com/apache/datafusion-comet/issues/6737
I've been working on runtime pruning for `IcebergScanExec` (#6588, #6641, follow-up to #4807/#4808). It works. On the benchmark in #6641, the sorted join and TopK cases cut reader bytes by 94-99%. But getting there meant adding a fair amount of adaptive-pruning machinery to iceberg-rust (tracked in apache/iceberg-rust#3343). That includes a runtime predicate provider, rejecting file tasks from retained stats, live row-group refresh, and keeping position/equality deletes correct through all of it.
While writing it I kept noticing that DataFusion's Parquet path already does most of this for plain Parquet scans. apache/datafusion#22450 (closing #22407) re-checks a live `DynamicFilterPhysicalExpr` at row-group boundaries, and the sort pushdown epic (apache/datafusion#23036) covers stats-based file/RG ordering for TopK. So for the same dynamic filters we end up with two parallel stacks:
```
native Parquet scan: DynamicFilterPhysicalExpr -> ParquetSource -> file / RG / page pruning
Iceberg scan: DynamicFilterPhysicalExpr -> translate -> iceberg-rust runtime predicate
-> ArrowReader -> file / RG / page pruning (reimplemented)
```
To be clear, I'm not proposing to rip anything out, and #6641 stands as is. I also don't want Comet re-planning tables in Rust. Iceberg Java should keep producing the `FileScanTask`s.
What I'm wondering about is `apache/datafusion-iceberg`, now that it lives under DataFusion. Long term, could it expose a source that takes already-planned `FileScanTask`s and runs them through `DataSourceExec`/`ParquetSource`, with the Iceberg-specific parts wrapped around it (field-ID schema mapping, defaults, position/equality deletes, DVs, encryption)?
```
Spark / Iceberg Java -> FileScanTask[] -> Comet -> datafusion-iceberg
-> DataSourceExec / ParquetSource -> Parquet
```
Comet would hand it the Java-planned tasks, and dynamic filters, TopK ordering and whatever DataFusion adds next would come for free instead of being redone for Iceberg.
I know the hard part is semantics. Position deletes need physical row positions before any filtering, equality deletes need extra columns pulled into the internal projection, and then there are DVs, split tasks and schema evolution. `ArrowReader` handles all of that today and Comet relies on it. There's also a smaller practical point in #6126: for the native Parquet scan Comet owns the `AsyncFileReader`, but for Iceberg it's iceberg-rust's `ArrowFileReader` and we can't swap it, which makes things like memory accounting harder.
## Questions
1. Is this a direction people would want, or is `ArrowReader` meant to stay the physical execution path for Iceberg in Comet?
2. If it is, is datafusion-iceberg the right home, and would a "from pre-planned tasks" entry point be reasonable there?
3. Meanwhile, is it fine to keep going with #6641 in its current shape, knowing parts of it might be replaced later? I mainly want to decide how much more adaptive logic (live refresh, ordering) to put into iceberg-rust.
I haven't dug into datafusion-iceberg's internals yet, so some of this may already exist or be planned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Design question: long-term, should native Iceberg scans run through DataFusion's ParquetSource instead of iceberg-rust's ArrowReader? [datafusion-comet]
Posted by "unikdahal (via GitHub)" <gi...@apache.org>.
unikdahal commented on issue #6737:
URL: https://github.com/apache/datafusion-comet/issues/6737#issuecomment-6026242683
Thanks @comphead, your diagram is right, and I worded it badly. iceberg-rust would still own the Iceberg semantics. I only meant: who owns the physical Parquet read.
`IcebergTableScan` builds a `TableScan` and calls `to_arrow()`, with filters fixed into an Iceberg `Predicate` at plan time. Comet's `IcebergScanExec` also reads through `ArrowReader`. So neither gets DataFusion's ParquetSource runtime filter pruning (join/TopK), which is why #6641 currently needs equivalent support on the iceberg-rust ArrowReader path.
By "pre-planned tasks" I meant the `FileScanTask`s Comet already gets from Iceberg Java. By "DataFusion-native source" I meant a `DataSource` behind `DataSourceExec`, so runtime filters have a standard place to land.
So I agree with @mbutrovich: start with a `DataSource` that still reads through `ArrowReader`. Decoding through `ParquetSource` itself can be a later question. Deletes, especially equality deletes, are the hard part there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Design question: long-term, should native Iceberg scans run through DataFusion's ParquetSource instead of iceberg-rust's ArrowReader? [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on issue #6737:
URL: https://github.com/apache/datafusion-comet/issues/6737#issuecomment-6024840468
I think my gut reaction is that I would be interested in implementing a DataSource in datafusion-iceberg that uses ArrowReader methods (making whatever we need ‘pub’ there to make that happen). I think we want to leave the really deep Iceberg-specific logic in iceberg-rust.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Design question: long-term, should native Iceberg scans run through DataFusion's ParquetSource instead of iceberg-rust's ArrowReader? [datafusion-comet]
Posted by "alexanderbianchi (via GitHub)" <gi...@apache.org>.
alexanderbianchi commented on issue #6737:
URL: https://github.com/apache/datafusion-comet/issues/6737#issuecomment-6114821853
I've done some preliminary investigation here. Sail has its own Iceberg integration that follows a similar native-DataFusion approach:
https://github.com/lakehq/sail/tree/main/crates/sail-iceberg
The main attraction for me is directly reusing DataFusion's Parquet configuration and optimization machinery, rather than maintaining parallel implementations.
For example, DataFusion already has CachedParquetFileReaderFactory for caching parsed footer and page metadata:
https://docs.rs/datafusion/latest/datafusion/datasource/physical_plan/parquet/struct.CachedParquetFileReaderFactory.html
For supported data-file filters, including dynamic filters, a native Parquet path could also evaluate DataFusion physical expressions without translating them into Iceberg predicates. Manifest and partition pruning would remain a separate concern.
We've already migrated IcebergTableScan to IcebergDataSource under DataSourceExec: https://github.com/apache/datafusion-iceberg/pull/50. That gives filters a standard place to land, execution still goes through iceberg-rust's ArrowReader.
I see two possible directions:
#### 1. Expose more Iceberg read preparation and use DataFusion's Parquet scan machinery
Iceberg metadata/file planning could still use iceberg-rust, but data-file execution would no longer go through ArrowReader.
The challenge is implementing Iceberg read semantics around the native Parquet scans: field-ID/schema evolution, equality and position deletes, delete vectors, and row lineage. That could involve exposing reusable helpers from iceberg-rust or implementing those pieces in the integration.
We would need to preserve per-file metadata and original row positions through pruning and filtering, rather than treating everything as an arbitrary stream of already-filtered batches.
#### 2. Keep ArrowReader and introduce narrower extension points
An injectable Parquet AsyncFileReader factory could let integrations reuse metadata caching and custom I/O while retaining Iceberg's existing schema and delete handling.
This would be analogous in spirit to the Storage abstraction: https://github.com/apache/iceberg-rust/issues/2051
That factory alone would not reuse DataFusion's filter evaluation or pruning. A separate, higher-level hook would be needed to supply engine-built row filters and pruning selections, with enough Iceberg schema context to compose them safely with deletes.
Neither direction is trivial, and the narrower hooks might be useful independently of a full native-Parquet integration.
FYI Thanks for the work on execution-time adaptive pruning. DataFusion Distributed also wants dynamic filtering for Iceberg, and we're interested in building on https://github.com/apache/iceberg-rust/issues/3343
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Design question: long-term, should native Iceberg scans run through DataFusion's ParquetSource instead of iceberg-rust's ArrowReader? [datafusion-comet]
Posted by "unikdahal (via GitHub)" <gi...@apache.org>.
unikdahal commented on issue #6737:
URL: https://github.com/apache/datafusion-comet/issues/6737#issuecomment-6129447709
A small update before I disappear for about a month: I don't want this design question, or the related epic #6641, to block anyone while I'm away.
My summary of where the discussion landed (please correct me if I've misread it):
- **Near term:** follow @mbutrovich's suggestion. Build a `DataSource` in datafusion-iceberg that still reads through iceberg-rust's `ArrowReader`, making whatever needs to be `pub` public, and keep the deep Iceberg semantics in iceberg-rust. @alexanderbianchi's apache/datafusion-iceberg#50 already moved `IcebergTableScan` onto `DataSourceExec`, so runtime filters have a standard place to land.
- **Longer term:** either run data files through DataFusion's Parquet scan with the Iceberg semantics wrapped around it, or add narrower hooks to `ArrowReader`, such as an injectable `AsyncFileReader` factory plus a way to pass engine-built row filters. Both are still open, and Alex laid out the trade-offs well above.
- **For Comet specifically:** my prototype for #6641 runs on the iceberg-rust `ArrowReader` path. If a datafusion-iceberg source that accepts pre-planned `FileScanTask`s shows up, I'd expect the producer side (join / Top-K / MIN-MAX filters) to carry over with little change, because it only publishes dynamic filters.
apache/datafusion-iceberg#68 now tracks the dynamic-filter piece with a concrete design, which seems like the natural place for that part to continue. The broader `ParquetSource`-vs-`ArrowReader` question can stay open here. If anyone wants to pick up the Comet side or prototype a pre-planned-tasks entry point, please go ahead; I'll catch up when I'm back.
Thanks @mbutrovich, @comphead and @alexanderbianchi for the thoughtful replies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Design question: long-term, should native Iceberg scans run through DataFusion's ParquetSource instead of iceberg-rust's ArrowReader? [datafusion-comet]
Posted by "unikdahal (via GitHub)" <gi...@apache.org>.
unikdahal commented on issue #6737:
URL: https://github.com/apache/datafusion-comet/issues/6737#issuecomment-6024616011
cc @mbutrovich @andygrove @parthchandra, would love your opinion on this, especially question
1: should `ArrowReader` stay the physical execution path for native Iceberg scans in Comet, or is a DataFusion-native source for pre-planned tasks worth exploring long term?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Design question: long-term, should native Iceberg scans run through DataFusion's ParquetSource instead of iceberg-rust's ArrowReader? [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #6737:
URL: https://github.com/apache/datafusion-comet/issues/6737#issuecomment-6025895235
> or is a DataFusion-native source for pre-planned tasks worth exploring long term?
Thanks @unikdahal
I prob missing something, what is `DataFusion-native source for pre-planned tasks`?
My understanding DataFusion still talking to iceberg through iceberg-rs, ArrowReader
```
DataFusion (SQL, DataFrame, optimizer)
└─ CatalogProvider / SchemaProvider / TableProvider / ExecutionPlan
└─ datafusion-iceberg: adapters plus custom plan nodes
└─ iceberg-rust: Catalog, TableScan, ArrowReader, writers, Transaction
└─ FileIO (StorageFactory, e.g. OpenDAL) → S3, GCS, local
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org