You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by mb...@apache.org on 2026/07/29 13:59:26 UTC

(datafusion-comet) branch main updated: docs: update post 1.0.0 roadmap (#5064)

This is an automated email from the ASF dual-hosted git repository.

mbutrovich pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git


The following commit(s) were added to refs/heads/main by this push:
     new e344f7490a docs: update post 1.0.0 roadmap (#5064)
e344f7490a is described below

commit e344f7490a3db8a07e147c57fdecc7bbe74f3dd6
Author: Matt Butrovich <mb...@users.noreply.github.com>
AuthorDate: Wed Jul 29 09:59:21 2026 -0400

    docs: update post 1.0.0 roadmap (#5064)
---
 docs/source/contributor-guide/roadmap.md | 111 ++++++++++++++++++++++---------
 1 file changed, 80 insertions(+), 31 deletions(-)

diff --git a/docs/source/contributor-guide/roadmap.md b/docs/source/contributor-guide/roadmap.md
index eaa6f0c097..4fd9e16902 100644
--- a/docs/source/contributor-guide/roadmap.md
+++ b/docs/source/contributor-guide/roadmap.md
@@ -41,32 +41,56 @@ supported functions, frames, and fallback cases.
 [#4836]: https://github.com/apache/datafusion-comet/issues/4836
 [#4837]: https://github.com/apache/datafusion-comet/issues/4837
 
-## Lambda Expressions
+## Native Lambda Evaluation
 
 Spark supports higher-order functions on arrays and maps that take a lambda, including `transform`, `exists`,
-`forall`, `aggregate`, `zip_with`, `map_filter`, and `map_zip_with`. Comet currently lacks a general mechanism
-for serializing lambda expressions and evaluating them in DataFusion. Adding this capability will unlock a
-significant family of Spark expressions in one effort.
-
-## Dynamic Partition Pruning
-
-Native Parquet scans (`CometNativeScanExec`) support Dynamic Partition Pruning (DPP) both with and without
-Adaptive Query Execution. Non-AQE DPP landed in [#4011] and AQE DPP with broadcast reuse landed in [#4112].
-Iceberg native scans support non-AQE DPP ([#3349], [#3511]) and, on Spark 3.5+, AQE DPP with broadcast reuse
-([#4215]); on Spark 3.4 Iceberg AQE DPP falls back to Spark without reuse.
-
-[#3349]: https://github.com/apache/datafusion-comet/pull/3349
-[#3511]: https://github.com/apache/datafusion-comet/pull/3511
-[#4011]: https://github.com/apache/datafusion-comet/pull/4011
-[#4112]: https://github.com/apache/datafusion-comet/pull/4112
-[#4215]: https://github.com/apache/datafusion-comet/pull/4215
+`forall`, `aggregate`, `zip_with`, `map_filter`, and `map_zip_with`. Comet evaluates these today through a JVM
+codegen-dispatch bridge (`CometScalaUDF`, `CometBatchKernelCodegen`) instead of falling back to Spark, but the
+lambda body is still interpreted row-at-a-time on the JVM rather than natively in DataFusion. DataFusion added
+native higher-order function support (`array_transform`, `array_filter`, `array_any_match`, etc.) that Comet
+does not yet use; it's not yet known whether their semantics are Spark-compatible. We'll explore whether
+DataFusion's implementations can replace the JVM codegen-dispatch bridge to remove that round-trip and let
+these expressions benefit from vectorized native execution.
+
+## Native Coverage for Codegen-Dispatched Expressions
+
+Beyond lambda bodies, a number of built-in Spark scalar expressions (some regular expression, JSON, and datetime
+functions, for example) and aggregate functions route through the same JVM codegen-dispatch bridge by default,
+either because their native DataFusion or `datafusion-spark` implementation has known semantic differences from
+Spark, or because no native implementation exists yet. See the [compatibility guide] for the current list of
+codegen-dispatched expressions. We're exploring closing these gaps so that more expressions run natively by
+default, which would reduce JVM round-trips beyond what the lambda and UDF work above already covers.
+
+[compatibility guide]: ../user-guide/latest/compatibility/index.md
+
+## Iceberg Table Format V3 Support
+
+Comet landed its first Iceberg table format V3 feature (native data file decryption, [#4991]), and we want to
+add more V3 features to Comet's native Iceberg scans so they don't fall back to Spark. The work is tracked
+phase-by-phase in [#3376]: detecting the V3 format and supporting new V3 data types, deletion vector reads, row
+lineage, and table encryption. Native deletion vector reads are prototyped in a draft PR ([#4887]), but are
+blocked on upstream `iceberg-rust` support, tracked in [iceberg-rust #2792] and [iceberg-rust #2411]. Native Iceberg
+scans don't support HDFS-backed tables today: the storage scheme match in `iceberg_scan.rs` only handles `file`,
+`s3`/`s3a`, `gs`, and `oss`, and would need an HDFS `StorageFactory` upstream in `iceberg-storage-opendal`. We're
+scoping what that work would take.
+
+[#3376]: https://github.com/apache/datafusion-comet/issues/3376
+[#4887]: https://github.com/apache/datafusion-comet/pull/4887
+[#4991]: https://github.com/apache/datafusion-comet/pull/4991
+[iceberg-rust #2411]: https://github.com/apache/iceberg-rust/issues/2411
+[iceberg-rust #2792]: https://github.com/apache/iceberg-rust/issues/2792
 
 ## TPC-H and TPC-DS Performance
 
-We regularly publish benchmark results derived from TPC-H and TPC-DS to track performance against Spark. Closing
-the remaining gaps and increasing the speedup on both benchmark suites is an ongoing focus, tracked under [#2004]
-(TPC-H) and [#858] (TPC-DS).
+Comet already delivers substantial speedups over vanilla Spark on both benchmark suites; we publish per-query
+results for [TPC-H] and [TPC-DS] with each release. An independent [AWS Labs benchmark] comparing Comet 0.16.0 with
+Gluten 1.6.0 on a 3TB TPC-DS workload found that the two accelerators deliver similar overall performance. Increasing
+the speedup further and closing the remaining per-query gaps is an ongoing focus, tracked under [#2004] (TPC-H) and
+[#858] (TPC-DS).
 
+[TPC-H]: benchmark-results/tpc-h.md
+[TPC-DS]: benchmark-results/tpc-ds.md
+[AWS Labs benchmark]: https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-gluten-velox-comet-benchmark
 [#858]: https://github.com/apache/datafusion-comet/issues/858
 [#2004]: https://github.com/apache/datafusion-comet/issues/2004
 
@@ -85,11 +109,13 @@ Comet's native hash join currently requires the build side to fit entirely in me
 support will allow Comet to handle larger joins without falling back to Spark, improving both reliability and
 performance for memory-intensive workloads.
 
-## Java/Scala Columnar and Arrow UDF Support
+## Java/Scala UDF Support
 
-Spark users frequently define custom UDFs in Java or Scala. Comet currently falls back to Spark when a query
-contains a JVM UDF. Adding support for calling Java/Scala UDFs that operate on columnar Arrow data directly
-from native execution will reduce fallbacks and allow more queries to run end-to-end in Comet.
+Spark users frequently define custom UDFs in Java or Scala. Comet now dispatches scalar `ScalaUDF` expressions
+through a JVM codegen bridge (`CometScalaUDF`) instead of always falling back to Spark. Aggregate UDFs, table
+UDFs/generators, Python/Pandas UDFs, and Hive `GenericUDF`/`SimpleUDF` still fall back to Spark entirely.
+Extending the codegen-dispatch approach to cover these remaining categories will reduce fallbacks further and
+allow more queries to run end-to-end in Comet.
 
 ## Memory Management Improvements
 
@@ -97,13 +123,6 @@ Comet coordinates memory between the JVM and native Rust execution through a cus
 memory accounting, reservation strategies, and spill integration will reduce out-of-memory errors and allow
 Comet to make better use of available resources, especially in multi-query and multi-task environments.
 
-## Prepare for 1.0.0 Release
-
-The project is working toward a 1.0.0 release. This effort includes finalizing configuration options,
-resolving known correctness issues, and improving documentation. Progress is tracked in [#4082].
-
-[#4082]: https://github.com/apache/datafusion-comet/issues/4082
-
 ## Native Parquet Writes
 
 Comet has experimental support for native Parquet writes via `InsertIntoHadoopFsRelationCommand`, currently
@@ -111,3 +130,33 @@ disabled by default. The goal is to reach correctness and performance parity wit
 enabled by default ([#1625]).
 
 [#1625]: https://github.com/apache/datafusion-comet/issues/1625
+
+## Iceberg Table Writes
+
+Comet's native scans accelerate Iceberg reads today, but writes still run entirely through Spark's Iceberg V2 write
+operator. [#4322] tracks exploring native acceleration of Iceberg writes, so an ETL job could run end to end in
+native code, falling back only where Comet can't match Iceberg Java's functionality. Draft proposals split the
+existing V2 write operator into separate "writer" and "committer" operators so the data-file-write portion can be
+planned and accelerated the same way as a native Parquet write ([#4658], [#4487]). No design here is committed yet;
+we're exploring whether this approach is worth pursuing.
+
+[#4322]: https://github.com/apache/datafusion-comet/issues/4322
+[#4487]: https://github.com/apache/datafusion-comet/pull/4487
+[#4658]: https://github.com/apache/datafusion-comet/pull/4658
+
+## Delta Lake Support
+
+Comet currently supports Spark's built-in file formats and Iceberg, but not Delta Lake ([#174]). A draft PR reuses
+Comet's existing native Parquet reader to accelerate scans of plain Delta tables that use neither deletion vectors
+nor column mapping ([#4669]); a separate, larger effort explores a full native Delta scan built on `delta-kernel-rs`
+as an out-of-tree contrib module ([#4366]), landing in inert, gated slices ([#4700], [#4952]). Generalizing Comet's
+scan-side APIs so that Delta and other non-Iceberg data sources can plug in more easily is tracked as a Table
+Provider API abstraction ([#4706]). None of this is committed yet; we're exploring which approach, if any, is worth
+pursuing.
+
+[#174]: https://github.com/apache/datafusion-comet/issues/174
+[#4366]: https://github.com/apache/datafusion-comet/pull/4366
+[#4669]: https://github.com/apache/datafusion-comet/pull/4669
+[#4700]: https://github.com/apache/datafusion-comet/pull/4700
+[#4706]: https://github.com/apache/datafusion-comet/issues/4706
+[#4952]: https://github.com/apache/datafusion-comet/pull/4952


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org