You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "mbutrovich (via GitHub)" <gi...@apache.org> on 2026/07/28 14:48:31 UTC
[PR] docs: update post 1.0.0 roadmap [datafusion-comet]
mbutrovich opened a new pull request, #5064:
URL: https://github.com/apache/datafusion-comet/pull/5064
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
Closes #.
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
Roadmap needs updating.
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
Put some thoughts about Iceberg table format V3, prune stale items.
## How are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: update post 1.0.0 roadmap [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #5064:
URL: https://github.com/apache/datafusion-comet/pull/5064#discussion_r3666975917
##########
docs/source/contributor-guide/roadmap.md:
##########
@@ -85,25 +88,20 @@ Comet's native hash join currently requires the build side to fit entirely in me
support will allow Comet to handle larger joins without falling back to Spark, improving both reliability and
performance for memory-intensive workloads.
-## Java/Scala Columnar and Arrow UDF Support
+## Java/Scala UDF Support
-Spark users frequently define custom UDFs in Java or Scala. Comet currently falls back to Spark when a query
-contains a JVM UDF. Adding support for calling Java/Scala UDFs that operate on columnar Arrow data directly
-from native execution will reduce fallbacks and allow more queries to run end-to-end in Comet.
+Spark users frequently define custom UDFs in Java or Scala. Comet now dispatches scalar `ScalaUDF` expressions
+through a JVM codegen bridge (`CometScalaUDF`) instead of always falling back to Spark. Aggregate UDFs, table
+UDFs/generators, Python/Pandas UDFs, and Hive `GenericUDF`/`SimpleUDF` still fall back to Spark entirely.
+Extending the codegen-dispatch approach to cover these remaining categories will reduce fallbacks further and
+allow more queries to run end-to-end in Comet.
## Memory Management Improvements
Comet coordinates memory between the JVM and native Rust execution through a custom memory pool. Improving
memory accounting, reservation strategies, and spill integration will reduce out-of-memory errors and allow
Comet to make better use of available resources, especially in multi-query and multi-task environments.
-## Prepare for 1.0.0 Release
-
-The project is working toward a 1.0.0 release. This effort includes finalizing configuration options,
-resolving known correctness issues, and improving documentation. Progress is tracked in [#4082].
-
-[#4082]: https://github.com/apache/datafusion-comet/issues/4082
-
## Native Parquet Writes
Review Comment:
Iceberg HDFS reads
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: update post 1.0.0 roadmap [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #5064:
URL: https://github.com/apache/datafusion-comet/pull/5064#discussion_r3666586997
##########
docs/source/contributor-guide/roadmap.md:
##########
@@ -85,25 +88,20 @@ Comet's native hash join currently requires the build side to fit entirely in me
support will allow Comet to handle larger joins without falling back to Spark, improving both reliability and
performance for memory-intensive workloads.
-## Java/Scala Columnar and Arrow UDF Support
+## Java/Scala UDF Support
-Spark users frequently define custom UDFs in Java or Scala. Comet currently falls back to Spark when a query
-contains a JVM UDF. Adding support for calling Java/Scala UDFs that operate on columnar Arrow data directly
-from native execution will reduce fallbacks and allow more queries to run end-to-end in Comet.
+Spark users frequently define custom UDFs in Java or Scala. Comet now dispatches scalar `ScalaUDF` expressions
+through a JVM codegen bridge (`CometScalaUDF`) instead of always falling back to Spark. Aggregate UDFs, table
+UDFs/generators, Python/Pandas UDFs, and Hive `GenericUDF`/`SimpleUDF` still fall back to Spark entirely.
+Extending the codegen-dispatch approach to cover these remaining categories will reduce fallbacks further and
+allow more queries to run end-to-end in Comet.
## Memory Management Improvements
Comet coordinates memory between the JVM and native Rust execution through a custom memory pool. Improving
memory accounting, reservation strategies, and spill integration will reduce out-of-memory errors and allow
Comet to make better use of available resources, especially in multi-query and multi-task environments.
-## Prepare for 1.0.0 Release
-
-The project is working toward a 1.0.0 release. This effort includes finalizing configuration options,
-resolving known correctness issues, and improving documentation. Progress is tracked in [#4082].
-
-[#4082]: https://github.com/apache/datafusion-comet/issues/4082
-
## Native Parquet Writes
Review Comment:
Should we mention native Iceberg writes? Delta support?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: update post 1.0.0 roadmap [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #5064:
URL: https://github.com/apache/datafusion-comet/pull/5064#discussion_r3666583434
##########
docs/source/contributor-guide/roadmap.md:
##########
@@ -41,25 +41,28 @@ supported functions, frames, and fallback cases.
[#4836]: https://github.com/apache/datafusion-comet/issues/4836
[#4837]: https://github.com/apache/datafusion-comet/issues/4837
-## Lambda Expressions
+## Native Lambda Evaluation
Review Comment:
Should we add a section about implementing other types of expression natively (scalar and aggregate)? We rely on codegen dispatch for many expressions today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: update post 1.0.0 roadmap [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich merged PR #5064:
URL: https://github.com/apache/datafusion-comet/pull/5064
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: update post 1.0.0 roadmap [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #5064:
URL: https://github.com/apache/datafusion-comet/pull/5064#discussion_r3666975917
##########
docs/source/contributor-guide/roadmap.md:
##########
@@ -85,25 +88,20 @@ Comet's native hash join currently requires the build side to fit entirely in me
support will allow Comet to handle larger joins without falling back to Spark, improving both reliability and
performance for memory-intensive workloads.
-## Java/Scala Columnar and Arrow UDF Support
+## Java/Scala UDF Support
-Spark users frequently define custom UDFs in Java or Scala. Comet currently falls back to Spark when a query
-contains a JVM UDF. Adding support for calling Java/Scala UDFs that operate on columnar Arrow data directly
-from native execution will reduce fallbacks and allow more queries to run end-to-end in Comet.
+Spark users frequently define custom UDFs in Java or Scala. Comet now dispatches scalar `ScalaUDF` expressions
+through a JVM codegen bridge (`CometScalaUDF`) instead of always falling back to Spark. Aggregate UDFs, table
+UDFs/generators, Python/Pandas UDFs, and Hive `GenericUDF`/`SimpleUDF` still fall back to Spark entirely.
+Extending the codegen-dispatch approach to cover these remaining categories will reduce fallbacks further and
+allow more queries to run end-to-end in Comet.
## Memory Management Improvements
Comet coordinates memory between the JVM and native Rust execution through a custom memory pool. Improving
memory accounting, reservation strategies, and spill integration will reduce out-of-memory errors and allow
Comet to make better use of available resources, especially in multi-query and multi-task environments.
-## Prepare for 1.0.0 Release
-
-The project is working toward a 1.0.0 release. This effort includes finalizing configuration options,
-resolving known correctness issues, and improving documentation. Progress is tracked in [#4082].
-
-[#4082]: https://github.com/apache/datafusion-comet/issues/4082
-
## Native Parquet Writes
Review Comment:
also Iceberg HDFS reads would be interesting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: update post 1.0.0 roadmap [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #5064:
URL: https://github.com/apache/datafusion-comet/pull/5064#discussion_r3666576048
##########
docs/source/contributor-guide/roadmap.md:
##########
@@ -41,25 +41,28 @@ supported functions, frames, and fallback cases.
[#4836]: https://github.com/apache/datafusion-comet/issues/4836
[#4837]: https://github.com/apache/datafusion-comet/issues/4837
-## Lambda Expressions
+## Native Lambda Evaluation
Spark supports higher-order functions on arrays and maps that take a lambda, including `transform`, `exists`,
-`forall`, `aggregate`, `zip_with`, `map_filter`, and `map_zip_with`. Comet currently lacks a general mechanism
-for serializing lambda expressions and evaluating them in DataFusion. Adding this capability will unlock a
-significant family of Spark expressions in one effort.
-
-## Dynamic Partition Pruning
-
-Native Parquet scans (`CometNativeScanExec`) support Dynamic Partition Pruning (DPP) both with and without
-Adaptive Query Execution. Non-AQE DPP landed in [#4011] and AQE DPP with broadcast reuse landed in [#4112].
-Iceberg native scans support non-AQE DPP ([#3349], [#3511]) and, on Spark 3.5+, AQE DPP with broadcast reuse
-([#4215]); on Spark 3.4 Iceberg AQE DPP falls back to Spark without reuse.
-
-[#3349]: https://github.com/apache/datafusion-comet/pull/3349
-[#3511]: https://github.com/apache/datafusion-comet/pull/3511
-[#4011]: https://github.com/apache/datafusion-comet/pull/4011
-[#4112]: https://github.com/apache/datafusion-comet/pull/4112
-[#4215]: https://github.com/apache/datafusion-comet/pull/4215
+`forall`, `aggregate`, `zip_with`, `map_filter`, and `map_zip_with`. Comet evaluates these today through a JVM
+codegen-dispatch bridge (`CometScalaUDF`, `CometBatchKernelCodegen`) instead of falling back to Spark, but the
+lambda body is still interpreted row-at-a-time on the JVM rather than natively in DataFusion. Moving evaluation
+into native Rust code would remove that JVM round-trip and let these expressions benefit from vectorized native
+execution.
+
+## Iceberg Table Format V3 Support
+
+Comet landed its first Iceberg table format V3 feature (native data file decryption, [#4991]), and we want to
+add more V3 features to Comet's native Iceberg scans so they don't fall back to Spark. The work is tracked
+phase-by-phase in [#3376]: detecting the V3 format and supporting new V3 data types, deletion vector reads, row
+lineage, and table encryption. Native deletion vector reads are prototyped in a draft PR ([#4887]), but are
+blocked on upstream `iceberg-rust` support, tracked in [iceberg-rust #2792] and [iceberg-rust #2411].
+
+[#3376]: https://github.com/apache/datafusion-comet/issues/3376
+[#4887]: https://github.com/apache/datafusion-comet/pull/4887
+[#4991]: https://github.com/apache/datafusion-comet/pull/4991
+[iceberg-rust #2411]: https://github.com/apache/iceberg-rust/issues/2411
+[iceberg-rust #2792]: https://github.com/apache/iceberg-rust/issues/2792
## TPC-H and TPC-DS Performance
Review Comment:
I think we could rewrite the TPC performance section to make it more positive. We can point out that current performance is already quite good, and share results from AWS Labs and then talk about ongoing optimization work?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org