You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/04/30 01:03:25 UTC
(datafusion-comet) branch asf-site updated: Publish built docs triggered by 4d01ce153ce920c0d20ab156f2bdbc7234c40549
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 09e3daa90 Publish built docs triggered by 4d01ce153ce920c0d20ab156f2bdbc7234c40549
09e3daa90 is described below
commit 09e3daa90e851bfc7e0bfd5c47a81c51d595f924
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Thu Apr 30 01:03:22 2026 +0000
Publish built docs triggered by 4d01ce153ce920c0d20ab156f2bdbc7234c40549
---
_sources/contributor-guide/roadmap.md.txt | 59 ++++++++++++++++++++-----------
contributor-guide/index.html | 8 +++--
contributor-guide/roadmap.html | 51 +++++++++++++++-----------
searchindex.js | 2 +-
4 files changed, 76 insertions(+), 44 deletions(-)
diff --git a/_sources/contributor-guide/roadmap.md.txt b/_sources/contributor-guide/roadmap.md.txt
index 3f9fc7eeb..fd376655f 100644
--- a/_sources/contributor-guide/roadmap.md.txt
+++ b/_sources/contributor-guide/roadmap.md.txt
@@ -22,25 +22,26 @@ under the License.
Comet is an open-source project and contributors are welcome to work on any issues at any time, but we find it
helpful to have a roadmap for some of the major items that require coordination between contributors.
-## Major Initiatives
+## Window Expressions
-### Iceberg Integration
+Native window execution is currently disabled by default due to known correctness issues ([#2721], [#2841]).
+In addition, dedicated window functions such as `rank`, `dense_rank`, `row_number`, `lag`, `lead`, `ntile`,
+`cume_dist`, `percent_rank`, and `nth_value` are not yet implemented and fall back to Spark ([#2705]). The
+goal is to enable windowed aggregates by default ([#4007]) and add the missing dedicated window functions.
-Reads of Iceberg tables with Parquet data files are fully native and enabled by default, powered by a scan operator
-backed by Iceberg-rust ([#2528]). We anticipate major improvements in the next few releases, including bringing Iceberg table format V3 features (_e.g._,
-encryption) to the reader.
+[#2705]: https://github.com/apache/datafusion-comet/issues/2705
+[#2721]: https://github.com/apache/datafusion-comet/issues/2721
+[#2841]: https://github.com/apache/datafusion-comet/issues/2841
+[#4007]: https://github.com/apache/datafusion-comet/issues/4007
-[#2528]: https://github.com/apache/datafusion-comet/pull/2528
+## Lambda Expressions
-### Spark 4.0 Support
+Spark supports higher-order functions on arrays and maps that take a lambda, including `transform`, `exists`,
+`forall`, `aggregate`, `zip_with`, `map_filter`, and `map_zip_with`. Comet currently lacks a general mechanism
+for serializing lambda expressions and evaluating them in DataFusion. Adding this capability will unlock a
+significant family of Spark expressions in one effort.
-Comet has experimental support for Spark 4.0, but there is more work to do ([#1637]), such as enabling
-more Spark SQL tests and fully implementing ANSI support ([#313]) for all supported expressions.
-
-[#313]: https://github.com/apache/datafusion-comet/issues/313
-[#1637]: https://github.com/apache/datafusion-comet/issues/1637
-
-### Dynamic Partition Pruning
+## Dynamic Partition Pruning
Both Iceberg table scans and Parquet V1 native scans (`CometNativeScanExec`) support non-AQE Dynamic Partition Pruning
(DPP) filters generated by Spark's `PlanDynamicPruningFilters` optimizer rule ([#3349], [#3511]). However, Spark's
@@ -51,11 +52,29 @@ requires a redesign of Comet's plan translation. This effort can be tracked at [
[#3510]: https://github.com/apache/datafusion-comet/issues/3510
[#3511]: https://github.com/apache/datafusion-comet/pull/3511
-## Ongoing Improvements
+## TPC-H and TPC-DS Performance
+
+We regularly publish benchmark results derived from TPC-H and TPC-DS to track performance against Spark. Closing
+the remaining gaps and increasing the speedup on both benchmark suites is an ongoing focus, tracked under [#2004]
+(TPC-H), [#858] (TPC-DS), and [#3799] (improving the awslabs published TPC-DS results).
+
+[#858]: https://github.com/apache/datafusion-comet/issues/858
+[#2004]: https://github.com/apache/datafusion-comet/issues/2004
+[#3799]: https://github.com/apache/datafusion-comet/issues/3799
+
+## Upstream Work in DataFusion
+
+A growing number of Spark-compatible expressions live in the `datafusion-spark` crate in the core DataFusion
+repository. Comet is migrating its expression implementations to that crate so that they can be shared by other
+DataFusion-based projects, tracked in [#2084]. Improvements to core DataFusion operators (joins, aggregates,
+window) made in support of Comet also benefit the wider ecosystem.
+
+[#2084]: https://github.com/apache/datafusion-comet/issues/2084
+
+## Native Parquet Writes
-In addition to the major initiatives above, we have the following ongoing areas of work:
+Comet has experimental support for native Parquet writes via `InsertIntoHadoopFsRelationCommand`, currently
+disabled by default. The goal is to reach correctness and performance parity with Spark's writer so it can be
+enabled by default ([#1625]).
-- Adding support for more Spark expressions
-- Moving more expressions to the `datafusion-spark` crate in the core DataFusion repository
-- Performance tuning
-- Nested type support improvements
+[#1625]: https://github.com/apache/datafusion-comet/issues/1625
diff --git a/contributor-guide/index.html b/contributor-guide/index.html
index 8f3914db6..c7ee87c56 100644
--- a/contributor-guide/index.html
+++ b/contributor-guide/index.html
@@ -623,8 +623,12 @@ under the License.
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="roadmap.html">Roadmap</a><ul>
-<li class="toctree-l2"><a class="reference internal" href="roadmap.html#major-initiatives">Major Initiatives</a></li>
-<li class="toctree-l2"><a class="reference internal" href="roadmap.html#ongoing-improvements">Ongoing Improvements</a></li>
+<li class="toctree-l2"><a class="reference internal" href="roadmap.html#window-expressions">Window Expressions</a></li>
+<li class="toctree-l2"><a class="reference internal" href="roadmap.html#lambda-expressions">Lambda Expressions</a></li>
+<li class="toctree-l2"><a class="reference internal" href="roadmap.html#dynamic-partition-pruning">Dynamic Partition Pruning</a></li>
+<li class="toctree-l2"><a class="reference internal" href="roadmap.html#tpc-h-and-tpc-ds-performance">TPC-H and TPC-DS Performance</a></li>
+<li class="toctree-l2"><a class="reference internal" href="roadmap.html#upstream-work-in-datafusion">Upstream Work in DataFusion</a></li>
+<li class="toctree-l2"><a class="reference internal" href="roadmap.html#native-parquet-writes">Native Parquet Writes</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="release_process.html">Release Process</a><ul>
diff --git a/contributor-guide/roadmap.html b/contributor-guide/roadmap.html
index f137dde77..b175853cd 100644
--- a/contributor-guide/roadmap.html
+++ b/contributor-guide/roadmap.html
@@ -461,36 +461,45 @@ under the License.
<h1>Comet Roadmap<a class="headerlink" href="#comet-roadmap" title="Link to this heading">#</a></h1>
<p>Comet is an open-source project and contributors are welcome to work on any issues at any time, but we find it
helpful to have a roadmap for some of the major items that require coordination between contributors.</p>
-<section id="major-initiatives">
-<h2>Major Initiatives<a class="headerlink" href="#major-initiatives" title="Link to this heading">#</a></h2>
-<section id="iceberg-integration">
-<h3>Iceberg Integration<a class="headerlink" href="#iceberg-integration" title="Link to this heading">#</a></h3>
-<p>Reads of Iceberg tables with Parquet data files are fully native and enabled by default, powered by a scan operator
-backed by Iceberg-rust (<a class="reference external" href="https://github.com/apache/datafusion-comet/pull/2528">#2528</a>). We anticipate major improvements in the next few releases, including bringing Iceberg table format V3 features (<em>e.g.</em>,
-encryption) to the reader.</p>
+<section id="window-expressions">
+<h2>Window Expressions<a class="headerlink" href="#window-expressions" title="Link to this heading">#</a></h2>
+<p>Native window execution is currently disabled by default due to known correctness issues (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/2721">#2721</a>, <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/2841">#2841</a>).
+In addition, dedicated window functions such as <code class="docutils literal notranslate"><span class="pre">rank</span></code>, <code class="docutils literal notranslate"><span class="pre">dense_rank</span></code>, <code class="docutils literal notranslate"><span class="pre">row_number</span></code>, <code class="docutils literal notranslate"><span class="pre">lag</span></code>, <code class="docutils literal notranslate"><span class="pre">lead</span></code>, <code class="docutils litera [...]
+<code class="docutils literal notranslate"><span class="pre">cume_dist</span></code>, <code class="docutils literal notranslate"><span class="pre">percent_rank</span></code>, and <code class="docutils literal notranslate"><span class="pre">nth_value</span></code> are not yet implemented and fall back to Spark (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/2705">#2705</a>). The
+goal is to enable windowed aggregates by default (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4007">#4007</a>) and add the missing dedicated window functions.</p>
</section>
-<section id="spark-4-0-support">
-<h3>Spark 4.0 Support<a class="headerlink" href="#spark-4-0-support" title="Link to this heading">#</a></h3>
-<p>Comet has experimental support for Spark 4.0, but there is more work to do (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/1637">#1637</a>), such as enabling
-more Spark SQL tests and fully implementing ANSI support (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/313">#313</a>) for all supported expressions.</p>
+<section id="lambda-expressions">
+<h2>Lambda Expressions<a class="headerlink" href="#lambda-expressions" title="Link to this heading">#</a></h2>
+<p>Spark supports higher-order functions on arrays and maps that take a lambda, including <code class="docutils literal notranslate"><span class="pre">transform</span></code>, <code class="docutils literal notranslate"><span class="pre">exists</span></code>,
+<code class="docutils literal notranslate"><span class="pre">forall</span></code>, <code class="docutils literal notranslate"><span class="pre">aggregate</span></code>, <code class="docutils literal notranslate"><span class="pre">zip_with</span></code>, <code class="docutils literal notranslate"><span class="pre">map_filter</span></code>, and <code class="docutils literal notranslate"><span class="pre">map_zip_with</span></code>. Comet currently lacks a general mechanism
+for serializing lambda expressions and evaluating them in DataFusion. Adding this capability will unlock a
+significant family of Spark expressions in one effort.</p>
</section>
<section id="dynamic-partition-pruning">
-<h3>Dynamic Partition Pruning<a class="headerlink" href="#dynamic-partition-pruning" title="Link to this heading">#</a></h3>
+<h2>Dynamic Partition Pruning<a class="headerlink" href="#dynamic-partition-pruning" title="Link to this heading">#</a></h2>
<p>Both Iceberg table scans and Parquet V1 native scans (<code class="docutils literal notranslate"><span class="pre">CometNativeScanExec</span></code>) support non-AQE Dynamic Partition Pruning
(DPP) filters generated by Spark’s <code class="docutils literal notranslate"><span class="pre">PlanDynamicPruningFilters</span></code> optimizer rule (<a class="reference external" href="https://github.com/apache/datafusion-comet/pull/3349">#3349</a>, <a class="reference external" href="https://github.com/apache/datafusion-comet/pull/3511">#3511</a>). However, Spark’s
<code class="docutils literal notranslate"><span class="pre">PlanAdaptiveDynamicPruningFilters</span></code> optimizer rule runs after Comet’s rules, so DPP with Adaptive Query Execution
requires a redesign of Comet’s plan translation. This effort can be tracked at <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/3510">#3510</a>.</p>
</section>
+<section id="tpc-h-and-tpc-ds-performance">
+<h2>TPC-H and TPC-DS Performance<a class="headerlink" href="#tpc-h-and-tpc-ds-performance" title="Link to this heading">#</a></h2>
+<p>We regularly publish benchmark results derived from TPC-H and TPC-DS to track performance against Spark. Closing
+the remaining gaps and increasing the speedup on both benchmark suites is an ongoing focus, tracked under <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/2004">#2004</a>
+(TPC-H), <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/858">#858</a> (TPC-DS), and <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/3799">#3799</a> (improving the awslabs published TPC-DS results).</p>
</section>
-<section id="ongoing-improvements">
-<h2>Ongoing Improvements<a class="headerlink" href="#ongoing-improvements" title="Link to this heading">#</a></h2>
-<p>In addition to the major initiatives above, we have the following ongoing areas of work:</p>
-<ul class="simple">
-<li><p>Adding support for more Spark expressions</p></li>
-<li><p>Moving more expressions to the <code class="docutils literal notranslate"><span class="pre">datafusion-spark</span></code> crate in the core DataFusion repository</p></li>
-<li><p>Performance tuning</p></li>
-<li><p>Nested type support improvements</p></li>
-</ul>
+<section id="upstream-work-in-datafusion">
+<h2>Upstream Work in DataFusion<a class="headerlink" href="#upstream-work-in-datafusion" title="Link to this heading">#</a></h2>
+<p>A growing number of Spark-compatible expressions live in the <code class="docutils literal notranslate"><span class="pre">datafusion-spark</span></code> crate in the core DataFusion
+repository. Comet is migrating its expression implementations to that crate so that they can be shared by other
+DataFusion-based projects, tracked in <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/2084">#2084</a>. Improvements to core DataFusion operators (joins, aggregates,
+window) made in support of Comet also benefit the wider ecosystem.</p>
+</section>
+<section id="native-parquet-writes">
+<h2>Native Parquet Writes<a class="headerlink" href="#native-parquet-writes" title="Link to this heading">#</a></h2>
+<p>Comet has experimental support for native Parquet writes via <code class="docutils literal notranslate"><span class="pre">InsertIntoHadoopFsRelationCommand</span></code>, currently
+disabled by default. The goal is to reach correctness and performance parity with Spark’s writer so it can be
+enabled by default (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/1625">#1625</a>).</p>
</section>
</section>
diff --git a/searchindex.js b/searchindex.js
index 6e1940beb..e9a779d2f 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"1. Format Your Code": [[14, "format-your-code"]], "1. Install Comet": [[16, "install-comet"], [24, "install-comet"]], "1. Native Operators (nativeExecs map)": [[4, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[14, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[16, "clone-iceberg-and-apply-diff"]], "2. Clone Spark and Apply Diff": [[24, "clone-spark-and-apply-diff"]], "2. Sink Operators (sinks map)": [[4, "sink-operators-sinks-m [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"1. Format Your Code": [[14, "format-your-code"]], "1. Install Comet": [[16, "install-comet"], [24, "install-comet"]], "1. Native Operators (nativeExecs map)": [[4, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[14, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[16, "clone-iceberg-and-apply-diff"]], "2. Clone Spark and Apply Diff": [[24, "clone-spark-and-apply-diff"]], "2. Sink Operators (sinks map)": [[4, "sink-operators-sinks-m [...]
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org