You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/09/30 12:58:05 UTC
(datafusion-comet) branch asf-site updated: Publish built docs triggered by cb2643bbfffdd41b1d85794fd4bd8b69b1faa576
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new eb7b17f8a7 Publish built docs triggered by cb2643bbfffdd41b1d85794fd4bd8b69b1faa576
eb7b17f8a7 is described below
commit eb7b17f8a7a5f4fc36e95d045542811f07e23adc
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Wed Sep 30 12:58:00 2026 +0000
Publish built docs triggered by cb2643bbfffdd41b1d85794fd4bd8b69b1faa576
---
...id-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg | 2 +-
_sources/contributor-guide/roadmap.md.txt | 88 +++++++++++++---------
contributor-guide/roadmap.html | 68 +++++++++--------
searchindex.js | 2 +-
4 files changed, 90 insertions(+), 70 deletions(-)
diff --git a/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg b/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg
index c6d3334b31..89234363d2 100644
--- a/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg
+++ b/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg
@@ -1 +1 @@
-<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 966" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
+<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 966" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
diff --git a/_sources/contributor-guide/roadmap.md.txt b/_sources/contributor-guide/roadmap.md.txt
index b5aa6be16f..d318c07a53 100644
--- a/_sources/contributor-guide/roadmap.md.txt
+++ b/_sources/contributor-guide/roadmap.md.txt
@@ -53,30 +53,29 @@ these expressions benefit from vectorized native execution.
## Native Coverage for Codegen-Dispatched Expressions
Beyond lambda bodies, a number of built-in Spark scalar expressions (some regular expression, JSON, and datetime
-functions, for example) and aggregate functions route through the same JVM codegen-dispatch bridge by default,
-either because their native DataFusion or `datafusion-spark` implementation has known semantic differences from
-Spark, or because no native implementation exists yet. See the [compatibility guide] for the current list of
-codegen-dispatched expressions. We're exploring closing these gaps so that more expressions run natively by
-default, which would reduce JVM round-trips beyond what the lambda and UDF work above already covers.
+functions, for example) route through the same JVM codegen-dispatch bridge by default, either because their native
+DataFusion or `datafusion-spark` implementation has known semantic differences from Spark, or because no native
+implementation exists yet. The [expression reference] records which expressions are codegen-dispatched today. The
+bridge covers only scalar expressions, so an aggregate function with no Spark-compatible native implementation falls
+back to Spark instead. We're exploring closing these gaps so that more expressions run natively by default, which
+would remove JVM round-trips beyond those the lambda work above addresses.
-[compatibility guide]: ../user-guide/latest/compatibility/index.md
+[expression reference]: ../user-guide/latest/expressions.md
## Iceberg Table Format V3 Support
-Comet landed its first Iceberg table format V3 feature (native data file decryption, [#4991]), and we want to
-add more V3 features to Comet's native Iceberg scans so they don't fall back to Spark. The work is tracked
-phase-by-phase in [#3376]: detecting the V3 format and supporting new V3 data types, deletion vector reads, row
-lineage, and table encryption. Native deletion vector reads are prototyped in a draft PR ([#4887]), but are
-blocked on upstream `iceberg-rust` support, tracked in [iceberg-rust #2792] and [iceberg-rust #2411]. Native Iceberg
-scans don't support HDFS-backed tables today: the storage scheme match in `iceberg_scan.rs` only handles `file`,
-`s3`/`s3a`, `gs`, and `oss`, and would need an HDFS `StorageFactory` upstream in `iceberg-storage-opendal`. We're
-scoping what that work would take.
+Comet's native Iceberg scans read V3 tables, including encrypted tables ([#4991]) and tables with deletion vectors
+([#5853]). We want to add the remaining V3 features so that these scans don't fall back to Spark: row lineage
+metadata columns, column default values, and the new V3 types (`variant`, `geometry`, `geography`, and `unknown`).
+The work is tracked in [#3376], and upstream `iceberg-rust` support in [iceberg-rust #2411]. Native Iceberg scans
+also don't support HDFS-backed tables today: Comet's native Iceberg storage layer handles only local files, S3 and
+S3-compatible stores, GCS, and OSS, and would need an HDFS `StorageFactory` upstream in `iceberg-storage-opendal`.
+We're scoping what that work would take.
[#3376]: https://github.com/apache/datafusion-comet/issues/3376
-[#4887]: https://github.com/apache/datafusion-comet/pull/4887
[#4991]: https://github.com/apache/datafusion-comet/pull/4991
+[#5853]: https://github.com/apache/datafusion-comet/pull/5853
[iceberg-rust #2411]: https://github.com/apache/iceberg-rust/issues/2411
-[iceberg-rust #2792]: https://github.com/apache/iceberg-rust/issues/2792
## TPC-H and TPC-DS Performance
@@ -84,18 +83,18 @@ Comet already delivers substantial speedups over vanilla Spark on TPC-H and TPC-
results for [TPC-DS] with each release. An independent [AWS Labs benchmark] comparing Comet 0.16.0 with
Gluten 1.6.0 on a 3TB TPC-DS workload found that the two accelerators deliver similar overall performance. Increasing
the speedup further and closing the remaining per-query gaps is an ongoing focus, tracked under [#2004] (TPC-H) and
-[#858] (TPC-DS).
+[#2551] (TPC-DS).
[TPC-DS]: benchmark-results/tpc-ds.md
[AWS Labs benchmark]: https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-gluten-velox-comet-benchmark
-[#858]: https://github.com/apache/datafusion-comet/issues/858
[#2004]: https://github.com/apache/datafusion-comet/issues/2004
+[#2551]: https://github.com/apache/datafusion-comet/issues/2551
## Upstream Work in DataFusion
A growing number of Spark-compatible expressions live in the `datafusion-spark` crate in the core DataFusion
repository. Comet is migrating its expression implementations to that crate so that they can be shared by other
-DataFusion-based projects, and is wiring up the functions that crate already provides ([#4150]). Improvements to
+DataFusion-based projects, and has wired up nearly every function that crate provides ([#4150]). Improvements to
core DataFusion operators (joins, aggregates, window) made in support of Comet also benefit the wider ecosystem.
[#4150]: https://github.com/apache/datafusion-comet/issues/4150
@@ -103,17 +102,25 @@ core DataFusion operators (joins, aggregates, window) made in support of Comet a
## Spillable Hash Join
Comet's native hash join currently requires the build side to fit entirely in memory. Adding spill-to-disk
-support will allow Comet to handle larger joins without falling back to Spark, improving both reliability and
-performance for memory-intensive workloads.
+support ([#2545]) will allow Comet to handle larger joins without falling back to Spark, improving both reliability
+and performance for memory-intensive workloads. Comet's native hash join uses DataFusion's `HashJoinExec`, and a
+design for spilling in that operator, behind a flag that is off by default, is proposed upstream in
+[datafusion #24768].
+
+[#2545]: https://github.com/apache/datafusion-comet/issues/2545
+[datafusion #24768]: https://github.com/apache/datafusion/issues/24768
## Java/Scala UDF Support
Spark users frequently define custom UDFs in Java or Scala. Comet now dispatches scalar `ScalaUDF` expressions
through a JVM codegen bridge (`CometScalaUDF`) instead of always falling back to Spark. Aggregate UDFs, table
-UDFs/generators, Python/Pandas UDFs, and Hive `GenericUDF`/`SimpleUDF` still fall back to Spark entirely.
+UDFs/generators, Hive `GenericUDF`/`SimpleUDF`, and Python UDFs other than `mapInArrow` and `mapInPandas` still fall
+back to Spark entirely; Comet's support for those two Python APIs is experimental and disabled by default ([#4234]).
Extending the codegen-dispatch approach to cover these remaining categories will reduce fallbacks further and
allow more queries to run end-to-end in Comet.
+[#4234]: https://github.com/apache/datafusion-comet/pull/4234
+
## Memory Management Improvements
Comet coordinates memory between the JVM and native Rust execution through a custom memory pool. Improving
@@ -130,30 +137,37 @@ enabled by default ([#1625]).
## Iceberg Table Writes
-Comet's native scans accelerate Iceberg reads today, but writes still run entirely through Spark's Iceberg V2 write
-operator. [#4322] tracks exploring native acceleration of Iceberg writes, so an ETL job could run end to end in
-native code, falling back only where Comet can't match Iceberg Java's functionality. Draft proposals split the
-existing V2 write operator into separate "writer" and "committer" operators so the data-file-write portion can be
-planned and accelerated the same way as a native Parquet write ([#4658], [#4487]). No design here is committed yet;
-we're exploring whether this approach is worth pursuing.
+Comet can now write Iceberg tables natively. The feature is experimental, disabled by default, and controlled by two
+settings, the second of which requires the first. The first splits Spark's Iceberg V2 write operator into separate
+writer and committer operators, so the query feeding the write becomes visible to AQE and to Comet's columnar rules
+([#4658]). The second delegates each task's Parquet write to `iceberg-rust` when the write passes an eligibility
+check ([#5361]); writes that don't pass keep Iceberg Java's writer. Merge-on-read writes are not intercepted yet
+([#6240]). The remaining work toward the original goal of [#4322], an ETL job that runs end to end in native code, is
+tracked in [#5649]: correctness fixes, failure handling that matches Iceberg Java, broader coverage, and enabling
+both settings by default ([#5644]). See [Iceberg Writes](iceberg-writes.md) for how the write path works.
[#4322]: https://github.com/apache/datafusion-comet/issues/4322
-[#4487]: https://github.com/apache/datafusion-comet/pull/4487
[#4658]: https://github.com/apache/datafusion-comet/pull/4658
+[#5361]: https://github.com/apache/datafusion-comet/pull/5361
+[#5644]: https://github.com/apache/datafusion-comet/issues/5644
+[#5649]: https://github.com/apache/datafusion-comet/issues/5649
+[#6240]: https://github.com/apache/datafusion-comet/issues/6240
## Delta Lake Support
-Comet currently supports Spark's built-in file formats and Iceberg, but not Delta Lake ([#174]). A draft PR reuses
-Comet's existing native Parquet reader to accelerate scans of plain Delta tables that use neither deletion vectors
-nor column mapping ([#4669]); a separate, larger effort explores a full native Delta scan built on `delta-kernel-rs`
-as an out-of-tree contrib module ([#4366]), landing in inert, gated slices ([#4700], [#4952]). Generalizing Comet's
-scan-side APIs so that Delta and other non-Iceberg data sources can plug in more easily is tracked as a Table
-Provider API abstraction ([#4706]). None of this is committed yet; we're exploring which approach, if any, is worth
-pursuing.
+Comet currently supports Spark's built-in file formats and Iceberg, but not Delta Lake ([#174]). The plugin boundary
+for an out-of-tree Delta module has landed in inert, gated slices ([#4700], [#4952]), and two read paths build on
+it. In the first, which is in review, delta-spark plans the scan and Comet's native Parquet reader executes it,
+applying deletion vectors inside the scan; it ships as an opt-in contrib module ([#5365]). The second, still a
+draft, is a full native Delta scan built on `delta-kernel-rs` ([#4366]). [#5411] proposes converging the two into
+one plugin, with the Parquet path as the default and the kernel path covering what it can't express yet, such as
+change data feed. Generalizing Comet's scan-side APIs so that Delta and other non-Iceberg data sources can plug in
+more easily is tracked as a Table Provider API abstraction ([#4706]).
[#174]: https://github.com/apache/datafusion-comet/issues/174
[#4366]: https://github.com/apache/datafusion-comet/pull/4366
-[#4669]: https://github.com/apache/datafusion-comet/pull/4669
[#4700]: https://github.com/apache/datafusion-comet/pull/4700
[#4706]: https://github.com/apache/datafusion-comet/issues/4706
[#4952]: https://github.com/apache/datafusion-comet/pull/4952
+[#5365]: https://github.com/apache/datafusion-comet/pull/5365
+[#5411]: https://github.com/apache/datafusion-comet/issues/5411
diff --git a/contributor-guide/roadmap.html b/contributor-guide/roadmap.html
index 56562585ec..788fe8c942 100644
--- a/contributor-guide/roadmap.html
+++ b/contributor-guide/roadmap.html
@@ -579,22 +579,22 @@ these expressions benefit from vectorized native execution.</p>
<section id="native-coverage-for-codegen-dispatched-expressions">
<h2>Native Coverage for Codegen-Dispatched Expressions<a class="headerlink" href="#native-coverage-for-codegen-dispatched-expressions" title="Link to this heading">#</a></h2>
<p>Beyond lambda bodies, a number of built-in Spark scalar expressions (some regular expression, JSON, and datetime
-functions, for example) and aggregate functions route through the same JVM codegen-dispatch bridge by default,
-either because their native DataFusion or <code class="docutils literal notranslate"><span class="pre">datafusion-spark</span></code> implementation has known semantic differences from
-Spark, or because no native implementation exists yet. See the <a class="reference internal" href="../user-guide/latest/compatibility/index.html"><span class="std std-doc">compatibility guide</span></a> for the current list of
-codegen-dispatched expressions. We’re exploring closing these gaps so that more expressions run natively by
-default, which would reduce JVM round-trips beyond what the lambda and UDF work above already covers.</p>
+functions, for example) route through the same JVM codegen-dispatch bridge by default, either because their native
+DataFusion or <code class="docutils literal notranslate"><span class="pre">datafusion-spark</span></code> implementation has known semantic differences from Spark, or because no native
+implementation exists yet. The <a class="reference internal" href="../user-guide/latest/expressions.html"><span class="std std-doc">expression reference</span></a> records which expressions are codegen-dispatched today. The
+bridge covers only scalar expressions, so an aggregate function with no Spark-compatible native implementation falls
+back to Spark instead. We’re exploring closing these gaps so that more expressions run natively by default, which
+would remove JVM round-trips beyond those the lambda work above addresses.</p>
</section>
<section id="iceberg-table-format-v3-support">
<h2>Iceberg Table Format V3 Support<a class="headerlink" href="#iceberg-table-format-v3-support" title="Link to this heading">#</a></h2>
-<p>Comet landed its first Iceberg table format V3 feature (native data file decryption, <a class="reference external" href="https://github.com/apache/datafusion-comet/pull/4991">#4991</a>), and we want to
-add more V3 features to Comet’s native Iceberg scans so they don’t fall back to Spark. The work is tracked
-phase-by-phase in <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/3376">#3376</a>: detecting the V3 format and supporting new V3 data types, deletion vector reads, row
-lineage, and table encryption. Native deletion vector reads are prototyped in a draft PR (<a class="reference external" href="https://github.com/apache/datafusion-comet/pull/4887">#4887</a>), but are
-blocked on upstream <code class="docutils literal notranslate"><span class="pre">iceberg-rust</span></code> support, tracked in <a class="reference external" href="https://github.com/apache/iceberg-rust/issues/2792">iceberg-rust #2792</a> and <a class="reference external" href="https://github.com/apache/iceberg-rust/issues/2411">iceberg-rust #2411</a>. Native Iceberg
-scans don’t support HDFS-backed tables today: the storage scheme match in <code class="docutils literal notranslate"><span class="pre">iceberg_scan.rs</span></code> only handles <code class="docutils literal notranslate"><span class="pre">file</span></code>,
-<code class="docutils literal notranslate"><span class="pre">s3</span></code>/<code class="docutils literal notranslate"><span class="pre">s3a</span></code>, <code class="docutils literal notranslate"><span class="pre">gs</span></code>, and <code class="docutils literal notranslate"><span class="pre">oss</span></code>, and would need an HDFS <code class="docutils literal notranslate"><span class="pre">StorageFactory</span></code> upstream in <code class="docutils literal notranslate"><sp [...]
-scoping what that work would take.</p>
+<p>Comet’s native Iceberg scans read V3 tables, including encrypted tables (<a class="reference external" href="https://github.com/apache/datafusion-comet/pull/4991">#4991</a>) and tables with deletion vectors
+(<a class="reference external" href="https://github.com/apache/datafusion-comet/pull/5853">#5853</a>). We want to add the remaining V3 features so that these scans don’t fall back to Spark: row lineage
+metadata columns, column default values, and the new V3 types (<code class="docutils literal notranslate"><span class="pre">variant</span></code>, <code class="docutils literal notranslate"><span class="pre">geometry</span></code>, <code class="docutils literal notranslate"><span class="pre">geography</span></code>, and <code class="docutils literal notranslate"><span class="pre">unknown</span></code>).
+The work is tracked in <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/3376">#3376</a>, and upstream <code class="docutils literal notranslate"><span class="pre">iceberg-rust</span></code> support in <a class="reference external" href="https://github.com/apache/iceberg-rust/issues/2411">iceberg-rust #2411</a>. Native Iceberg scans
+also don’t support HDFS-backed tables today: Comet’s native Iceberg storage layer handles only local files, S3 and
+S3-compatible stores, GCS, and OSS, and would need an HDFS <code class="docutils literal notranslate"><span class="pre">StorageFactory</span></code> upstream in <code class="docutils literal notranslate"><span class="pre">iceberg-storage-opendal</span></code>.
+We’re scoping what that work would take.</p>
</section>
<section id="tpc-h-and-tpc-ds-performance">
<h2>TPC-H and TPC-DS Performance<a class="headerlink" href="#tpc-h-and-tpc-ds-performance" title="Link to this heading">#</a></h2>
@@ -602,26 +602,29 @@ scoping what that work would take.</p>
results for <a class="reference internal" href="benchmark-results/tpc-ds.html"><span class="std std-doc">TPC-DS</span></a> with each release. An independent <a class="reference external" href="https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-gluten-velox-comet-benchmark">AWS Labs benchmark</a> comparing Comet 0.16.0 with
Gluten 1.6.0 on a 3TB TPC-DS workload found that the two accelerators deliver similar overall performance. Increasing
the speedup further and closing the remaining per-query gaps is an ongoing focus, tracked under <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/2004">#2004</a> (TPC-H) and
-<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/858">#858</a> (TPC-DS).</p>
+<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/2551">#2551</a> (TPC-DS).</p>
</section>
<section id="upstream-work-in-datafusion">
<h2>Upstream Work in DataFusion<a class="headerlink" href="#upstream-work-in-datafusion" title="Link to this heading">#</a></h2>
<p>A growing number of Spark-compatible expressions live in the <code class="docutils literal notranslate"><span class="pre">datafusion-spark</span></code> crate in the core DataFusion
repository. Comet is migrating its expression implementations to that crate so that they can be shared by other
-DataFusion-based projects, and is wiring up the functions that crate already provides (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4150">#4150</a>). Improvements to
+DataFusion-based projects, and has wired up nearly every function that crate provides (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4150">#4150</a>). Improvements to
core DataFusion operators (joins, aggregates, window) made in support of Comet also benefit the wider ecosystem.</p>
</section>
<section id="spillable-hash-join">
<h2>Spillable Hash Join<a class="headerlink" href="#spillable-hash-join" title="Link to this heading">#</a></h2>
<p>Comet’s native hash join currently requires the build side to fit entirely in memory. Adding spill-to-disk
-support will allow Comet to handle larger joins without falling back to Spark, improving both reliability and
-performance for memory-intensive workloads.</p>
+support (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/2545">#2545</a>) will allow Comet to handle larger joins without falling back to Spark, improving both reliability
+and performance for memory-intensive workloads. Comet’s native hash join uses DataFusion’s <code class="docutils literal notranslate"><span class="pre">HashJoinExec</span></code>, and a
+design for spilling in that operator, behind a flag that is off by default, is proposed upstream in
+<a class="reference external" href="https://github.com/apache/datafusion/issues/24768">datafusion #24768</a>.</p>
</section>
<section id="java-scala-udf-support">
<h2>Java/Scala UDF Support<a class="headerlink" href="#java-scala-udf-support" title="Link to this heading">#</a></h2>
<p>Spark users frequently define custom UDFs in Java or Scala. Comet now dispatches scalar <code class="docutils literal notranslate"><span class="pre">ScalaUDF</span></code> expressions
through a JVM codegen bridge (<code class="docutils literal notranslate"><span class="pre">CometScalaUDF</span></code>) instead of always falling back to Spark. Aggregate UDFs, table
-UDFs/generators, Python/Pandas UDFs, and Hive <code class="docutils literal notranslate"><span class="pre">GenericUDF</span></code>/<code class="docutils literal notranslate"><span class="pre">SimpleUDF</span></code> still fall back to Spark entirely.
+UDFs/generators, Hive <code class="docutils literal notranslate"><span class="pre">GenericUDF</span></code>/<code class="docutils literal notranslate"><span class="pre">SimpleUDF</span></code>, and Python UDFs other than <code class="docutils literal notranslate"><span class="pre">mapInArrow</span></code> and <code class="docutils literal notranslate"><span class="pre">mapInPandas</span></code> still fall
+back to Spark entirely; Comet’s support for those two Python APIs is experimental and disabled by default (<a class="reference external" href="https://github.com/apache/datafusion-comet/pull/4234">#4234</a>).
Extending the codegen-dispatch approach to cover these remaining categories will reduce fallbacks further and
allow more queries to run end-to-end in Comet.</p>
</section>
@@ -639,22 +642,25 @@ enabled by default (<a class="reference external" href="https://github.com/apach
</section>
<section id="iceberg-table-writes">
<h2>Iceberg Table Writes<a class="headerlink" href="#iceberg-table-writes" title="Link to this heading">#</a></h2>
-<p>Comet’s native scans accelerate Iceberg reads today, but writes still run entirely through Spark’s Iceberg V2 write
-operator. <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4322">#4322</a> tracks exploring native acceleration of Iceberg writes, so an ETL job could run end to end in
-native code, falling back only where Comet can’t match Iceberg Java’s functionality. Draft proposals split the
-existing V2 write operator into separate “writer” and “committer” operators so the data-file-write portion can be
-planned and accelerated the same way as a native Parquet write (<a class="reference external" href="https://github.com/apache/datafusion-comet/pull/4658">#4658</a>, <a class="reference external" href="https://github.com/apache/datafusion-comet/pull/4487">#4487</a>). No design here is committed yet;
-we’re exploring whether this approach is worth pursuing.</p>
+<p>Comet can now write Iceberg tables natively. The feature is experimental, disabled by default, and controlled by two
+settings, the second of which requires the first. The first splits Spark’s Iceberg V2 write operator into separate
+writer and committer operators, so the query feeding the write becomes visible to AQE and to Comet’s columnar rules
+(<a class="reference external" href="https://github.com/apache/datafusion-comet/pull/4658">#4658</a>). The second delegates each task’s Parquet write to <code class="docutils literal notranslate"><span class="pre">iceberg-rust</span></code> when the write passes an eligibility
+check (<a class="reference external" href="https://github.com/apache/datafusion-comet/pull/5361">#5361</a>); writes that don’t pass keep Iceberg Java’s writer. Merge-on-read writes are not intercepted yet
+(<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/6240">#6240</a>). The remaining work toward the original goal of <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4322">#4322</a>, an ETL job that runs end to end in native code, is
+tracked in <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/5649">#5649</a>: correctness fixes, failure handling that matches Iceberg Java, broader coverage, and enabling
+both settings by default (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/5644">#5644</a>). See <a class="reference internal" href="iceberg-writes.html"><span class="std std-doc">Iceberg Writes</span></a> for how the write path works.</p>
</section>
<section id="delta-lake-support">
<h2>Delta Lake Support<a class="headerlink" href="#delta-lake-support" title="Link to this heading">#</a></h2>
-<p>Comet currently supports Spark’s built-in file formats and Iceberg, but not Delta Lake (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/174">#174</a>). A draft PR reuses
-Comet’s existing native Parquet reader to accelerate scans of plain Delta tables that use neither deletion vectors
-nor column mapping (<a class="reference external" href="https://github.com/apache/datafusion-comet/pull/4669">#4669</a>); a separate, larger effort explores a full native Delta scan built on <code class="docutils literal notranslate"><span class="pre">delta-kernel-rs</span></code>
-as an out-of-tree contrib module (<a class="reference external" href="https://github.com/apache/datafusion-comet/pull/4366">#4366</a>), landing in inert, gated slices (<a class="reference external" href="https://github.com/apache/datafusion-comet/pull/4700">#4700</a>, <a class="reference external" href="https://github.com/apache/datafusion-comet/pull/4952">#4952</a>). Generalizing Comet’s
-scan-side APIs so that Delta and other non-Iceberg data sources can plug in more easily is tracked as a Table
-Provider API abstraction (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4706">#4706</a>). None of this is committed yet; we’re exploring which approach, if any, is worth
-pursuing.</p>
+<p>Comet currently supports Spark’s built-in file formats and Iceberg, but not Delta Lake (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/174">#174</a>). The plugin boundary
+for an out-of-tree Delta module has landed in inert, gated slices (<a class="reference external" href="https://github.com/apache/datafusion-comet/pull/4700">#4700</a>, <a class="reference external" href="https://github.com/apache/datafusion-comet/pull/4952">#4952</a>), and two read paths build on
+it. In the first, which is in review, delta-spark plans the scan and Comet’s native Parquet reader executes it,
+applying deletion vectors inside the scan; it ships as an opt-in contrib module (<a class="reference external" href="https://github.com/apache/datafusion-comet/pull/5365">#5365</a>). The second, still a
+draft, is a full native Delta scan built on <code class="docutils literal notranslate"><span class="pre">delta-kernel-rs</span></code> (<a class="reference external" href="https://github.com/apache/datafusion-comet/pull/4366">#4366</a>). <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/5411">#5411</a> proposes converging the two into
+one plugin, with the Parquet path as the default and the kernel path covering what it can’t express yet, such as
+change data feed. Generalizing Comet’s scan-side APIs so that Delta and other non-Iceberg data sources can plug in
+more easily is tracked as a Table Provider API abstraction (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4706">#4706</a>).</p>
</section>
</section>
diff --git a/searchindex.js b/searchindex.js
index 5b4d3b8905..30329611fe 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[57, "id1"]], "%": [[55, "id1"]], "&": [[45, "id1"]], "*": [[55, "id2"]], "+": [[55, "id3"]], "-": [[55, "id4"]], "/": [[55, "id5"]], "1. Format Your Code": [[42, "format-your-code"]], "1. Install Comet": [[63, "install-comet"], [75, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[42, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[63, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[57, "id1"]], "%": [[55, "id1"]], "&": [[45, "id1"]], "*": [[55, "id2"]], "+": [[55, "id3"]], "-": [[55, "id4"]], "/": [[55, "id5"]], "1. Format Your Code": [[42, "format-your-code"]], "1. Install Comet": [[63, "install-comet"], [75, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[42, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[63, "clone-iceberg-and-apply- [...]
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org