You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/06/16 11:31:45 UTC
(datafusion-comet) branch asf-site updated (5ae57211fc -> 1972be160d)
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
discard 5ae57211fc Publish built docs triggered by 1bd90a78610b2a4b335635738cbc671478b4d720
new 1972be160d Publish built docs triggered by e5c59fda5d27d8186d9cd713315997c14d98bbf2
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (5ae57211fc)
\
N -- N -- N refs/heads/asf-site (1972be160d)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
_sources/user-guide/latest/scala_java_udfs.md.txt | 12 ++++++------
searchindex.js | 2 +-
user-guide/latest/scala_java_udfs.html | 8 ++++----
3 files changed, 11 insertions(+), 11 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion-comet) 01/01: Publish built docs triggered by e5c59fda5d27d8186d9cd713315997c14d98bbf2
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit 1972be160d48ee6bff32ca70ce9c4a6896cf9dc9
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Tue Jun 16 11:31:38 2026 +0000
Publish built docs triggered by e5c59fda5d27d8186d9cd713315997c14d98bbf2
---
_sources/user-guide/latest/scala_java_udfs.md.txt | 12 ++--
.../latest/understanding-comet-plans.md.txt | 69 +++++++++++-----------
searchindex.js | 2 +-
user-guide/latest/scala_java_udfs.html | 8 +--
user-guide/latest/understanding-comet-plans.html | 49 +++++++--------
5 files changed, 71 insertions(+), 69 deletions(-)
diff --git a/_sources/user-guide/latest/scala_java_udfs.md.txt b/_sources/user-guide/latest/scala_java_udfs.md.txt
index 89b0c3e941..1ebf79250f 100644
--- a/_sources/user-guide/latest/scala_java_udfs.md.txt
+++ b/_sources/user-guide/latest/scala_java_udfs.md.txt
@@ -19,7 +19,7 @@
# Scala UDF and Java UDF Support
-Comet executes Spark's Scala and Java [scalar user-defined functions (UDFs)](https://spark.apache.org/docs/latest/sql-ref-functions-udf-scalar.html) on the native Comet path. The presence of a UDF does not force the enclosing operator off the native path; surrounding native operators stay native.
+Comet executes Spark's Scala and Java [scalar user-defined functions (UDFs)](https://spark.apache.org/docs/latest/sql-ref-functions-udf-scalar.html) within the Comet pipeline. The presence of a UDF does not force the enclosing operator out of the Comet pipeline; surrounding Rust-implemented operators stay in the pipeline.
This page covers Spark's `ScalaUDF` (Scala `udf(...)`, `spark.udf.register(...)` over Scala or Java functional interfaces, and SQL `CREATE FUNCTION ... AS 'com.example.MyUDF'`). Other UDF kinds (Python / Pandas, Hive, aggregate) are out of scope and continue to fall back to Spark.
@@ -27,16 +27,16 @@ This feature is enabled by default. Set `spark.comet.exec.scalaUDF.codegen.enabl
## Configuration
-| Key | Default | Description |
-| ------------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------ |
-| `spark.comet.exec.scalaUDF.codegen.enabled` | `true` | When `true`, eligible `ScalaUDF`s run on the Comet path. When `false`, the enclosing operator falls back to Spark. |
+| Key | Default | Description |
+| ------------------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------- |
+| `spark.comet.exec.scalaUDF.codegen.enabled` | `true` | When `true`, eligible `ScalaUDF`s run in the Comet pipeline. When `false`, the enclosing operator falls back to Spark. |
## Supported
- User functions registered via `udf(...)`, `spark.udf.register(...)` (Scala or Java functional interfaces), or SQL `CREATE FUNCTION ... AS 'com.example.MyUDF'`.
- Scalar input/output types: `Boolean`, `Byte`, `Short`, `Int`, `Long`, `Float`, `Double`, `Decimal`, `String`, `Binary`, `Date`, `Timestamp`, `TimestampNTZ`.
- Complex input/output types with arbitrary nesting: `ArrayType`, `StructType`, `MapType`.
-- Composition with other Catalyst expressions inside the argument tree (e.g. `myUdf(upper(s))` runs as one native unit).
+- Composition with other Catalyst expressions inside the argument tree (e.g. `myUdf(upper(s))` runs as one unit in the Comet pipeline).
- Higher-order functions (`transform`, `filter`, `exists`, `aggregate`, `zip_with`, `map_filter`, `map_zip_with`, etc.) inside the argument tree.
## Not supported
@@ -45,7 +45,7 @@ This feature is enabled by default. Set `spark.comet.exec.scalaUDF.codegen.enabl
- Table UDFs and generators.
- Python `@udf` and Pandas `@pandas_udf`.
- Hive `GenericUDF` and `SimpleUDF`.
-- `CalendarIntervalType`, `NullType`, and `UserDefinedType` arguments and return types. UDT-typed columns fall back to Spark; for native execution, store and read the underlying representation directly (e.g. write MLlib `Vector` outputs as `Struct<type: Byte, size: Int, indices: Array<Int>, values: Array<Double>>` rather than `VectorUDT`).
+- `CalendarIntervalType`, `NullType`, and `UserDefinedType` arguments and return types. UDT-typed columns fall back to Spark; to keep execution in the Comet pipeline, store and read the underlying representation directly (e.g. write MLlib `Vector` outputs as `Struct<type: Byte, size: Int, indices: Array<Int>, values: Array<Double>>` rather than `VectorUDT`).
- Trees whose total nested-field count (output plus all input columns the UDF tree references) exceeds `spark.sql.codegen.maxFields` (default 100). Comet refuses these at plan time and the operator falls back to Spark.
When a UDF is rejected, the reason surfaces through Comet's standard fallback diagnostics; the query still runs on Spark.
diff --git a/_sources/user-guide/latest/understanding-comet-plans.md.txt b/_sources/user-guide/latest/understanding-comet-plans.md.txt
index 8adccc0929..07c1086624 100644
--- a/_sources/user-guide/latest/understanding-comet-plans.md.txt
+++ b/_sources/user-guide/latest/understanding-comet-plans.md.txt
@@ -27,13 +27,14 @@ inspect that behavior.
When Comet is enabled, the `CometSparkSessionExtensions` rules walk the
physical plan bottom-up and replace Spark operators with Comet equivalents
-where possible. Consecutive native operators are combined into a single block
-that is serialized as protobuf and executed by DataFusion on the executor.
+where possible. Consecutive Rust-implemented operators are combined into a
+single block that is serialized as protobuf and executed by DataFusion on the
+executor.
Operators that Comet does not support remain as their original Spark form.
As a result, a plan can mix three kinds of nodes:
-- **`Comet*` nodes** that run natively in Rust (for example `CometProject`,
+- **`Comet*` nodes** that are implemented in Rust (for example `CometProject`,
`CometHashAggregate`).
- **`Comet*` nodes that run on the JVM** but are still part of the Comet
pipeline (for example `CometBroadcastExchange`, `CometColumnarExchange`).
@@ -62,9 +63,9 @@ the same plan is shown in the Spark SQL UI. When reading a plan, look for:
## Fallback
-A "fallback" happens when Comet cannot translate part of a plan into native
-execution. Fallback can be partial (a subtree falls back while the rest stays
-native) or full (no Comet nodes appear).
+A "fallback" happens when Comet cannot run part of a plan in the Comet
+pipeline. Fallback can be partial (a subtree falls back to Spark while the rest
+stays in the Comet pipeline) or full (no Comet nodes appear).
Common reasons:
@@ -88,15 +89,15 @@ They serve different purposes and produce output in different places.
| Config | Output destination | What you see |
| ---------------------------------------- | ---------------------------------- | --------------------------------------------------------------------------------------------- |
-| `spark.comet.explainFallback.enabled` | Driver log (only when fallback) | A WARN with the list of reasons each query stage could not run natively. |
+| `spark.comet.explainFallback.enabled` | Driver log (only when fallback) | A WARN with the list of reasons each query stage could not run in Comet. |
| `spark.comet.logFallbackReasons.enabled` | Driver log | One WARN per fallback reason as it is encountered, without surrounding plan context. |
| `spark.comet.explain.format` | Spark SQL UI (Spark 4.0 and newer) | Annotated plan or fallback-reason list, depending on `verbose` (default) or `fallback` value. |
-| `spark.comet.explain.native.enabled` | Executor logs, per task | The DataFusion native plan with metrics, useful for inspecting native execution. |
+| `spark.comet.explain.native.enabled` | Executor logs, per task | The DataFusion plan with metrics, useful for inspecting Rust execution. |
### `spark.comet.explainFallback.enabled`
Logs a single WARN listing the reasons each query stage could not be executed
-natively. Nothing is logged when the entire stage runs in Comet. Useful as a
+in Comet. Nothing is logged when the entire stage runs in Comet. Useful as a
low-noise check that fallback is or is not happening.
### `spark.comet.logFallbackReasons.enabled`
@@ -131,12 +132,12 @@ the config has no effect there.
### `spark.comet.explain.native.enabled`
-When enabled, each executor task logs the DataFusion native plan it executes,
+When enabled, each executor task logs the DataFusion plan it executes,
along with metrics. This is verbose because there is one plan per task, but it
-is the only way to see the native plan as DataFusion sees it (including how
+is the only way to see the plan as DataFusion sees it (including how
operators were arranged after Comet's serialization). See the
-[Metrics Guide](metrics.md) for details on the native metrics that appear in
-this output.
+[Metrics Guide](metrics.md) for details on the DataFusion metrics that appear
+in this output.
## Programmatic Access to Fallback Reasons
@@ -193,14 +194,14 @@ by role. Names match what is shown in the plan output.
| Node | Description |
| ------------------------ | --------------------------------------------------------------------------------------------- |
| `CometBatchScan` | DataSource V2 scan, including Iceberg Parquet, that produces Arrow batches consumed by Comet. |
-| `CometNativeScan` | Fully native Parquet scan that runs entirely in DataFusion. |
-| `CometIcebergNativeScan` | Fully native Iceberg Parquet scan. |
-| `CometCsvNativeScan` | Fully native CSV scan (experimental). |
+| `CometNativeScan` | Parquet scan that runs entirely in Rust via DataFusion. |
+| `CometIcebergNativeScan` | Iceberg Parquet scan that runs entirely in Rust via DataFusion. |
+| `CometCsvNativeScan` | CSV scan that runs entirely in Rust via DataFusion (experimental). |
-### Native Execution Operators
+### Native Rust Operators
-These run natively in DataFusion. When several appear consecutively in a plan,
-they execute as a single fused native block.
+These are implemented in Rust and run in DataFusion. When several appear
+consecutively in a plan, they execute as a single fused block.
| Node | Spark equivalent |
| ------------------------------ | ----------------------------------------------- |
@@ -223,13 +224,13 @@ they execute as a single fused native block.
These keep their data on the JVM but participate in the Comet pipeline.
-| Node | Notes |
-| ------------------------ | ------------------------------------------------------------------------------------- |
-| `CometUnion` | JVM-side union of Comet inputs. The native side reads each branch as a separate scan. |
-| `CometCoalesce` | JVM-side partition coalesce. |
-| `CometCollectLimit` | JVM-side collect limit, equivalent to `CollectLimitExec`. |
-| `CometBroadcastExchange` | Broadcast exchange producing serialized Arrow batches that the consumer can decode. |
-| `CometSubqueryBroadcast` | Companion to `CometBroadcastExchange` for dynamic partition pruning subqueries. |
+| Node | Notes |
+| ------------------------ | ----------------------------------------------------------------------------------- |
+| `CometUnion` | JVM-side union of Comet inputs. The Rust side reads each branch as a separate scan. |
+| `CometCoalesce` | JVM-side partition coalesce. |
+| `CometCollectLimit` | JVM-side collect limit, equivalent to `CollectLimitExec`. |
+| `CometBroadcastExchange` | Broadcast exchange producing serialized Arrow batches that the consumer can decode. |
+| `CometSubqueryBroadcast` | Companion to `CometBroadcastExchange` for dynamic partition pruning subqueries. |
### Shuffle Operators
@@ -239,8 +240,8 @@ use:
- **`CometExchange`** is the **native shuffle** path. The child must already
be a Comet operator producing columnar Arrow batches; the node calls
`executeColumnar()` on its child and the partition, encode, and compress
- steps run in native code. Hash and range partitioning **keys** must be
- primitive types because native hashing and ordering do not support complex
+ steps run in Rust. Hash and range partitioning **keys** must be
+ primitive types because the Rust hashing and ordering do not support complex
types, but the data columns themselves can include `StructType`,
`ArrayType`, and `MapType` since batches are serialized via the Arrow IPC
writer.
@@ -265,12 +266,12 @@ Comet inserts these nodes wherever data has to cross the columnar/row boundary.
Multiple implementations exist because the optimal strategy depends on what
produced the columnar data.
-| Node | Direction | Notes |
-| ------------------------------ | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `CometColumnarToRow` | columnar → row | JVM-based row conversion. A fork of Spark's `ColumnarToRowExec` that includes the SPARK-50235 fix. |
-| `CometNativeColumnarToRow` | columnar → row | Native row conversion that decodes broadcast Arrow batches via `NativeColumnarToRowConverter`. Used downstream of `CometBroadcastExchange`. Zero-copy for variable-length types and avoids an extra JVM materialization step. |
-| `CometSparkColumnarToColumnar` | columnar → columnar | Converts a Spark columnar input (a non-Comet `ColumnarBatch`) into Comet's Arrow batches. |
-| `CometSparkRowToColumnar` | row → columnar | Converts a Spark row input into Comet's Arrow batches. |
+| Node | Direction | Notes |
+| ------------------------------ | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `CometColumnarToRow` | columnar → row | JVM-based row conversion. A fork of Spark's `ColumnarToRowExec` that includes the SPARK-50235 fix. |
+| `CometNativeColumnarToRow` | columnar → row | Rust-based row conversion that decodes broadcast Arrow batches via `NativeColumnarToRowConverter`. Used downstream of `CometBroadcastExchange`. Zero-copy for variable-length types and avoids an extra JVM materialization step. |
+| `CometSparkColumnarToColumnar` | columnar → columnar | Converts a Spark columnar input (a non-Comet `ColumnarBatch`) into Comet's Arrow batches. |
+| `CometSparkRowToColumnar` | row → columnar | Converts a Spark row input into Comet's Arrow batches. |
The two `CometSpark*` names come from a single `CometSparkToColumnarExec`
operator that picks the node name based on whether its child supports
diff --git a/searchindex.js b/searchindex.js
index bd76c1a240..a2f3e6e5cb 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[49, "id1"]], "%": [[47, "id1"]], "&": [[37, "id1"]], "*": [[47, "id2"]], "+": [[47, "id3"]], "-": [[47, "id4"]], "/": [[47, "id5"]], "1. Format Your Code": [[34, "format-your-code"]], "1. Install Comet": [[55, "install-comet"], [64, "install-comet"]], "1. Native Operators (nativeExecs map)": [[23, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[34, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[55, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[49, "id1"]], "%": [[47, "id1"]], "&": [[37, "id1"]], "*": [[47, "id2"]], "+": [[47, "id3"]], "-": [[47, "id4"]], "/": [[47, "id5"]], "1. Format Your Code": [[34, "format-your-code"]], "1. Install Comet": [[55, "install-comet"], [64, "install-comet"]], "1. Native Operators (nativeExecs map)": [[23, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[34, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[55, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/scala_java_udfs.html b/user-guide/latest/scala_java_udfs.html
index cbd81b3c50..b8654cd4cc 100644
--- a/user-guide/latest/scala_java_udfs.html
+++ b/user-guide/latest/scala_java_udfs.html
@@ -568,7 +568,7 @@ under the License.
-->
<section id="scala-udf-and-java-udf-support">
<h1>Scala UDF and Java UDF Support<a class="headerlink" href="#scala-udf-and-java-udf-support" title="Link to this heading">#</a></h1>
-<p>Comet executes Spark’s Scala and Java <a class="reference external" href="https://spark.apache.org/docs/latest/sql-ref-functions-udf-scalar.html">scalar user-defined functions (UDFs)</a> on the native Comet path. The presence of a UDF does not force the enclosing operator off the native path; surrounding native operators stay native.</p>
+<p>Comet executes Spark’s Scala and Java <a class="reference external" href="https://spark.apache.org/docs/latest/sql-ref-functions-udf-scalar.html">scalar user-defined functions (UDFs)</a> within the Comet pipeline. The presence of a UDF does not force the enclosing operator out of the Comet pipeline; surrounding Rust-implemented operators stay in the pipeline.</p>
<p>This page covers Spark’s <code class="docutils literal notranslate"><span class="pre">ScalaUDF</span></code> (Scala <code class="docutils literal notranslate"><span class="pre">udf(...)</span></code>, <code class="docutils literal notranslate"><span class="pre">spark.udf.register(...)</span></code> over Scala or Java functional interfaces, and SQL <code class="docutils literal notranslate"><span class="pre">CREATE</span> <span class="pre">FUNCTION</span> <span class="pre">...</span> < [...]
<p>This feature is enabled by default. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.scalaUDF.codegen.enabled</span></code> to <code class="docutils literal notranslate"><span class="pre">false</span></code> to route plans containing a <code class="docutils literal notranslate"><span class="pre">ScalaUDF</span></code> back to Spark for the enclosing operator.</p>
<section id="configuration">
@@ -583,7 +583,7 @@ under the License.
<tbody>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.scalaUDF.codegen.enabled</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">true</span></code></p></td>
-<td><p>When <code class="docutils literal notranslate"><span class="pre">true</span></code>, eligible <code class="docutils literal notranslate"><span class="pre">ScalaUDF</span></code>s run on the Comet path. When <code class="docutils literal notranslate"><span class="pre">false</span></code>, the enclosing operator falls back to Spark.</p></td>
+<td><p>When <code class="docutils literal notranslate"><span class="pre">true</span></code>, eligible <code class="docutils literal notranslate"><span class="pre">ScalaUDF</span></code>s run in the Comet pipeline. When <code class="docutils literal notranslate"><span class="pre">false</span></code>, the enclosing operator falls back to Spark.</p></td>
</tr>
</tbody>
</table>
@@ -595,7 +595,7 @@ under the License.
<li><p>User functions registered via <code class="docutils literal notranslate"><span class="pre">udf(...)</span></code>, <code class="docutils literal notranslate"><span class="pre">spark.udf.register(...)</span></code> (Scala or Java functional interfaces), or SQL <code class="docutils literal notranslate"><span class="pre">CREATE</span> <span class="pre">FUNCTION</span> <span class="pre">...</span> <span class="pre">AS</span> <span class="pre">'com.example.MyUDF'</span></code>.</p></li>
<li><p>Scalar input/output types: <code class="docutils literal notranslate"><span class="pre">Boolean</span></code>, <code class="docutils literal notranslate"><span class="pre">Byte</span></code>, <code class="docutils literal notranslate"><span class="pre">Short</span></code>, <code class="docutils literal notranslate"><span class="pre">Int</span></code>, <code class="docutils literal notranslate"><span class="pre">Long</span></code>, <code class="docutils literal notranslate"><span c [...]
<li><p>Complex input/output types with arbitrary nesting: <code class="docutils literal notranslate"><span class="pre">ArrayType</span></code>, <code class="docutils literal notranslate"><span class="pre">StructType</span></code>, <code class="docutils literal notranslate"><span class="pre">MapType</span></code>.</p></li>
-<li><p>Composition with other Catalyst expressions inside the argument tree (e.g. <code class="docutils literal notranslate"><span class="pre">myUdf(upper(s))</span></code> runs as one native unit).</p></li>
+<li><p>Composition with other Catalyst expressions inside the argument tree (e.g. <code class="docutils literal notranslate"><span class="pre">myUdf(upper(s))</span></code> runs as one unit in the Comet pipeline).</p></li>
<li><p>Higher-order functions (<code class="docutils literal notranslate"><span class="pre">transform</span></code>, <code class="docutils literal notranslate"><span class="pre">filter</span></code>, <code class="docutils literal notranslate"><span class="pre">exists</span></code>, <code class="docutils literal notranslate"><span class="pre">aggregate</span></code>, <code class="docutils literal notranslate"><span class="pre">zip_with</span></code>, <code class="docutils literal notransl [...]
</ul>
</section>
@@ -606,7 +606,7 @@ under the License.
<li><p>Table UDFs and generators.</p></li>
<li><p>Python <code class="docutils literal notranslate"><span class="pre">@udf</span></code> and Pandas <code class="docutils literal notranslate"><span class="pre">@pandas_udf</span></code>.</p></li>
<li><p>Hive <code class="docutils literal notranslate"><span class="pre">GenericUDF</span></code> and <code class="docutils literal notranslate"><span class="pre">SimpleUDF</span></code>.</p></li>
-<li><p><code class="docutils literal notranslate"><span class="pre">CalendarIntervalType</span></code>, <code class="docutils literal notranslate"><span class="pre">NullType</span></code>, and <code class="docutils literal notranslate"><span class="pre">UserDefinedType</span></code> arguments and return types. UDT-typed columns fall back to Spark; for native execution, store and read the underlying representation directly (e.g. write MLlib <code class="docutils literal notranslate"><span [...]
+<li><p><code class="docutils literal notranslate"><span class="pre">CalendarIntervalType</span></code>, <code class="docutils literal notranslate"><span class="pre">NullType</span></code>, and <code class="docutils literal notranslate"><span class="pre">UserDefinedType</span></code> arguments and return types. UDT-typed columns fall back to Spark; to keep execution in the Comet pipeline, store and read the underlying representation directly (e.g. write MLlib <code class="docutils literal [...]
<li><p>Trees whose total nested-field count (output plus all input columns the UDF tree references) exceeds <code class="docutils literal notranslate"><span class="pre">spark.sql.codegen.maxFields</span></code> (default 100). Comet refuses these at plan time and the operator falls back to Spark.</p></li>
</ul>
<p>When a UDF is rejected, the reason surfaces through Comet’s standard fallback diagnostics; the query still runs on Spark.</p>
diff --git a/user-guide/latest/understanding-comet-plans.html b/user-guide/latest/understanding-comet-plans.html
index 3bc9943586..c3c09c7438 100644
--- a/user-guide/latest/understanding-comet-plans.html
+++ b/user-guide/latest/understanding-comet-plans.html
@@ -575,12 +575,13 @@ inspect that behavior.</p>
<h2>Overview<a class="headerlink" href="#overview" title="Link to this heading">#</a></h2>
<p>When Comet is enabled, the <code class="docutils literal notranslate"><span class="pre">CometSparkSessionExtensions</span></code> rules walk the
physical plan bottom-up and replace Spark operators with Comet equivalents
-where possible. Consecutive native operators are combined into a single block
-that is serialized as protobuf and executed by DataFusion on the executor.
+where possible. Consecutive Rust-implemented operators are combined into a
+single block that is serialized as protobuf and executed by DataFusion on the
+executor.
Operators that Comet does not support remain as their original Spark form.</p>
<p>As a result, a plan can mix three kinds of nodes:</p>
<ul class="simple">
-<li><p><strong><code class="docutils literal notranslate"><span class="pre">Comet*</span></code> nodes</strong> that run natively in Rust (for example <code class="docutils literal notranslate"><span class="pre">CometProject</span></code>,
+<li><p><strong><code class="docutils literal notranslate"><span class="pre">Comet*</span></code> nodes</strong> that are implemented in Rust (for example <code class="docutils literal notranslate"><span class="pre">CometProject</span></code>,
<code class="docutils literal notranslate"><span class="pre">CometHashAggregate</span></code>).</p></li>
<li><p><strong><code class="docutils literal notranslate"><span class="pre">Comet*</span></code> nodes that run on the JVM</strong> but are still part of the Comet
pipeline (for example <code class="docutils literal notranslate"><span class="pre">CometBroadcastExchange</span></code>, <code class="docutils literal notranslate"><span class="pre">CometColumnarExchange</span></code>).</p></li>
@@ -610,9 +611,9 @@ below.</p></li>
</section>
<section id="fallback">
<h2>Fallback<a class="headerlink" href="#fallback" title="Link to this heading">#</a></h2>
-<p>A “fallback” happens when Comet cannot translate part of a plan into native
-execution. Fallback can be partial (a subtree falls back while the rest stays
-native) or full (no Comet nodes appear).</p>
+<p>A “fallback” happens when Comet cannot run part of a plan in the Comet
+pipeline. Fallback can be partial (a subtree falls back to Spark while the rest
+stays in the Comet pipeline) or full (no Comet nodes appear).</p>
<p>Common reasons:</p>
<ul class="simple">
<li><p>The Spark operator is not supported by Comet.</p></li>
@@ -642,7 +643,7 @@ They serve different purposes and produce output in different places.</p>
<tbody>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.explainFallback.enabled</span></code></p></td>
<td><p>Driver log (only when fallback)</p></td>
-<td><p>A WARN with the list of reasons each query stage could not run natively.</p></td>
+<td><p>A WARN with the list of reasons each query stage could not run in Comet.</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.logFallbackReasons.enabled</span></code></p></td>
<td><p>Driver log</p></td>
@@ -654,7 +655,7 @@ They serve different purposes and produce output in different places.</p>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.explain.native.enabled</span></code></p></td>
<td><p>Executor logs, per task</p></td>
-<td><p>The DataFusion native plan with metrics, useful for inspecting native execution.</p></td>
+<td><p>The DataFusion plan with metrics, useful for inspecting Rust execution.</p></td>
</tr>
</tbody>
</table>
@@ -662,7 +663,7 @@ They serve different purposes and produce output in different places.</p>
<section id="spark-comet-explainfallback-enabled">
<h3><code class="docutils literal notranslate"><span class="pre">spark.comet.explainFallback.enabled</span></code><a class="headerlink" href="#spark-comet-explainfallback-enabled" title="Link to this heading">#</a></h3>
<p>Logs a single WARN listing the reasons each query stage could not be executed
-natively. Nothing is logged when the entire stage runs in Comet. Useful as a
+in Comet. Nothing is logged when the entire stage runs in Comet. Useful as a
low-noise check that fallback is or is not happening.</p>
</section>
<section id="spark-comet-logfallbackreasons-enabled">
@@ -695,12 +696,12 @@ the config has no effect there.</p>
</section>
<section id="spark-comet-explain-native-enabled">
<h3><code class="docutils literal notranslate"><span class="pre">spark.comet.explain.native.enabled</span></code><a class="headerlink" href="#spark-comet-explain-native-enabled" title="Link to this heading">#</a></h3>
-<p>When enabled, each executor task logs the DataFusion native plan it executes,
+<p>When enabled, each executor task logs the DataFusion plan it executes,
along with metrics. This is verbose because there is one plan per task, but it
-is the only way to see the native plan as DataFusion sees it (including how
+is the only way to see the plan as DataFusion sees it (including how
operators were arranged after Comet’s serialization). See the
-<a class="reference internal" href="metrics.html"><span class="std std-doc">Metrics Guide</span></a> for details on the native metrics that appear in
-this output.</p>
+<a class="reference internal" href="metrics.html"><span class="std std-doc">Metrics Guide</span></a> for details on the DataFusion metrics that appear
+in this output.</p>
</section>
</section>
<section id="programmatic-access-to-fallback-reasons">
@@ -760,22 +761,22 @@ by role. Names match what is shown in the plan output.</p>
<td><p>DataSource V2 scan, including Iceberg Parquet, that produces Arrow batches consumed by Comet.</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">CometNativeScan</span></code></p></td>
-<td><p>Fully native Parquet scan that runs entirely in DataFusion.</p></td>
+<td><p>Parquet scan that runs entirely in Rust via DataFusion.</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">CometIcebergNativeScan</span></code></p></td>
-<td><p>Fully native Iceberg Parquet scan.</p></td>
+<td><p>Iceberg Parquet scan that runs entirely in Rust via DataFusion.</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">CometCsvNativeScan</span></code></p></td>
-<td><p>Fully native CSV scan (experimental).</p></td>
+<td><p>CSV scan that runs entirely in Rust via DataFusion (experimental).</p></td>
</tr>
</tbody>
</table>
</div>
</section>
-<section id="native-execution-operators">
-<h3>Native Execution Operators<a class="headerlink" href="#native-execution-operators" title="Link to this heading">#</a></h3>
-<p>These run natively in DataFusion. When several appear consecutively in a plan,
-they execute as a single fused native block.</p>
+<section id="native-rust-operators">
+<h3>Native Rust Operators<a class="headerlink" href="#native-rust-operators" title="Link to this heading">#</a></h3>
+<p>These are implemented in Rust and run in DataFusion. When several appear
+consecutively in a plan, they execute as a single fused block.</p>
<div class="pst-scrollable-table-container"><table class="table">
<thead>
<tr class="row-odd"><th class="head"><p>Node</p></th>
@@ -840,7 +841,7 @@ they execute as a single fused native block.</p>
</thead>
<tbody>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">CometUnion</span></code></p></td>
-<td><p>JVM-side union of Comet inputs. The native side reads each branch as a separate scan.</p></td>
+<td><p>JVM-side union of Comet inputs. The Rust side reads each branch as a separate scan.</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">CometCoalesce</span></code></p></td>
<td><p>JVM-side partition coalesce.</p></td>
@@ -866,8 +867,8 @@ use:</p>
<li><p><strong><code class="docutils literal notranslate"><span class="pre">CometExchange</span></code></strong> is the <strong>native shuffle</strong> path. The child must already
be a Comet operator producing columnar Arrow batches; the node calls
<code class="docutils literal notranslate"><span class="pre">executeColumnar()</span></code> on its child and the partition, encode, and compress
-steps run in native code. Hash and range partitioning <strong>keys</strong> must be
-primitive types because native hashing and ordering do not support complex
+steps run in Rust. Hash and range partitioning <strong>keys</strong> must be
+primitive types because the Rust hashing and ordering do not support complex
types, but the data columns themselves can include <code class="docutils literal notranslate"><span class="pre">StructType</span></code>,
<code class="docutils literal notranslate"><span class="pre">ArrayType</span></code>, and <code class="docutils literal notranslate"><span class="pre">MapType</span></code> since batches are serialized via the Arrow IPC
writer.</p></li>
@@ -904,7 +905,7 @@ produced the columnar data.</p>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">CometNativeColumnarToRow</span></code></p></td>
<td><p>columnar → row</p></td>
-<td><p>Native row conversion that decodes broadcast Arrow batches via <code class="docutils literal notranslate"><span class="pre">NativeColumnarToRowConverter</span></code>. Used downstream of <code class="docutils literal notranslate"><span class="pre">CometBroadcastExchange</span></code>. Zero-copy for variable-length types and avoids an extra JVM materialization step.</p></td>
+<td><p>Rust-based row conversion that decodes broadcast Arrow batches via <code class="docutils literal notranslate"><span class="pre">NativeColumnarToRowConverter</span></code>. Used downstream of <code class="docutils literal notranslate"><span class="pre">CometBroadcastExchange</span></code>. Zero-copy for variable-length types and avoids an extra JVM materialization step.</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">CometSparkColumnarToColumnar</span></code></p></td>
<td><p>columnar → columnar</p></td>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org