You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/07/14 16:16:34 UTC
(datafusion-comet) branch asf-site updated: Publish built docs triggered by 32caa13d000dbfc2458b68a7052ae4114f59c4d1
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 98b41d5156 Publish built docs triggered by 32caa13d000dbfc2458b68a7052ae4114f59c4d1
98b41d5156 is described below
commit 98b41d5156558c12abbdc6e99b45b814fe497c29
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Tue Jul 14 16:16:30 2026 +0000
Publish built docs triggered by 32caa13d000dbfc2458b68a7052ae4114f59c4d1
---
_sources/user-guide/latest/configs.md.txt | 1 +
.../latest/understanding-comet-plans.md.txt | 57 +++++++++++++++++++---
searchindex.js | 2 +-
user-guide/latest/configs.html | 8 ++-
user-guide/latest/understanding-comet-plans.html | 48 ++++++++++++++++--
5 files changed, 103 insertions(+), 13 deletions(-)
diff --git a/_sources/user-guide/latest/configs.md.txt b/_sources/user-guide/latest/configs.md.txt
index eeecf2cb4b..cf2618feed 100644
--- a/_sources/user-guide/latest/configs.md.txt
+++ b/_sources/user-guide/latest/configs.md.txt
@@ -92,6 +92,7 @@ These settings can be used to determine which parts of the plan are accelerated
| `spark.comet.explain.format` | Choose extended explain output. The default format of 'verbose' will provide the full query plan annotated with fallback reasons as well as a summary of how much of the plan was accelerated by Comet. The format 'fallback' provides a list of fallback reasons instead. | verbose |
| `spark.comet.explain.native.enabled` | When this setting is enabled, Comet will provide a tree representation of the native query plan before execution and again after execution, with metrics. | false |
| `spark.comet.explain.rules` | When this setting is enabled, Comet will log all plan transformations performed in physical optimizer rules. Default: false | false |
+| `spark.comet.explainCodegen.enabled` | When enabled, Comet annotates the surrounding Comet operator with a `[COMET-INFO: JVM codegen dispatcher: <names>]` segment listing every expression it routed through the JVM codegen dispatcher. Disabled by default. | false |
| `spark.comet.explainFallback.enabled` | When this setting is enabled, Comet will provide logging explaining the reason(s) why a query stage cannot be executed natively. Set this to false to reduce the amount of logging. | false |
| `spark.comet.logFallbackReasons.enabled` | When this setting is enabled, Comet will log warnings for all fallback reasons. It can be overridden by the environment variable `ENABLE_COMET_LOG_FALLBACK_REASONS`. | false |
<!-- prettier-ignore-end -->
diff --git a/_sources/user-guide/latest/understanding-comet-plans.md.txt b/_sources/user-guide/latest/understanding-comet-plans.md.txt
index 07c1086624..8eac031540 100644
--- a/_sources/user-guide/latest/understanding-comet-plans.md.txt
+++ b/_sources/user-guide/latest/understanding-comet-plans.md.txt
@@ -84,15 +84,16 @@ incompatibility details.
## Configs for Inspecting Plans and Fallback
-Comet provides four configs for understanding what is happening in a plan.
+Comet provides five configs for understanding what is happening in a plan.
They serve different purposes and produce output in different places.
-| Config | Output destination | What you see |
-| ---------------------------------------- | ---------------------------------- | --------------------------------------------------------------------------------------------- |
-| `spark.comet.explainFallback.enabled` | Driver log (only when fallback) | A WARN with the list of reasons each query stage could not run in Comet. |
-| `spark.comet.logFallbackReasons.enabled` | Driver log | One WARN per fallback reason as it is encountered, without surrounding plan context. |
-| `spark.comet.explain.format` | Spark SQL UI (Spark 4.0 and newer) | Annotated plan or fallback-reason list, depending on `verbose` (default) or `fallback` value. |
-| `spark.comet.explain.native.enabled` | Executor logs, per task | The DataFusion plan with metrics, useful for inspecting Rust execution. |
+| Config | Output destination | What you see |
+| ---------------------------------------- | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `spark.comet.explainFallback.enabled` | Driver log (only when fallback) | A WARN with the list of reasons each query stage could not run in Comet. |
+| `spark.comet.logFallbackReasons.enabled` | Driver log | One WARN per fallback reason as it is encountered, without surrounding plan context. |
+| `spark.comet.explainCodegen.enabled` | `ExtendedExplainInfo` / SQL UI | A single `[COMET-INFO: JVM codegen dispatcher: <name1>, <name2>, ...]` segment naming each expression on the operator that was routed through the JVM codegen dispatcher. |
+| `spark.comet.explain.format` | Spark SQL UI (Spark 4.0 and newer) | Annotated plan or fallback-reason list, depending on `verbose` (default) or `fallback` value. |
+| `spark.comet.explain.native.enabled` | Executor logs, per task | The DataFusion plan with metrics, useful for inspecting Rust execution. |
### `spark.comet.explainFallback.enabled`
@@ -108,6 +109,48 @@ when you want to see all reasons, including ones that
not include the surrounding plan, so it is best for accumulating diagnostics
across many queries.
+### `spark.comet.explainCodegen.enabled`
+
+Disabled by default. When enabled, Comet annotates the surrounding Comet
+operator (`CometProject`, `CometFilter`, etc.) with a single combined
+`[COMET-INFO: JVM codegen dispatcher: <name1>, <name2>, ...]` segment listing
+every expression on that operator that was routed through the JVM codegen
+dispatcher (Spark's own `doGenCode` running inside a Comet kernel), gated by
+`spark.comet.exec.scalaUDF.codegen.enabled=true`. The projection stays
+Comet-native — this is informational only, useful for distinguishing "runs
+natively in DataFusion" from "runs Spark's generated code inside a Comet
+kernel". The annotation only appears for expressions Comet actually routed
+through the dispatcher on this plan — either because no native DataFusion
+implementation exists (`CometCodegenDispatch` serdes such as `hypot`,
+`levenshtein`, `pmod`), or because a native path exists but declined this
+specific input (`CodegenDispatchFallback` mixins). It is not a general "this
+expression ran here" marker.
+
+Example:
+
+```scala
+spark.conf.set("spark.comet.exec.scalaUDF.codegen.enabled", "true")
+spark.conf.set("spark.comet.explainCodegen.enabled", "true")
+
+val df = spark.sql("SELECT hypot(a, b), levenshtein(s1, s2) FROM t")
+println(new org.apache.comet.ExtendedExplainInfo()
+ .generateExtendedInfo(df.queryExecution.executedPlan))
+```
+
+Output:
+
+```
+CometNativeColumnarToRow
++- CometProject [COMET-INFO: JVM codegen dispatcher: hypot, levenshtein]
+ +- CometNativeScan parquet spark_catalog.default.t
+
+Comet accelerated 2 out of 2 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet.
+```
+
+Note that the operator is still `CometProject` (Comet-accelerated); only the
+per-expression evaluation for `hypot` and `levenshtein` takes the JVM codegen
+path.
+
### `spark.comet.explain.format`
This config is read by `org.apache.comet.ExtendedExplainInfo`, which Spark
diff --git a/searchindex.js b/searchindex.js
index 1de3872ee8..2ba442ef5c 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[51, "id1"]], "%": [[49, "id1"]], "&": [[39, "id1"]], "*": [[49, "id2"]], "+": [[49, "id3"]], "-": [[49, "id4"]], "/": [[49, "id5"]], "1. Format Your Code": [[36, "format-your-code"]], "1. Install Comet": [[57, "install-comet"], [66, "install-comet"]], "1. Native Operators (nativeExecs map)": [[25, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[36, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[57, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[51, "id1"]], "%": [[49, "id1"]], "&": [[39, "id1"]], "*": [[49, "id2"]], "+": [[49, "id3"]], "-": [[49, "id4"]], "/": [[49, "id5"]], "1. Format Your Code": [[36, "format-your-code"]], "1. Install Comet": [[57, "install-comet"], [66, "install-comet"]], "1. Native Operators (nativeExecs map)": [[25, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[36, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[57, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/configs.html b/user-guide/latest/configs.html
index 9114bd6bfb..eb2a43364a 100644
--- a/user-guide/latest/configs.html
+++ b/user-guide/latest/configs.html
@@ -953,11 +953,15 @@ under the License.
<td><p>When this setting is enabled, Comet will log all plan transformations performed in physical optimizer rules. Default: false</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.explainFallback.enabled</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.explainCodegen.enabled</span></code></p></td>
+<td><p>When enabled, Comet annotates the surrounding Comet operator with a <code class="docutils literal notranslate"><span class="pre">[COMET-INFO:</span> <span class="pre">JVM</span> <span class="pre">codegen</span> <span class="pre">dispatcher:</span> <span class="pre"><names>]</span></code> segment listing every expression it routed through the JVM codegen dispatcher. Disabled by default.</p></td>
+<td><p>false</p></td>
+</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.explainFallback.enabled</span></code></p></td>
<td><p>When this setting is enabled, Comet will provide logging explaining the reason(s) why a query stage cannot be executed natively. Set this to false to reduce the amount of logging.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.logFallbackReasons.enabled</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.logFallbackReasons.enabled</span></code></p></td>
<td><p>When this setting is enabled, Comet will log warnings for all fallback reasons. It can be overridden by the environment variable <code class="docutils literal notranslate"><span class="pre">ENABLE_COMET_LOG_FALLBACK_REASONS</span></code>.</p></td>
<td><p>false</p></td>
</tr>
diff --git a/user-guide/latest/understanding-comet-plans.html b/user-guide/latest/understanding-comet-plans.html
index 951a13b0b2..20189ce10f 100644
--- a/user-guide/latest/understanding-comet-plans.html
+++ b/user-guide/latest/understanding-comet-plans.html
@@ -804,7 +804,7 @@ incompatibility details.</p>
</section>
<section id="configs-for-inspecting-plans-and-fallback">
<h2>Configs for Inspecting Plans and Fallback<a class="headerlink" href="#configs-for-inspecting-plans-and-fallback" title="Link to this heading">#</a></h2>
-<p>Comet provides four configs for understanding what is happening in a plan.
+<p>Comet provides five configs for understanding what is happening in a plan.
They serve different purposes and produce output in different places.</p>
<div class="pst-scrollable-table-container"><table class="table">
<thead>
@@ -822,11 +822,15 @@ They serve different purposes and produce output in different places.</p>
<td><p>Driver log</p></td>
<td><p>One WARN per fallback reason as it is encountered, without surrounding plan context.</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.explain.format</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.explainCodegen.enabled</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">ExtendedExplainInfo</span></code> / SQL UI</p></td>
+<td><p>A single <code class="docutils literal notranslate"><span class="pre">[COMET-INFO:</span> <span class="pre">JVM</span> <span class="pre">codegen</span> <span class="pre">dispatcher:</span> <span class="pre"><name1>,</span> <span class="pre"><name2>,</span> <span class="pre">...]</span></code> segment naming each expression on the operator that was routed through the JVM codegen dispatcher.</p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.explain.format</span></code></p></td>
<td><p>Spark SQL UI (Spark 4.0 and newer)</p></td>
<td><p>Annotated plan or fallback-reason list, depending on <code class="docutils literal notranslate"><span class="pre">verbose</span></code> (default) or <code class="docutils literal notranslate"><span class="pre">fallback</span></code> value.</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.explain.native.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.explain.native.enabled</span></code></p></td>
<td><p>Executor logs, per task</p></td>
<td><p>The DataFusion plan with metrics, useful for inspecting Rust execution.</p></td>
</tr>
@@ -847,6 +851,43 @@ when you want to see all reasons, including ones that
not include the surrounding plan, so it is best for accumulating diagnostics
across many queries.</p>
</section>
+<section id="spark-comet-explaincodegen-enabled">
+<h3><code class="docutils literal notranslate"><span class="pre">spark.comet.explainCodegen.enabled</span></code><a class="headerlink" href="#spark-comet-explaincodegen-enabled" title="Link to this heading">#</a></h3>
+<p>Disabled by default. When enabled, Comet annotates the surrounding Comet
+operator (<code class="docutils literal notranslate"><span class="pre">CometProject</span></code>, <code class="docutils literal notranslate"><span class="pre">CometFilter</span></code>, etc.) with a single combined
+<code class="docutils literal notranslate"><span class="pre">[COMET-INFO:</span> <span class="pre">JVM</span> <span class="pre">codegen</span> <span class="pre">dispatcher:</span> <span class="pre"><name1>,</span> <span class="pre"><name2>,</span> <span class="pre">...]</span></code> segment listing
+every expression on that operator that was routed through the JVM codegen
+dispatcher (Spark’s own <code class="docutils literal notranslate"><span class="pre">doGenCode</span></code> running inside a Comet kernel), gated by
+<code class="docutils literal notranslate"><span class="pre">spark.comet.exec.scalaUDF.codegen.enabled=true</span></code>. The projection stays
+Comet-native — this is informational only, useful for distinguishing “runs
+natively in DataFusion” from “runs Spark’s generated code inside a Comet
+kernel”. The annotation only appears for expressions Comet actually routed
+through the dispatcher on this plan — either because no native DataFusion
+implementation exists (<code class="docutils literal notranslate"><span class="pre">CometCodegenDispatch</span></code> serdes such as <code class="docutils literal notranslate"><span class="pre">hypot</span></code>,
+<code class="docutils literal notranslate"><span class="pre">levenshtein</span></code>, <code class="docutils literal notranslate"><span class="pre">pmod</span></code>), or because a native path exists but declined this
+specific input (<code class="docutils literal notranslate"><span class="pre">CodegenDispatchFallback</span></code> mixins). It is not a general “this
+expression ran here” marker.</p>
+<p>Example:</p>
+<div class="highlight-scala notranslate"><div class="highlight"><pre><span></span><span class="n">spark</span><span class="p">.</span><span class="n">conf</span><span class="p">.</span><span class="n">set</span><span class="p">(</span><span class="s">"spark.comet.exec.scalaUDF.codegen.enabled"</span><span class="p">,</span><span class="w"> </span><span class="s">"true"</span><span class="p">)</span>
+<span class="n">spark</span><span class="p">.</span><span class="n">conf</span><span class="p">.</span><span class="n">set</span><span class="p">(</span><span class="s">"spark.comet.explainCodegen.enabled"</span><span class="p">,</span><span class="w"> </span><span class="s">"true"</span><span class="p">)</span>
+
+<span class="kd">val</span><span class="w"> </span><span class="n">df</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">spark</span><span class="p">.</span><span class="n">sql</span><span class="p">(</span><span class="s">"SELECT hypot(a, b), levenshtein(s1, s2) FROM t"</span><span class="p">)</span>
+<span class="n">println</span><span class="p">(</span><span class="k">new</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">apache</span><span class="p">.</span><span class="n">comet</span><span class="p">.</span><span class="nc">ExtendedExplainInfo</span><span class="p">()</span>
+<span class="w"> </span><span class="p">.</span><span class="n">generateExtendedInfo</span><span class="p">(</span><span class="n">df</span><span class="p">.</span><span class="n">queryExecution</span><span class="p">.</span><span class="n">executedPlan</span><span class="p">))</span>
+</pre></div>
+</div>
+<p>Output:</p>
+<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">CometNativeColumnarToRow</span>
+<span class="o">+-</span> <span class="n">CometProject</span> <span class="p">[</span><span class="n">COMET</span><span class="o">-</span><span class="n">INFO</span><span class="p">:</span> <span class="n">JVM</span> <span class="n">codegen</span> <span class="n">dispatcher</span><span class="p">:</span> <span class="n">hypot</span><span class="p">,</span> <span class="n">levenshtein</span><span class="p">]</span>
+ <span class="o">+-</span> <span class="n">CometNativeScan</span> <span class="n">parquet</span> <span class="n">spark_catalog</span><span class="o">.</span><span class="n">default</span><span class="o">.</span><span class="n">t</span>
+
+<span class="n">Comet</span> <span class="n">accelerated</span> <span class="mi">2</span> <span class="n">out</span> <span class="n">of</span> <span class="mi">2</span> <span class="n">eligible</span> <span class="n">operators</span> <span class="p">(</span><span class="mi">100</span><span class="o">%</span><span class="p">)</span><span class="o">.</span> <span class="n">Final</span> <span class="n">plan</span> <span class="n">contains</span> <span class="mi">1</span> <span class="n">tra [...]
+</pre></div>
+</div>
+<p>Note that the operator is still <code class="docutils literal notranslate"><span class="pre">CometProject</span></code> (Comet-accelerated); only the
+per-expression evaluation for <code class="docutils literal notranslate"><span class="pre">hypot</span></code> and <code class="docutils literal notranslate"><span class="pre">levenshtein</span></code> takes the JVM codegen
+path.</p>
+</section>
<section id="spark-comet-explain-format">
<h3><code class="docutils literal notranslate"><span class="pre">spark.comet.explain.format</span></code><a class="headerlink" href="#spark-comet-explain-format" title="Link to this heading">#</a></h3>
<p>This config is read by <code class="docutils literal notranslate"><span class="pre">org.apache.comet.ExtendedExplainInfo</span></code>, which Spark
@@ -1151,6 +1192,7 @@ columnar.</p>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#configs-for-inspecting-plans-and-fallback">Configs for Inspecting Plans and Fallback</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#spark-comet-explainfallback-enabled"><code class="docutils literal notranslate"><span class="pre">spark.comet.explainFallback.enabled</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#spark-comet-logfallbackreasons-enabled"><code class="docutils literal notranslate"><span class="pre">spark.comet.logFallbackReasons.enabled</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#spark-comet-explaincodegen-enabled"><code class="docutils literal notranslate"><span class="pre">spark.comet.explainCodegen.enabled</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#spark-comet-explain-format"><code class="docutils literal notranslate"><span class="pre">spark.comet.explain.format</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#spark-comet-explain-native-enabled"><code class="docutils literal notranslate"><span class="pre">spark.comet.explain.native.enabled</span></code></a></li>
</ul>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org