You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/08/25 17:31:53 UTC
(datafusion-comet) branch asf-site updated: Publish built docs triggered by f1717826468574369bbe1271f711f03d900b6ffb
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 8de5e3b454 Publish built docs triggered by f1717826468574369bbe1271f711f03d900b6ffb
8de5e3b454 is described below
commit 8de5e3b45496923cd905661e5231a1fa56eecd1a
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Tue Aug 25 17:31:49 2026 +0000
Publish built docs triggered by f1717826468574369bbe1271f711f03d900b6ffb
---
_sources/user-guide/latest/tuning.md.txt | 11 +++++++++++
searchindex.js | 2 +-
user-guide/latest/tuning.html | 16 +++++++++++++++-
3 files changed, 27 insertions(+), 2 deletions(-)
diff --git a/_sources/user-guide/latest/tuning.md.txt b/_sources/user-guide/latest/tuning.md.txt
index 37b6f03669..5499052464 100644
--- a/_sources/user-guide/latest/tuning.md.txt
+++ b/_sources/user-guide/latest/tuning.md.txt
@@ -273,6 +273,17 @@ when the number of columnar-to-row transitions exceeds
`spark.comet.exec.transitionRevert.maxTransitions` (default `2`). This trades native execution of a small
subset of operators for eliminating conversion overhead across the stage.
+### Entire-Plan Fallback for Wide or Deeply Nested Schemas
+
+The cost of each conversion also grows sharply with schema shape: for wide or deeply nested schemas,
+columnar-to-row conversion is especially expensive because the conversion work scales with the number of
+columns and nested fields. If profiling shows these conversions dominating a query over such a schema, set
+`spark.comet.exec.transitionRevert.enabled=true` and lower
+`spark.comet.exec.transitionRevert.maxTransitions` (default `2`) to `1`. Note that this causes the entire
+plan to fall back to Spark row-based execution — Comet removes its native operators rather than running a
+mix of native and fallback operators joined by repeated conversions — which can be cheaper than paying the
+expensive conversions again and again.
+
## Metrics Overhead
Comet exposes rich native operator metrics for observability (see [Metrics](metrics.md)), but they are
diff --git a/searchindex.js b/searchindex.js
index 71e3ea6db1..75b9e7c9a5 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[55, "id1"]], "%": [[53, "id1"]], "&": [[43, "id1"]], "*": [[53, "id2"]], "+": [[53, "id3"]], "-": [[53, "id4"]], "/": [[53, "id5"]], "1. Format Your Code": [[40, "format-your-code"]], "1. Install Comet": [[61, "install-comet"], [71, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[40, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[61, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[55, "id1"]], "%": [[53, "id1"]], "&": [[43, "id1"]], "*": [[53, "id2"]], "+": [[53, "id3"]], "-": [[53, "id4"]], "/": [[53, "id5"]], "1. Format Your Code": [[40, "format-your-code"]], "1. Install Comet": [[61, "install-comet"], [71, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[40, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[61, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/tuning.html b/user-guide/latest/tuning.html
index e225aa01da..f3b51bba55 100644
--- a/user-guide/latest/tuning.html
+++ b/user-guide/latest/tuning.html
@@ -998,6 +998,17 @@ repeated columnar-to-row and row-to-columnar conversions that dominate stage run
when the number of columnar-to-row transitions exceeds
<code class="docutils literal notranslate"><span class="pre">spark.comet.exec.transitionRevert.maxTransitions</span></code> (default <code class="docutils literal notranslate"><span class="pre">2</span></code>). This trades native execution of a small
subset of operators for eliminating conversion overhead across the stage.</p>
+<section id="entire-plan-fallback-for-wide-or-deeply-nested-schemas">
+<h3>Entire-Plan Fallback for Wide or Deeply Nested Schemas<a class="headerlink" href="#entire-plan-fallback-for-wide-or-deeply-nested-schemas" title="Link to this heading">#</a></h3>
+<p>The cost of each conversion also grows sharply with schema shape: for wide or deeply nested schemas,
+columnar-to-row conversion is especially expensive because the conversion work scales with the number of
+columns and nested fields. If profiling shows these conversions dominating a query over such a schema, set
+<code class="docutils literal notranslate"><span class="pre">spark.comet.exec.transitionRevert.enabled=true</span></code> and lower
+<code class="docutils literal notranslate"><span class="pre">spark.comet.exec.transitionRevert.maxTransitions</span></code> (default <code class="docutils literal notranslate"><span class="pre">2</span></code>) to <code class="docutils literal notranslate"><span class="pre">1</span></code>. Note that this causes the entire
+plan to fall back to Spark row-based execution — Comet removes its native operators rather than running a
+mix of native and fallback operators joined by repeated conversions — which can be cheaper than paying the
+expensive conversions again and again.</p>
+</section>
</section>
<section id="metrics-overhead">
<h2>Metrics Overhead<a class="headerlink" href="#metrics-overhead" title="Link to this heading">#</a></h2>
@@ -1087,7 +1098,10 @@ fallback to Spark works, see <a class="reference internal" href="understanding-c
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#shuffle-compression">Shuffle Compression</a></li>
</ul>
</li>
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#reducing-row-columnar-conversion-overhead">Reducing Row/Columnar Conversion Overhead</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#reducing-row-columnar-conversion-overhead">Reducing Row/Columnar Conversion Overhead</a><ul class="nav section-nav flex-column">
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#entire-plan-fallback-for-wide-or-deeply-nested-schemas">Entire-Plan Fallback for Wide or Deeply Nested Schemas</a></li>
+</ul>
+</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#metrics-overhead">Metrics Overhead</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#explain-plan">Explain Plan</a></li>
</ul>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org