(datafusion-comet) branch asf-site updated: Publish built docs triggered by 964e57892a3397faf216e781e71c35352c6e4745

[gi...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git


The following commit(s) were added to refs/heads/asf-site by this push:
     new 604a20304 Publish built docs triggered by 964e57892a3397faf216e781e71c35352c6e4745
604a20304 is described below

commit 604a2030459d0742868a5ba7dbc1cdc82171ddbd
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Sat Apr 25 00:59:22 2026 +0000

    Publish built docs triggered by 964e57892a3397faf216e781e71c35352c6e4745
---
 _sources/user-guide/latest/configs.md.txt |  1 +
 _sources/user-guide/latest/tuning.md.txt  | 18 ++++++++++++++++++
 searchindex.js                            |  2 +-
 user-guide/latest/configs.html            | 18 +++++++++++-------
 user-guide/latest/tuning.html             | 14 ++++++++++++++
 5 files changed, 45 insertions(+), 8 deletions(-)

diff --git a/_sources/user-guide/latest/configs.md.txt b/_sources/user-guide/latest/configs.md.txt
index 76de6ac2f..497228d46 100644
--- a/_sources/user-guide/latest/configs.md.txt
+++ b/_sources/user-guide/latest/configs.md.txt
@@ -108,6 +108,7 @@ These settings can be used to determine which parts of the plan are accelerated
 | `spark.comet.exec.shuffle.compression.zstd.level` | The compression level to use when compressing shuffle files with zstd. | 1 |
 | `spark.comet.exec.shuffle.directRead.enabled` | When enabled, native operators that consume shuffle output will read compressed shuffle blocks directly in native code, bypassing Arrow FFI. Applies to both native shuffle and JVM columnar shuffle. Requires spark.comet.exec.shuffle.enabled to be true. | true |
 | `spark.comet.exec.shuffle.enabled` | Whether to enable Comet native shuffle. Note that this requires setting `spark.shuffle.manager` to `org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager`. `spark.shuffle.manager` must be set before starting the Spark application and cannot be changed during the application. | true |
+| `spark.comet.exec.shuffle.revertRedundantColumnar.enabled` | When enabled, Comet reverts a `CometShuffleExchangeExec` with `CometColumnarShuffle` back to Spark's `ShuffleExchangeExec` when both its parent and child are non-Comet hash aggregate operators. This avoids a redundant row -> Arrow -> shuffle -> Arrow -> row conversion when no Comet operator on either side can consume columnar output. Disable to keep Comet columnar shuffle even in that case, which preserves Comet's off-heap sh [...]
 | `spark.comet.exec.shuffle.writeBufferSize` | Size of the write buffer in bytes used by the native shuffle writer when writing shuffle data to disk. Larger values may improve write performance by reducing the number of system calls, but will use more memory. The default is 1MB which provides a good balance between performance and memory usage. | 1048576b |
 | `spark.comet.native.shuffle.partitioning.hash.enabled` | Whether to enable hash partitioning for Comet native shuffle. | true |
 | `spark.comet.native.shuffle.partitioning.range.enabled` | Whether to enable range partitioning for Comet native shuffle. | true |
diff --git a/_sources/user-guide/latest/tuning.md.txt b/_sources/user-guide/latest/tuning.md.txt
index ff9acee1f..fbbc2ecd7 100644
--- a/_sources/user-guide/latest/tuning.md.txt
+++ b/_sources/user-guide/latest/tuning.md.txt
@@ -154,6 +154,24 @@ partitioning keys. Columns that are not partitioning keys may contain complex ty
 Comet Columnar shuffle is JVM-based and supports `HashPartitioning`, `RoundRobinPartitioning`, `RangePartitioning`, and
 `SinglePartitioning`. This shuffle implementation supports complex data types as partitioning keys.
 
+#### Automatic Revert to Spark Shuffle
+
+When a Comet columnar shuffle ends up between two non-Comet operators (for example, a partial/final hash aggregate
+pair that Comet could not convert), Comet reverts it to Spark's built-in shuffle. Keeping columnar shuffle between
+two row-based operators would add `row -> Arrow -> shuffle -> Arrow -> row` conversions with no Comet consumer on
+either side to benefit from columnar output.
+
+This shifts the affected shuffles from Comet's off-heap memory pool back to the JVM execution memory pool. Clusters
+tuned for a small JVM heap may see `ExternalSorter` spills on queries where this revert fires. Shuffle I/O may also
+grow marginally because Spark's row-based serializer generally compresses less well than Comet's Arrow IPC format.
+
+Each revert is logged at `INFO` level on the driver as `Reverting Comet columnar shuffle to Spark shuffle between
+<parent> and <child>`, which lets you correlate any unexpected behavior with this optimization.
+
+This optimization is enabled by default and can be disabled by setting
+`spark.comet.exec.shuffle.revertRedundantColumnar.enabled=false`, in which case Comet will keep the columnar shuffle
+even when both its parent and child are non-Comet operators.
+
 ### Shuffle Compression
 
 By default, Spark compresses shuffle files using LZ4 compression. Comet overrides this behavior with ZSTD compression.
diff --git a/searchindex.js b/searchindex.js
index fd5eea5fb..32d7af7b4 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"1. Format Your Code": [[14, "format-your-code"]], "1. Install Comet": [[16, "install-comet"], [24, "install-comet"]], "1. Native Operators (nativeExecs map)": [[4, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[14, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[16, "clone-iceberg-and-apply-diff"]], "2. Clone Spark and Apply Diff": [[24, "clone-spark-and-apply-diff"]], "2. Sink Operators (sinks map)": [[4, "sink-operators-sinks-m [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"1. Format Your Code": [[14, "format-your-code"]], "1. Install Comet": [[16, "install-comet"], [24, "install-comet"]], "1. Native Operators (nativeExecs map)": [[4, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[14, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[16, "clone-iceberg-and-apply-diff"]], "2. Clone Spark and Apply Diff": [[24, "clone-spark-and-apply-diff"]], "2. Sink Operators (sinks map)": [[4, "sink-operators-sinks-m [...]
\ No newline at end of file
diff --git a/user-guide/latest/configs.html b/user-guide/latest/configs.html
index 85379d90b..925cdff60 100644
--- a/user-guide/latest/configs.html
+++ b/user-guide/latest/configs.html
@@ -718,31 +718,35 @@ under the License.
 <td><p>Whether to enable Comet native shuffle. Note that this requires setting <code class="docutils literal notranslate"><span class="pre">spark.shuffle.manager</span></code> to <code class="docutils literal notranslate"><span class="pre">org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager</span></code>. <code class="docutils literal notranslate"><span class="pre">spark.shuffle.manager</span></code> must be set before starting the Spark application and cannot be changed dur [...]
 <td><p>true</p></td>
 </tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.shuffle.writeBufferSize</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.shuffle.revertRedundantColumnar.enabled</span></code></p></td>
+<td><p>When enabled, Comet reverts a <code class="docutils literal notranslate"><span class="pre">CometShuffleExchangeExec</span></code> with <code class="docutils literal notranslate"><span class="pre">CometColumnarShuffle</span></code> back to Spark’s <code class="docutils literal notranslate"><span class="pre">ShuffleExchangeExec</span></code> when both its parent and child are non-Comet hash aggregate operators. This avoids a redundant row -&gt; Arrow -&gt; shuffle -&gt; Arrow -&gt;  [...]
+<td><p>true</p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.shuffle.writeBufferSize</span></code></p></td>
 <td><p>Size of the write buffer in bytes used by the native shuffle writer when writing shuffle data to disk. Larger values may improve write performance by reducing the number of system calls, but will use more memory. The default is 1MB which provides a good balance between performance and memory usage.</p></td>
 <td><p>1048576b</p></td>
 </tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.native.shuffle.partitioning.hash.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.native.shuffle.partitioning.hash.enabled</span></code></p></td>
 <td><p>Whether to enable hash partitioning for Comet native shuffle.</p></td>
 <td><p>true</p></td>
 </tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.native.shuffle.partitioning.range.enabled</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.native.shuffle.partitioning.range.enabled</span></code></p></td>
 <td><p>Whether to enable range partitioning for Comet native shuffle.</p></td>
 <td><p>true</p></td>
 </tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.native.shuffle.partitioning.roundrobin.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.native.shuffle.partitioning.roundrobin.enabled</span></code></p></td>
 <td><p>Whether to enable round robin partitioning for Comet native shuffle. This is disabled by default because Comet’s round-robin produces different partition assignments than Spark. Spark sorts rows by their binary UnsafeRow representation before assigning partitions, but Comet uses Arrow format which has a different binary layout. Instead, Comet implements round-robin as hash partitioning on all columns, which achieves the same goals: even distribution, deterministic output (for faul [...]
 <td><p>false</p></td>
 </tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.native.shuffle.partitioning.roundrobin.maxHashColumns</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.native.shuffle.partitioning.roundrobin.maxHashColumns</span></code></p></td>
 <td><p>The maximum number of columns to hash for round robin partitioning. When set to 0 (the default), all columns are hashed. When set to a positive value, only the first N columns are used for hashing, which can improve performance for wide tables while still providing reasonable distribution.</p></td>
 <td><p>0</p></td>
 </tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.shuffle.preferDictionary.ratio</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.shuffle.preferDictionary.ratio</span></code></p></td>
 <td><p>The ratio of total values to distinct values in a string column to decide whether to prefer dictionary encoding when shuffling the column. If the ratio is higher than this config, dictionary encoding will be used on shuffling string column. This config is effective if it is higher than 1.0. Note that this config is only used when <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.shuffle.mode</span></code> is <code class="docutils literal notranslate"><s [...]
 <td><p>10.0</p></td>
 </tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.shuffle.sizeInBytesMultiplier</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.shuffle.sizeInBytesMultiplier</span></code></p></td>
 <td><p>Comet reports smaller sizes for shuffle due to using Arrow’s columnar memory format and this can result in Spark choosing a different join strategy due to the estimated size of the exchange being smaller. Comet will multiple sizeInBytes by this amount to avoid regressions in join strategy.</p></td>
 <td><p>1.0</p></td>
 </tr>
diff --git a/user-guide/latest/tuning.html b/user-guide/latest/tuning.html
index ce503f78b..30af63929 100644
--- a/user-guide/latest/tuning.html
+++ b/user-guide/latest/tuning.html
@@ -571,6 +571,20 @@ partitioning keys. Columns that are not partitioning keys may contain complex ty
 <p>Comet Columnar shuffle is JVM-based and supports <code class="docutils literal notranslate"><span class="pre">HashPartitioning</span></code>, <code class="docutils literal notranslate"><span class="pre">RoundRobinPartitioning</span></code>, <code class="docutils literal notranslate"><span class="pre">RangePartitioning</span></code>, and
 <code class="docutils literal notranslate"><span class="pre">SinglePartitioning</span></code>. This shuffle implementation supports complex data types as partitioning keys.</p>
 </section>
+<section id="automatic-revert-to-spark-shuffle">
+<h4>Automatic Revert to Spark Shuffle<a class="headerlink" href="#automatic-revert-to-spark-shuffle" title="Link to this heading">#</a></h4>
+<p>When a Comet columnar shuffle ends up between two non-Comet operators (for example, a partial/final hash aggregate
+pair that Comet could not convert), Comet reverts it to Spark’s built-in shuffle. Keeping columnar shuffle between
+two row-based operators would add <code class="docutils literal notranslate"><span class="pre">row</span> <span class="pre">-&gt;</span> <span class="pre">Arrow</span> <span class="pre">-&gt;</span> <span class="pre">shuffle</span> <span class="pre">-&gt;</span> <span class="pre">Arrow</span> <span class="pre">-&gt;</span> <span class="pre">row</span></code> conversions with no Comet consumer on
+either side to benefit from columnar output.</p>
+<p>This shifts the affected shuffles from Comet’s off-heap memory pool back to the JVM execution memory pool. Clusters
+tuned for a small JVM heap may see <code class="docutils literal notranslate"><span class="pre">ExternalSorter</span></code> spills on queries where this revert fires. Shuffle I/O may also
+grow marginally because Spark’s row-based serializer generally compresses less well than Comet’s Arrow IPC format.</p>
+<p>Each revert is logged at <code class="docutils literal notranslate"><span class="pre">INFO</span></code> level on the driver as <code class="docutils literal notranslate"><span class="pre">Reverting</span> <span class="pre">Comet</span> <span class="pre">columnar</span> <span class="pre">shuffle</span> <span class="pre">to</span> <span class="pre">Spark</span> <span class="pre">shuffle</span> <span class="pre">between</span> <span class="pre">&lt;parent&gt;</span> <span class="pre">an [...]
+<p>This optimization is enabled by default and can be disabled by setting
+<code class="docutils literal notranslate"><span class="pre">spark.comet.exec.shuffle.revertRedundantColumnar.enabled=false</span></code>, in which case Comet will keep the columnar shuffle
+even when both its parent and child are non-Comet operators.</p>
+</section>
 </section>
 <section id="shuffle-compression">
 <h3>Shuffle Compression<a class="headerlink" href="#shuffle-compression" title="Link to this heading">#</a></h3>


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org