You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/07 11:39:03 UTC
(datafusion-comet) branch asf-site updated: Publish built docs triggered by e038cf05fa665dc61fdecbdde1c07b8b6a1ce3cb
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 8decac148f Publish built docs triggered by e038cf05fa665dc61fdecbdde1c07b8b6a1ce3cb
8decac148f is described below
commit 8decac148f7b800f1b7a60c164983a20d33ab005
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Wed Oct 7 11:38:58 2026 +0000
Publish built docs triggered by e038cf05fa665dc61fdecbdde1c07b8b6a1ce3cb
---
...id-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg | 2 +-
_sources/contributor-guide/jvm_shuffle.md.txt | 6 ++-
_sources/contributor-guide/native_shuffle.md.txt | 21 +++++---
_sources/user-guide/latest/configs.md.txt | 3 +-
_sources/user-guide/latest/tuning/shuffle.md.txt | 25 +++++++---
contributor-guide/jvm_shuffle.html | 6 ++-
contributor-guide/native_shuffle.html | 21 +++++---
searchindex.js | 2 +-
user-guide/latest/configs.html | 56 ++++++++++++----------
user-guide/latest/tuning/shuffle.html | 22 ++++++---
10 files changed, 106 insertions(+), 58 deletions(-)
diff --git a/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg b/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg
index 6823f15fc9..ec81c0c263 100644
--- a/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg
+++ b/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg
@@ -1 +1 @@
-<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 966" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
+<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 966" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
diff --git a/_sources/contributor-guide/jvm_shuffle.md.txt b/_sources/contributor-guide/jvm_shuffle.md.txt
index 11014f3490..dca7144a01 100644
--- a/_sources/contributor-guide/jvm_shuffle.md.txt
+++ b/_sources/contributor-guide/jvm_shuffle.md.txt
@@ -43,8 +43,10 @@ JVM shuffle (`CometColumnarExchange`) is used instead of native shuffle (`CometE
1. **Shuffle mode is explicitly set to "jvm"**: When `spark.comet.shuffle.mode` is set to `jvm`.
2. **Child plan is not a Comet native operator**: When the child plan is a Spark row-based operator
- (not a `CometPlan`), JVM shuffle is the only option since native shuffle requires columnar input
- from Comet operators.
+ (not a `CometPlan`), JVM shuffle is used, since native shuffle requires columnar input from
+ Comet operators. The exception is `spark.comet.convert.shuffleInput.enabled`, which converts
+ the child's rows to Arrow with `CometSparkToColumnarExec` so that native shuffle can take the
+ shuffle instead. See [When Native Shuffle is Used](native_shuffle.md#when-native-shuffle-is-used).
3. **Unsupported partition key types**: `RangePartitioning` keys must be primitive, so a complex
range key always falls back here. `HashPartitioning` keys must be primitive only by default:
diff --git a/_sources/contributor-guide/native_shuffle.md.txt b/_sources/contributor-guide/native_shuffle.md.txt
index e9cf777441..704fa2a0ea 100644
--- a/_sources/contributor-guide/native_shuffle.md.txt
+++ b/_sources/contributor-guide/native_shuffle.md.txt
@@ -52,7 +52,13 @@ Native shuffle (`CometExchange`) is selected when all of the following condition
1. **Shuffle mode allows native**: `spark.comet.shuffle.mode` is `native` or `auto`.
2. **Child plan is a Comet native operator**: The child must be a `CometPlan` that produces
- columnar output. Row-based Spark operators require JVM shuffle.
+ columnar output. Row-based Spark operators require JVM shuffle, unless
+ `spark.comet.convert.shuffleInput.enabled` is set. Then a shuffle that JVM shuffle would take
+ uses native shuffle instead, provided native shuffle supports the partitioning and the
+ columns, and a `CometSparkToColumnarExec` converts the child's rows to Arrow for it. A shuffle
+ that hashes a string, a value computed from a string, or a decimal wider than 18 digits stays
+ on JVM shuffle, because native shuffle would not put every row in the partition Spark's
+ partitioner does (`CometShuffleExchangeExec.convertsInputForNativeShuffle`).
3. **Supported partitioning type**: Native shuffle supports:
- `HashPartitioning`
@@ -166,11 +172,14 @@ The native shuffle implementation is its own workspace crate, `datafusion-comet-
- The child plan's `nativeOp` directly, when `CometShuffleExchangeExec`'s child is a
`CometNativeExec` subtree. The upstream operators run inside the same `CometExecIterator`
as the writer, with no JVM-to-native batch boundary between them.
- - A synthetic `Scan("ShuffleWriterInput")` placeholder, when the dep was built via the
- convenience `prepareShuffleDependency(rdd, ...)` overload (used by
- `CometCollectLimitExec` and `CometTakeOrderedAndProjectExec`, or when the
- exchange's child is a non-native `CometPlan` such as `CometSparkToColumnarExec`). Native
- code reads `ColumnarBatch`es from the JVM input iterator via Arrow C Stream Interface.
+ - A synthetic `Scan("ShuffleWriterInput")` placeholder, when the exchange's child is a
+ non-native `CometPlan`, and for the dependencies that `CometCollectLimitExec` and
+ `CometTakeOrderedAndProjectExec` build. Native code reads the input through the Arrow C
+ Stream Interface. A `CometNativeArrowSource` child, such as `CometSparkToColumnarExec`,
+ exports its own Arrow stream from `doExecuteAsArrowStream()`, which
+ `prepareArrowStreamShuffleDependency` hands to native. Any other input is an
+ `RDD[ColumnarBatch]`, which the convenience `prepareShuffleDependency(rdd, ...)` overload
+ wraps in a stream.
2. **Native execution**: A single `CometExecIterator` per partition runs the unified plan.
diff --git a/_sources/user-guide/latest/configs.md.txt b/_sources/user-guide/latest/configs.md.txt
index 1a51b9a253..bf6af5aa7e 100644
--- a/_sources/user-guide/latest/configs.md.txt
+++ b/_sources/user-guide/latest/configs.md.txt
@@ -62,6 +62,7 @@ Comet provides the following configuration settings.
| `spark.comet.convert.range.enabled` | When enabled, the rows of `spark.range` and SQL `range()` that Comet does not generate natively (see `spark.comet.exec.range.enabled`) will be converted to Arrow format. | false |
| `spark.comet.convert.rdd.enabled` | When enabled, data from a DataFrame created from an RDD of rows, for example with `spark.createDataFrame(rdd, schema)`, will be converted to Arrow format. | false |
| `spark.comet.convert.rowDataSource.enabled` | When enabled, data from Data Source V1 relations that are not file-based, such as JDBC tables, will be converted to Arrow format. Spark scans these relations with `RowDataSourceScanExec`. | false |
+| `spark.comet.convert.shuffleInput.enabled` | When enabled, a shuffle that would use Comet's JVM columnar shuffle because its input comes from a Spark operator converts that input to Arrow format and uses Comet native shuffle instead. This applies only where native shuffle supports the partitioning and Comet can convert all of the input's columns, which rules out calendar intervals. A shuffle that hashes a decimal wider than 18 digits, or a string or a value computed from a string, stay [...]
| `spark.comet.convert.typedDataset.enabled` | When enabled, the output of typed Dataset operations, such as `map`, `flatMap`, `mapPartitions` and `groupByKey(...).mapGroups`, will be converted to Arrow format so that the operators above them can run natively. The user function still runs in Spark. This pays off when the operators above do enough work, such as an aggregation over many groups, and can be slower when they are cheap, such as an aggregation over a few groups after a selectiv [...]
| `spark.comet.debug.enabled` | Whether to enable debug mode for Comet. When enabled, Comet will do additional checks for debugging purpose. For example, validating array when importing arrays from JVM at native side. Note that these checks may be expensive in performance and should only be enabled for debugging purpose. | false |
| `spark.comet.enabled` | Whether to enable Comet extension for Spark. When this is turned on, Spark will use Comet to read Parquet data source. Note that to enable native vectorized execution, both this config and `spark.comet.exec.enabled` need to be enabled. It can be overridden by the environment variable `ENABLE_COMET`. | true |
@@ -137,7 +138,7 @@ interact, including encoding workspace and local fallback for oversized rows.
| `spark.comet.shuffle.native.partitioning.roundrobin.positional.enabled` | When true, Comet's native round-robin shuffle places rows by position rather than by hashing their contents, sending each map task's rows to the output partitions in turn, in contiguous groups. This skips a murmur3 pass over every column of every row and replaces the per-row gather on flush with a bulk copy per group, which is what dominates the shuffle write on wide nested schemas, and it spreads duplicate rows [...]
| `spark.comet.shuffle.native.partitioning.roundrobin.positional.groupRows` | Rows per contiguous group under positional round robin. Smaller groups balance better and larger ones are cheaper to copy. Within one map task, output partitions differ by at most this many rows, but a reducer receives groups from every map task, so the stage is only evenly balanced when each task emits many more groups than there are output partitions; a group approaching a task's whole input skews the stage a [...]
| `spark.comet.shuffle.native.writeBufferSize` | Size of the write buffer in bytes used by the native shuffle writer when writing shuffle data to disk. Larger values may improve write performance by reducing the number of system calls, but will use more memory. The default is 1MB which provides a good balance between performance and memory usage. | 1048576b |
-| `spark.comet.shuffle.revertRedundantColumnar.enabled` | When enabled, Comet reverts a `CometShuffleExchangeExec` with `CometColumnarShuffle` back to Spark's `ShuffleExchangeExec` when both its parent and child are non-Comet hash aggregate operators. This avoids a redundant row -> Arrow -> shuffle -> Arrow -> row conversion when no Comet operator on either side can consume columnar output. Disable to keep Comet columnar shuffle even in that case, which preserves Comet's off-heap shuffle [...]
+| `spark.comet.shuffle.revertRedundantColumnar.enabled` | When enabled, Comet reverts a `CometShuffleExchangeExec` with `CometColumnarShuffle` back to Spark's `ShuffleExchangeExec` when both its parent and child are non-Comet aggregate operators. The same applies to a native shuffle whose input `spark.comet.convert.shuffleInput.enabled` converted. This avoids a redundant row -> Arrow -> shuffle -> Arrow -> row conversion when no Comet operator on either side can consume columnar output. [...]
| `spark.comet.shuffle.rss.maxFrameBytes` | Maximum encoded size of one complete native shuffle frame sent to a remote shuffle service. Frames are never split across remote push requests. | 67108864b |
| `spark.comet.shuffle.rss.maxInFlightBytes` | Maximum shuffle bytes admitted concurrently by native Comet map attempts sharing an executor-side remote shuffle client. Admission includes native encoding scratch and overlapping native, JNI, and remote shuffle frame copies. A frame must fit its codec and Arrow workspace as well as its encoded bytes; ordinary uncompressed frames need approximately seven times their size plus schema and transport overhead. Compressed frames also reserve work [...]
| `spark.comet.shuffle.sizeInBytesMultiplier` | Comet reports smaller sizes for shuffle due to using Arrow's columnar memory format and this can result in Spark choosing a different join strategy due to the estimated size of the exchange being smaller. Comet will multiple sizeInBytes by this amount to avoid regressions in join strategy. | 1.0 |
diff --git a/_sources/user-guide/latest/tuning/shuffle.md.txt b/_sources/user-guide/latest/tuning/shuffle.md.txt
index 71c9c431ab..b7ad3d2f09 100644
--- a/_sources/user-guide/latest/tuning/shuffle.md.txt
+++ b/_sources/user-guide/latest/tuning/shuffle.md.txt
@@ -66,23 +66,34 @@ write side. To restrict columnar shuffle to cases where the child is already a C
as native Spark shuffles, which avoids the row to columnar conversion but means the downstream stage will also start
on Spark.
+Setting `spark.comet.convert.shuffleInput.enabled=true` makes such a shuffle use native shuffle instead. Comet
+converts the child's rows to Arrow, as the `spark.comet.convert.*` settings do for Spark scans, and shuffles the
+batches natively. With hash partitioning that is usually faster than columnar shuffle, while with range partitioning
+the two come out about even. This applies only where native shuffle supports the partitioning and Comet can convert
+all of the child's columns, which rules out calendar intervals, and arrays and maps other than `array<string>` and
+`map<string,string>`. A shuffle that hashes a decimal wider than 18 digits, or a string or a value computed from a
+string, stays on columnar shuffle, so that it partitions rows as Spark does. The converted batches are allocated on
+the JVM outside Comet's memory pools. The setting is disabled by default.
+
### Automatic Revert to Spark Shuffle
-When a Comet columnar shuffle ends up between a partial and a final aggregate that Comet could not convert (both
-remain Spark `HashAggregateExec` or `ObjectHashAggregateExec` operators), Comet reverts it to Spark's built-in shuffle.
+When a Comet columnar shuffle, or a native shuffle over rows that `spark.comet.convert.shuffleInput.enabled` converted,
+ends up between a partial and a final aggregate that Comet could not convert (both remain Spark `HashAggregateExec`,
+`ObjectHashAggregateExec` or `SortAggregateExec` operators), Comet reverts it to Spark's built-in shuffle.
Keeping columnar shuffle between the two row-based aggregates would add `row -> Arrow -> shuffle -> Arrow -> row`
-conversions with no Comet consumer on either side to benefit from columnar output. Other shuffles between non-Comet
-operators are not reverted.
+conversions with no Comet consumer on either side to benefit from columnar output. The final aggregate has to read the
+shuffle directly, which a `SortAggregateExec` with grouping keys does not: it reads the shuffle through a sort, so that
+shuffle is not reverted. Other shuffles between non-Comet operators are not reverted either.
This shifts the affected shuffles from Comet's off-heap memory pool back to the JVM execution memory pool. Clusters
tuned for a small JVM heap may see `ExternalSorter` spills on queries where this revert fires. Shuffle I/O may also
grow marginally because Spark's row-based serializer generally compresses less well than Comet's Arrow IPC format.
-Each revert is logged at `INFO` level on the driver as `Reverting Comet columnar shuffle to Spark shuffle between
-<parent> and <child>`, which lets you correlate any unexpected behavior with this optimization.
+Each revert is logged at `INFO` level on the driver as `Reverting Comet shuffle to Spark shuffle between <parent>
+and <child>`, which lets you correlate any unexpected behavior with this optimization.
This optimization is enabled by default and can be disabled by setting
-`spark.comet.shuffle.revertRedundantColumnar.enabled=false`, in which case Comet will keep the columnar shuffle
+`spark.comet.shuffle.revertRedundantColumnar.enabled=false`, in which case Comet will keep its shuffle
even when both of those aggregates run on Spark.
## Shuffle Compression
diff --git a/contributor-guide/jvm_shuffle.html b/contributor-guide/jvm_shuffle.html
index c753ff43f7..71617dd972 100644
--- a/contributor-guide/jvm_shuffle.html
+++ b/contributor-guide/jvm_shuffle.html
@@ -573,8 +573,10 @@ Uses Spark’s partitioner for partition assignment. Can accept either row-based
<ol class="arabic simple">
<li><p><strong>Shuffle mode is explicitly set to “jvm”</strong>: When <code class="docutils literal notranslate"><span class="pre">spark.comet.shuffle.mode</span></code> is set to <code class="docutils literal notranslate"><span class="pre">jvm</span></code>.</p></li>
<li><p><strong>Child plan is not a Comet native operator</strong>: When the child plan is a Spark row-based operator
-(not a <code class="docutils literal notranslate"><span class="pre">CometPlan</span></code>), JVM shuffle is the only option since native shuffle requires columnar input
-from Comet operators.</p></li>
+(not a <code class="docutils literal notranslate"><span class="pre">CometPlan</span></code>), JVM shuffle is used, since native shuffle requires columnar input from
+Comet operators. The exception is <code class="docutils literal notranslate"><span class="pre">spark.comet.convert.shuffleInput.enabled</span></code>, which converts
+the child’s rows to Arrow with <code class="docutils literal notranslate"><span class="pre">CometSparkToColumnarExec</span></code> so that native shuffle can take the
+shuffle instead. See <a class="reference internal" href="native_shuffle.html#when-native-shuffle-is-used"><span class="std std-ref">When Native Shuffle is Used</span></a>.</p></li>
<li><p><strong>Unsupported partition key types</strong>: <code class="docutils literal notranslate"><span class="pre">RangePartitioning</span></code> keys must be primitive, so a complex
range key always falls back here. <code class="docutils literal notranslate"><span class="pre">HashPartitioning</span></code> keys must be primitive only by default:
setting <code class="docutils literal notranslate"><span class="pre">spark.comet.shuffle.native.partitioning.hash.nested.enabled</span></code> to <code class="docutils literal notranslate"><span class="pre">true</span></code> keeps struct and
diff --git a/contributor-guide/native_shuffle.html b/contributor-guide/native_shuffle.html
index fbf10662dc..a3ff5ada48 100644
--- a/contributor-guide/native_shuffle.html
+++ b/contributor-guide/native_shuffle.html
@@ -578,7 +578,13 @@ them. Spark’s <code class="docutils literal notranslate"><span class="pre">Row
<ol class="arabic simple">
<li><p><strong>Shuffle mode allows native</strong>: <code class="docutils literal notranslate"><span class="pre">spark.comet.shuffle.mode</span></code> is <code class="docutils literal notranslate"><span class="pre">native</span></code> or <code class="docutils literal notranslate"><span class="pre">auto</span></code>.</p></li>
<li><p><strong>Child plan is a Comet native operator</strong>: The child must be a <code class="docutils literal notranslate"><span class="pre">CometPlan</span></code> that produces
-columnar output. Row-based Spark operators require JVM shuffle.</p></li>
+columnar output. Row-based Spark operators require JVM shuffle, unless
+<code class="docutils literal notranslate"><span class="pre">spark.comet.convert.shuffleInput.enabled</span></code> is set. Then a shuffle that JVM shuffle would take
+uses native shuffle instead, provided native shuffle supports the partitioning and the
+columns, and a <code class="docutils literal notranslate"><span class="pre">CometSparkToColumnarExec</span></code> converts the child’s rows to Arrow for it. A shuffle
+that hashes a string, a value computed from a string, or a decimal wider than 18 digits stays
+on JVM shuffle, because native shuffle would not put every row in the partition Spark’s
+partitioner does (<code class="docutils literal notranslate"><span class="pre">CometShuffleExchangeExec.convertsInputForNativeShuffle</span></code>).</p></li>
<li><p><strong>Supported partitioning type</strong>: Native shuffle supports:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">HashPartitioning</span></code></p></li>
@@ -752,11 +758,14 @@ one of two shapes:</p>
<li><p>The child plan’s <code class="docutils literal notranslate"><span class="pre">nativeOp</span></code> directly, when <code class="docutils literal notranslate"><span class="pre">CometShuffleExchangeExec</span></code>’s child is a
<code class="docutils literal notranslate"><span class="pre">CometNativeExec</span></code> subtree. The upstream operators run inside the same <code class="docutils literal notranslate"><span class="pre">CometExecIterator</span></code>
as the writer, with no JVM-to-native batch boundary between them.</p></li>
-<li><p>A synthetic <code class="docutils literal notranslate"><span class="pre">Scan("ShuffleWriterInput")</span></code> placeholder, when the dep was built via the
-convenience <code class="docutils literal notranslate"><span class="pre">prepareShuffleDependency(rdd,</span> <span class="pre">...)</span></code> overload (used by
-<code class="docutils literal notranslate"><span class="pre">CometCollectLimitExec</span></code> and <code class="docutils literal notranslate"><span class="pre">CometTakeOrderedAndProjectExec</span></code>, or when the
-exchange’s child is a non-native <code class="docutils literal notranslate"><span class="pre">CometPlan</span></code> such as <code class="docutils literal notranslate"><span class="pre">CometSparkToColumnarExec</span></code>). Native
-code reads <code class="docutils literal notranslate"><span class="pre">ColumnarBatch</span></code>es from the JVM input iterator via Arrow C Stream Interface.</p></li>
+<li><p>A synthetic <code class="docutils literal notranslate"><span class="pre">Scan("ShuffleWriterInput")</span></code> placeholder, when the exchange’s child is a
+non-native <code class="docutils literal notranslate"><span class="pre">CometPlan</span></code>, and for the dependencies that <code class="docutils literal notranslate"><span class="pre">CometCollectLimitExec</span></code> and
+<code class="docutils literal notranslate"><span class="pre">CometTakeOrderedAndProjectExec</span></code> build. Native code reads the input through the Arrow C
+Stream Interface. A <code class="docutils literal notranslate"><span class="pre">CometNativeArrowSource</span></code> child, such as <code class="docutils literal notranslate"><span class="pre">CometSparkToColumnarExec</span></code>,
+exports its own Arrow stream from <code class="docutils literal notranslate"><span class="pre">doExecuteAsArrowStream()</span></code>, which
+<code class="docutils literal notranslate"><span class="pre">prepareArrowStreamShuffleDependency</span></code> hands to native. Any other input is an
+<code class="docutils literal notranslate"><span class="pre">RDD[ColumnarBatch]</span></code>, which the convenience <code class="docutils literal notranslate"><span class="pre">prepareShuffleDependency(rdd,</span> <span class="pre">...)</span></code> overload
+wraps in a stream.</p></li>
</ul>
</li>
<li><p><strong>Native execution</strong>: A single <code class="docutils literal notranslate"><span class="pre">CometExecIterator</span></code> per partition runs the unified plan.</p></li>
diff --git a/searchindex.js b/searchindex.js
index 7214386ce9..8d2b7e0567 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[58, "id1"]], "%": [[56, "id1"]], "&": [[46, "id1"]], "*": [[56, "id2"]], "+": [[56, "id3"]], "-": [[56, "id4"]], "/": [[56, "id5"]], "1. Format Your Code": [[43, "format-your-code"]], "1. Install Comet": [[64, "install-comet"], [76, "install-comet"]], "1. Native Operators (nativeExecs map)": [[29, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[43, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[64, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[58, "id1"]], "%": [[56, "id1"]], "&": [[46, "id1"]], "*": [[56, "id2"]], "+": [[56, "id3"]], "-": [[56, "id4"]], "/": [[56, "id5"]], "1. Format Your Code": [[43, "format-your-code"]], "1. Install Comet": [[64, "install-comet"], [76, "install-comet"]], "1. Native Operators (nativeExecs map)": [[29, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[43, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[64, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/configs.html b/user-guide/latest/configs.html
index dbf0f1018c..3458f193a9 100644
--- a/user-guide/latest/configs.html
+++ b/user-guide/latest/configs.html
@@ -962,103 +962,107 @@ under the License.
<td><p>When enabled, data from Data Source V1 relations that are not file-based, such as JDBC tables, will be converted to Arrow format. Spark scans these relations with <code class="docutils literal notranslate"><span class="pre">RowDataSourceScanExec</span></code>.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.convert.typedDataset.enabled</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.convert.shuffleInput.enabled</span></code></p></td>
+<td><p>When enabled, a shuffle that would use Comet’s JVM columnar shuffle because its input comes from a Spark operator converts that input to Arrow format and uses Comet native shuffle instead. This applies only where native shuffle supports the partitioning and Comet can convert all of the input’s columns, which rules out calendar intervals. A shuffle that hashes a decimal wider than 18 digits, or a string or a value computed from a string, stays on the JVM columnar shuffle, so that i [...]
+<td><p>false</p></td>
+</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.convert.typedDataset.enabled</span></code></p></td>
<td><p>When enabled, the output of typed Dataset operations, such as <code class="docutils literal notranslate"><span class="pre">map</span></code>, <code class="docutils literal notranslate"><span class="pre">flatMap</span></code>, <code class="docutils literal notranslate"><span class="pre">mapPartitions</span></code> and <code class="docutils literal notranslate"><span class="pre">groupByKey(...).mapGroups</span></code>, will be converted to Arrow format so that the operators above th [...]
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.debug.enabled</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.debug.enabled</span></code></p></td>
<td><p>Whether to enable debug mode for Comet. When enabled, Comet will do additional checks for debugging purpose. For example, validating array when importing arrays from JVM at native side. Note that these checks may be expensive in performance and should only be enabled for debugging purpose.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.enabled</span></code></p></td>
<td><p>Whether to enable Comet extension for Spark. When this is turned on, Spark will use Comet to read Parquet data source. Note that to enable native vectorized execution, both this config and <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.enabled</span></code> need to be enabled. It can be overridden by the environment variable <code class="docutils literal notranslate"><span class="pre">ENABLE_COMET</span></code>.</p></td>
<td><p>true</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.aggregate.skipPartial.enabled</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.aggregate.skipPartial.enabled</span></code></p></td>
<td><p>Experimental opt-in: let a native partial aggregate stop aggregating once its input looks mostly distinct, and send the rest of the task’s rows to the shuffle unaggregated. Only applies to partial aggregates that feed Comet native shuffle and whose aggregate functions, if any, are all single-argument COUNT. The check starts after the first 100,000 input rows of a task, and once aggregation stops it does not resume, so a task whose keys repeat after a mostly distinct start can shuf [...]
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.columnarToRow.native.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.columnarToRow.native.enabled</span></code></p></td>
<td><p>Whether to enable native columnar to row conversion. When enabled, Comet will use native Rust code to convert Arrow columnar data to Spark UnsafeRow format instead of the JVM implementation. The native conversion carries a fixed JNI cost per batch and is slower than the JVM implementation for small batches.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.enabled</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.enabled</span></code></p></td>
<td><p>Whether to enable Comet native vectorized execution for Spark. This controls whether Spark should convert operators into their Comet counterparts and execute them in native space. Note: each operator is associated with a separate config in the format of <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.<operator_name>.enabled</span></code> at the moment, and both the config and this need to be turned on, in order for the operator to be executed in [...]
<td><p>true</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.forceShuffledHashJoin</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.forceShuffledHashJoin</span></code></p></td>
<td><p>Experimental feature to force Spark to replace SortMergeJoin with ShuffledHashJoin for improved performance. This feature is not stable yet. For more information, refer to the <a class="reference external" href="https://datafusion.apache.org/comet/user-guide/latest/tuning.html">Comet Tuning Guide</a>.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.inMemoryCache.compression.codec</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.inMemoryCache.compression.codec</span></code></p></td>
<td><p>The Arrow IPC compression codec used when Comet’s cache serializer writes cached data. Unlike spark.io.compression.codec, this compresses each Arrow buffer separately rather than the batch as a whole, which is what lets a projected scan decompress only the columns it selected. zstd is the default for footprint rather than speed: it stores cached data in a fraction of the memory, but makes materializing slower, and a read slower in proportion to how many columns it selects. Set to [...]
<td><p>zstd</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.inMemoryCache.compression.zstd.level</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.inMemoryCache.compression.zstd.level</span></code></p></td>
<td><p>The compression level to use when Comet’s cache serializer compresses cached data with zstd. Ignored for other codecs.</p></td>
<td><p>1</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.inMemoryCache.enabled</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.inMemoryCache.enabled</span></code></p></td>
<td><p>Whether to enable Comet native scans and fused Spark reads of in-memory cached tables. Requires spark.comet.enabled=true. At startup, this setting also decides whether CometDriverPlugin installs Comet’s cache serializer, which stores cached data in Arrow format. The plugin installs it only if spark.comet.enabled and spark.comet.exec.enabled are also enabled at startup, and only with one of Comet’s shuffle managers while Comet shuffle is enabled. Because spark.sql.cache.serializer [...]
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.join.dynamicFilter.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.join.dynamicFilter.enabled</span></code></p></td>
<td><p>Experimental opt-in: use a completed native hash join build’s key domain to prune eligible native Parquet row groups and filter probe batches before the hash probe. Supports inner joins with one direct signed integer key and one native partition per input, including both Spark build sides. The probe filter remains active when reader pruning is unavailable. Unsupported joins retain their existing execution path. Filters do not cross Spark exchanges or JVM/Arrow boundaries.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.pyarrowUDF.enabled</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.pyarrowUDF.enabled</span></code></p></td>
<td><p>Experimental: whether to enable optimized execution of PyArrow UDFs (mapInArrow/mapInPandas). When enabled, Comet passes Arrow columnar data directly to Python UDFs without the intermediate Arrow-to-Row-to-Arrow conversion that Spark normally performs. Disabled by default while the feature stabilizes.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.scalaUDF.codegen.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.scalaUDF.codegen.enabled</span></code></p></td>
<td><p>Whether to route Spark <code class="docutils literal notranslate"><span class="pre">ScalaUDF</span></code> expressions through Comet’s Arrow-direct codegen dispatcher. When enabled, a supported ScalaUDF is compiled into a per-batch kernel that reads and writes Arrow vectors directly from native execution. When disabled, plans containing a ScalaUDF fall back to Spark for the enclosing operator. The same dispatcher backs the regex family (<code class="docutils literal notranslate">< [...]
<td><p>true</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint</span></code></p></td>
<td><p>When enabled, fall back to Spark for floating-point operations that may differ from Spark, such as comparing -0.0 and 0.0. <code class="docutils literal notranslate"><span class="pre">ORDER</span> <span class="pre">BY</span></code>, window ordering and range partitioning keys are unaffected, including floating-point values nested in arrays and structs, because Comet normalizes those comparison keys to match Spark. The exception is a nested key whose type can hold a null element or [...]
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.topK.dynamicFilter.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.topK.dynamicFilter.enabled</span></code></p></td>
<td><p>Experimental opt-in: use a local TopK heap’s improving threshold to prune eligible native Parquet input. Requires spark.comet.exec.topK.fusion.enabled. Supports one direct signed integer sort key. Thresholds are local to each native execution and do not cross Spark exchanges or JVM/Arrow boundaries. Unsupported readers retain ordinary TopK execution.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.topK.fusion.enabled</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.topK.fusion.enabled</span></code></p></td>
<td><p>Experimental opt-in: run an eligible local TopK in the same native execution as its Parquet scan. Supports one direct signed integer sort key. This changes the local execution pipeline and can reduce scan/TopK overlap, so it may be slower for some workloads.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.transitionRevert.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.transitionRevert.enabled</span></code></p></td>
<td><p>When enabled, Comet reverts a query stage to Spark row-based execution if the number of columnar-to-row (C2R) transitions in the stage exceeds the configured threshold. This avoids the overhead of repeated format conversions in stages where many operators fall back to row-based execution.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.transitionRevert.maxTransitions</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.transitionRevert.maxTransitions</span></code></p></td>
<td><p>The maximum number of columnar-to-row (C2R) transitions allowed in a single query stage before Comet reverts the entire stage to Spark row-based execution. When columnar shuffle is enabled, each such C2R typically implies a corresponding row-to-columnar conversion to feed back into the columnar shuffle, so each counted C2R is a useful proxy for the conversion overhead in the stage. Set to 0 to revert any stage with transitions. Only effective when spark.comet.exec.transitionRevert [...]
<td><p>2</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.metrics.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.metrics.enabled</span></code></p></td>
<td><p>Whether to enable Comet metrics reporting through Spark’s external monitoring system. When enabled, Comet exposes metrics such as native operators, Spark operators, queries planned, transitions, and acceleration ratio. These metrics can be visualized through tools like Grafana when a metrics sink (e.g., Prometheus) is configured. Disabled by default because Spark plan traversal adds overhead and metrics require a sink to be useful. This config must be set before the SparkSession i [...]
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.metrics.updateInterval</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.metrics.updateInterval</span></code></p></td>
<td><p>The interval in milliseconds to update metrics. If interval is negative, metrics will be updated upon task completion.</p></td>
<td><p>3000</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.nativeLoadRequired</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.nativeLoadRequired</span></code></p></td>
<td><p>Whether to require Comet native library to load successfully when Comet is enabled. If not, Comet will silently fallback to Spark when it fails to load the native lib. Otherwise, an error will be thrown and the Spark job will be aborted.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.operator.DataWritingCommandExec.allowIncompatible</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.operator.DataWritingCommandExec.allowIncompatible</span></code></p></td>
<td><p>Whether to allow incompatibility for operator: DataWritingCommandExec. False by default. Can be overridden with SPARK_COMET_OPERATOR_DATAWRITINGCOMMANDEXEC_ALLOWINCOMPATIBLE env variable It can be overridden by the environment variable <code class="docutils literal notranslate"><span class="pre">SPARK_COMET_OPERATOR_DATAWRITINGCOMMANDEXEC_ALLOWINCOMPATIBLE</span></code>.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.operator.WriteFilesExec.allowIncompatible</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.operator.WriteFilesExec.allowIncompatible</span></code></p></td>
<td><p>Whether to allow incompatibility for operator: WriteFilesExec. False by default. Can be overridden with SPARK_COMET_OPERATOR_WRITEFILESEXEC_ALLOWINCOMPATIBLE env variable It can be overridden by the environment variable <code class="docutils literal notranslate"><span class="pre">SPARK_COMET_OPERATOR_WRITEFILESEXEC_ALLOWINCOMPATIBLE</span></code>.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.enabled</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.enabled</span></code></p></td>
<td><p>Whether to convert the output of the leaf operators named in <code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.supportedOperatorList</span></code> to Arrow format. Ranges, in-memory cached tables, RDDs and queries without a FROM clause have <code class="docutils literal notranslate"><span class="pre">spark.comet.convert</span></code> configs of their own. When the list is not set, this converts those four instead, which is deprecated and will [...]
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.supportedOperatorList</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.supportedOperatorList</span></code></p></td>
<td><p>A comma-separated list of leaf operators to convert to Arrow format when <code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.enabled</span></code> is true, each named by its Spark class name without the <code class="docutils literal notranslate"><span class="pre">Exec</span></code> suffix. It is for operators that have no <code class="docutils literal notranslate"><span class="pre">spark.comet.convert</span></code> config of their own, such as [...]
<td><p></p></td>
</tr>
@@ -1204,7 +1208,7 @@ interact, including encoding workspace and local fallback for oversized rows.</p
<td><p>1048576b</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.shuffle.revertRedundantColumnar.enabled</span></code></p></td>
-<td><p>When enabled, Comet reverts a <code class="docutils literal notranslate"><span class="pre">CometShuffleExchangeExec</span></code> with <code class="docutils literal notranslate"><span class="pre">CometColumnarShuffle</span></code> back to Spark’s <code class="docutils literal notranslate"><span class="pre">ShuffleExchangeExec</span></code> when both its parent and child are non-Comet hash aggregate operators. This avoids a redundant row -> Arrow -> shuffle -> Arrow -> [...]
+<td><p>When enabled, Comet reverts a <code class="docutils literal notranslate"><span class="pre">CometShuffleExchangeExec</span></code> with <code class="docutils literal notranslate"><span class="pre">CometColumnarShuffle</span></code> back to Spark’s <code class="docutils literal notranslate"><span class="pre">ShuffleExchangeExec</span></code> when both its parent and child are non-Comet aggregate operators. The same applies to a native shuffle whose input <code class="docutils litera [...]
<td><p>true</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.shuffle.rss.maxFrameBytes</span></code></p></td>
diff --git a/user-guide/latest/tuning/shuffle.html b/user-guide/latest/tuning/shuffle.html
index 754bcc266c..f72db5984d 100644
--- a/user-guide/latest/tuning/shuffle.html
+++ b/user-guide/latest/tuning/shuffle.html
@@ -884,20 +884,30 @@ write side. To restrict columnar shuffle to cases where the child is already a C
<code class="docutils literal notranslate"><span class="pre">spark.comet.shuffle.convertFromSparkPlan.enabled=false</span></code>. Shuffles whose child is a Spark plan will then be left
as native Spark shuffles, which avoids the row to columnar conversion but means the downstream stage will also start
on Spark.</p>
+<p>Setting <code class="docutils literal notranslate"><span class="pre">spark.comet.convert.shuffleInput.enabled=true</span></code> makes such a shuffle use native shuffle instead. Comet
+converts the child’s rows to Arrow, as the <code class="docutils literal notranslate"><span class="pre">spark.comet.convert.*</span></code> settings do for Spark scans, and shuffles the
+batches natively. With hash partitioning that is usually faster than columnar shuffle, while with range partitioning
+the two come out about even. This applies only where native shuffle supports the partitioning and Comet can convert
+all of the child’s columns, which rules out calendar intervals, and arrays and maps other than <code class="docutils literal notranslate"><span class="pre">array<string></span></code> and
+<code class="docutils literal notranslate"><span class="pre">map<string,string></span></code>. A shuffle that hashes a decimal wider than 18 digits, or a string or a value computed from a
+string, stays on columnar shuffle, so that it partitions rows as Spark does. The converted batches are allocated on
+the JVM outside Comet’s memory pools. The setting is disabled by default.</p>
</section>
<section id="automatic-revert-to-spark-shuffle">
<h3>Automatic Revert to Spark Shuffle<a class="headerlink" href="#automatic-revert-to-spark-shuffle" title="Link to this heading">#</a></h3>
-<p>When a Comet columnar shuffle ends up between a partial and a final aggregate that Comet could not convert (both
-remain Spark <code class="docutils literal notranslate"><span class="pre">HashAggregateExec</span></code> or <code class="docutils literal notranslate"><span class="pre">ObjectHashAggregateExec</span></code> operators), Comet reverts it to Spark’s built-in shuffle.
+<p>When a Comet columnar shuffle, or a native shuffle over rows that <code class="docutils literal notranslate"><span class="pre">spark.comet.convert.shuffleInput.enabled</span></code> converted,
+ends up between a partial and a final aggregate that Comet could not convert (both remain Spark <code class="docutils literal notranslate"><span class="pre">HashAggregateExec</span></code>,
+<code class="docutils literal notranslate"><span class="pre">ObjectHashAggregateExec</span></code> or <code class="docutils literal notranslate"><span class="pre">SortAggregateExec</span></code> operators), Comet reverts it to Spark’s built-in shuffle.
Keeping columnar shuffle between the two row-based aggregates would add <code class="docutils literal notranslate"><span class="pre">row</span> <span class="pre">-></span> <span class="pre">Arrow</span> <span class="pre">-></span> <span class="pre">shuffle</span> <span class="pre">-></span> <span class="pre">Arrow</span> <span class="pre">-></span> <span class="pre">row</span></code>
-conversions with no Comet consumer on either side to benefit from columnar output. Other shuffles between non-Comet
-operators are not reverted.</p>
+conversions with no Comet consumer on either side to benefit from columnar output. The final aggregate has to read the
+shuffle directly, which a <code class="docutils literal notranslate"><span class="pre">SortAggregateExec</span></code> with grouping keys does not: it reads the shuffle through a sort, so that
+shuffle is not reverted. Other shuffles between non-Comet operators are not reverted either.</p>
<p>This shifts the affected shuffles from Comet’s off-heap memory pool back to the JVM execution memory pool. Clusters
tuned for a small JVM heap may see <code class="docutils literal notranslate"><span class="pre">ExternalSorter</span></code> spills on queries where this revert fires. Shuffle I/O may also
grow marginally because Spark’s row-based serializer generally compresses less well than Comet’s Arrow IPC format.</p>
-<p>Each revert is logged at <code class="docutils literal notranslate"><span class="pre">INFO</span></code> level on the driver as <code class="docutils literal notranslate"><span class="pre">Reverting</span> <span class="pre">Comet</span> <span class="pre">columnar</span> <span class="pre">shuffle</span> <span class="pre">to</span> <span class="pre">Spark</span> <span class="pre">shuffle</span> <span class="pre">between</span> <span class="pre"><parent></span> <span class="pre">an [...]
+<p>Each revert is logged at <code class="docutils literal notranslate"><span class="pre">INFO</span></code> level on the driver as <code class="docutils literal notranslate"><span class="pre">Reverting</span> <span class="pre">Comet</span> <span class="pre">shuffle</span> <span class="pre">to</span> <span class="pre">Spark</span> <span class="pre">shuffle</span> <span class="pre">between</span> <span class="pre"><parent></span> <span class="pre">and</span> <span class="pre"><chi [...]
<p>This optimization is enabled by default and can be disabled by setting
-<code class="docutils literal notranslate"><span class="pre">spark.comet.shuffle.revertRedundantColumnar.enabled=false</span></code>, in which case Comet will keep the columnar shuffle
+<code class="docutils literal notranslate"><span class="pre">spark.comet.shuffle.revertRedundantColumnar.enabled=false</span></code>, in which case Comet will keep its shuffle
even when both of those aggregates run on Spark.</p>
</section>
</section>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org