You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/08/28 17:27:35 UTC
(datafusion-comet) branch asf-site updated: Publish built docs triggered by c067e4e0db4e2fcae0cf99e60c54a673f59c75eb
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new c420ccf4ce Publish built docs triggered by c067e4e0db4e2fcae0cf99e60c54a673f59c75eb
c420ccf4ce is described below
commit c420ccf4cebfba40f9808fba1fe55ae22421a076
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Fri Aug 28 17:27:29 2026 +0000
Publish built docs triggered by c067e4e0db4e2fcae0cf99e60c54a673f59c75eb
---
_sources/user-guide/latest/configs.md.txt | 2 ++
searchindex.js | 2 +-
user-guide/latest/configs.html | 8 ++++++++
3 files changed, 11 insertions(+), 1 deletion(-)
diff --git a/_sources/user-guide/latest/configs.md.txt b/_sources/user-guide/latest/configs.md.txt
index 00814d8cf1..c94e02ab0e 100644
--- a/_sources/user-guide/latest/configs.md.txt
+++ b/_sources/user-guide/latest/configs.md.txt
@@ -112,6 +112,8 @@ These settings can be used to determine which parts of the plan are accelerated
| `spark.comet.shuffle.native.partitioning.roundrobin.maxHashColumns` | The maximum number of columns to hash for round robin partitioning. When set to 0 (the default), all columns are hashed. When set to a positive value, only the first N columns are used for hashing, which can improve performance for wide tables while still providing reasonable distribution. | 0 |
| `spark.comet.shuffle.native.writeBufferSize` | Size of the write buffer in bytes used by the native shuffle writer when writing shuffle data to disk. Larger values may improve write performance by reducing the number of system calls, but will use more memory. The default is 1MB which provides a good balance between performance and memory usage. | 1048576b |
| `spark.comet.shuffle.revertRedundantColumnar.enabled` | When enabled, Comet reverts a `CometShuffleExchangeExec` with `CometColumnarShuffle` back to Spark's `ShuffleExchangeExec` when both its parent and child are non-Comet hash aggregate operators. This avoids a redundant row -> Arrow -> shuffle -> Arrow -> row conversion when no Comet operator on either side can consume columnar output. Disable to keep Comet columnar shuffle even in that case, which preserves Comet's off-heap shuffle [...]
+| `spark.comet.shuffle.rss.maxFrameBytes` | Maximum encoded size of one complete native shuffle frame sent to a remote shuffle service. Frames are never split across remote push requests. | 67108864b |
+| `spark.comet.shuffle.rss.maxInFlightBytes` | Maximum shuffle bytes admitted concurrently by native Comet map attempts sharing an executor-side remote shuffle client. Admission includes native encoding scratch and overlapping native, JNI, and remote shuffle frame copies. A frame must fit its codec and Arrow workspace as well as its encoded bytes; too-small limits fail before encoding. Encrypted native RSS is not supported; use ordinary Spark shuffle when spark.io.encryption.enabled is t [...]
| `spark.comet.shuffle.sizeInBytesMultiplier` | Comet reports smaller sizes for shuffle due to using Arrow's columnar memory format and this can result in Spark choosing a different join strategy due to the estimated size of the exchange being smaller. Comet will multiple sizeInBytes by this amount to avoid regressions in join strategy. | 1.0 |
<!-- prettier-ignore-end -->
<!--END:CONFIG_TABLE-->
diff --git a/searchindex.js b/searchindex.js
index b91f9b1b6b..763cf06a9d 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[55, "id1"]], "%": [[53, "id1"]], "&": [[43, "id1"]], "*": [[53, "id2"]], "+": [[53, "id3"]], "-": [[53, "id4"]], "/": [[53, "id5"]], "1. Format Your Code": [[40, "format-your-code"]], "1. Install Comet": [[61, "install-comet"], [71, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[40, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[61, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[55, "id1"]], "%": [[53, "id1"]], "&": [[43, "id1"]], "*": [[53, "id2"]], "+": [[53, "id3"]], "-": [[53, "id4"]], "/": [[53, "id5"]], "1. Format Your Code": [[40, "format-your-code"]], "1. Install Comet": [[61, "install-comet"], [71, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[40, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[61, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/configs.html b/user-guide/latest/configs.html
index 3a43763822..32cb885067 100644
--- a/user-guide/latest/configs.html
+++ b/user-guide/latest/configs.html
@@ -1050,6 +1050,14 @@ under the License.
<td><p>When enabled, Comet reverts a <code class="docutils literal notranslate"><span class="pre">CometShuffleExchangeExec</span></code> with <code class="docutils literal notranslate"><span class="pre">CometColumnarShuffle</span></code> back to Spark’s <code class="docutils literal notranslate"><span class="pre">ShuffleExchangeExec</span></code> when both its parent and child are non-Comet hash aggregate operators. This avoids a redundant row -> Arrow -> shuffle -> Arrow -> [...]
<td><p>true</p></td>
</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.shuffle.rss.maxFrameBytes</span></code></p></td>
+<td><p>Maximum encoded size of one complete native shuffle frame sent to a remote shuffle service. Frames are never split across remote push requests.</p></td>
+<td><p>67108864b</p></td>
+</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.shuffle.rss.maxInFlightBytes</span></code></p></td>
+<td><p>Maximum shuffle bytes admitted concurrently by native Comet map attempts sharing an executor-side remote shuffle client. Admission includes native encoding scratch and overlapping native, JNI, and remote shuffle frame copies. A frame must fit its codec and Arrow workspace as well as its encoded bytes; too-small limits fail before encoding. Encrypted native RSS is not supported; use ordinary Spark shuffle when spark.io.encryption.enabled is true.</p></td>
+<td><p>268435456b</p></td>
+</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.shuffle.sizeInBytesMultiplier</span></code></p></td>
<td><p>Comet reports smaller sizes for shuffle due to using Arrow’s columnar memory format and this can result in Spark choosing a different join strategy due to the estimated size of the exchange being smaller. Comet will multiple sizeInBytes by this amount to avoid regressions in join strategy.</p></td>
<td><p>1.0</p></td>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org