You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/07/21 13:11:08 UTC
(datafusion-comet) branch asf-site updated: Publish built docs triggered by 32e2d7c4a82fafd70ba023dd848c6076609f3c9a
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 611d3d898e Publish built docs triggered by 32e2d7c4a82fafd70ba023dd848c6076609f3c9a
611d3d898e is described below
commit 611d3d898eed755d20f7312d1a8e35ebdb4b2a1d
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Tue Jul 21 13:11:03 2026 +0000
Publish built docs triggered by 32e2d7c4a82fafd70ba023dd848c6076609f3c9a
---
_sources/contributor-guide/jvm_shuffle.md.txt | 13 +++++--------
_sources/user-guide/latest/configs.md.txt | 4 +---
contributor-guide/jvm_shuffle.html | 7 -------
searchindex.js | 2 +-
user-guide/latest/configs.html | 18 +++++-------------
5 files changed, 12 insertions(+), 32 deletions(-)
diff --git a/_sources/contributor-guide/jvm_shuffle.md.txt b/_sources/contributor-guide/jvm_shuffle.md.txt
index 2145c82eba..e67f79a524 100644
--- a/_sources/contributor-guide/jvm_shuffle.md.txt
+++ b/_sources/contributor-guide/jvm_shuffle.md.txt
@@ -184,14 +184,11 @@ Selection logic in `CometShuffleManager.shouldBypassMergeSort()`:
- `CometShuffleMemoryAllocator`: Custom allocator for off-heap memory pages
- Memory is allocated in pages; when allocation fails, writers spill to disk
- `CometDiskBlockWriter` coordinates spilling across all partition writers (largest first)
-- Async spilling is supported via `ShuffleThreadPool`
## Configuration
-| Config | Description |
-| ----------------------------------------------- | ----------------------------------- |
-| `spark.comet.columnar.shuffle.async.enabled` | Enable async spill writes |
-| `spark.comet.columnar.shuffle.async.thread.num` | Threads per writer for async |
-| `spark.comet.columnar.shuffle.batch.size` | Rows per Arrow batch |
-| `spark.comet.columnar.shuffle.spill.threshold` | Row count threshold for spill |
-| `spark.comet.exec.shuffle.compression.codec` | Compression codec (zstd, lz4, etc.) |
+| Config | Description |
+| ---------------------------------------------- | ----------------------------------- |
+| `spark.comet.columnar.shuffle.batch.size` | Rows per Arrow batch |
+| `spark.comet.columnar.shuffle.spill.threshold` | Row count threshold for spill |
+| `spark.comet.exec.shuffle.compression.codec` | Compression codec (zstd, lz4, etc.) |
diff --git a/_sources/user-guide/latest/configs.md.txt b/_sources/user-guide/latest/configs.md.txt
index f79e45df39..7e5f520e4f 100644
--- a/_sources/user-guide/latest/configs.md.txt
+++ b/_sources/user-guide/latest/configs.md.txt
@@ -99,10 +99,8 @@ These settings can be used to determine which parts of the plan are accelerated
<!-- prettier-ignore-start -->
| Config | Description | Default Value |
|--------|-------------|---------------|
-| `spark.comet.columnar.shuffle.async.enabled` | Whether to enable asynchronous shuffle for Arrow-based shuffle. | false |
-| `spark.comet.columnar.shuffle.async.max.thread.num` | Maximum number of threads on an executor used for Comet async columnar shuffle. This is the upper bound of total number of shuffle threads per executor. In other words, if the number of cores * the number of shuffle threads per task `spark.comet.columnar.shuffle.async.thread.num` is larger than this config. Comet will use this config as the number of shuffle threads per executor instead. | 100 |
-| `spark.comet.columnar.shuffle.async.thread.num` | Number of threads used for Comet async columnar shuffle per shuffle task. Note that more threads means more memory requirement to buffer shuffle data before flushing to disk. Also, more threads may not always improve performance, and should be set based on the number of cores available. | 3 |
| `spark.comet.columnar.shuffle.batch.size` | Batch size when writing out sorted spill files on the native side. Note that this should not be larger than batch size (i.e., `spark.comet.batchSize`). Otherwise it will produce larger batches than expected in the native operator after shuffle. | 8192 |
+| `spark.comet.columnar.shuffle.max.writers.per.executor` | Maximum number of concurrent hash-based shuffle writers per executor. Comet's hash-based (bypass merge sort) columnar shuffle allocates one writer per partition, so `executor cores * numPartitions` writers can be active concurrently. When that product exceeds this cap, Comet falls back to sort-based shuffle to avoid OOM. | 100 |
| `spark.comet.exec.shuffle.compression.codec` | The codec of Comet native shuffle used to compress shuffle data. lz4, zstd, and snappy are supported. Compression can be disabled by setting spark.shuffle.compress=false. | lz4 |
| `spark.comet.exec.shuffle.compression.zstd.level` | The compression level to use when compressing shuffle files with zstd. | 1 |
| `spark.comet.exec.shuffle.convertFromSparkPlan.enabled` | When enabled, Comet will convert a Spark `ShuffleExchangeExec` to a Comet columnar shuffle even when its child is a non-Comet (Spark) plan. Disable to leave such shuffles as native Spark shuffles, restricting Comet shuffle to cases where the child is already a Comet plan. | true |
diff --git a/contributor-guide/jvm_shuffle.html b/contributor-guide/jvm_shuffle.html
index b4238f24e5..84b9dfce31 100644
--- a/contributor-guide/jvm_shuffle.html
+++ b/contributor-guide/jvm_shuffle.html
@@ -780,7 +780,6 @@ to protobuf and implements the partitioning logic in native code.</p>
<li><p><code class="docutils literal notranslate"><span class="pre">CometShuffleMemoryAllocator</span></code>: Custom allocator for off-heap memory pages</p></li>
<li><p>Memory is allocated in pages; when allocation fails, writers spill to disk</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">CometDiskBlockWriter</span></code> coordinates spilling across all partition writers (largest first)</p></li>
-<li><p>Async spilling is supported via <code class="docutils literal notranslate"><span class="pre">ShuffleThreadPool</span></code></p></li>
</ul>
</section>
<section id="configuration">
@@ -792,12 +791,6 @@ to protobuf and implements the partitioning logic in native code.</p>
</tr>
</thead>
<tbody>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.columnar.shuffle.async.enabled</span></code></p></td>
-<td><p>Enable async spill writes</p></td>
-</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.columnar.shuffle.async.thread.num</span></code></p></td>
-<td><p>Threads per writer for async</p></td>
-</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.columnar.shuffle.batch.size</span></code></p></td>
<td><p>Rows per Arrow batch</p></td>
</tr>
diff --git a/searchindex.js b/searchindex.js
index 75694070db..8d48050344 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[53, "id1"]], "%": [[51, "id1"]], "&": [[41, "id1"]], "*": [[51, "id2"]], "+": [[51, "id3"]], "-": [[51, "id4"]], "/": [[51, "id5"]], "1. Format Your Code": [[38, "format-your-code"]], "1. Install Comet": [[59, "install-comet"], [69, "install-comet"]], "1. Native Operators (nativeExecs map)": [[26, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[38, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[59, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[53, "id1"]], "%": [[51, "id1"]], "&": [[41, "id1"]], "*": [[51, "id2"]], "+": [[51, "id3"]], "-": [[51, "id4"]], "/": [[51, "id5"]], "1. Format Your Code": [[38, "format-your-code"]], "1. Install Comet": [[59, "install-comet"], [69, "install-comet"]], "1. Native Operators (nativeExecs map)": [[26, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[38, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[59, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/configs.html b/user-guide/latest/configs.html
index f8df7199f1..c278deab22 100644
--- a/user-guide/latest/configs.html
+++ b/user-guide/latest/configs.html
@@ -963,22 +963,14 @@ under the License.
</tr>
</thead>
<tbody>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.columnar.shuffle.async.enabled</span></code></p></td>
-<td><p>Whether to enable asynchronous shuffle for Arrow-based shuffle.</p></td>
-<td><p>false</p></td>
-</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.columnar.shuffle.async.max.thread.num</span></code></p></td>
-<td><p>Maximum number of threads on an executor used for Comet async columnar shuffle. This is the upper bound of total number of shuffle threads per executor. In other words, if the number of cores * the number of shuffle threads per task <code class="docutils literal notranslate"><span class="pre">spark.comet.columnar.shuffle.async.thread.num</span></code> is larger than this config. Comet will use this config as the number of shuffle threads per executor instead.</p></td>
-<td><p>100</p></td>
-</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.columnar.shuffle.async.thread.num</span></code></p></td>
-<td><p>Number of threads used for Comet async columnar shuffle per shuffle task. Note that more threads means more memory requirement to buffer shuffle data before flushing to disk. Also, more threads may not always improve performance, and should be set based on the number of cores available.</p></td>
-<td><p>3</p></td>
-</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.columnar.shuffle.batch.size</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.columnar.shuffle.batch.size</span></code></p></td>
<td><p>Batch size when writing out sorted spill files on the native side. Note that this should not be larger than batch size (i.e., <code class="docutils literal notranslate"><span class="pre">spark.comet.batchSize</span></code>). Otherwise it will produce larger batches than expected in the native operator after shuffle.</p></td>
<td><p>8192</p></td>
</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.columnar.shuffle.max.writers.per.executor</span></code></p></td>
+<td><p>Maximum number of concurrent hash-based shuffle writers per executor. Comet’s hash-based (bypass merge sort) columnar shuffle allocates one writer per partition, so <code class="docutils literal notranslate"><span class="pre">executor</span> <span class="pre">cores</span> <span class="pre">*</span> <span class="pre">numPartitions</span></code> writers can be active concurrently. When that product exceeds this cap, Comet falls back to sort-based shuffle to avoid OOM.</p></td>
+<td><p>100</p></td>
+</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.shuffle.compression.codec</span></code></p></td>
<td><p>The codec of Comet native shuffle used to compress shuffle data. lz4, zstd, and snappy are supported. Compression can be disabled by setting spark.shuffle.compress=false.</p></td>
<td><p>lz4</p></td>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org