You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/04/30 14:28:28 UTC
(datafusion-ballista) branch asf-site updated: Publish built docs triggered by 90d0606b5e9e4d41b8d5f67ed5b2f17f91802e81
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-ballista.git
The following commit(s) were added to refs/heads/asf-site by this push:
new cd1147afd Publish built docs triggered by 90d0606b5e9e4d41b8d5f67ed5b2f17f91802e81
cd1147afd is described below
commit cd1147afd398eaa5b7d1461e02770d5cb69ab51a
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Thu Apr 30 14:28:23 2026 +0000
Publish built docs triggered by 90d0606b5e9e4d41b8d5f67ed5b2f17f91802e81
---
_sources/user-guide/configs.md.txt | 4 +--
_sources/user-guide/tuning-guide.md.txt | 32 ++++++++++++------------
searchindex.js | 2 +-
user-guide/configs.html | 4 +--
user-guide/tuning-guide.html | 43 ++++++++++++++++-----------------
5 files changed, 41 insertions(+), 44 deletions(-)
diff --git a/_sources/user-guide/configs.md.txt b/_sources/user-guide/configs.md.txt
index cd0a790ff..b9e44f6dc 100644
--- a/_sources/user-guide/configs.md.txt
+++ b/_sources/user-guide/configs.md.txt
@@ -92,14 +92,14 @@ let expected = [
The following session-level keys control Ballista's shuffle behavior. See
the [tuning guide](tuning-guide.md#shuffle-implementation) for an
-explanation of the hash-based (default) and sort-based shuffle writers.
+explanation of the sort-based (default) and hash-based shuffle writers.
| key | type | default | description |
| --------------------------------------------- | ------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| ballista.shuffle.max_concurrent_read_requests | UInt64 | 64 | Maximum number of concurrent fetch requests the shuffle reader will issue. |
| ballista.shuffle.force_remote_read | Boolean | false | Forces the shuffle reader to fetch every partition through Arrow Flight, even when the data is local. Intended for testing. |
| ballista.shuffle.remote_read_prefer_flight | Boolean | false | For remote reads, prefer the Arrow Flight reader over the block reader. The block reader is generally faster. |
-| ballista.shuffle.sort_based.enabled | Boolean | false | Enables the sort-based shuffle writer (consolidated data file per input partition with an index, instead of one file per (input partition, output partition) pair). |
+| ballista.shuffle.sort_based.enabled | Boolean | true | Enables the sort-based shuffle writer (consolidated data file per input partition with an index, instead of one file per (input partition, output partition) pair). |
| ballista.shuffle.sort_based.buffer_size | UInt64 | 1048576 | Per-partition buffer size in bytes for the sort-based writer (1 MiB default). |
| ballista.shuffle.sort_based.memory_limit | UInt64 | 268435456 | Total in-memory budget across all output-partition buffers for the sort-based writer (256 MiB default). |
| ballista.shuffle.sort_based.spill_threshold | Utf8 | "0.8" | Fraction of `memory_limit` at which the largest buffers spill to disk. Must be in the range 0–1. |
diff --git a/_sources/user-guide/tuning-guide.md.txt b/_sources/user-guide/tuning-guide.md.txt
index 9a6334d22..0354c1835 100644
--- a/_sources/user-guide/tuning-guide.md.txt
+++ b/_sources/user-guide/tuning-guide.md.txt
@@ -99,18 +99,7 @@ upstream task to local files, which downstream tasks read either from disk
available, with different trade-offs around file count, memory use, and
write latency.
-### Hash-based shuffle (default)
-
-The default writer hashes each incoming `RecordBatch` and immediately
-encodes the per-partition slices to Arrow IPC, streaming them into one
-file per `(input_partition, output_partition)` pair. Nothing is buffered
-in memory across batches.
-
-This is simple and low latency, but for `N` input partitions and `M`
-output partitions it produces `N × M` files. Wide shuffles can therefore
-generate a large number of small files.
-
-### Sort-based shuffle (opt-in)
+### Sort-based shuffle (default)
The sort-based writer accumulates batches in a per-output-partition
in-memory buffer. When the total buffered size crosses a threshold, the
@@ -122,21 +111,30 @@ an index file that lets readers seek directly to a given output partition.
This produces `2 × N` files instead of `N × M`, coalesces small batches
to a target size before writing, and bounds shuffle memory use via
spilling — at the cost of higher write latency than the hash writer.
-Consider enabling it for queries with high partition fan-out or when
-file-count pressure on local storage is a concern.
-The sort-based writer is disabled by default and is enabled per session:
+### Hash-based shuffle (opt-in)
+
+The hash-based writer hashes each incoming `RecordBatch` and immediately
+encodes the per-partition slices to Arrow IPC, streaming them into one
+file per `(input_partition, output_partition)` pair. Nothing is buffered
+in memory across batches.
+
+This is simple and low latency, but for `N` input partitions and `M`
+output partitions it produces `N × M` files. Wide shuffles can therefore
+generate a large number of small files. Consider switching to the
+hash-based writer for narrow shuffles where the additional buffering
+and merging of the sort-based writer is unnecessary overhead:
```rust
let session_config = SessionConfig::new_with_ballista()
- .set_bool("ballista.shuffle.sort_based.enabled", true);
+ .set_bool("ballista.shuffle.sort_based.enabled", false);
```
The following session-level keys tune its behavior:
| key | type | default | description |
| ------------------------------------------- | ------- | --------- | --------------------------------------------------------------------------------------------------------- |
-| ballista.shuffle.sort_based.enabled | Boolean | false | Enables the sort-based shuffle writer. |
+| ballista.shuffle.sort_based.enabled | Boolean | true | Enables the sort-based shuffle writer. |
| ballista.shuffle.sort_based.buffer_size | UInt64 | 1048576 | Per-partition buffer size in bytes (1 MiB default). |
| ballista.shuffle.sort_based.memory_limit | UInt64 | 268435456 | Total in-memory budget across all output-partition buffers (256 MiB default). |
| ballista.shuffle.sort_based.spill_threshold | Utf8 | "0.8" | Fraction of `memory_limit` at which the largest buffers begin spilling to disk. Must be in the range 0–1. |
diff --git a/searchindex.js b/searchindex.js
index 711c14425..f242726cb 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"Aggregate Functions": [[24, "aggregate-functions"]], "Apache DataFusion Ballista": [[4, null]], "Arrow-native": [[1, "arrow-native"]], "Autoscaling Executors": [[11, "autoscaling-executors"]], "Ballista Architecture": [[1, null]], "Ballista Code Organization": [[2, null]], "Ballista Command-line Interface": [[5, null]], "Ballista Configuration Settings": [[6, "ballista-configuration-settings"]], "Ballista Development": [[3, null]], "Ballista Python Binding [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"Aggregate Functions": [[24, "aggregate-functions"]], "Apache DataFusion Ballista": [[4, null]], "Arrow-native": [[1, "arrow-native"]], "Autoscaling Executors": [[11, "autoscaling-executors"]], "Ballista Architecture": [[1, null]], "Ballista Code Organization": [[2, null]], "Ballista Command-line Interface": [[5, null]], "Ballista Configuration Settings": [[6, "ballista-configuration-settings"]], "Ballista Development": [[3, null]], "Ballista Python Binding [...]
\ No newline at end of file
diff --git a/user-guide/configs.html b/user-guide/configs.html
index cdefe8ab3..ec2fb6e5e 100644
--- a/user-guide/configs.html
+++ b/user-guide/configs.html
@@ -414,7 +414,7 @@
<h2>Shuffle Settings<a class="headerlink" href="#shuffle-settings" title="Link to this heading">¶</a></h2>
<p>The following session-level keys control Ballista’s shuffle behavior. See
the <a class="reference internal" href="tuning-guide.html#shuffle-implementation"><span class="std std-ref">tuning guide</span></a> for an
-explanation of the hash-based (default) and sort-based shuffle writers.</p>
+explanation of the sort-based (default) and hash-based shuffle writers.</p>
<table class="table">
<thead>
<tr class="row-odd"><th class="head"><p>key</p></th>
@@ -441,7 +441,7 @@ explanation of the hash-based (default) and sort-based shuffle writers.</p>
</tr>
<tr class="row-odd"><td><p>ballista.shuffle.sort_based.enabled</p></td>
<td><p>Boolean</p></td>
-<td><p>false</p></td>
+<td><p>true</p></td>
<td><p>Enables the sort-based shuffle writer (consolidated data file per input partition with an index, instead of one file per (input partition, output partition) pair).</p></td>
</tr>
<tr class="row-even"><td><p>ballista.shuffle.sort_based.buffer_size</p></td>
diff --git a/user-guide/tuning-guide.html b/user-guide/tuning-guide.html
index 7b8893ca5..7bad440e5 100644
--- a/user-guide/tuning-guide.html
+++ b/user-guide/tuning-guide.html
@@ -308,13 +308,13 @@
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
- <a class="reference internal nav-link" href="#hash-based-shuffle-default">
- Hash-based shuffle (default)
+ <a class="reference internal nav-link" href="#sort-based-shuffle-default">
+ Sort-based shuffle (default)
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
- <a class="reference internal nav-link" href="#sort-based-shuffle-opt-in">
- Sort-based shuffle (opt-in)
+ <a class="reference internal nav-link" href="#hash-based-shuffle-opt-in">
+ Hash-based shuffle (opt-in)
</a>
</li>
</ul>
@@ -445,18 +445,8 @@ upstream task to local files, which downstream tasks read either from disk
(when co-located) or over Arrow Flight. Two shuffle implementations are
available, with different trade-offs around file count, memory use, and
write latency.</p>
-<section id="hash-based-shuffle-default">
-<h3>Hash-based shuffle (default)<a class="headerlink" href="#hash-based-shuffle-default" title="Link to this heading">¶</a></h3>
-<p>The default writer hashes each incoming <code class="docutils literal notranslate"><span class="pre">RecordBatch</span></code> and immediately
-encodes the per-partition slices to Arrow IPC, streaming them into one
-file per <code class="docutils literal notranslate"><span class="pre">(input_partition,</span> <span class="pre">output_partition)</span></code> pair. Nothing is buffered
-in memory across batches.</p>
-<p>This is simple and low latency, but for <code class="docutils literal notranslate"><span class="pre">N</span></code> input partitions and <code class="docutils literal notranslate"><span class="pre">M</span></code>
-output partitions it produces <code class="docutils literal notranslate"><span class="pre">N</span> <span class="pre">×</span> <span class="pre">M</span></code> files. Wide shuffles can therefore
-generate a large number of small files.</p>
-</section>
-<section id="sort-based-shuffle-opt-in">
-<h3>Sort-based shuffle (opt-in)<a class="headerlink" href="#sort-based-shuffle-opt-in" title="Link to this heading">¶</a></h3>
+<section id="sort-based-shuffle-default">
+<h3>Sort-based shuffle (default)<a class="headerlink" href="#sort-based-shuffle-default" title="Link to this heading">¶</a></h3>
<p>The sort-based writer accumulates batches in a per-output-partition
in-memory buffer. When the total buffered size crosses a threshold, the
largest buffers are spilled to disk. After the input stream finishes, the
@@ -465,12 +455,21 @@ into a single consolidated Arrow IPC file per input partition, alongside
an index file that lets readers seek directly to a given output partition.</p>
<p>This produces <code class="docutils literal notranslate"><span class="pre">2</span> <span class="pre">×</span> <span class="pre">N</span></code> files instead of <code class="docutils literal notranslate"><span class="pre">N</span> <span class="pre">×</span> <span class="pre">M</span></code>, coalesces small batches
to a target size before writing, and bounds shuffle memory use via
-spilling — at the cost of higher write latency than the hash writer.
-Consider enabling it for queries with high partition fan-out or when
-file-count pressure on local storage is a concern.</p>
-<p>The sort-based writer is disabled by default and is enabled per session:</p>
+spilling — at the cost of higher write latency than the hash writer.</p>
+</section>
+<section id="hash-based-shuffle-opt-in">
+<h3>Hash-based shuffle (opt-in)<a class="headerlink" href="#hash-based-shuffle-opt-in" title="Link to this heading">¶</a></h3>
+<p>The hash-based writer hashes each incoming <code class="docutils literal notranslate"><span class="pre">RecordBatch</span></code> and immediately
+encodes the per-partition slices to Arrow IPC, streaming them into one
+file per <code class="docutils literal notranslate"><span class="pre">(input_partition,</span> <span class="pre">output_partition)</span></code> pair. Nothing is buffered
+in memory across batches.</p>
+<p>This is simple and low latency, but for <code class="docutils literal notranslate"><span class="pre">N</span></code> input partitions and <code class="docutils literal notranslate"><span class="pre">M</span></code>
+output partitions it produces <code class="docutils literal notranslate"><span class="pre">N</span> <span class="pre">×</span> <span class="pre">M</span></code> files. Wide shuffles can therefore
+generate a large number of small files. Consider switching to the
+hash-based writer for narrow shuffles where the additional buffering
+and merging of the sort-based writer is unnecessary overhead:</p>
<div class="highlight-rust notranslate"><div class="highlight"><pre><span></span><span class="kd">let</span><span class="w"> </span><span class="n">session_config</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">SessionConfig</span><span class="p">::</span><span class="n">new_with_ballista</span><span class="p">()</span>
-<span class="w"> </span><span class="p">.</span><span class="n">set_bool</span><span class="p">(</span><span class="s">"ballista.shuffle.sort_based.enabled"</span><span class="p">,</span><span class="w"> </span><span class="kc">true</span><span class="p">);</span>
+<span class="w"> </span><span class="p">.</span><span class="n">set_bool</span><span class="p">(</span><span class="s">"ballista.shuffle.sort_based.enabled"</span><span class="p">,</span><span class="w"> </span><span class="kc">false</span><span class="p">);</span>
</pre></div>
</div>
<p>The following session-level keys tune its behavior:</p>
@@ -485,7 +484,7 @@ file-count pressure on local storage is a concern.</p>
<tbody>
<tr class="row-even"><td><p>ballista.shuffle.sort_based.enabled</p></td>
<td><p>Boolean</p></td>
-<td><p>false</p></td>
+<td><p>true</p></td>
<td><p>Enables the sort-based shuffle writer.</p></td>
</tr>
<tr class="row-odd"><td><p>ballista.shuffle.sort_based.buffer_size</p></td>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org