You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/04/29 19:08:13 UTC
(datafusion-ballista) branch asf-site updated: Publish built docs triggered by e5f2216411692a4bb34e281263035ca9ce521ada
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-ballista.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 6daecc793 Publish built docs triggered by e5f2216411692a4bb34e281263035ca9ce521ada
6daecc793 is described below
commit 6daecc7939713a70c3fef9a54d9be370526b5d8c
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Wed Apr 29 19:08:08 2026 +0000
Publish built docs triggered by e5f2216411692a4bb34e281263035ca9ce521ada
---
_sources/user-guide/tuning-guide.md.txt | 29 +++++++++++++++++++++++++++--
searchindex.js | 2 +-
user-guide/tuning-guide.html | 30 ++++++++++++++++++++++++++++--
3 files changed, 56 insertions(+), 5 deletions(-)
diff --git a/_sources/user-guide/tuning-guide.md.txt b/_sources/user-guide/tuning-guide.md.txt
index dfe3c2937..9a6334d22 100644
--- a/_sources/user-guide/tuning-guide.md.txt
+++ b/_sources/user-guide/tuning-guide.md.txt
@@ -63,8 +63,33 @@ Increasing this configuration setting will increase the number of tasks that eac
this will also mean that the executor will use more memory. If executors are failing due to out-of-memory errors then
decreasing the number of concurrent tasks may help.
-In the future, Ballista will have better support for tracking memory usage and allocating tasks based on available
-memory, as well as supporting spill-to-disk to reduce memory pressure.
+## Configuring Executor Memory Pool
+
+By default the executor uses DataFusion's unbounded memory pool, so spillable
+operators (sort, hash join, hash aggregate) grow until the host runs out of
+memory. To bound executor memory and let those operators spill to disk under
+pressure, pass `--memory-pool-size` when starting the executor:
+
+```sh
+ballista-executor --memory-pool-size 8GB --concurrent-tasks 8
+```
+
+The argument accepts human-readable sizes (`8GB`, `512MiB`) or a plain byte
+count. SI suffixes (`KB`/`MB`/`GB`) are powers of 10; IEC suffixes
+(`KiB`/`MiB`/`GiB`) are powers of 2.
+
+The total budget is divided equally across concurrent task slots: each task
+receives its own `FairSpillPool` of size `memory_pool_size / concurrent_tasks`.
+With `--memory-pool-size 8GB --concurrent-tasks 8`, every task sees a 1 GB
+pool, fully isolated from other tasks. Idle slots do not lend their share to
+busy ones, which keeps task memory predictable at the cost of some unused
+capacity when the executor is under-utilized.
+
+The executor refuses to start if the per-task share would round to zero (i.e.
+`memory_pool_size < concurrent_tasks`).
+
+When `--memory-pool-size` is not set, the executor behaves as before with no
+memory pool installed.
## Shuffle Implementation
diff --git a/searchindex.js b/searchindex.js
index e4552c905..711c14425 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"Aggregate Functions": [[24, "aggregate-functions"]], "Apache DataFusion Ballista": [[4, null]], "Arrow-native": [[1, "arrow-native"]], "Autoscaling Executors": [[11, "autoscaling-executors"]], "Ballista Architecture": [[1, null]], "Ballista Code Organization": [[2, null]], "Ballista Command-line Interface": [[5, null]], "Ballista Configuration Settings": [[6, "ballista-configuration-settings"]], "Ballista Development": [[3, null]], "Ballista Python Binding [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"Aggregate Functions": [[24, "aggregate-functions"]], "Apache DataFusion Ballista": [[4, null]], "Arrow-native": [[1, "arrow-native"]], "Autoscaling Executors": [[11, "autoscaling-executors"]], "Ballista Architecture": [[1, null]], "Ballista Code Organization": [[2, null]], "Ballista Command-line Interface": [[5, null]], "Ballista Configuration Settings": [[6, "ballista-configuration-settings"]], "Ballista Development": [[3, null]], "Ballista Python Binding [...]
\ No newline at end of file
diff --git a/user-guide/tuning-guide.html b/user-guide/tuning-guide.html
index 338953d17..7b8893ca5 100644
--- a/user-guide/tuning-guide.html
+++ b/user-guide/tuning-guide.html
@@ -297,6 +297,11 @@
Configuring Executor Concurrency Levels
</a>
</li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#configuring-executor-memory-pool">
+ Configuring Executor Memory Pool
+ </a>
+ </li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#shuffle-implementation">
Shuffle Implementation
@@ -409,8 +414,29 @@ the source operation has fewer partitions than this setting.</p>
<p>Increasing this configuration setting will increase the number of tasks that each executor can run concurrently but
this will also mean that the executor will use more memory. If executors are failing due to out-of-memory errors then
decreasing the number of concurrent tasks may help.</p>
-<p>In the future, Ballista will have better support for tracking memory usage and allocating tasks based on available
-memory, as well as supporting spill-to-disk to reduce memory pressure.</p>
+</section>
+<section id="configuring-executor-memory-pool">
+<h2>Configuring Executor Memory Pool<a class="headerlink" href="#configuring-executor-memory-pool" title="Link to this heading">¶</a></h2>
+<p>By default the executor uses DataFusion’s unbounded memory pool, so spillable
+operators (sort, hash join, hash aggregate) grow until the host runs out of
+memory. To bound executor memory and let those operators spill to disk under
+pressure, pass <code class="docutils literal notranslate"><span class="pre">--memory-pool-size</span></code> when starting the executor:</p>
+<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>ballista-executor<span class="w"> </span>--memory-pool-size<span class="w"> </span>8GB<span class="w"> </span>--concurrent-tasks<span class="w"> </span><span class="m">8</span>
+</pre></div>
+</div>
+<p>The argument accepts human-readable sizes (<code class="docutils literal notranslate"><span class="pre">8GB</span></code>, <code class="docutils literal notranslate"><span class="pre">512MiB</span></code>) or a plain byte
+count. SI suffixes (<code class="docutils literal notranslate"><span class="pre">KB</span></code>/<code class="docutils literal notranslate"><span class="pre">MB</span></code>/<code class="docutils literal notranslate"><span class="pre">GB</span></code>) are powers of 10; IEC suffixes
+(<code class="docutils literal notranslate"><span class="pre">KiB</span></code>/<code class="docutils literal notranslate"><span class="pre">MiB</span></code>/<code class="docutils literal notranslate"><span class="pre">GiB</span></code>) are powers of 2.</p>
+<p>The total budget is divided equally across concurrent task slots: each task
+receives its own <code class="docutils literal notranslate"><span class="pre">FairSpillPool</span></code> of size <code class="docutils literal notranslate"><span class="pre">memory_pool_size</span> <span class="pre">/</span> <span class="pre">concurrent_tasks</span></code>.
+With <code class="docutils literal notranslate"><span class="pre">--memory-pool-size</span> <span class="pre">8GB</span> <span class="pre">--concurrent-tasks</span> <span class="pre">8</span></code>, every task sees a 1 GB
+pool, fully isolated from other tasks. Idle slots do not lend their share to
+busy ones, which keeps task memory predictable at the cost of some unused
+capacity when the executor is under-utilized.</p>
+<p>The executor refuses to start if the per-task share would round to zero (i.e.
+<code class="docutils literal notranslate"><span class="pre">memory_pool_size</span> <span class="pre"><</span> <span class="pre">concurrent_tasks</span></code>).</p>
+<p>When <code class="docutils literal notranslate"><span class="pre">--memory-pool-size</span></code> is not set, the executor behaves as before with no
+memory pool installed.</p>
</section>
<section id="shuffle-implementation">
<h2>Shuffle Implementation<a class="headerlink" href="#shuffle-implementation" title="Link to this heading">¶</a></h2>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org