You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/07/16 20:58:06 UTC
(datafusion-ballista) branch asf-site updated: Publish built docs triggered by f3fc9d0cab7222aa2dd67379380463f175ce7e91
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-ballista.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 31371e07a Publish built docs triggered by f3fc9d0cab7222aa2dd67379380463f175ce7e91
31371e07a is described below
commit 31371e07a19e5307ffb08a77d0887775d2e00f94
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Thu Jul 16 20:58:01 2026 +0000
Publish built docs triggered by f3fc9d0cab7222aa2dd67379380463f175ce7e91
---
_sources/user-guide/cli.md.txt | 2 +-
_sources/user-guide/deployment/docker.md.txt | 2 +-
_sources/user-guide/python/quickstart.md.txt | 6 +++---
_sources/user-guide/tuning-guide.md.txt | 16 ++++++++--------
searchindex.js | 2 +-
user-guide/cli.html | 2 +-
user-guide/deployment/docker.html | 2 +-
user-guide/python/quickstart.html | 6 +++---
user-guide/tuning-guide.html | 16 ++++++++--------
9 files changed, 27 insertions(+), 27 deletions(-)
diff --git a/_sources/user-guide/cli.md.txt b/_sources/user-guide/cli.md.txt
index 777e68d92..b4fec1faa 100644
--- a/_sources/user-guide/cli.md.txt
+++ b/_sources/user-guide/cli.md.txt
@@ -158,7 +158,7 @@ ballista-cli
The TUI provides the following views:
-- **Executors**: Lists all registered executors with their host, port, task slots, memory usage, and last seen time.
+- **Executors**: Lists all registered executors with their host, port, vcores, memory usage, and last seen time.
Supports sorting by any column.
- **Executor details**: Select an executor and press `Enter` to show extra details about it.
- **Jobs**: Displays active and completed jobs with their status, start time, and duration. Supports sorting, job
diff --git a/_sources/user-guide/deployment/docker.md.txt b/_sources/user-guide/deployment/docker.md.txt
index ad3571c50..7b0e1f1e1 100644
--- a/_sources/user-guide/deployment/docker.md.txt
+++ b/_sources/user-guide/deployment/docker.md.txt
@@ -98,7 +98,7 @@ Use `docker logs CONTAINER_ID` to check the output from the executor(s):
$ docker logs fb8b530cee6d
INFO ballista_executor::executor_process: Running with config:
INFO ballista_executor::executor_process: work_dir: /tmp/.tmpAkP3pZ
-INFO ballista_executor::executor_process: concurrent_tasks: 48
+INFO ballista_executor::executor_process: Executor vcores (default: available CPU cores): 48
INFO ballista_executor::executor_process: Ballista v52.0.0 Rust Executor Flight Server listening on 0.0.0.0:50051
INFO ballista_executor::execution_loop: Starting poll work loop with scheduler
```
diff --git a/_sources/user-guide/python/quickstart.md.txt b/_sources/user-guide/python/quickstart.md.txt
index 012e4006d..6dfa6e53d 100644
--- a/_sources/user-guide/python/quickstart.md.txt
+++ b/_sources/user-guide/python/quickstart.md.txt
@@ -51,7 +51,7 @@ ctx = BallistaSessionContext(f"df://{host}:{port}")
### Target Partitions
Set `datafusion.execution.target_partitions` to match your cluster capacity
-(`executors × concurrent_tasks_per_executor`) by passing `cluster_config` to the
+(`executors × vcores_per_executor`) by passing `cluster_config` to the
constructor. The default inherits from DataFusion and is based on the client's CPU
count, which is far too low for distributed execution:
@@ -59,8 +59,8 @@ count, which is far too low for distributed execution:
from ballista import BallistaSessionContext
executors = 4
-concurrent_tasks = 8
-target_partitions = executors * concurrent_tasks
+vcores = 8
+target_partitions = executors * vcores
ctx = BallistaSessionContext(
"df://localhost:50050",
diff --git a/_sources/user-guide/tuning-guide.md.txt b/_sources/user-guide/tuning-guide.md.txt
index b7906b166..c8845ba22 100644
--- a/_sources/user-guide/tuning-guide.md.txt
+++ b/_sources/user-guide/tuning-guide.md.txt
@@ -56,12 +56,12 @@ let ctx: SessionContext = SessionContext::remote_with_state(&url,state).await?;
## Configuring Executor Concurrency Levels
-Each executor instance has a fixed number of tasks that it can process concurrently. This is specified by passing a
-`concurrent_tasks` command-line parameter. The default setting is to use all available CPU cores.
+Each executor instance advertises a fixed number of virtual cores (vcores) to the scheduler. This is specified by
+passing a `--vcores` command-line parameter. The default setting is to use all available CPU cores.
Increasing this configuration setting will increase the number of tasks that each executor can run concurrently but
this will also mean that the executor will use more memory. If executors are failing due to out-of-memory errors then
-decreasing the number of concurrent tasks may help.
+decreasing the vcore count may help.
## Configuring Executor Memory Pool
@@ -71,22 +71,22 @@ memory. To bound executor memory and let those operators spill to disk under
pressure, pass `--memory-pool-size` when starting the executor:
```sh
-ballista-executor --memory-pool-size 8GB --concurrent-tasks 8
+ballista-executor --memory-pool-size 8GB --vcores 8
```
The argument accepts human-readable sizes (`8GB`, `512MiB`) or a plain byte
count. SI suffixes (`KB`/`MB`/`GB`) are powers of 10; IEC suffixes
(`KiB`/`MiB`/`GiB`) are powers of 2.
-The total budget is divided equally across concurrent task slots: each task
-receives its own `FairSpillPool` of size `memory_pool_size / concurrent_tasks`.
-With `--memory-pool-size 8GB --concurrent-tasks 8`, every task sees a 1 GB
+The total budget is divided equally across the executor's vcores: each task
+receives its own `FairSpillPool` of size `memory_pool_size / vcores`.
+With `--memory-pool-size 8GB --vcores 8`, every task sees a 1 GB
pool, fully isolated from other tasks. Idle slots do not lend their share to
busy ones, which keeps task memory predictable at the cost of some unused
capacity when the executor is under-utilized.
The executor refuses to start if the per-task share would round to zero (i.e.
-`memory_pool_size < concurrent_tasks`).
+`memory_pool_size < vcores`).
When `--memory-pool-size` is not set, the executor behaves as before with no
memory pool installed.
diff --git a/searchindex.js b/searchindex.js
index d4b6d095b..1c4f4ed29 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles":{"AQE join behavior (opt-in)":[[26,"aqe-join-behavior-opt-in"]],"Adaptive Query Execution (AQE)":[[21,"adaptive-query-execution-aqe"]],"Adaptive Query Execution (Experimental)":[[48,"adaptive-query-execution-experimental"]],"Adding a persona":[[24,"adding-a-persona"]],"Aggregate Functions":[[47,"aggregate-functions"]],"Apache DataFusion Ballista":[[25,null]],"Apache DataFusion Ballista 0.10.0 Changelog":[[0,null]],"Apache DataFusion Ballista 0.11.0 Changelog" [...]
\ No newline at end of file
+Search.setIndex({"alltitles":{"AQE join behavior (opt-in)":[[26,"aqe-join-behavior-opt-in"]],"Adaptive Query Execution (AQE)":[[21,"adaptive-query-execution-aqe"]],"Adaptive Query Execution (Experimental)":[[48,"adaptive-query-execution-experimental"]],"Adding a persona":[[24,"adding-a-persona"]],"Aggregate Functions":[[47,"aggregate-functions"]],"Apache DataFusion Ballista":[[25,null]],"Apache DataFusion Ballista 0.10.0 Changelog":[[0,null]],"Apache DataFusion Ballista 0.11.0 Changelog" [...]
\ No newline at end of file
diff --git a/user-guide/cli.html b/user-guide/cli.html
index 0de853711..55aeaeb7d 100644
--- a/user-guide/cli.html
+++ b/user-guide/cli.html
@@ -520,7 +520,7 @@ that provides a visual overview of the Ballista cluster.</p>
<h3>TUI Features<a class="headerlink" href="#tui-features" title="Link to this heading">#</a></h3>
<p>The TUI provides the following views:</p>
<ul class="simple">
-<li><p><strong>Executors</strong>: Lists all registered executors with their host, port, task slots, memory usage, and last seen time.
+<li><p><strong>Executors</strong>: Lists all registered executors with their host, port, vcores, memory usage, and last seen time.
Supports sorting by any column.</p></li>
<li><p><strong>Executor details</strong>: Select an executor and press <code class="docutils literal notranslate"><span class="pre">Enter</span></code> to show extra details about it.</p></li>
<li><p><strong>Jobs</strong>: Displays active and completed jobs with their status, start time, and duration. Supports sorting, job
diff --git a/user-guide/deployment/docker.html b/user-guide/deployment/docker.html
index 745745f40..e5735a631 100644
--- a/user-guide/deployment/docker.html
+++ b/user-guide/deployment/docker.html
@@ -456,7 +456,7 @@ a756055576f3 apache/datafusion-ballista-scheduler:latest "/root/schedul
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ docker logs fb8b530cee6d
INFO ballista_executor::executor_process: Running with config:
INFO ballista_executor::executor_process: work_dir: /tmp/.tmpAkP3pZ
-INFO ballista_executor::executor_process: concurrent_tasks: 48
+INFO ballista_executor::executor_process: Executor vcores (default: available CPU cores): 48
INFO ballista_executor::executor_process: Ballista v52.0.0 Rust Executor Flight Server listening on 0.0.0.0:50051
INFO ballista_executor::execution_loop: Starting poll work loop with scheduler
</pre></div>
diff --git a/user-guide/python/quickstart.html b/user-guide/python/quickstart.html
index 917fb20ef..f129a4855 100644
--- a/user-guide/python/quickstart.html
+++ b/user-guide/python/quickstart.html
@@ -418,14 +418,14 @@ Useful for development, testing, and notebooks:</p>
<section id="target-partitions">
<h3>Target Partitions<a class="headerlink" href="#target-partitions" title="Link to this heading">#</a></h3>
<p>Set <code class="docutils literal notranslate"><span class="pre">datafusion.execution.target_partitions</span></code> to match your cluster capacity
-(<code class="docutils literal notranslate"><span class="pre">executors</span> <span class="pre">×</span> <span class="pre">concurrent_tasks_per_executor</span></code>) by passing <code class="docutils literal notranslate"><span class="pre">cluster_config</span></code> to the
+(<code class="docutils literal notranslate"><span class="pre">executors</span> <span class="pre">×</span> <span class="pre">vcores_per_executor</span></code>) by passing <code class="docutils literal notranslate"><span class="pre">cluster_config</span></code> to the
constructor. The default inherits from DataFusion and is based on the client’s CPU
count, which is far too low for distributed execution:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span><span class="w"> </span><span class="nn">ballista</span><span class="w"> </span><span class="kn">import</span> <span class="n">BallistaSessionContext</span>
<span class="n">executors</span> <span class="o">=</span> <span class="mi">4</span>
-<span class="n">concurrent_tasks</span> <span class="o">=</span> <span class="mi">8</span>
-<span class="n">target_partitions</span> <span class="o">=</span> <span class="n">executors</span> <span class="o">*</span> <span class="n">concurrent_tasks</span>
+<span class="n">vcores</span> <span class="o">=</span> <span class="mi">8</span>
+<span class="n">target_partitions</span> <span class="o">=</span> <span class="n">executors</span> <span class="o">*</span> <span class="n">vcores</span>
<span class="n">ctx</span> <span class="o">=</span> <span class="n">BallistaSessionContext</span><span class="p">(</span>
<span class="s2">"df://localhost:50050"</span><span class="p">,</span>
diff --git a/user-guide/tuning-guide.html b/user-guide/tuning-guide.html
index e73c3312b..7a31691b9 100644
--- a/user-guide/tuning-guide.html
+++ b/user-guide/tuning-guide.html
@@ -420,11 +420,11 @@ the source operation has fewer partitions than this setting.</p>
</section>
<section id="configuring-executor-concurrency-levels">
<h2>Configuring Executor Concurrency Levels<a class="headerlink" href="#configuring-executor-concurrency-levels" title="Link to this heading">#</a></h2>
-<p>Each executor instance has a fixed number of tasks that it can process concurrently. This is specified by passing a
-<code class="docutils literal notranslate"><span class="pre">concurrent_tasks</span></code> command-line parameter. The default setting is to use all available CPU cores.</p>
+<p>Each executor instance advertises a fixed number of virtual cores (vcores) to the scheduler. This is specified by
+passing a <code class="docutils literal notranslate"><span class="pre">--vcores</span></code> command-line parameter. The default setting is to use all available CPU cores.</p>
<p>Increasing this configuration setting will increase the number of tasks that each executor can run concurrently but
this will also mean that the executor will use more memory. If executors are failing due to out-of-memory errors then
-decreasing the number of concurrent tasks may help.</p>
+decreasing the vcore count may help.</p>
</section>
<section id="configuring-executor-memory-pool">
<h2>Configuring Executor Memory Pool<a class="headerlink" href="#configuring-executor-memory-pool" title="Link to this heading">#</a></h2>
@@ -432,20 +432,20 @@ decreasing the number of concurrent tasks may help.</p>
operators (sort, hash join, hash aggregate) grow until the host runs out of
memory. To bound executor memory and let those operators spill to disk under
pressure, pass <code class="docutils literal notranslate"><span class="pre">--memory-pool-size</span></code> when starting the executor:</p>
-<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>ballista-executor<span class="w"> </span>--memory-pool-size<span class="w"> </span>8GB<span class="w"> </span>--concurrent-tasks<span class="w"> </span><span class="m">8</span>
+<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>ballista-executor<span class="w"> </span>--memory-pool-size<span class="w"> </span>8GB<span class="w"> </span>--vcores<span class="w"> </span><span class="m">8</span>
</pre></div>
</div>
<p>The argument accepts human-readable sizes (<code class="docutils literal notranslate"><span class="pre">8GB</span></code>, <code class="docutils literal notranslate"><span class="pre">512MiB</span></code>) or a plain byte
count. SI suffixes (<code class="docutils literal notranslate"><span class="pre">KB</span></code>/<code class="docutils literal notranslate"><span class="pre">MB</span></code>/<code class="docutils literal notranslate"><span class="pre">GB</span></code>) are powers of 10; IEC suffixes
(<code class="docutils literal notranslate"><span class="pre">KiB</span></code>/<code class="docutils literal notranslate"><span class="pre">MiB</span></code>/<code class="docutils literal notranslate"><span class="pre">GiB</span></code>) are powers of 2.</p>
-<p>The total budget is divided equally across concurrent task slots: each task
-receives its own <code class="docutils literal notranslate"><span class="pre">FairSpillPool</span></code> of size <code class="docutils literal notranslate"><span class="pre">memory_pool_size</span> <span class="pre">/</span> <span class="pre">concurrent_tasks</span></code>.
-With <code class="docutils literal notranslate"><span class="pre">--memory-pool-size</span> <span class="pre">8GB</span> <span class="pre">--concurrent-tasks</span> <span class="pre">8</span></code>, every task sees a 1 GB
+<p>The total budget is divided equally across the executor’s vcores: each task
+receives its own <code class="docutils literal notranslate"><span class="pre">FairSpillPool</span></code> of size <code class="docutils literal notranslate"><span class="pre">memory_pool_size</span> <span class="pre">/</span> <span class="pre">vcores</span></code>.
+With <code class="docutils literal notranslate"><span class="pre">--memory-pool-size</span> <span class="pre">8GB</span> <span class="pre">--vcores</span> <span class="pre">8</span></code>, every task sees a 1 GB
pool, fully isolated from other tasks. Idle slots do not lend their share to
busy ones, which keeps task memory predictable at the cost of some unused
capacity when the executor is under-utilized.</p>
<p>The executor refuses to start if the per-task share would round to zero (i.e.
-<code class="docutils literal notranslate"><span class="pre">memory_pool_size</span> <span class="pre"><</span> <span class="pre">concurrent_tasks</span></code>).</p>
+<code class="docutils literal notranslate"><span class="pre">memory_pool_size</span> <span class="pre"><</span> <span class="pre">vcores</span></code>).</p>
<p>When <code class="docutils literal notranslate"><span class="pre">--memory-pool-size</span></code> is not set, the executor behaves as before with no
memory pool installed.</p>
</section>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org