You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by mi...@apache.org on 2026/09/23 09:08:27 UTC
(datafusion-ballista) branch main updated: docs: say vcores instead of task slots and concurrency levels (#2482)
This is an automated email from the ASF dual-hosted git repository.
milenkovicm pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-ballista.git
The following commit(s) were added to refs/heads/main by this push:
new 9ff3fe0a8 docs: say vcores instead of task slots and concurrency levels (#2482)
9ff3fe0a8 is described below
commit 9ff3fe0a89205fc827a132359a53300d045da76d
Author: NoahKusaba <97...@users.noreply.github.com>
AuthorDate: Wed Sep 23 05:08:23 2026 -0400
docs: say vcores instead of task slots and concurrency levels (#2482)
Standardise on the term the executor flag and config already use. Includes a
wording change to Persona 2 in user-personas.md: Ballista packs several
partitions per task by default (max_partitions_per_task = 0), so the page no
longer promises one task per partition, and says how to get that back.
Co-authored-by: Claude Opus 5.5 <no...@anthropic.com>
---
README.md | 2 +-
ballista/core/src/config.rs | 2 +-
docs/source/contributors-guide/benchmarking.md | 6 +++---
docs/source/contributors-guide/shuffle.md | 18 +++++++++---------
docs/source/contributors-guide/user-personas.md | 14 ++++++++------
docs/source/user-guide/configs.md | 2 +-
docs/source/user-guide/tuning-guide.md | 2 +-
7 files changed, 24 insertions(+), 22 deletions(-)
diff --git a/README.md b/README.md
index 773ed5a75..e6fba3cce 100644
--- a/README.md
+++ b/README.md
@@ -94,7 +94,7 @@ For documentation or more examples, please refer to the [Ballista User Guide][us
Ballista serves several distinct audiences:
- **DataFusion users going multi-node** — you already use [Apache DataFusion](https://github.com/apache/datafusion) on a single machine and have outgrown it. Ballista runs the same SQL and DataFrame workloads across a cluster with minimal code changes and the same results.
-- **Spark users wanting the same execution model** — you run Spark SQL or batch jobs and want a lighter, Rust-native alternative without relearning a new paradigm. Ballista keeps the familiar model: plans split into stages at shuffle boundaries, one task per partition, executors with vcores, and adaptive query execution (AQE).
+- **Spark users wanting the same execution model** — you run Spark SQL or batch jobs and want a lighter, Rust-native alternative without relearning a new paradigm. Ballista keeps the familiar model: plans split into stages at shuffle boundaries, tasks over partitions, executors with vcores, and adaptive query execution (AQE).
- **Library users building a specialized engine** — you are building a bespoke distributed query engine and want reusable scheduler, executor, and plan-serialization building blocks with extension points, instead of writing distributed execution from scratch.
These audiences are documented in more detail, along with the guarantees each relies on, in the [User Personas](docs/source/contributors-guide/user-personas.md) guide.
diff --git a/ballista/core/src/config.rs b/ballista/core/src/config.rs
index d2f5475d5..fc7dac2eb 100644
--- a/ballista/core/src/config.rs
+++ b/ballista/core/src/config.rs
@@ -211,7 +211,7 @@ static CONFIG_ENTRIES: LazyLock<HashMap<String, ConfigEntry>> = LazyLock::new(||
"Sets the job name that will appear in the web user interface for any submitted jobs".to_string(),
DataType::Utf8, None),
ConfigEntry::new(BALLISTA_STANDALONE_PARALLELISM.to_string(),
- "Number of concurrent tasks a standalone in-process executor will run.".to_string(),
+ "Number of vcores advertised by the standalone in-process executor.".to_string(),
DataType::UInt16, Some(std::thread::available_parallelism().map(|v| v.get()).unwrap_or(1).to_string()))
.with_doc_default("number of available CPU cores"),
ConfigEntry::new(BALLISTA_CACHE_NOOP.to_string(),
diff --git a/docs/source/contributors-guide/benchmarking.md b/docs/source/contributors-guide/benchmarking.md
index cebb33aec..fc1051cdd 100644
--- a/docs/source/contributors-guide/benchmarking.md
+++ b/docs/source/contributors-guide/benchmarking.md
@@ -74,7 +74,7 @@ Spark on the same cluster has always used this pattern via
| Flag / config key | Value |
| ------------------------------------------------------------- | ----------------------------------------------------------------------- |
-| `--concurrent-tasks` | `8` |
+| `--vcores` | `8` |
| `--memory-pool-size` (bytes; ≈70 % of the 64 GiB container) | `48103633715` |
| `--work-dir` | `/data` (dedicated gp3 PVC — see [Executor storage](#executor-storage)) |
| `--grpc-server-max-decoding-message-size` | `134217728` |
@@ -120,7 +120,7 @@ Note that `spark.executor.cores=16` is Spark's **task parallelism** setting,
not a CPU allocation — each executor pod is given only **8 physical vCPU**
via `spark.kubernetes.executor.limit.cores` / `.request.cores`, so Spark
schedules 16 concurrent tasks onto 8 physical cores (2× oversubscription).
-The matching Ballista executor runs `--concurrent-tasks=8` on the same
+The matching Ballista executor runs `--vcores=8` on the same
8 physical vCPU (1:1).
## Queries
@@ -172,7 +172,7 @@ client. Executor sizing on each node:
ballista-executor \
--bind-host 0.0.0.0 --bind-port 50051 \
--scheduler-host <scheduler> --scheduler-port 50050 \
- --concurrent-tasks 8 \
+ --vcores 8 \
--memory-pool-size 48103633715 \
--work-dir /data \
--grpc-server-max-decoding-message-size 134217728 \
diff --git a/docs/source/contributors-guide/shuffle.md b/docs/source/contributors-guide/shuffle.md
index 10d33d214..c9ac93a19 100644
--- a/docs/source/contributors-guide/shuffle.md
+++ b/docs/source/contributors-guide/shuffle.md
@@ -69,17 +69,17 @@ that one fact.
## Why the barrier is there
-### Producers and consumers never compete for slots
+### Producers and consumers never compete for vcores
-Each executor advertises a fixed number of task slots. Under a blocking
-shuffle, a stage's tasks hold slots only while that stage runs, so a stage with
-1000 tasks executes perfectly well on a cluster with 32 slots — the scheduler
-simply feeds tasks in as slots free up.
+Each executor advertises a fixed number of vcores. Under a blocking
+shuffle, a stage's tasks hold vcores only while that stage runs, so a stage with
+1000 tasks executes perfectly well on a cluster with 32 vcores — the scheduler
+simply feeds tasks in as vcores free up.
Pipelined shuffle removes that freedom. If a consumer task streams from a
producer task, both must be resident at the same time, which turns stage
scheduling into a co-scheduling (gang scheduling) problem. Get it wrong and the
-cluster deadlocks: every slot is held by a consumer waiting on a producer that
+cluster deadlocks: every vcore is held by a consumer waiting on a producer that
cannot be scheduled. Engines that pipeline either require enough capacity for
the whole pipeline region, provision workers on demand, or fall back to
materialization under pressure.
@@ -126,12 +126,12 @@ and one that never does.
### The filesystem absorbs producer/consumer skew
In a blocking shuffle, the buffer between stages is the disk. A producer never
-waits for a consumer: it writes its output, reports, and releases its slot. That
+waits for a consumer: it writes its output, reports, and releases its vcores. That
decoupling is what lets the sort-based writer bound its own memory by spilling
and still guarantee forward progress.
In a pipelined shuffle, the buffer is memory plus network, and backpressure is
-end-to-end. A slow consumer stalls its producer, which holds a slot and its
+end-to-end. A slow consumer stalls its producer, which holds its vcores and its
working set while stalled. Shuffles far larger than cluster memory — the case
Ballista is built for — need either a spill path or a remote shuffle service to
stay safe, which is most of the blocking machinery reintroduced.
@@ -252,7 +252,7 @@ behavior it would be config-gated, leaving today's model as the default:
today. A consumer task could begin once some producer tasks have finished,
learning about additional `PartitionLocation`s incrementally. Where the
cluster has spare capacity this overlaps stages and shortens the straggler
- stall, and it keeps files, retries, and slot accounting exactly as they are.
+ stall, and it keeps files, retries, and vcore accounting exactly as they are.
The general form of this — deciding how much of the DAG is in flight at once
rather than always running exactly one stage — is sometimes called bubble
execution ([#2320], [#408]).
diff --git a/docs/source/contributors-guide/user-personas.md b/docs/source/contributors-guide/user-personas.md
index a54d03854..c7d56da97 100644
--- a/docs/source/contributors-guide/user-personas.md
+++ b/docs/source/contributors-guide/user-personas.md
@@ -39,7 +39,7 @@ persona lists concrete "red flags" — the kinds of change that would quietly ta
functionality away from that audience. A red flag does not mean a change is
forbidden, but it does mean the change needs explicit discussion and maintainer
sign-off, and usually a migration path (for example an entry in the
-[upgrade guide](../upgrading/index)), rather than landing silently.
+[upgrade guide](../upgrading/index.rst)), rather than landing silently.
These personas are deliberately in tension with one another. One audience wants
distributed execution to be a transparent implementation detail; another wants
@@ -82,9 +82,11 @@ to make sure a change that delights one audience does not quietly break another.
paradigm.
- **Coming from**: Apache Spark.
- **Why Ballista**: Keep the Spark mental model — plans split into **stages** at
- shuffle boundaries, one **task** per partition, shuffle files, executors with
- task slots, and adaptive query execution (**AQE**) for runtime adaptivity — on
- top of DataFusion and Arrow.
+ shuffle boundaries, **tasks** over partitions, shuffle files, executors with
+ vcores, and adaptive query execution (**AQE**) for runtime adaptivity — on
+ top of DataFusion and Arrow. Ballista packs several partitions into one task
+ by default rather than Spark's strict one-task-per-partition dispatch; set
+ `ballista.scheduler.max_partitions_per_task` to `1` for the Spark model.
- **Depends on** (must not regress):
- The stage/task execution model at shuffle boundaries.
- The AQE / adaptive planner path (partition coalescing, dynamic join
@@ -92,7 +94,7 @@ to make sure a change that delights one audience does not quietly break another.
- Broadcast joins for small build sides, to avoid shuffles and skew (the
equivalent of Spark's broadcast hash join).
- The operational surface Spark users expect: a scheduler plus executors,
- task-slot concurrency, a tunable `target_partitions`, and stage/task
+ vcore-bounded concurrency, a tunable `target_partitions`, and stage/task
observability (task timings, a history server / UI) for debugging skew.
- Behavior driven by configuration and `SET`, not hard-coded planner choices.
- **Red flags in a pull request**:
@@ -125,7 +127,7 @@ to make sure a change that delights one audience does not quietly break another.
- Behavior that stays _configurable rather than hard-coded_, so an embedder can
swap the planner, join strategy, partitioning, or runtime environment.
- Backward-compatible public APIs, with breaking changes signaled in the
- [upgrade guide](../upgrading/index).
+ [upgrade guide](../upgrading/index.rst).
- No assumptions baked into the core that only fit the built-in clients or the
TPC-H benchmark.
- **Red flags in a pull request**:
diff --git a/docs/source/user-guide/configs.md b/docs/source/user-guide/configs.md
index 55b1ed687..92f7667b7 100644
--- a/docs/source/user-guide/configs.md
+++ b/docs/source/user-guide/configs.md
@@ -135,7 +135,7 @@ standard DataFusion settings.
| ballista.planner.parallel_window.enabled | Boolean | false | Enables the AQE parallel-window rule (ParallelWindowRule), which rewrites bounded-RANGE-frame windows into a distributed range-shuffle so BoundedWindowAggExec's single-partition constraint is not a serial bottleneck. Disabled by default — opt in when the workload contains matching window shapes. [...]
| ballista.planner.propagate_empty.enabled | Boolean | true | Enables the AQE propagate-empty-relation rule. Injects EmptyExec into the plan where an input is known to be empty, such as one side of a join, allowing downstream work to be skipped. [...]
| ballista.scheduler.max_partitions_per_task | UInt64 | 0 | Upper bound on the number of input partitions packed into a single task's `partition_slice`. `0` (default) means unbounded — the scheduler fills each task up to the executor's free vcore count. `1` means one task per input partition. Does not apply to collapse stages, which must pack their full pending queue into a single task for correctness. [...]
-| ballista.standalone.parallelism | UInt16 | number of available CPU cores | Number of concurrent tasks a standalone in-process executor will run. [...]
+| ballista.standalone.parallelism | UInt16 | number of available CPU cores | Number of vcores advertised by the standalone in-process executor. [...]
<!-- END GENERATED CONFIG REFERENCE -->
diff --git a/docs/source/user-guide/tuning-guide.md b/docs/source/user-guide/tuning-guide.md
index 9c238a503..09e67be63 100644
--- a/docs/source/user-guide/tuning-guide.md
+++ b/docs/source/user-guide/tuning-guide.md
@@ -91,7 +91,7 @@ them would produce partial results that downstream stages cannot merge.
Each task's plan is rewritten before dispatch so that its scan sees only the file groups belonging to its own slice.
-## Configuring Executor Concurrency Levels
+## Configuring Executor vcores
Each executor instance advertises a fixed number of virtual cores (vcores) to the scheduler. This is specified by
passing a `--vcores` command-line parameter. The default setting is to use all available CPU cores.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org