You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/07/23 22:26:27 UTC
(datafusion-ballista) branch asf-site updated: Publish built docs triggered by d738ca9bd6b86aecaf9587af9025ae0dd12c5af1
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-ballista.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 5c05df167 Publish built docs triggered by d738ca9bd6b86aecaf9587af9025ae0dd12c5af1
5c05df167 is described below
commit 5c05df16764ad823b765777a9360b2cce31c90bb
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Thu Jul 23 22:26:23 2026 +0000
Publish built docs triggered by d738ca9bd6b86aecaf9587af9025ae0dd12c5af1
---
_sources/contributors-guide/benchmarking.md.txt | 476 +++++------------
contributors-guide/benchmarking.html | 661 ++++++++----------------
searchindex.js | 2 +-
3 files changed, 361 insertions(+), 778 deletions(-)
diff --git a/_sources/contributors-guide/benchmarking.md.txt b/_sources/contributors-guide/benchmarking.md.txt
index 94b608623..fb3b911bf 100644
--- a/_sources/contributors-guide/benchmarking.md.txt
+++ b/_sources/contributors-guide/benchmarking.md.txt
@@ -19,384 +19,176 @@
# Benchmarking
-This page describes how Ballista is benchmarked at scale, and records the current
-results. It is aimed at contributors who want to reproduce a number, understand why
-a comparison is set up the way it is, or add a result of their own.
+Current TPC-H **SF1000** results for Ballista, compared against a vanilla
+**Spark 3.4** baseline running on the same cluster shape.
-The benchmark used here is derived from TPC-H. For running TPC-H locally at small
-scale factors, see [`benchmarks/README.md`][bench-readme] in the repository; this
-page covers the multi-node, large-scale-factor setup and the cross-engine
-comparisons.
+## Versions under test
-[bench-readme]: https://github.com/apache/datafusion-ballista/blob/main/benchmarks/README.md
-
-## What we measure, and why
-
-Ballista's central performance question is **how evenly work is spread across
-executor tasks**. A distributed query is only as fast as its slowest stage, and a
-stage is only as fast as its slowest task, so a query whose work concentrates onto
-a few partitions will underperform no matter how fast the underlying operators are.
-Benchmarks are therefore run at a scale where that imbalance actually shows up.
-
-These benchmarks are run with **adaptive query execution (AQE) enabled**
-(`ballista.planner.adaptive.enabled=true`). This is the configuration Ballista is
-being actively developed against, so it is the one measured here.
-
-AQE selects the adaptive planner, which can re-plan stages using runtime statistics —
-coalescing partitions, re-optimising joins, and promoting a small join side to a
-broadcast at runtime. The alternative, AQE off, selects the static
-`DefaultDistributedPlanner`; it is a different planner with materially different join
-behaviour, and it is no longer benchmarked on this page.
+| Engine | Version |
+| -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Ballista | [`696ca29b`](https://github.com/apache/datafusion-ballista/commit/696ca29b3c1fe3013238822b7f5d8cdb918fdaa3) (`main`, 2026-07-23), Cargo pkg `54.0.0`, DataFusion `54.1.0` |
+| Spark | 3.4 (vanilla, no acceleration plugin) |
## Environment
-The results on this page currently come from a **small homelab cluster**: two
-bare-metal nodes running Kubernetes, with the TPC-H data staged on node-local disk.
-This is a deliberate starting point rather than a final destination. A two-node
-cluster with local disk removes as many confounds as possible — no object-store
-latency, no cloud network variance, no noisy neighbours — so that when a query is
-slow, the cause is Ballista and not the environment.
-
-The intent is to **move these benchmarks to AWS with data in S3** once the results
-here are good. That environment is the one users actually run, and it exercises
-things a homelab cannot: object-store reads instead of local disk, higher and more
-variable network latency between executors, and larger executor counts. Some of
-Ballista's behaviour is expected to change there — a shuffle that is cheap over a
-local link is not cheap over a cloud network, and object-store reads make scan
-parallelism matter differently.
-
-So results on this page should be read as **relative comparisons on controlled
-hardware**, useful for "did this change help", not as absolute throughput numbers
-for a cloud deployment.
-
-## Reference cluster
-
-All results on this page use the following shape. It is a reference point, not a
-requirement — the commands below work on any cluster, but numbers are only
-comparable when the shape matches.
-
-| | |
-| ------------------- | ------------------------------------------- |
-| Executors | 2, one per physical node |
-| Per executor | 16 cores, 56 GiB, `--memory-pool-size=48GB` |
-| Per task slot | 16 concurrent tasks → 3 GB pool each |
-| Scheduler | 1 |
-| Data | TPC-H SF1000 Parquet, node-local disk |
-| `target_partitions` | 64 |
-
-Two details matter more than they look:
-
-- **Executors are spread one per node.** Packing two executors onto one node makes
- them contend for the same disk and memory bandwidth, which is measuring the host,
- not the engine.
-- **Data is on node-local disk, not object storage.** With a modest network between
- nodes, reading from object storage makes the interconnect the bottleneck and the
- engine comparison becomes an I/O comparison.
-
-The memory pool is deliberately set below the container limit. Ballista splits
-`--memory-pool-size` into one `FairSpillPool` per task slot, and that accounting
-does not cover every allocation, so leaving headroom between the pool and the
-container limit avoids the container being killed outright instead of reporting a
-graceful resource error.
+- **Cluster:** Kubernetes on AWS (`us-west-2`); one driver/scheduler pod and
+ 32 executor pods for each engine, launched on the same node pool.
+- **Executor pod (Ballista):** x86_64, 8 vCPU, 64 GiB memory.
+- **Executor pod (Spark):** x86_64, 8 vCPU, 64 GiB + 10 GiB overhead.
+- **Client pod (Ballista):** the Python benchmark runner submits SQL to
+ Ballista via `BallistaSessionContext` and collects results locally.
+ Requires 64 GiB memory limit to complete the full 22-query suite —
+ smaller limits (16 GiB) OOM the client partway through, even though the
+ executor cluster is healthy.
+- **Data:** TPC-H SF1000 Parquet on S3 (`us-west-2`), ZSTD compression,
+ ~512 MiB row groups, one directory per table.
+
+## Ballista configuration
+
+| Flag / config key | Value |
+| ------------------------------------------------------------- | ------------- |
+| `--concurrent-tasks` | `8` |
+| `--memory-pool-size` (bytes; ≈70 % of the 64 GiB container) | `48103633715` |
+| `datafusion.execution.target_partitions` | `256` |
+| `datafusion.execution.collect_statistics` | `true` |
+| `datafusion.execution.listing_table_factory_infer_partitions` | `false` |
+| `datafusion.catalog.information_schema` | `true` |
+| `ballista.planner.adaptive.enabled` | `true` (AQE) |
+| `ballista.shuffle.sort_based.memory_limit_per_task_bytes` | `0` |
+
+`datafusion.optimizer.prefer_hash_join` is left at its default; under AQE the
+join strategy is selected at runtime by `DelayJoinSelectionRule` /
+`DynamicJoinSelectionExec` from runtime statistics and the broadcast /
+`ballista.optimizer.hash_join_max_build_partition_bytes` thresholds.
+
+## Spark configuration (highlights)
+
+Vanilla Spark 3.4 — no Comet plugin, stock `SortShuffleManager`.
+
+| Key | Value |
+| ---------------------------------------------------------- | ------------------------------------ |
+| `spark.executor.instances` | `32` |
+| `spark.executor.cores` | `16` (task parallelism per executor) |
+| `spark.kubernetes.executor.limit.cores` / `.request.cores` | `8` (physical vCPU) |
+| `spark.executor.memory` | `64G` |
+| `spark.executor.memoryOverhead` | `10G` |
+| `spark.memory.fraction` | `0.6` |
+| `spark.memory.storageFraction` | `0.2` |
+| `spark.sql.shuffle.partitions` | `512` |
+| `spark.sql.broadcastTimeout` | `900` |
+| `spark.serializer` | `KryoSerializer` |
+| `spark.io.compression.codec` | `zstd` |
+
+Spark AQE is left at its Spark 3.4 defaults. Shuffle spills to a `gp3`-backed
+per-executor volume (`spark.kubernetes.executor.volumes...spark-local-dir-1`).
+
+Note that `spark.executor.cores=16` is Spark's **task parallelism** setting,
+not a CPU allocation — each executor pod is given only **8 physical vCPU**
+via `spark.kubernetes.executor.limit.cores` / `.request.cores`, so Spark
+schedules 16 concurrent tasks onto 8 physical cores (2× oversubscription).
+The matching Ballista executor runs `--concurrent-tasks=8` on the same
+8 physical vCPU (1:1).
## Queries
-All engines run the **same SQL**, which is what makes a cross-engine comparison
-apples-to-apples.
+The SQLBench-H phrasing of the 22 TPC-H queries from
+[apache/datafusion-benchmarks](https://github.com/apache/datafusion-benchmarks).
-The shared set is the TPC-H queries from [apache/datafusion-benchmarks][dfb]
-(SQLBench-H). Ballista's bundled queries in `benchmarks/queries/` are the classic
-TPC-H phrasing and differ in places, so the shared set is overlaid over them for a
-comparison run.
-
-Spark and Comet are driven through Comet's benchmark harness, which reads its own
-bundled copy of the queries (labelled CometBench-H). That copy is textually
-identical to the SQLBench-H set — all 22 queries match once the licence header
-comment is ignored — so the engines are executing the same statements even though
-they load them from different paths. Worth re-checking if either set is ever
-regenerated.
-
-[dfb]: https://github.com/apache/datafusion-benchmarks
+## Results
-## Running the benchmark
+**Times in seconds; lower is better.** Ballista: single iteration. Spark:
+mean of 3 iterations (cold iteration dropped by the harness).
+
+| Query | Ballista (s) | Spark 3.4 (s) |
+| -----------------: | -----------: | ------------: |
+| 1 | 21.16 | 67.58 |
+| 2 | 32.37 | 29.80 |
+| 3 | 32.69 | 25.13 |
+| 4 | 27.75 | 21.19 |
+| 5 | 49.37 | 54.12 |
+| 6 | 16.17 | 1.23 |
+| 7 | 48.24 | 19.57 |
+| 8 | 189.30 | 48.60 |
+| 9 | 154.88 | 69.38 |
+| 10 | 67.72 | 35.92 |
+| 11 | 26.72 | 30.88 |
+| 12 | 24.46 | 10.78 |
+| 13 | 18.36 | 20.45 |
+| 14 | 19.93 | 7.00 |
+| 15 | 27.56 | 23.75 |
+| 16 | 18.03 | 23.41 |
+| 17 | 60.46 | 82.30 |
+| 18 | 282.79 | 129.40 |
+| 19 | 23.28 | 11.26 |
+| 20 | 88.93 | 19.22 |
+| 21 | 110.87 | 101.53 |
+| 22 | 12.17 | 12.71 |
+| **Total (Q1–Q22)** | **1353.21** | **845.21** |
+
+Row counts agree across engines for every query. Ballista completed the
+full 22-query suite in a single continuous run.
+
+## Reproducing
### Ballista
-Start a scheduler and one executor per node:
+Bring up the cluster (one scheduler, N executors), then run the suite from a
+client. Executor sizing on each node:
```sh
-# scheduler
-ballista-scheduler --bind-host 0.0.0.0 --bind-port 50050
-
-# executor (one per node; --concurrent-tasks defaults to the detected core count)
ballista-executor \
- --bind-host 0.0.0.0 \
+ --bind-host 0.0.0.0 --bind-port 50051 \
--scheduler-host <scheduler> --scheduler-port 50050 \
- --memory-pool-size=48GB \
- --work-dir /work \
- --client-ttl=60
+ --concurrent-tasks 8 \
+ --memory-pool-size 48103633715
```
-`--client-ttl=60` enables shuffle-client connection caching. Without it every
-shuffle fetch opens a new connection, and a high `target_partitions` can exhaust
-ephemeral ports.
-
-Run a query:
+Run all 22 queries:
```sh
tpch benchmark ballista \
--host <scheduler> --port 50050 \
- --query 18 \
- --path /mnt/bigdata/tpch/sf1000 --format parquet \
- --partitions 64 --iterations 1 \
- -c ballista.planner.adaptive.enabled=true
+ --path s3://<bucket>/tpch/sf1000 --format parquet \
+ --partitions 256 --iterations 2 \
+ -c ballista.planner.adaptive.enabled=true \
+ -c datafusion.execution.collect_statistics=true \
+ -c ballista.shuffle.sort_based.memory_limit_per_task_bytes=0
```
-Omit `--query` to run all 22, and add `--skip <n>` (repeatable) to leave individual
-queries out of that run — `--skip 18` is what produced the Ballista column below.
-`ballista.planner.adaptive.enabled=true` is the AQE-on configuration these results
-use.
-
-Note that `datafusion.optimizer.prefer_hash_join` is deliberately left at its default.
-Under AQE it does not select the join strategy: `DelayJoinSelectionRule` folds both
-`HashJoinExec` and `SortMergeJoinExec` into a single `DynamicJoinSelectionExec`, and
-the strategy is then chosen from runtime statistics — the broadcast thresholds, and
-`ballista.optimizer.hash_join_max_build_partition_bytes` (64 MiB by default), which
-lowers a Partitioned hash join to the spillable `SortMergeJoin` when the build side's
-largest partition would not fit. Setting `prefer_hash_join=false` does not force
-sort-merge; it only changes which conversion path the plan takes, and the
-`SortMergeJoinExec` path discards the join's projection, so it measures a slower plan
-rather than a different join strategy.
-
### Spark
-Spark runs the same queries via `tpcbench.py` from
-[apache/datafusion-benchmarks][dfb]:
+Runs the same queries via `tpcbench.py` from
+[apache/datafusion-benchmarks](https://github.com/apache/datafusion-benchmarks),
+with the highlights above and stock Spark 3.4 defaults for everything else:
```sh
spark-submit \
--master <master> \
- --conf spark.executor.instances=2 \
+ --conf spark.executor.instances=32 \
--conf spark.executor.cores=16 \
- --conf spark.executor.memory=32G \
- --conf spark.executor.memoryOverhead=8G \
- --conf spark.memory.offHeap.enabled=true \
- --conf spark.memory.offHeap.size=16g \
- --conf spark.comet.enabled=false \
- --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.SortShuffleManager \
+ --conf spark.executor.memory=64G \
+ --conf spark.executor.memoryOverhead=10G \
+ --conf spark.sql.shuffle.partitions=512 \
tpcbench.py \
--benchmark tpch \
- --data /mnt/bigdata/tpch/sf1000 \
+ --data s3a://<bucket>/tpch/sf1000 \
--format parquet \
- --iterations 1 \
- --query 18
+ --iterations 3
```
-`spark.comet.enabled=false` and the stock `SortShuffleManager` are what make this a
-**vanilla** Spark baseline, in case the image being used ships Comet.
-
-### Comet
-
-[Apache DataFusion Comet][comet] accelerates Spark by translating supported
-operators to DataFusion. Same queries, same sizing; the difference is the plugin,
-the Comet shuffle manager, and the jar on the classpath:
-
-```sh
-spark-submit \
- --master <master> \
- --jars $COMET_JAR --driver-class-path $COMET_JAR \
- --conf spark.executor.extraClassPath=$COMET_JAR \
- --conf spark.plugins=org.apache.spark.CometPlugin \
- --conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \
- --conf spark.comet.exec.replaceSortMergeJoin=false \
- --conf spark.comet.exec.memoryPool=fair_unified \
- --conf spark.comet.exec.memoryPool.fraction=0.8 \
- --conf spark.executor.instances=2 \
- --conf spark.executor.cores=16 \
- --conf spark.executor.memory=32G \
- --conf spark.executor.memoryOverhead=8G \
- --conf spark.memory.offHeap.enabled=true \
- --conf spark.memory.offHeap.size=16g \
- tpcbench.py \
- --benchmark tpch \
- --data /mnt/bigdata/tpch/sf1000 \
- --format parquet \
- --iterations 1 \
- --query 18
-```
-
-`spark.comet.exec.replaceSortMergeJoin=false` keeps Comet on Spark's `SortMergeJoin`
-instead of converting it to a shuffled hash join.
-
-This no longer pins the two engines to the same join strategy. Ballista under AQE
-chooses per join at runtime — broadcast, Partitioned hash, or sort-merge — and there
-is no supported way to force it to sort-merge throughout (see the note in the Ballista
-section above). So a Ballista-vs-Comet gap may partly reflect different join
-strategies, not only different execution. Read the comparison with that in mind.
-
-[comet]: https://datafusion.apache.org/comet/
-
-## Why compare against Comet
-
-Comet is the most informative comparison available to Ballista, because **Comet and
-Ballista execute DataFusion physical plans using the same DataFusion operators**.
-The scan, filter, join, and aggregate implementations doing the work are largely
-shared code. When Ballista and Comet diverge on a query, the difference therefore
-points at what is _not_ shared — how work is distributed, scheduled, shuffled, and
-bounded by memory — rather than at the speed of the operators themselves. That is
-precisely the surface Ballista is trying to improve, which makes the comparison
-diagnostic rather than merely competitive.
-
-That said, **the two do not necessarily run the same plan shape**, and the numbers
-should not be read as an operator-level A/B:
-
-- **Different planners produce the plan.** Comet accelerates a plan that _Spark's_
- optimizer produced: Spark chooses the join order and join strategies, and
- **Spark's AQE** coalesces shuffle partitions, converts joins, and splits skewed
- partitions at runtime. Ballista plans with DataFusion's optimizer and, when
- enabled, its own experimental AQE. Spark's AQE is a mature implementation and
- Ballista's is not, so the same SQL can arrive at execution with materially
- different plans.
-- **Comet falls back to Spark.** Operators and expressions Comet does not support
- stay on the JVM, so a Comet run is generally a mix of DataFusion and Spark
- execution rather than an all-DataFusion one.
-- **The distribution models differ.** Comet executes within Spark's task model and
- shuffle service; Ballista has its own scheduler, stage/task model, and shuffle.
-
-So a Comet-vs-Ballista gap is best read as a question — _what is Spark's planner or
-execution model doing here that Ballista's is not?_ — rather than as a verdict on
-DataFusion. Vanilla Spark is included as the third data point, since it isolates
-how much of any Comet result comes from DataFusion acceleration versus from Spark's
-planner.
-
-## Results
-
-TPC-H **SF1000**, reference cluster above, **AQE on**, 1 iteration,
-`target_partitions=64`, all Ballista join settings at their defaults. Ballista picks
-each join's strategy at runtime; Spark and Comet are held to `SortMergeJoin`. Times in
-seconds; lower is better.
-
-Versions under test:
-
-| Engine | Version |
-| -------- | ----------------------------------------------- |
-| Ballista | `#2124` @ `5290436d` |
-| Spark | 3.5.3 (vanilla, Comet disabled) |
-| Comet | `main` @ `b0165552` (1.0.0-SNAPSHOT, Spark 3.5) |
-
-Pin the **exact commit** the numbers came from, not "main": `main` moves, and a row
-that mixes numbers from different commits silently misattributes a regression.
-
-The Ballista column comes from a **single suite run of 21 queries** on freshly
-started executors, using `--skip 18` to leave out the one query that exhausts the
-memory pool and OOM-kills the executors (see below), which would otherwise wedge
-the rest of the suite. Comet and Spark each completed the full suite in one run.
-
-Keep the whole column to one continuous run wherever possible. Driving queries as
-individual single-query jobs measures something different: each job starts against
-a cold page cache, which on this cluster cost the two join-free queries (Q1, Q6)
-roughly 20 s and 20 s respectively when the two methods were compared directly. Use
-`--skip` rather than a loop of `--query` runs.
-
-Single iteration means **run-to-run variance is not quantified here**. Comparing the
-same commit across the two methods above showed spreads of up to ~20% on individual
-queries, so treat a per-query difference under about 20% as noise and read the total
-row instead.
-
-| Query | Spark | Comet | Ballista (AQE on) | Rows |
-| --------------------: | ---------: | ---------: | ----------------: | -----: |
-| 1 | 427.1 | 46.6 | 54.2 | 4 |
-| 2 | 75.9 | 39.2 | 60.3 | 100 |
-| 3 | 154.1 | 195.4 | 140.0 | 10 |
-| 4 | 82.0 | 42.6 | 46.3 | 5 |
-| 5 | 336.4 | 493.4 | 327.2 | 5 |
-| 6 | 28.9 | 14.6 | 27.2 | 1 |
-| 7 | 180.6 | 149.5 | 364.9 | 4 |
-| 8 | 391.8 | 642.1 | 481.1 | 2 |
-| 9 | 509.8 | 843.5 | 606.4 | 175 |
-| 10 | 151.1 | 104.8 | 148.4 | 20 |
-| 11 | 44.7 | 44.1 | 62.0 | 0 [1] |
-| 12 | 74.1 | 49.9 | 58.3 | 2 |
-| 13 | 98.7 | 58.3 | 100.4 | 30 |
-| 14 | 43.0 | 27.4 | 71.1 | 1 |
-| 15 | 121.5 | 65.6 | 53.8 | 1 [2] |
-| 16 | 24.5 | 17.5 | 34.3 | 27840 |
-| 17 | 406.7 | 285.8 | 417.8 | 1 |
-| 18 | 428.8 | 370.5 | TBD | 100 |
-| 19 | 58.9 | 35.5 | 68.5 | 1 |
-| 20 | 105.9 | 67.6 | 93.7 | 110759 |
-| 21 | 562.2 | 460.1 | 698.0 | 100 |
-| 22 | 36.8 | 21.5 | 22.9 | 7 |
-| **Total (excl. Q18)** | **3914.7** | **3705.0** | **3936.8** | |
-
-The **Total** row sums the 21 queries **excluding Q18**, because Q18 has no Ballista
-figure at this commit (below), so a 22-query total would not be comparable across
-engines. Spark's and Comet's Q18 times are in its row.
-
-Row counts agree across all three engines on every query for which Ballista produced
-a result.
-
-The **AQE-off** Ballista column has been removed pending a re-run at a matched core
-count; the numbers above for AQE off were taken at a different executor size and are
-not comparable to a fresh AQE-off run, so they were dropped rather than left stale.
-
-[1] Q11 returns 0 rows for every engine at this scale factor: the query's threshold
-constant is tuned for SF1.
-
-[2] Q15 is a multi-statement query (`CREATE VIEW` / `SELECT` / `DROP VIEW`); all three
-engines report 1 row here. (A previous result set saw Spark report 0 for Q15 depending
-on which statement the harness took as the result; it does not recur in this run.)
-
-**Q18 is not measured at this commit.** Q18's hash-join build side is `Partitioned`
-and does not spill; with 16 task slots sharing the 48 GB pool (3 GB per slot), the
-per-task build side exceeds the container limit and the executor is OOM-killed
-([#2025](https://github.com/apache/datafusion-ballista/issues/2025)). At the previous
-8-slot sizing (6 GB per slot) Q18 completed, so this is a direct consequence of the
-denser packing.
-
-It was excluded from the suite run above with `--skip 18` so that one query could not
-wedge the other 21, and it has not since been run to completion at this commit — hence
-`TBD` rather than `OOM`. An earlier attempt on this branch did OOM-kill an executor,
-but under a different configuration (`prefer_hash_join=false`, which sends the plan
-down a conversion path that discards the join's projection and so materialises more
-columns than the configuration documented above). That result does not transfer, so it
-is not recorded here.
-
-The `Rows` column is the row count the query returned, recorded so a time is never
-read without the answer it produced.
-
-This table records **one current result set**. When results are refreshed, the
-table and the pinned versions above are replaced together — a row must never mix
-numbers from different commits, because a stale row silently misattributes a
-regression.
-
-`TBD` means not yet measured on this cluster at this commit; a query that ran but
-did not produce an answer is recorded as `FAIL`, or `OOM` where the failure is a
-known memory exhaustion.
-
-### Recording a result
+## Recording a new result set
- Pin the **exact commit** the numbers came from, not a branch name.
-- Report the **AQE-on** number (`ballista.planner.adaptive.enabled=true`) — the
- configuration this page measures — from the same commit and cluster.
-- Prefer the figure from a **full suite run**, not a standalone single-query run: a
- long-lived executor deep into a suite is not in the same state as a freshly started
- one, and a single-query job pays a cold-cache penalty that is large enough to swamp
- the effect being measured. When a query cannot complete in-suite (e.g. Q18's OOM
- currently wedges the run), exclude it with `--skip <n>` so the rest of the column
- still comes from one continuous run, and measure that query separately.
-- Note the **row count** each query returned. A fast wrong answer is not a result,
- and distributed execution has produced silently wrong row counts before.
-- Flag any stage whose runtime is dominated by a few partitions — that is the
- imbalance this page exists to surface.
-
-## Known issues found by benchmarking
-
-Benchmarking at SF1000 is how most of the following were found. They are worth
-knowing about before interpreting a number:
-
-| Issue | Summary |
-| ------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| [#1359](https://github.com/apache/datafusion-ballista/issues/1359) | Umbrella issue for adaptive (AQE) query execution. |
-| [#2025](https://github.com/apache/datafusion-ballista/issues/2025) | Q18's hash-join build side exhausts the memory pool at SF1000. DataFusion's hash-join build side does not spill, so a per-partition build side larger than one task slot's pool fails the task outright. |
-| [#2063](https://github.com/apache/datafusion-ballista/issues/2063) | AQE can hang when a re-plan cancels an in-flight stage; the job never reports completion. |
+- Replace the version, environment, config, and results tables together — a
+ row that mixes numbers from different commits silently misattributes a
+ regression.
+- Prefer a single continuous suite run: a long-lived executor deep into a
+ suite is not in the same state as a freshly started one.
+- Report `FAIL` for a query that ran but did not produce an answer, and
+ `OOM` when the failure is a known memory exhaustion. See the tracker for
+ open issues found by benchmarking: [#1359][aqe],
+ [#2025][q18], [#2063][aqe-hang].
+
+[aqe]: https://github.com/apache/datafusion-ballista/issues/1359
+[q18]: https://github.com/apache/datafusion-ballista/issues/2025
+[aqe-hang]: https://github.com/apache/datafusion-ballista/issues/2063
diff --git a/contributors-guide/benchmarking.html b/contributors-guide/benchmarking.html
index 384390bfe..d3f16c2b0 100644
--- a/contributors-guide/benchmarking.html
+++ b/contributors-guide/benchmarking.html
@@ -390,519 +390,314 @@
-->
<section id="benchmarking">
<h1>Benchmarking<a class="headerlink" href="#benchmarking" title="Link to this heading">#</a></h1>
-<p>This page describes how Ballista is benchmarked at scale, and records the current
-results. It is aimed at contributors who want to reproduce a number, understand why
-a comparison is set up the way it is, or add a result of their own.</p>
-<p>The benchmark used here is derived from TPC-H. For running TPC-H locally at small
-scale factors, see <a class="reference external" href="https://github.com/apache/datafusion-ballista/blob/main/benchmarks/README.md"><code class="docutils literal notranslate"><span class="pre">benchmarks/README.md</span></code></a> in the repository; this
-page covers the multi-node, large-scale-factor setup and the cross-engine
-comparisons.</p>
-<section id="what-we-measure-and-why">
-<h2>What we measure, and why<a class="headerlink" href="#what-we-measure-and-why" title="Link to this heading">#</a></h2>
-<p>Ballista’s central performance question is <strong>how evenly work is spread across
-executor tasks</strong>. A distributed query is only as fast as its slowest stage, and a
-stage is only as fast as its slowest task, so a query whose work concentrates onto
-a few partitions will underperform no matter how fast the underlying operators are.
-Benchmarks are therefore run at a scale where that imbalance actually shows up.</p>
-<p>These benchmarks are run with <strong>adaptive query execution (AQE) enabled</strong>
-(<code class="docutils literal notranslate"><span class="pre">ballista.planner.adaptive.enabled=true</span></code>). This is the configuration Ballista is
-being actively developed against, so it is the one measured here.</p>
-<p>AQE selects the adaptive planner, which can re-plan stages using runtime statistics —
-coalescing partitions, re-optimising joins, and promoting a small join side to a
-broadcast at runtime. The alternative, AQE off, selects the static
-<code class="docutils literal notranslate"><span class="pre">DefaultDistributedPlanner</span></code>; it is a different planner with materially different join
-behaviour, and it is no longer benchmarked on this page.</p>
+<p>Current TPC-H <strong>SF1000</strong> results for Ballista, compared against a vanilla
+<strong>Spark 3.4</strong> baseline running on the same cluster shape.</p>
+<section id="versions-under-test">
+<h2>Versions under test<a class="headerlink" href="#versions-under-test" title="Link to this heading">#</a></h2>
+<div class="pst-scrollable-table-container"><table class="table">
+<thead>
+<tr class="row-odd"><th class="head"><p>Engine</p></th>
+<th class="head"><p>Version</p></th>
+</tr>
+</thead>
+<tbody>
+<tr class="row-even"><td><p>Ballista</p></td>
+<td><p><a class="reference external" href="https://github.com/apache/datafusion-ballista/commit/696ca29b3c1fe3013238822b7f5d8cdb918fdaa3"><code class="docutils literal notranslate"><span class="pre">696ca29b</span></code></a> (<code class="docutils literal notranslate"><span class="pre">main</span></code>, 2026-07-23), Cargo pkg <code class="docutils literal notranslate"><span class="pre">54.0.0</span></code>, DataFusion <code class="docutils literal notranslate"><span class="pre">54.1.0 [...]
+</tr>
+<tr class="row-odd"><td><p>Spark</p></td>
+<td><p>3.4 (vanilla, no acceleration plugin)</p></td>
+</tr>
+</tbody>
+</table>
+</div>
</section>
<section id="environment">
<h2>Environment<a class="headerlink" href="#environment" title="Link to this heading">#</a></h2>
-<p>The results on this page currently come from a <strong>small homelab cluster</strong>: two
-bare-metal nodes running Kubernetes, with the TPC-H data staged on node-local disk.
-This is a deliberate starting point rather than a final destination. A two-node
-cluster with local disk removes as many confounds as possible — no object-store
-latency, no cloud network variance, no noisy neighbours — so that when a query is
-slow, the cause is Ballista and not the environment.</p>
-<p>The intent is to <strong>move these benchmarks to AWS with data in S3</strong> once the results
-here are good. That environment is the one users actually run, and it exercises
-things a homelab cannot: object-store reads instead of local disk, higher and more
-variable network latency between executors, and larger executor counts. Some of
-Ballista’s behaviour is expected to change there — a shuffle that is cheap over a
-local link is not cheap over a cloud network, and object-store reads make scan
-parallelism matter differently.</p>
-<p>So results on this page should be read as <strong>relative comparisons on controlled
-hardware</strong>, useful for “did this change help”, not as absolute throughput numbers
-for a cloud deployment.</p>
+<ul class="simple">
+<li><p><strong>Cluster:</strong> Kubernetes on AWS (<code class="docutils literal notranslate"><span class="pre">us-west-2</span></code>); one driver/scheduler pod and
+32 executor pods for each engine, launched on the same node pool.</p></li>
+<li><p><strong>Executor pod (Ballista):</strong> x86_64, 8 vCPU, 64 GiB memory.</p></li>
+<li><p><strong>Executor pod (Spark):</strong> x86_64, 8 vCPU, 64 GiB + 10 GiB overhead.</p></li>
+<li><p><strong>Client pod (Ballista):</strong> the Python benchmark runner submits SQL to
+Ballista via <code class="docutils literal notranslate"><span class="pre">BallistaSessionContext</span></code> and collects results locally.
+Requires 64 GiB memory limit to complete the full 22-query suite —
+smaller limits (16 GiB) OOM the client partway through, even though the
+executor cluster is healthy.</p></li>
+<li><p><strong>Data:</strong> TPC-H SF1000 Parquet on S3 (<code class="docutils literal notranslate"><span class="pre">us-west-2</span></code>), ZSTD compression,
+~512 MiB row groups, one directory per table.</p></li>
+</ul>
</section>
-<section id="reference-cluster">
-<h2>Reference cluster<a class="headerlink" href="#reference-cluster" title="Link to this heading">#</a></h2>
-<p>All results on this page use the following shape. It is a reference point, not a
-requirement — the commands below work on any cluster, but numbers are only
-comparable when the shape matches.</p>
+<section id="ballista-configuration">
+<h2>Ballista configuration<a class="headerlink" href="#ballista-configuration" title="Link to this heading">#</a></h2>
<div class="pst-scrollable-table-container"><table class="table">
<thead>
-<tr class="row-odd"><th class="head"><p></p></th>
-<th class="head"><p></p></th>
+<tr class="row-odd"><th class="head"><p>Flag / config key</p></th>
+<th class="head"><p>Value</p></th>
</tr>
</thead>
<tbody>
-<tr class="row-even"><td><p>Executors</p></td>
-<td><p>2, one per physical node</p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">--concurrent-tasks</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">8</span></code></p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">--memory-pool-size</span></code> (bytes; ≈70 % of the 64 GiB container)</p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">48103633715</span></code></p></td>
</tr>
-<tr class="row-odd"><td><p>Per executor</p></td>
-<td><p>16 cores, 56 GiB, <code class="docutils literal notranslate"><span class="pre">--memory-pool-size=48GB</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">datafusion.execution.target_partitions</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">256</span></code></p></td>
</tr>
-<tr class="row-even"><td><p>Per task slot</p></td>
-<td><p>16 concurrent tasks → 3 GB pool each</p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">datafusion.execution.collect_statistics</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">true</span></code></p></td>
</tr>
-<tr class="row-odd"><td><p>Scheduler</p></td>
-<td><p>1</p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">datafusion.execution.listing_table_factory_infer_partitions</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">false</span></code></p></td>
</tr>
-<tr class="row-even"><td><p>Data</p></td>
-<td><p>TPC-H SF1000 Parquet, node-local disk</p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">datafusion.catalog.information_schema</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">true</span></code></p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">target_partitions</span></code></p></td>
-<td><p>64</p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">ballista.planner.adaptive.enabled</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">true</span></code> (AQE)</p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">ballista.shuffle.sort_based.memory_limit_per_task_bytes</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">0</span></code></p></td>
</tr>
</tbody>
</table>
</div>
-<p>Two details matter more than they look:</p>
-<ul class="simple">
-<li><p><strong>Executors are spread one per node.</strong> Packing two executors onto one node makes
-them contend for the same disk and memory bandwidth, which is measuring the host,
-not the engine.</p></li>
-<li><p><strong>Data is on node-local disk, not object storage.</strong> With a modest network between
-nodes, reading from object storage makes the interconnect the bottleneck and the
-engine comparison becomes an I/O comparison.</p></li>
-</ul>
-<p>The memory pool is deliberately set below the container limit. Ballista splits
-<code class="docutils literal notranslate"><span class="pre">--memory-pool-size</span></code> into one <code class="docutils literal notranslate"><span class="pre">FairSpillPool</span></code> per task slot, and that accounting
-does not cover every allocation, so leaving headroom between the pool and the
-container limit avoids the container being killed outright instead of reporting a
-graceful resource error.</p>
-</section>
-<section id="queries">
-<h2>Queries<a class="headerlink" href="#queries" title="Link to this heading">#</a></h2>
-<p>All engines run the <strong>same SQL</strong>, which is what makes a cross-engine comparison
-apples-to-apples.</p>
-<p>The shared set is the TPC-H queries from <a class="reference external" href="https://github.com/apache/datafusion-benchmarks">apache/datafusion-benchmarks</a>
-(SQLBench-H). Ballista’s bundled queries in <code class="docutils literal notranslate"><span class="pre">benchmarks/queries/</span></code> are the classic
-TPC-H phrasing and differ in places, so the shared set is overlaid over them for a
-comparison run.</p>
-<p>Spark and Comet are driven through Comet’s benchmark harness, which reads its own
-bundled copy of the queries (labelled CometBench-H). That copy is textually
-identical to the SQLBench-H set — all 22 queries match once the licence header
-comment is ignored — so the engines are executing the same statements even though
-they load them from different paths. Worth re-checking if either set is ever
-regenerated.</p>
-</section>
-<section id="running-the-benchmark">
-<h2>Running the benchmark<a class="headerlink" href="#running-the-benchmark" title="Link to this heading">#</a></h2>
-<section id="ballista">
-<h3>Ballista<a class="headerlink" href="#ballista" title="Link to this heading">#</a></h3>
-<p>Start a scheduler and one executor per node:</p>
-<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span><span class="c1"># scheduler</span>
-ballista-scheduler<span class="w"> </span>--bind-host<span class="w"> </span><span class="m">0</span>.0.0.0<span class="w"> </span>--bind-port<span class="w"> </span><span class="m">50050</span>
-
-<span class="c1"># executor (one per node; --concurrent-tasks defaults to the detected core count)</span>
-ballista-executor<span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--bind-host<span class="w"> </span><span class="m">0</span>.0.0.0<span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--scheduler-host<span class="w"> </span><scheduler><span class="w"> </span>--scheduler-port<span class="w"> </span><span class="m">50050</span><span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--memory-pool-size<span class="o">=</span>48GB<span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--work-dir<span class="w"> </span>/work<span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--client-ttl<span class="o">=</span><span class="m">60</span>
-</pre></div>
-</div>
-<p><code class="docutils literal notranslate"><span class="pre">--client-ttl=60</span></code> enables shuffle-client connection caching. Without it every
-shuffle fetch opens a new connection, and a high <code class="docutils literal notranslate"><span class="pre">target_partitions</span></code> can exhaust
-ephemeral ports.</p>
-<p>Run a query:</p>
-<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>tpch<span class="w"> </span>benchmark<span class="w"> </span>ballista<span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--host<span class="w"> </span><scheduler><span class="w"> </span>--port<span class="w"> </span><span class="m">50050</span><span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--query<span class="w"> </span><span class="m">18</span><span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--path<span class="w"> </span>/mnt/bigdata/tpch/sf1000<span class="w"> </span>--format<span class="w"> </span>parquet<span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--partitions<span class="w"> </span><span class="m">64</span><span class="w"> </span>--iterations<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>-c<span class="w"> </span>ballista.planner.adaptive.enabled<span class="o">=</span><span class="nb">true</span>
-</pre></div>
-</div>
-<p>Omit <code class="docutils literal notranslate"><span class="pre">--query</span></code> to run all 22, and add <code class="docutils literal notranslate"><span class="pre">--skip</span> <span class="pre"><n></span></code> (repeatable) to leave individual
-queries out of that run — <code class="docutils literal notranslate"><span class="pre">--skip</span> <span class="pre">18</span></code> is what produced the Ballista column below.
-<code class="docutils literal notranslate"><span class="pre">ballista.planner.adaptive.enabled=true</span></code> is the AQE-on configuration these results
-use.</p>
-<p>Note that <code class="docutils literal notranslate"><span class="pre">datafusion.optimizer.prefer_hash_join</span></code> is deliberately left at its default.
-Under AQE it does not select the join strategy: <code class="docutils literal notranslate"><span class="pre">DelayJoinSelectionRule</span></code> folds both
-<code class="docutils literal notranslate"><span class="pre">HashJoinExec</span></code> and <code class="docutils literal notranslate"><span class="pre">SortMergeJoinExec</span></code> into a single <code class="docutils literal notranslate"><span class="pre">DynamicJoinSelectionExec</span></code>, and
-the strategy is then chosen from runtime statistics — the broadcast thresholds, and
-<code class="docutils literal notranslate"><span class="pre">ballista.optimizer.hash_join_max_build_partition_bytes</span></code> (64 MiB by default), which
-lowers a Partitioned hash join to the spillable <code class="docutils literal notranslate"><span class="pre">SortMergeJoin</span></code> when the build side’s
-largest partition would not fit. Setting <code class="docutils literal notranslate"><span class="pre">prefer_hash_join=false</span></code> does not force
-sort-merge; it only changes which conversion path the plan takes, and the
-<code class="docutils literal notranslate"><span class="pre">SortMergeJoinExec</span></code> path discards the join’s projection, so it measures a slower plan
-rather than a different join strategy.</p>
-</section>
-<section id="spark">
-<h3>Spark<a class="headerlink" href="#spark" title="Link to this heading">#</a></h3>
-<p>Spark runs the same queries via <code class="docutils literal notranslate"><span class="pre">tpcbench.py</span></code> from
-<a class="reference external" href="https://github.com/apache/datafusion-benchmarks">apache/datafusion-benchmarks</a>:</p>
-<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>spark-submit<span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--master<span class="w"> </span><master><span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--conf<span class="w"> </span>spark.executor.instances<span class="o">=</span><span class="m">2</span><span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--conf<span class="w"> </span>spark.executor.cores<span class="o">=</span><span class="m">16</span><span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--conf<span class="w"> </span>spark.executor.memory<span class="o">=</span>32G<span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--conf<span class="w"> </span>spark.executor.memoryOverhead<span class="o">=</span>8G<span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--conf<span class="w"> </span>spark.memory.offHeap.enabled<span class="o">=</span><span class="nb">true</span><span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--conf<span class="w"> </span>spark.memory.offHeap.size<span class="o">=</span>16g<span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--conf<span class="w"> </span>spark.comet.enabled<span class="o">=</span><span class="nb">false</span><span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--conf<span class="w"> </span>spark.shuffle.manager<span class="o">=</span>org.apache.spark.shuffle.sort.SortShuffleManager<span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>tpcbench.py<span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--benchmark<span class="w"> </span>tpch<span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--data<span class="w"> </span>/mnt/bigdata/tpch/sf1000<span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--format<span class="w"> </span>parquet<span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--iterations<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--query<span class="w"> </span><span class="m">18</span>
-</pre></div>
-</div>
-<p><code class="docutils literal notranslate"><span class="pre">spark.comet.enabled=false</span></code> and the stock <code class="docutils literal notranslate"><span class="pre">SortShuffleManager</span></code> are what make this a
-<strong>vanilla</strong> Spark baseline, in case the image being used ships Comet.</p>
-</section>
-<section id="comet">
-<h3>Comet<a class="headerlink" href="#comet" title="Link to this heading">#</a></h3>
-<p><a class="reference external" href="https://datafusion.apache.org/comet/">Apache DataFusion Comet</a> accelerates Spark by translating supported
-operators to DataFusion. Same queries, same sizing; the difference is the plugin,
-the Comet shuffle manager, and the jar on the classpath:</p>
-<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>spark-submit<span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--master<span class="w"> </span><master><span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--jars<span class="w"> </span><span class="nv">$COMET_JAR</span><span class="w"> </span>--driver-class-path<span class="w"> </span><span class="nv">$COMET_JAR</span><span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--conf<span class="w"> </span>spark.executor.extraClassPath<span class="o">=</span><span class="nv">$COMET_JAR</span><span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--conf<span class="w"> </span>spark.plugins<span class="o">=</span>org.apache.spark.CometPlugin<span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--conf<span class="w"> </span>spark.shuffle.manager<span class="o">=</span>org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager<span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--conf<span class="w"> </span>spark.comet.exec.replaceSortMergeJoin<span class="o">=</span><span class="nb">false</span><span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--conf<span class="w"> </span>spark.comet.exec.memoryPool<span class="o">=</span>fair_unified<span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--conf<span class="w"> </span>spark.comet.exec.memoryPool.fraction<span class="o">=</span><span class="m">0</span>.8<span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--conf<span class="w"> </span>spark.executor.instances<span class="o">=</span><span class="m">2</span><span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--conf<span class="w"> </span>spark.executor.cores<span class="o">=</span><span class="m">16</span><span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--conf<span class="w"> </span>spark.executor.memory<span class="o">=</span>32G<span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--conf<span class="w"> </span>spark.executor.memoryOverhead<span class="o">=</span>8G<span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--conf<span class="w"> </span>spark.memory.offHeap.enabled<span class="o">=</span><span class="nb">true</span><span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--conf<span class="w"> </span>spark.memory.offHeap.size<span class="o">=</span>16g<span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>tpcbench.py<span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--benchmark<span class="w"> </span>tpch<span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--data<span class="w"> </span>/mnt/bigdata/tpch/sf1000<span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--format<span class="w"> </span>parquet<span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--iterations<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--query<span class="w"> </span><span class="m">18</span>
-</pre></div>
-</div>
-<p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.replaceSortMergeJoin=false</span></code> keeps Comet on Spark’s <code class="docutils literal notranslate"><span class="pre">SortMergeJoin</span></code>
-instead of converting it to a shuffled hash join.</p>
-<p>This no longer pins the two engines to the same join strategy. Ballista under AQE
-chooses per join at runtime — broadcast, Partitioned hash, or sort-merge — and there
-is no supported way to force it to sort-merge throughout (see the note in the Ballista
-section above). So a Ballista-vs-Comet gap may partly reflect different join
-strategies, not only different execution. Read the comparison with that in mind.</p>
-</section>
-</section>
-<section id="why-compare-against-comet">
-<h2>Why compare against Comet<a class="headerlink" href="#why-compare-against-comet" title="Link to this heading">#</a></h2>
-<p>Comet is the most informative comparison available to Ballista, because <strong>Comet and
-Ballista execute DataFusion physical plans using the same DataFusion operators</strong>.
-The scan, filter, join, and aggregate implementations doing the work are largely
-shared code. When Ballista and Comet diverge on a query, the difference therefore
-points at what is <em>not</em> shared — how work is distributed, scheduled, shuffled, and
-bounded by memory — rather than at the speed of the operators themselves. That is
-precisely the surface Ballista is trying to improve, which makes the comparison
-diagnostic rather than merely competitive.</p>
-<p>That said, <strong>the two do not necessarily run the same plan shape</strong>, and the numbers
-should not be read as an operator-level A/B:</p>
-<ul class="simple">
-<li><p><strong>Different planners produce the plan.</strong> Comet accelerates a plan that <em>Spark’s</em>
-optimizer produced: Spark chooses the join order and join strategies, and
-<strong>Spark’s AQE</strong> coalesces shuffle partitions, converts joins, and splits skewed
-partitions at runtime. Ballista plans with DataFusion’s optimizer and, when
-enabled, its own experimental AQE. Spark’s AQE is a mature implementation and
-Ballista’s is not, so the same SQL can arrive at execution with materially
-different plans.</p></li>
-<li><p><strong>Comet falls back to Spark.</strong> Operators and expressions Comet does not support
-stay on the JVM, so a Comet run is generally a mix of DataFusion and Spark
-execution rather than an all-DataFusion one.</p></li>
-<li><p><strong>The distribution models differ.</strong> Comet executes within Spark’s task model and
-shuffle service; Ballista has its own scheduler, stage/task model, and shuffle.</p></li>
-</ul>
-<p>So a Comet-vs-Ballista gap is best read as a question — <em>what is Spark’s planner or
-execution model doing here that Ballista’s is not?</em> — rather than as a verdict on
-DataFusion. Vanilla Spark is included as the third data point, since it isolates
-how much of any Comet result comes from DataFusion acceleration versus from Spark’s
-planner.</p>
+<p><code class="docutils literal notranslate"><span class="pre">datafusion.optimizer.prefer_hash_join</span></code> is left at its default; under AQE the
+join strategy is selected at runtime by <code class="docutils literal notranslate"><span class="pre">DelayJoinSelectionRule</span></code> /
+<code class="docutils literal notranslate"><span class="pre">DynamicJoinSelectionExec</span></code> from runtime statistics and the broadcast /
+<code class="docutils literal notranslate"><span class="pre">ballista.optimizer.hash_join_max_build_partition_bytes</span></code> thresholds.</p>
</section>
-<section id="results">
-<h2>Results<a class="headerlink" href="#results" title="Link to this heading">#</a></h2>
-<p>TPC-H <strong>SF1000</strong>, reference cluster above, <strong>AQE on</strong>, 1 iteration,
-<code class="docutils literal notranslate"><span class="pre">target_partitions=64</span></code>, all Ballista join settings at their defaults. Ballista picks
-each join’s strategy at runtime; Spark and Comet are held to <code class="docutils literal notranslate"><span class="pre">SortMergeJoin</span></code>. Times in
-seconds; lower is better.</p>
-<p>Versions under test:</p>
+<section id="spark-configuration-highlights">
+<h2>Spark configuration (highlights)<a class="headerlink" href="#spark-configuration-highlights" title="Link to this heading">#</a></h2>
+<p>Vanilla Spark 3.4 — no Comet plugin, stock <code class="docutils literal notranslate"><span class="pre">SortShuffleManager</span></code>.</p>
<div class="pst-scrollable-table-container"><table class="table">
<thead>
-<tr class="row-odd"><th class="head"><p>Engine</p></th>
-<th class="head"><p>Version</p></th>
+<tr class="row-odd"><th class="head"><p>Key</p></th>
+<th class="head"><p>Value</p></th>
</tr>
</thead>
<tbody>
-<tr class="row-even"><td><p>Ballista</p></td>
-<td><p><code class="docutils literal notranslate"><span class="pre">#2124</span></code> @ <code class="docutils literal notranslate"><span class="pre">5290436d</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.executor.instances</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">32</span></code></p></td>
</tr>
-<tr class="row-odd"><td><p>Spark</p></td>
-<td><p>3.5.3 (vanilla, Comet disabled)</p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.executor.cores</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">16</span></code> (task parallelism per executor)</p></td>
+</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.kubernetes.executor.limit.cores</span></code> / <code class="docutils literal notranslate"><span class="pre">.request.cores</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">8</span></code> (physical vCPU)</p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.executor.memory</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">64G</span></code></p></td>
</tr>
-<tr class="row-even"><td><p>Comet</p></td>
-<td><p><code class="docutils literal notranslate"><span class="pre">main</span></code> @ <code class="docutils literal notranslate"><span class="pre">b0165552</span></code> (1.0.0-SNAPSHOT, Spark 3.5)</p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.executor.memoryOverhead</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">10G</span></code></p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.memory.fraction</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">0.6</span></code></p></td>
+</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.memory.storageFraction</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">0.2</span></code></p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.sql.shuffle.partitions</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">512</span></code></p></td>
+</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.sql.broadcastTimeout</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">900</span></code></p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.serializer</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">KryoSerializer</span></code></p></td>
+</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.io.compression.codec</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">zstd</span></code></p></td>
</tr>
</tbody>
</table>
</div>
-<p>Pin the <strong>exact commit</strong> the numbers came from, not “main”: <code class="docutils literal notranslate"><span class="pre">main</span></code> moves, and a row
-that mixes numbers from different commits silently misattributes a regression.</p>
-<p>The Ballista column comes from a <strong>single suite run of 21 queries</strong> on freshly
-started executors, using <code class="docutils literal notranslate"><span class="pre">--skip</span> <span class="pre">18</span></code> to leave out the one query that exhausts the
-memory pool and OOM-kills the executors (see below), which would otherwise wedge
-the rest of the suite. Comet and Spark each completed the full suite in one run.</p>
-<p>Keep the whole column to one continuous run wherever possible. Driving queries as
-individual single-query jobs measures something different: each job starts against
-a cold page cache, which on this cluster cost the two join-free queries (Q1, Q6)
-roughly 20 s and 20 s respectively when the two methods were compared directly. Use
-<code class="docutils literal notranslate"><span class="pre">--skip</span></code> rather than a loop of <code class="docutils literal notranslate"><span class="pre">--query</span></code> runs.</p>
-<p>Single iteration means <strong>run-to-run variance is not quantified here</strong>. Comparing the
-same commit across the two methods above showed spreads of up to ~20% on individual
-queries, so treat a per-query difference under about 20% as noise and read the total
-row instead.</p>
+<p>Spark AQE is left at its Spark 3.4 defaults. Shuffle spills to a <code class="docutils literal notranslate"><span class="pre">gp3</span></code>-backed
+per-executor volume (<code class="docutils literal notranslate"><span class="pre">spark.kubernetes.executor.volumes...spark-local-dir-1</span></code>).</p>
+<p>Note that <code class="docutils literal notranslate"><span class="pre">spark.executor.cores=16</span></code> is Spark’s <strong>task parallelism</strong> setting,
+not a CPU allocation — each executor pod is given only <strong>8 physical vCPU</strong>
+via <code class="docutils literal notranslate"><span class="pre">spark.kubernetes.executor.limit.cores</span></code> / <code class="docutils literal notranslate"><span class="pre">.request.cores</span></code>, so Spark
+schedules 16 concurrent tasks onto 8 physical cores (2× oversubscription).
+The matching Ballista executor runs <code class="docutils literal notranslate"><span class="pre">--concurrent-tasks=8</span></code> on the same
+8 physical vCPU (1:1).</p>
+</section>
+<section id="queries">
+<h2>Queries<a class="headerlink" href="#queries" title="Link to this heading">#</a></h2>
+<p>The SQLBench-H phrasing of the 22 TPC-H queries from
+<a class="reference external" href="https://github.com/apache/datafusion-benchmarks">apache/datafusion-benchmarks</a>.</p>
+</section>
+<section id="results">
+<h2>Results<a class="headerlink" href="#results" title="Link to this heading">#</a></h2>
+<p><strong>Times in seconds; lower is better.</strong> Ballista: single iteration. Spark:
+mean of 3 iterations (cold iteration dropped by the harness).</p>
<div class="pst-scrollable-table-container"><table class="table">
<thead>
<tr class="row-odd"><th class="head text-right"><p>Query</p></th>
-<th class="head text-right"><p>Spark</p></th>
-<th class="head text-right"><p>Comet</p></th>
-<th class="head text-right"><p>Ballista (AQE on)</p></th>
-<th class="head text-right"><p>Rows</p></th>
+<th class="head text-right"><p>Ballista (s)</p></th>
+<th class="head text-right"><p>Spark 3.4 (s)</p></th>
</tr>
</thead>
<tbody>
<tr class="row-even"><td class="text-right"><p>1</p></td>
-<td class="text-right"><p>427.1</p></td>
-<td class="text-right"><p>46.6</p></td>
-<td class="text-right"><p>54.2</p></td>
-<td class="text-right"><p>4</p></td>
+<td class="text-right"><p>21.16</p></td>
+<td class="text-right"><p>67.58</p></td>
</tr>
<tr class="row-odd"><td class="text-right"><p>2</p></td>
-<td class="text-right"><p>75.9</p></td>
-<td class="text-right"><p>39.2</p></td>
-<td class="text-right"><p>60.3</p></td>
-<td class="text-right"><p>100</p></td>
+<td class="text-right"><p>32.37</p></td>
+<td class="text-right"><p>29.80</p></td>
</tr>
<tr class="row-even"><td class="text-right"><p>3</p></td>
-<td class="text-right"><p>154.1</p></td>
-<td class="text-right"><p>195.4</p></td>
-<td class="text-right"><p>140.0</p></td>
-<td class="text-right"><p>10</p></td>
+<td class="text-right"><p>32.69</p></td>
+<td class="text-right"><p>25.13</p></td>
</tr>
<tr class="row-odd"><td class="text-right"><p>4</p></td>
-<td class="text-right"><p>82.0</p></td>
-<td class="text-right"><p>42.6</p></td>
-<td class="text-right"><p>46.3</p></td>
-<td class="text-right"><p>5</p></td>
+<td class="text-right"><p>27.75</p></td>
+<td class="text-right"><p>21.19</p></td>
</tr>
<tr class="row-even"><td class="text-right"><p>5</p></td>
-<td class="text-right"><p>336.4</p></td>
-<td class="text-right"><p>493.4</p></td>
-<td class="text-right"><p>327.2</p></td>
-<td class="text-right"><p>5</p></td>
+<td class="text-right"><p>49.37</p></td>
+<td class="text-right"><p>54.12</p></td>
</tr>
<tr class="row-odd"><td class="text-right"><p>6</p></td>
-<td class="text-right"><p>28.9</p></td>
-<td class="text-right"><p>14.6</p></td>
-<td class="text-right"><p>27.2</p></td>
-<td class="text-right"><p>1</p></td>
+<td class="text-right"><p>16.17</p></td>
+<td class="text-right"><p>1.23</p></td>
</tr>
<tr class="row-even"><td class="text-right"><p>7</p></td>
-<td class="text-right"><p>180.6</p></td>
-<td class="text-right"><p>149.5</p></td>
-<td class="text-right"><p>364.9</p></td>
-<td class="text-right"><p>4</p></td>
+<td class="text-right"><p>48.24</p></td>
+<td class="text-right"><p>19.57</p></td>
</tr>
<tr class="row-odd"><td class="text-right"><p>8</p></td>
-<td class="text-right"><p>391.8</p></td>
-<td class="text-right"><p>642.1</p></td>
-<td class="text-right"><p>481.1</p></td>
-<td class="text-right"><p>2</p></td>
+<td class="text-right"><p>189.30</p></td>
+<td class="text-right"><p>48.60</p></td>
</tr>
<tr class="row-even"><td class="text-right"><p>9</p></td>
-<td class="text-right"><p>509.8</p></td>
-<td class="text-right"><p>843.5</p></td>
-<td class="text-right"><p>606.4</p></td>
-<td class="text-right"><p>175</p></td>
+<td class="text-right"><p>154.88</p></td>
+<td class="text-right"><p>69.38</p></td>
</tr>
<tr class="row-odd"><td class="text-right"><p>10</p></td>
-<td class="text-right"><p>151.1</p></td>
-<td class="text-right"><p>104.8</p></td>
-<td class="text-right"><p>148.4</p></td>
-<td class="text-right"><p>20</p></td>
+<td class="text-right"><p>67.72</p></td>
+<td class="text-right"><p>35.92</p></td>
</tr>
<tr class="row-even"><td class="text-right"><p>11</p></td>
-<td class="text-right"><p>44.7</p></td>
-<td class="text-right"><p>44.1</p></td>
-<td class="text-right"><p>62.0</p></td>
-<td class="text-right"><p>0 [1]</p></td>
+<td class="text-right"><p>26.72</p></td>
+<td class="text-right"><p>30.88</p></td>
</tr>
<tr class="row-odd"><td class="text-right"><p>12</p></td>
-<td class="text-right"><p>74.1</p></td>
-<td class="text-right"><p>49.9</p></td>
-<td class="text-right"><p>58.3</p></td>
-<td class="text-right"><p>2</p></td>
+<td class="text-right"><p>24.46</p></td>
+<td class="text-right"><p>10.78</p></td>
</tr>
<tr class="row-even"><td class="text-right"><p>13</p></td>
-<td class="text-right"><p>98.7</p></td>
-<td class="text-right"><p>58.3</p></td>
-<td class="text-right"><p>100.4</p></td>
-<td class="text-right"><p>30</p></td>
+<td class="text-right"><p>18.36</p></td>
+<td class="text-right"><p>20.45</p></td>
</tr>
<tr class="row-odd"><td class="text-right"><p>14</p></td>
-<td class="text-right"><p>43.0</p></td>
-<td class="text-right"><p>27.4</p></td>
-<td class="text-right"><p>71.1</p></td>
-<td class="text-right"><p>1</p></td>
+<td class="text-right"><p>19.93</p></td>
+<td class="text-right"><p>7.00</p></td>
</tr>
<tr class="row-even"><td class="text-right"><p>15</p></td>
-<td class="text-right"><p>121.5</p></td>
-<td class="text-right"><p>65.6</p></td>
-<td class="text-right"><p>53.8</p></td>
-<td class="text-right"><p>1 [2]</p></td>
+<td class="text-right"><p>27.56</p></td>
+<td class="text-right"><p>23.75</p></td>
</tr>
<tr class="row-odd"><td class="text-right"><p>16</p></td>
-<td class="text-right"><p>24.5</p></td>
-<td class="text-right"><p>17.5</p></td>
-<td class="text-right"><p>34.3</p></td>
-<td class="text-right"><p>27840</p></td>
+<td class="text-right"><p>18.03</p></td>
+<td class="text-right"><p>23.41</p></td>
</tr>
<tr class="row-even"><td class="text-right"><p>17</p></td>
-<td class="text-right"><p>406.7</p></td>
-<td class="text-right"><p>285.8</p></td>
-<td class="text-right"><p>417.8</p></td>
-<td class="text-right"><p>1</p></td>
+<td class="text-right"><p>60.46</p></td>
+<td class="text-right"><p>82.30</p></td>
</tr>
<tr class="row-odd"><td class="text-right"><p>18</p></td>
-<td class="text-right"><p>428.8</p></td>
-<td class="text-right"><p>370.5</p></td>
-<td class="text-right"><p>TBD</p></td>
-<td class="text-right"><p>100</p></td>
+<td class="text-right"><p>282.79</p></td>
+<td class="text-right"><p>129.40</p></td>
</tr>
<tr class="row-even"><td class="text-right"><p>19</p></td>
-<td class="text-right"><p>58.9</p></td>
-<td class="text-right"><p>35.5</p></td>
-<td class="text-right"><p>68.5</p></td>
-<td class="text-right"><p>1</p></td>
+<td class="text-right"><p>23.28</p></td>
+<td class="text-right"><p>11.26</p></td>
</tr>
<tr class="row-odd"><td class="text-right"><p>20</p></td>
-<td class="text-right"><p>105.9</p></td>
-<td class="text-right"><p>67.6</p></td>
-<td class="text-right"><p>93.7</p></td>
-<td class="text-right"><p>110759</p></td>
+<td class="text-right"><p>88.93</p></td>
+<td class="text-right"><p>19.22</p></td>
</tr>
<tr class="row-even"><td class="text-right"><p>21</p></td>
-<td class="text-right"><p>562.2</p></td>
-<td class="text-right"><p>460.1</p></td>
-<td class="text-right"><p>698.0</p></td>
-<td class="text-right"><p>100</p></td>
+<td class="text-right"><p>110.87</p></td>
+<td class="text-right"><p>101.53</p></td>
</tr>
<tr class="row-odd"><td class="text-right"><p>22</p></td>
-<td class="text-right"><p>36.8</p></td>
-<td class="text-right"><p>21.5</p></td>
-<td class="text-right"><p>22.9</p></td>
-<td class="text-right"><p>7</p></td>
+<td class="text-right"><p>12.17</p></td>
+<td class="text-right"><p>12.71</p></td>
</tr>
-<tr class="row-even"><td class="text-right"><p><strong>Total (excl. Q18)</strong></p></td>
-<td class="text-right"><p><strong>3914.7</strong></p></td>
-<td class="text-right"><p><strong>3705.0</strong></p></td>
-<td class="text-right"><p><strong>3936.8</strong></p></td>
-<td class="text-right"><p></p></td>
+<tr class="row-even"><td class="text-right"><p><strong>Total (Q1–Q22)</strong></p></td>
+<td class="text-right"><p><strong>1353.21</strong></p></td>
+<td class="text-right"><p><strong>845.21</strong></p></td>
</tr>
</tbody>
</table>
</div>
-<p>The <strong>Total</strong> row sums the 21 queries <strong>excluding Q18</strong>, because Q18 has no Ballista
-figure at this commit (below), so a 22-query total would not be comparable across
-engines. Spark’s and Comet’s Q18 times are in its row.</p>
-<p>Row counts agree across all three engines on every query for which Ballista produced
-a result.</p>
-<p>The <strong>AQE-off</strong> Ballista column has been removed pending a re-run at a matched core
-count; the numbers above for AQE off were taken at a different executor size and are
-not comparable to a fresh AQE-off run, so they were dropped rather than left stale.</p>
-<p>[1] Q11 returns 0 rows for every engine at this scale factor: the query’s threshold
-constant is tuned for SF1.</p>
-<p>[2] Q15 is a multi-statement query (<code class="docutils literal notranslate"><span class="pre">CREATE</span> <span class="pre">VIEW</span></code> / <code class="docutils literal notranslate"><span class="pre">SELECT</span></code> / <code class="docutils literal notranslate"><span class="pre">DROP</span> <span class="pre">VIEW</span></code>); all three
-engines report 1 row here. (A previous result set saw Spark report 0 for Q15 depending
-on which statement the harness took as the result; it does not recur in this run.)</p>
-<p><strong>Q18 is not measured at this commit.</strong> Q18’s hash-join build side is <code class="docutils literal notranslate"><span class="pre">Partitioned</span></code>
-and does not spill; with 16 task slots sharing the 48 GB pool (3 GB per slot), the
-per-task build side exceeds the container limit and the executor is OOM-killed
-(<a class="reference external" href="https://github.com/apache/datafusion-ballista/issues/2025">#2025</a>). At the previous
-8-slot sizing (6 GB per slot) Q18 completed, so this is a direct consequence of the
-denser packing.</p>
-<p>It was excluded from the suite run above with <code class="docutils literal notranslate"><span class="pre">--skip</span> <span class="pre">18</span></code> so that one query could not
-wedge the other 21, and it has not since been run to completion at this commit — hence
-<code class="docutils literal notranslate"><span class="pre">TBD</span></code> rather than <code class="docutils literal notranslate"><span class="pre">OOM</span></code>. An earlier attempt on this branch did OOM-kill an executor,
-but under a different configuration (<code class="docutils literal notranslate"><span class="pre">prefer_hash_join=false</span></code>, which sends the plan
-down a conversion path that discards the join’s projection and so materialises more
-columns than the configuration documented above). That result does not transfer, so it
-is not recorded here.</p>
-<p>The <code class="docutils literal notranslate"><span class="pre">Rows</span></code> column is the row count the query returned, recorded so a time is never
-read without the answer it produced.</p>
-<p>This table records <strong>one current result set</strong>. When results are refreshed, the
-table and the pinned versions above are replaced together — a row must never mix
-numbers from different commits, because a stale row silently misattributes a
-regression.</p>
-<p><code class="docutils literal notranslate"><span class="pre">TBD</span></code> means not yet measured on this cluster at this commit; a query that ran but
-did not produce an answer is recorded as <code class="docutils literal notranslate"><span class="pre">FAIL</span></code>, or <code class="docutils literal notranslate"><span class="pre">OOM</span></code> where the failure is a
-known memory exhaustion.</p>
-<section id="recording-a-result">
-<h3>Recording a result<a class="headerlink" href="#recording-a-result" title="Link to this heading">#</a></h3>
-<ul class="simple">
-<li><p>Pin the <strong>exact commit</strong> the numbers came from, not a branch name.</p></li>
-<li><p>Report the <strong>AQE-on</strong> number (<code class="docutils literal notranslate"><span class="pre">ballista.planner.adaptive.enabled=true</span></code>) — the
-configuration this page measures — from the same commit and cluster.</p></li>
-<li><p>Prefer the figure from a <strong>full suite run</strong>, not a standalone single-query run: a
-long-lived executor deep into a suite is not in the same state as a freshly started
-one, and a single-query job pays a cold-cache penalty that is large enough to swamp
-the effect being measured. When a query cannot complete in-suite (e.g. Q18’s OOM
-currently wedges the run), exclude it with <code class="docutils literal notranslate"><span class="pre">--skip</span> <span class="pre"><n></span></code> so the rest of the column
-still comes from one continuous run, and measure that query separately.</p></li>
-<li><p>Note the <strong>row count</strong> each query returned. A fast wrong answer is not a result,
-and distributed execution has produced silently wrong row counts before.</p></li>
-<li><p>Flag any stage whose runtime is dominated by a few partitions — that is the
-imbalance this page exists to surface.</p></li>
-</ul>
+<p>Row counts agree across engines for every query. Ballista completed the
+full 22-query suite in a single continuous run.</p>
</section>
+<section id="reproducing">
+<h2>Reproducing<a class="headerlink" href="#reproducing" title="Link to this heading">#</a></h2>
+<section id="ballista">
+<h3>Ballista<a class="headerlink" href="#ballista" title="Link to this heading">#</a></h3>
+<p>Bring up the cluster (one scheduler, N executors), then run the suite from a
+client. Executor sizing on each node:</p>
+<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>ballista-executor<span class="w"> </span><span class="se">\</span>
+<span class="w"> </span>--bind-host<span class="w"> </span><span class="m">0</span>.0.0.0<span class="w"> </span>--bind-port<span class="w"> </span><span class="m">50051</span><span class="w"> </span><span class="se">\</span>
+<span class="w"> </span>--scheduler-host<span class="w"> </span><scheduler><span class="w"> </span>--scheduler-port<span class="w"> </span><span class="m">50050</span><span class="w"> </span><span class="se">\</span>
+<span class="w"> </span>--concurrent-tasks<span class="w"> </span><span class="m">8</span><span class="w"> </span><span class="se">\</span>
+<span class="w"> </span>--memory-pool-size<span class="w"> </span><span class="m">48103633715</span>
+</pre></div>
+</div>
+<p>Run all 22 queries:</p>
+<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>tpch<span class="w"> </span>benchmark<span class="w"> </span>ballista<span class="w"> </span><span class="se">\</span>
+<span class="w"> </span>--host<span class="w"> </span><scheduler><span class="w"> </span>--port<span class="w"> </span><span class="m">50050</span><span class="w"> </span><span class="se">\</span>
+<span class="w"> </span>--path<span class="w"> </span>s3://<bucket>/tpch/sf1000<span class="w"> </span>--format<span class="w"> </span>parquet<span class="w"> </span><span class="se">\</span>
+<span class="w"> </span>--partitions<span class="w"> </span><span class="m">256</span><span class="w"> </span>--iterations<span class="w"> </span><span class="m">2</span><span class="w"> </span><span class="se">\</span>
+<span class="w"> </span>-c<span class="w"> </span>ballista.planner.adaptive.enabled<span class="o">=</span><span class="nb">true</span><span class="w"> </span><span class="se">\</span>
+<span class="w"> </span>-c<span class="w"> </span>datafusion.execution.collect_statistics<span class="o">=</span><span class="nb">true</span><span class="w"> </span><span class="se">\</span>
+<span class="w"> </span>-c<span class="w"> </span>ballista.shuffle.sort_based.memory_limit_per_task_bytes<span class="o">=</span><span class="m">0</span>
+</pre></div>
+</div>
</section>
-<section id="known-issues-found-by-benchmarking">
-<h2>Known issues found by benchmarking<a class="headerlink" href="#known-issues-found-by-benchmarking" title="Link to this heading">#</a></h2>
-<p>Benchmarking at SF1000 is how most of the following were found. They are worth
-knowing about before interpreting a number:</p>
-<div class="pst-scrollable-table-container"><table class="table">
-<thead>
-<tr class="row-odd"><th class="head"><p>Issue</p></th>
-<th class="head"><p>Summary</p></th>
-</tr>
-</thead>
-<tbody>
-<tr class="row-even"><td><p><a class="reference external" href="https://github.com/apache/datafusion-ballista/issues/1359">#1359</a></p></td>
-<td><p>Umbrella issue for adaptive (AQE) query execution.</p></td>
-</tr>
-<tr class="row-odd"><td><p><a class="reference external" href="https://github.com/apache/datafusion-ballista/issues/2025">#2025</a></p></td>
-<td><p>Q18’s hash-join build side exhausts the memory pool at SF1000. DataFusion’s hash-join build side does not spill, so a per-partition build side larger than one task slot’s pool fails the task outright.</p></td>
-</tr>
-<tr class="row-even"><td><p><a class="reference external" href="https://github.com/apache/datafusion-ballista/issues/2063">#2063</a></p></td>
-<td><p>AQE can hang when a re-plan cancels an in-flight stage; the job never reports completion.</p></td>
-</tr>
-</tbody>
-</table>
+<section id="spark">
+<h3>Spark<a class="headerlink" href="#spark" title="Link to this heading">#</a></h3>
+<p>Runs the same queries via <code class="docutils literal notranslate"><span class="pre">tpcbench.py</span></code> from
+<a class="reference external" href="https://github.com/apache/datafusion-benchmarks">apache/datafusion-benchmarks</a>,
+with the highlights above and stock Spark 3.4 defaults for everything else:</p>
+<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>spark-submit<span class="w"> </span><span class="se">\</span>
+<span class="w"> </span>--master<span class="w"> </span><master><span class="w"> </span><span class="se">\</span>
+<span class="w"> </span>--conf<span class="w"> </span>spark.executor.instances<span class="o">=</span><span class="m">32</span><span class="w"> </span><span class="se">\</span>
+<span class="w"> </span>--conf<span class="w"> </span>spark.executor.cores<span class="o">=</span><span class="m">16</span><span class="w"> </span><span class="se">\</span>
+<span class="w"> </span>--conf<span class="w"> </span>spark.executor.memory<span class="o">=</span>64G<span class="w"> </span><span class="se">\</span>
+<span class="w"> </span>--conf<span class="w"> </span>spark.executor.memoryOverhead<span class="o">=</span>10G<span class="w"> </span><span class="se">\</span>
+<span class="w"> </span>--conf<span class="w"> </span>spark.sql.shuffle.partitions<span class="o">=</span><span class="m">512</span><span class="w"> </span><span class="se">\</span>
+<span class="w"> </span>tpcbench.py<span class="w"> </span><span class="se">\</span>
+<span class="w"> </span>--benchmark<span class="w"> </span>tpch<span class="w"> </span><span class="se">\</span>
+<span class="w"> </span>--data<span class="w"> </span>s3a://<bucket>/tpch/sf1000<span class="w"> </span><span class="se">\</span>
+<span class="w"> </span>--format<span class="w"> </span>parquet<span class="w"> </span><span class="se">\</span>
+<span class="w"> </span>--iterations<span class="w"> </span><span class="m">3</span>
+</pre></div>
</div>
</section>
+</section>
+<section id="recording-a-new-result-set">
+<h2>Recording a new result set<a class="headerlink" href="#recording-a-new-result-set" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Pin the <strong>exact commit</strong> the numbers came from, not a branch name.</p></li>
+<li><p>Replace the version, environment, config, and results tables together — a
+row that mixes numbers from different commits silently misattributes a
+regression.</p></li>
+<li><p>Prefer a single continuous suite run: a long-lived executor deep into a
+suite is not in the same state as a freshly started one.</p></li>
+<li><p>Report <code class="docutils literal notranslate"><span class="pre">FAIL</span></code> for a query that ran but did not produce an answer, and
+<code class="docutils literal notranslate"><span class="pre">OOM</span></code> when the failure is a known memory exhaustion. See the tracker for
+open issues found by benchmarking: <a class="reference external" href="https://github.com/apache/datafusion-ballista/issues/1359">#1359</a>,
+<a class="reference external" href="https://github.com/apache/datafusion-ballista/issues/2025">#2025</a>, <a class="reference external" href="https://github.com/apache/datafusion-ballista/issues/2063">#2063</a>.</p></li>
+</ul>
+</section>
</section>
@@ -952,22 +747,18 @@ knowing about before interpreting a number:</p>
</div>
<nav id="pst-page-toc-nav" class="page-toc" aria-labelledby="pst-page-navigation-heading-2">
<ul class="pst-show_toc_level nav section-nav flex-column">
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#what-we-measure-and-why">What we measure, and why</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#versions-under-test">Versions under test</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#environment">Environment</a></li>
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#reference-cluster">Reference cluster</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#ballista-configuration">Ballista configuration</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#spark-configuration-highlights">Spark configuration (highlights)</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#queries">Queries</a></li>
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#running-the-benchmark">Running the benchmark</a><ul class="nav section-nav flex-column">
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#results">Results</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#reproducing">Reproducing</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#ballista">Ballista</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#spark">Spark</a></li>
-<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#comet">Comet</a></li>
-</ul>
-</li>
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#why-compare-against-comet">Why compare against Comet</a></li>
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#results">Results</a><ul class="nav section-nav flex-column">
-<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#recording-a-result">Recording a result</a></li>
</ul>
</li>
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#known-issues-found-by-benchmarking">Known issues found by benchmarking</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#recording-a-new-result-set">Recording a new result set</a></li>
</ul>
</nav></div>
diff --git a/searchindex.js b/searchindex.js
index 9c8447e7b..188213351 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles":{"AQE join behavior (opt-in)":[[27,"aqe-join-behavior-opt-in"]],"Adaptive Query Execution (AQE)":[[21,"adaptive-query-execution-aqe"]],"Adaptive Query Execution (Experimental)":[[50,"adaptive-query-execution-experimental"]],"Adding a persona":[[25,"adding-a-persona"]],"Aggregate Functions":[[49,"aggregate-functions"]],"Apache DataFusion Ballista":[[26,null]],"Apache DataFusion Ballista 0.10.0 Changelog":[[0,null]],"Apache DataFusion Ballista 0.11.0 Changelog" [...]
\ No newline at end of file
+Search.setIndex({"alltitles":{"AQE join behavior (opt-in)":[[27,"aqe-join-behavior-opt-in"]],"Adaptive Query Execution (AQE)":[[21,"adaptive-query-execution-aqe"]],"Adaptive Query Execution (Experimental)":[[50,"adaptive-query-execution-experimental"]],"Adding a persona":[[25,"adding-a-persona"]],"Aggregate Functions":[[49,"aggregate-functions"]],"Apache DataFusion Ballista":[[26,null]],"Apache DataFusion Ballista 0.10.0 Changelog":[[0,null]],"Apache DataFusion Ballista 0.11.0 Changelog" [...]
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org