You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by ag...@apache.org on 2026/10/12 16:10:38 UTC

(datafusion-ballista) branch main updated: docs: split benchmarking page into results and per-engine pages (#2587)

This is an automated email from the ASF dual-hosted git repository.

andygrove pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-ballista.git


The following commit(s) were added to refs/heads/main by this push:
     new ea1126c06 docs: split benchmarking page into results and per-engine pages (#2587)
ea1126c06 is described below

commit ea1126c0649b2bc08e1e233b1972b64318821c16
Author: Andy Grove <ag...@apache.org>
AuthorDate: Mon Oct 12 10:10:33 2026 -0600

    docs: split benchmarking page into results and per-engine pages (#2587)
---
 docs/source/contributors-guide/benchmarking.md     | 337 ---------------------
 .../contributors-guide/benchmarking/ballista.md    |  79 +++++
 .../contributors-guide/benchmarking/comet.md       |  56 ++++
 .../contributors-guide/benchmarking/environment.md |  95 ++++++
 .../contributors-guide/benchmarking/index.md       | 107 +++++++
 .../contributors-guide/benchmarking/spark.md       |  75 +++++
 .../contributors-guide/benchmarking/trino.md       |  64 ++++
 docs/source/index.rst                              |   2 +-
 8 files changed, 477 insertions(+), 338 deletions(-)

diff --git a/docs/source/contributors-guide/benchmarking.md b/docs/source/contributors-guide/benchmarking.md
deleted file mode 100644
index 13f540c10..000000000
--- a/docs/source/contributors-guide/benchmarking.md
+++ /dev/null
@@ -1,337 +0,0 @@
-<!---
-  Licensed to the Apache Software Foundation (ASF) under one
-  or more contributor license agreements.  See the NOTICE file
-  distributed with this work for additional information
-  regarding copyright ownership.  The ASF licenses this file
-  to you under the Apache License, Version 2.0 (the
-  "License"); you may not use this file except in compliance
-  with the License.  You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-  Unless required by applicable law or agreed to in writing,
-  software distributed under the License is distributed on an
-  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-  KIND, either express or implied.  See the License for the
-  specific language governing permissions and limitations
-  under the License.
--->
-
-# Benchmarking
-
-Current TPC-H **SF1000** results for Ballista, compared against vanilla
-**Spark 4.1.3** and **Spark 4.1.3 with Apache DataFusion Comet 1.1.0-rc2**,
-running on the same cluster shape. A **Trino 483** column is included for
-reference; it is not an apples-to-apples comparison (see
-[Trino configuration](#trino-configuration)).
-
-## Versions under test
-
-| Engine   | Version                                                                                                                                                                                                                |
-| -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| Ballista | [`c76278cd`](https://github.com/apache/datafusion-ballista/commit/c76278cd48583504821ac4d71ea74b44e54ed4ea) (`main`, 2026-10-07), Cargo pkg `55.0.0`, DataFusion `55.1.0`                                              |
-| Spark    | `4.1.3` (`apache/spark:4.1.3` image, vanilla, no acceleration plugin)                                                                                                                                                  |
-| Comet    | [Apache DataFusion Comet](https://github.com/apache/datafusion-comet) `1.1.0-rc2` ([`992c806a`](https://github.com/apache/datafusion-comet/commit/992c806a7e38c2e88bd018aa5774164b0850e1fa)) on the same Spark `4.1.3` |
-| Trino    | `483` (`trinodb/trino:483` image, Hive connector)                                                                                                                                                                      |
-
-## Environment
-
-- **Cluster:** Kubernetes on AWS (`us-west-2`); one driver/scheduler pod and
-  32 executor pods for each engine, launched on the same node pool.
-- **K8s worker nodes:** a mix of `r6i.24xlarge` (96 vCPU, 768 GiB memory) and
-  `r6i.16xlarge` (64 vCPU, 512 GiB memory), EBS-only — no local
-  instance-store. Pods are not pinned to an instance type.
-- **Executor pod (Ballista):** x86_64, 8 vCPU, 64 GiB memory, plus a
-  dedicated 1000 GiB `gp3` EBS PVC mounted at `/data` for the executor's
-  shuffle work-dir (see [Executor storage](#executor-storage)).
-- **Executor pod (Spark):** x86_64, 8 vCPU, 64 GiB + 10 GiB overhead, plus a
-  dedicated `gp3` PVC via `spark-local-dir-1`.
-- **Executor pod (Spark + Comet):** the same as Spark, plus 32 GiB of
-  off-heap memory for Comet's native execution (about 106 GiB per pod in
-  total, compared with 64 GiB for a Ballista executor).
-- **Worker pod (Trino):** x86_64, 8 vCPU, 64 GiB memory (the image's
-  default JVM heap of 80 %, about 51 GiB), no spill volume. A separate
-  coordinator pod plans and schedules but does not scan data.
-- **Driver (Spark and Spark + Comet):** runs the queries through a Spark
-  benchmark harness that is not in this repository yet. See
-  [Spark and Spark + Comet](#spark-and-spark--comet) under Reproducing.
-- **Client pod (Ballista):** the `tpch` Rust benchmark runner from
-  [`benchmarks/`](https://github.com/apache/datafusion-ballista/tree/main/benchmarks)
-  in this repo (`cargo run --release --bin tpch -- benchmark ballista ...`),
-  which submits SQL through a Ballista `SessionContext` and collects
-  results locally.
-- **Data:** TPC-H SF1000 Parquet on S3 (`us-west-2`), ZSTD compression,
-  ~512 MiB row groups, one directory per table. `lineitem`, `orders`,
-  `customer` and `part` are Hive-style partitioned (e.g.
-  `lineitem/l_shipdate=YYYY-MM-DD/`). The `tpch` runner declares the
-  partition columns (see
-  [Hive-partitioned data](https://github.com/apache/datafusion-ballista/tree/main/benchmarks#hive-partitioned-data)),
-  so Ballista skips the partitions a filter excludes, as Spark, Comet and
-  Trino do. Results from before
-  [#2553](https://github.com/apache/datafusion-ballista/pull/2553) were
-  collected without partition columns and pruned these scans only through
-  Parquet statistics.
-
-## Executor storage
-
-Each Ballista executor pod is attached to a **fresh 1000 GiB `gp3` EBS
-volume** (generic-ephemeral PVC, `storageClassName: gp3`), mounted at
-`/data`, and the executor is launched with `--work-dir /data`. All shuffle
-temp files land on this dedicated volume.
-
-Without it, `--work-dir` defaults to a random directory under `/tmp` on the
-container overlay filesystem, i.e. onto the node's root EBS volume, which
-is shared with container images, `kubelet`, and every other pod on the
-same node. On `r6i.24xlarge` that shared bandwidth becomes the binding
-constraint under sustained shuffle-write pressure — `EXPLAIN ANALYZE`
-observed Q8's `SortShuffleWriter.write_time` inflate ~5× on the second
-run of a suite compared to a fresh cluster, despite the same 313 GB of
-shuffle output and zero spilling, because Q1–Q7's dirty pages force
-synchronous flushes to EBS at cap. Attaching a dedicated PVC removes that
-contention and brings in-suite per-query times in line with the standalone
-number.
-
-Spark on the same cluster has always used this pattern via
-`spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1`.
-
-## Ballista configuration
-
-| Flag / config key                                           | Value                                                                   |
-| ----------------------------------------------------------- | ----------------------------------------------------------------------- |
-| `--vcores`                                                  | `8`                                                                     |
-| `--memory-pool-size` (bytes; ≈70 % of the 64 GiB container) | `48103633715`                                                           |
-| `--work-dir`                                                | `/data` (dedicated gp3 PVC — see [Executor storage](#executor-storage)) |
-| `--grpc-server-max-decoding-message-size`                   | `134217728`                                                             |
-| `--grpc-server-max-encoding-message-size`                   | `134217728`                                                             |
-| `datafusion.execution.target_partitions`                    | `256`                                                                   |
-| `datafusion.execution.collect_statistics`                   | `true`                                                                  |
-| `ballista.planner.adaptive.enabled`                         | `true` (AQE; default)                                                   |
-| `ballista.shuffle.sort_based.memory_limit_per_task_bytes`   | `268435456` (256 MiB; default)                                          |
-
-`datafusion.optimizer.prefer_hash_join` is left at its default; under AQE the
-join strategy is selected at runtime by `DelayJoinSelectionRule` /
-`DynamicJoinSelectionExec` from runtime statistics and the broadcast /
-`ballista.optimizer.hash_join_max_build_partition_bytes` thresholds.
-
-The gRPC message-size ceiling is raised from the 16 MiB default to 128 MiB;
-some SF1000 physical plans (Q11, Q21, Q22) encode above 16 MiB and hit
-`OutOfRange` errors otherwise.
-
-## Spark configuration (highlights)
-
-Both Spark runs use these settings. The vanilla Spark run has no Comet
-plugin and uses the stock `SortShuffleManager`.
-
-| Key                                                        | Value                                |
-| ---------------------------------------------------------- | ------------------------------------ |
-| `spark.executor.instances`                                 | `32`                                 |
-| `spark.executor.cores`                                     | `16` (task parallelism per executor) |
-| `spark.kubernetes.executor.limit.cores` / `.request.cores` | `8` (physical vCPU)                  |
-| `spark.executor.memory`                                    | `64G`                                |
-| `spark.executor.memoryOverhead`                            | `10G`                                |
-| `spark.memory.fraction`                                    | `0.6`                                |
-| `spark.memory.storageFraction`                             | `0.2`                                |
-| `spark.sql.shuffle.partitions`                             | `512`                                |
-| `spark.sql.broadcastTimeout`                               | `900`                                |
-| `spark.serializer`                                         | `KryoSerializer`                     |
-| `spark.io.compression.codec`                               | `zstd`                               |
-
-Spark AQE is left at its Spark 4.1 defaults. Shuffle spills to a `gp3`-backed
-per-executor volume (`spark.kubernetes.executor.volumes...spark-local-dir-1`).
-
-Note that `spark.executor.cores=16` is Spark's **task parallelism** setting,
-not a CPU allocation — each executor pod is given only **8 physical vCPU**
-via `spark.kubernetes.executor.limit.cores` / `.request.cores`, so Spark
-schedules 16 concurrent tasks onto 8 physical cores (2× oversubscription).
-The matching Ballista executor runs `--vcores=8` on the same
-8 physical vCPU (1:1).
-
-## Comet configuration
-
-The Spark + Comet run adds these settings to the Spark configuration above:
-
-| Key                                    | Value                                                              |
-| -------------------------------------- | ------------------------------------------------------------------ |
-| `spark.plugins`                        | `org.apache.spark.CometPlugin`                                     |
-| `spark.comet.enabled`                  | `true`                                                             |
-| `spark.shuffle.manager`                | `org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager` |
-| `spark.memory.offHeap.enabled`         | `true`                                                             |
-| `spark.memory.offHeap.size`            | `32G`                                                              |
-| `spark.comet.exec.memoryPool.fraction` | `0.8`                                                              |
-
-The run also enabled Comet's explain and fallback logging. Everything else is
-left at Comet's defaults.
-
-## Trino configuration
-
-Trino is a useful reference point, but the setup differs from the other
-engines in ways that matter:
-
-- **Warm timings.** Each query runs once untimed, then the reported number
-  is the mean of 3 timed iterations. The other engines include the cold
-  first iteration in their mean.
-- **In-memory exchanges.** Trino streams data between stages over the
-  network and never writes shuffle files, while Ballista, Spark, and Comet
-  materialize every shuffle on disk. Spilling was disabled, so a query
-  that does not fit in memory would fail rather than slow down (none did).
-- **Partition pruning.** Tables are external Hive tables over the same
-  Parquet files with the partition columns declared, so Trino prunes
-  partitions like Spark and Comet do.
-- **No table statistics.** `ANALYZE` was not run, so the cost-based
-  optimizer had no row counts or column statistics.
-- **Queries.** The same SQL text as the Spark harness.
-
-| Key                                                              | Value                                |
-| ---------------------------------------------------------------- | ------------------------------------ |
-| Workers                                                          | `32` (`include-coordinator=false`)   |
-| `query.max-memory-per-node`                                      | `38GB`                               |
-| `memory.heap-headroom-per-node`                                  | `6GB`                                |
-| `spill-enabled`                                                  | `false`                              |
-| `hive.metastore`                                                 | `file` (stored in S3)                |
-| `hive.metastore-cache-ttl`, `hive.file-status-cache-expire-time` | `24h` (metadata cached across a run) |
-| `hive.dynamic-filtering.wait-timeout`                            | `1s`                                 |
-
-Everything else is left at Trino's defaults, with no session properties
-set. No tuning was done beyond this, and the configuration has not been
-reviewed by Trino experts.
-
-## Queries
-
-The SQLBench-H phrasing of the 22 TPC-H queries from
-[apache/datafusion-benchmarks](https://github.com/apache/datafusion-benchmarks).
-
-The Spark harness uses the same queries with the same substitution
-parameters. The only differences are in wording: it writes some dates as
-arithmetic (for example `date '1998-12-01' - interval '90' day` instead of
-`date '1998-09-02'`), and it phrases Q15 as a CTE instead of a view.
-
-## Results
-
-**Times in seconds; lower is better.** Ballista, Spark, and Spark + Comet:
-mean of 2 iterations, including the cold first iteration. Trino: mean of 3
-iterations after an untimed warm-up run (see
-[Trino configuration](#trino-configuration)).
-
-|     Query | Ballista (s) | Spark 4.1.3 (s) | Spark 4.1.3 + Comet 1.1.0-rc2 (s) | Trino 483 (s) |
-| --------: | -----------: | --------------: | --------------------------------: | ------------: |
-|         1 |        14.07 |           71.45 |                             10.66 |          5.32 |
-|         2 |        25.81 |           37.98 |                             21.66 |         15.34 |
-|         3 |        17.22 |           30.61 |                             16.05 |         13.47 |
-|         4 |        15.31 |           23.86 |                              8.11 |          8.01 |
-|         5 |        46.88 |           57.39 |                             38.32 |         25.98 |
-|         6 |         1.59 |            1.73 |                              0.99 |          0.84 |
-|         7 |        37.26 |           25.34 |                             16.60 |         16.19 |
-|         8 |        20.13 |           58.17 |                             46.04 |         30.93 |
-|         9 |        35.94 |           74.90 |                             55.78 |         41.71 |
-|        10 |        43.36 |           35.31 |                             21.31 |         20.41 |
-|        11 |        15.01 |           31.66 |                             14.79 |          5.73 |
-|        12 |        10.49 |           13.24 |                              5.33 |          4.40 |
-|        13 |        11.05 |           22.73 |                             11.31 |         10.56 |
-|        14 |         5.54 |            7.25 |                              2.50 |          1.66 |
-|        15 |        10.58 |           22.45 |                             11.85 |          8.30 |
-|        16 |        12.78 |           24.34 |                              8.94 |          6.98 |
-|        17 |        20.74 |           81.36 |                             25.14 |         24.76 |
-|        18 |        45.63 |          134.11 |                             46.42 |         66.00 |
-|        19 |        14.59 |           13.27 |                              9.07 |          5.93 |
-|        20 |        14.46 |           16.08 |                              6.57 |          8.57 |
-|        21 |        75.74 |           98.04 |                             68.43 |         83.22 |
-|        22 |         8.25 |           16.90 |                              9.22 |          7.29 |
-| **Total** |   **502.43** |      **898.17** |                        **455.09** |    **411.62** |
-
-## Reproducing
-
-The Ballista numbers come from the `tpch` runner in this repository. The Spark,
-Spark + Comet, and Trino numbers were collected with a benchmark harness that
-is not in this repository yet. We are working on adding scripts so that anyone
-can reproduce all of these results.
-
-### Ballista
-
-Bring up the cluster (one scheduler, N executors), then run the suite from a
-client. Executor sizing on each node:
-
-```sh
-ballista-executor \
-  --bind-host 0.0.0.0 --bind-port 50051 \
-  --scheduler-host <scheduler> --scheduler-port 50050 \
-  --vcores 8 \
-  --memory-pool-size 48103633715 \
-  --work-dir /data \
-  --grpc-server-max-decoding-message-size 134217728 \
-  --grpc-server-max-encoding-message-size 134217728
-```
-
-`/data` should be a dedicated volume (e.g. a `gp3` PVC) sized for the
-suite's shuffle output — see [Executor storage](#executor-storage).
-
-Run all 22 queries with the `tpch` Rust runner from
-[`benchmarks/`](https://github.com/apache/datafusion-ballista/tree/main/benchmarks):
-
-```sh
-cargo run --release --bin tpch -- benchmark ballista \
-  --host <scheduler> --port 50050 \
-  --path s3://<bucket>/tpch/sf1000 --format parquet \
-  --partitions 256 --iterations 2
-```
-
-The runner sets `target_partitions` from `--partitions` and enables
-`collect_statistics`; everything else is left at its default.
-
-### Spark and Spark + Comet
-
-Until the harness is added, the closest public equivalent is
-`tpcbench.py` from
-[apache/datafusion-benchmarks](https://github.com/apache/datafusion-benchmarks),
-with the settings above and stock Spark 4.1 defaults for everything else:
-
-```sh
-spark-submit \
-  --master <master> \
-  --conf spark.executor.instances=32 \
-  --conf spark.executor.cores=16 \
-  --conf spark.executor.memory=64G \
-  --conf spark.executor.memoryOverhead=10G \
-  --conf spark.sql.shuffle.partitions=512 \
-  tpcbench.py \
-    --benchmark tpch \
-    --data s3a://<bucket>/tpch/sf1000 \
-    --format parquet \
-    --iterations 2
-```
-
-For Spark + Comet, add the Comet JAR to the driver and executor classpath
-(see the
-[Comet installation guide](https://datafusion.apache.org/comet/user-guide/latest/installation.html))
-and the settings in [Comet configuration](#comet-configuration):
-
-```sh
-  --conf spark.plugins=org.apache.spark.CometPlugin \
-  --conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \
-  --conf spark.memory.offHeap.enabled=true \
-  --conf spark.memory.offHeap.size=32G \
-  --conf spark.comet.exec.memoryPool.fraction=0.8
-```
-
-### Trino
-
-The Trino numbers were collected with the same harness. It creates
-external Hive tables over the Parquet files, syncs their partitions, and runs
-each query once untimed and then three times, one query at a time, through the
-Trino client, with the settings in [Trino configuration](#trino-configuration).
-
-## Recording a new result set
-
-- Pin the **exact commit** the numbers came from, not a branch name.
-- Replace the version, environment, config, and results tables together — a
-  row that mixes numbers from different commits silently misattributes a
-  regression.
-- Prefer a single continuous suite run: a long-lived executor deep into a
-  suite is not in the same state as a freshly started one.
-- Report `FAIL` for a query that ran but did not produce an answer, and
-  `OOM` when the failure is a known memory exhaustion. See the tracker for
-  open issues found by benchmarking: [#1359][aqe],
-  [#2025][q18], [#2063][aqe-hang].
-
-[aqe]: https://github.com/apache/datafusion-ballista/issues/1359
-[q18]: https://github.com/apache/datafusion-ballista/issues/2025
-[aqe-hang]: https://github.com/apache/datafusion-ballista/issues/2063
diff --git a/docs/source/contributors-guide/benchmarking/ballista.md b/docs/source/contributors-guide/benchmarking/ballista.md
new file mode 100644
index 000000000..9ce4366cb
--- /dev/null
+++ b/docs/source/contributors-guide/benchmarking/ballista.md
@@ -0,0 +1,79 @@
+<!---
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+
+# Ballista Benchmark Configuration
+
+How Ballista was configured for the TPC-H SF1000 results on the
+[benchmarking page](index.md). The cluster, data and queries are described in
+[Benchmark Environment](environment.md).
+
+## Ballista configuration
+
+| Flag / config key                                           | Value                                                                |
+| ----------------------------------------------------------- | -------------------------------------------------------------------- |
+| `--vcores`                                                  | `8`                                                                  |
+| `--memory-pool-size` (bytes; ≈70 % of the 64 GiB container) | `48103633715`                                                        |
+| `--work-dir`                                                | `/data` (dedicated gp3 PVC — see [Executor storage](environment.md)) |
+| `--grpc-server-max-decoding-message-size`                   | `134217728`                                                          |
+| `--grpc-server-max-encoding-message-size`                   | `134217728`                                                          |
+| `datafusion.execution.target_partitions`                    | `256`                                                                |
+| `datafusion.execution.collect_statistics`                   | `true`                                                               |
+| `ballista.planner.adaptive.enabled`                         | `true` (AQE; default)                                                |
+| `ballista.shuffle.sort_based.memory_limit_per_task_bytes`   | `268435456` (256 MiB; default)                                       |
+
+`datafusion.optimizer.prefer_hash_join` is left at its default; under AQE the
+join strategy is selected at runtime by `DelayJoinSelectionRule` /
+`DynamicJoinSelectionExec` from runtime statistics and the broadcast /
+`ballista.optimizer.hash_join_max_build_partition_bytes` thresholds.
+
+The gRPC message-size ceiling is raised from the 16 MiB default to 128 MiB;
+some SF1000 physical plans (Q11, Q21, Q22) encode above 16 MiB and hit
+`OutOfRange` errors otherwise.
+
+## Reproducing
+
+Bring up the cluster (one scheduler, N executors), then run the suite from a
+client. Executor sizing on each node:
+
+```sh
+ballista-executor \
+  --bind-host 0.0.0.0 --bind-port 50051 \
+  --scheduler-host <scheduler> --scheduler-port 50050 \
+  --vcores 8 \
+  --memory-pool-size 48103633715 \
+  --work-dir /data \
+  --grpc-server-max-decoding-message-size 134217728 \
+  --grpc-server-max-encoding-message-size 134217728
+```
+
+`/data` should be a dedicated volume (e.g. a `gp3` PVC) sized for the
+suite's shuffle output — see [Executor storage](environment.md).
+
+Run all 22 queries with the `tpch` Rust runner from
+[`benchmarks/`](https://github.com/apache/datafusion-ballista/tree/main/benchmarks):
+
+```sh
+cargo run --release --bin tpch -- benchmark ballista \
+  --host <scheduler> --port 50050 \
+  --path s3://<bucket>/tpch/sf1000 --format parquet \
+  --partitions 256 --iterations 2
+```
+
+The runner sets `target_partitions` from `--partitions` and enables
+`collect_statistics`; everything else is left at its default.
diff --git a/docs/source/contributors-guide/benchmarking/comet.md b/docs/source/contributors-guide/benchmarking/comet.md
new file mode 100644
index 000000000..c57295015
--- /dev/null
+++ b/docs/source/contributors-guide/benchmarking/comet.md
@@ -0,0 +1,56 @@
+<!---
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+
+# Spark + Comet Benchmark Configuration
+
+How Spark 4.1.3 with [Apache DataFusion Comet](https://github.com/apache/datafusion-comet)
+1.1.0-rc2 was configured for the TPC-H SF1000 results on the
+[benchmarking page](index.md). The cluster, data and queries are described in
+[Benchmark Environment](environment.md).
+
+## Comet configuration
+
+The Spark + Comet run adds these settings to the [Spark configuration](spark.md):
+
+| Key                                    | Value                                                              |
+| -------------------------------------- | ------------------------------------------------------------------ |
+| `spark.plugins`                        | `org.apache.spark.CometPlugin`                                     |
+| `spark.comet.enabled`                  | `true`                                                             |
+| `spark.shuffle.manager`                | `org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager` |
+| `spark.memory.offHeap.enabled`         | `true`                                                             |
+| `spark.memory.offHeap.size`            | `32G`                                                              |
+| `spark.comet.exec.memoryPool.fraction` | `0.8`                                                              |
+
+The run also enabled Comet's explain and fallback logging. Everything else is
+left at Comet's defaults.
+
+## Reproducing
+
+Start from the [Spark](spark.md) `spark-submit` command, add the Comet JAR to
+the driver and executor classpath (see the
+[Comet installation guide](https://datafusion.apache.org/comet/user-guide/latest/installation.html))
+and add the settings above:
+
+```sh
+  --conf spark.plugins=org.apache.spark.CometPlugin \
+  --conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \
+  --conf spark.memory.offHeap.enabled=true \
+  --conf spark.memory.offHeap.size=32G \
+  --conf spark.comet.exec.memoryPool.fraction=0.8
+```
diff --git a/docs/source/contributors-guide/benchmarking/environment.md b/docs/source/contributors-guide/benchmarking/environment.md
new file mode 100644
index 000000000..78ad8062d
--- /dev/null
+++ b/docs/source/contributors-guide/benchmarking/environment.md
@@ -0,0 +1,95 @@
+<!---
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+
+# Benchmark Environment
+
+How the TPC-H SF1000 results on the [benchmarking page](index.md) were
+collected: cluster, data, and queries. Per-engine settings are on the
+[Ballista](ballista.md), [Spark](spark.md), [Spark + Comet](comet.md) and
+[Trino](trino.md) pages.
+
+## Cluster
+
+- **Cluster:** Kubernetes on AWS (`us-west-2`); one driver/scheduler pod and
+  32 executor pods for each engine, launched on the same node pool.
+- **K8s worker nodes:** a mix of `r6i.24xlarge` (96 vCPU, 768 GiB memory) and
+  `r6i.16xlarge` (64 vCPU, 512 GiB memory), EBS-only — no local
+  instance-store. Pods are not pinned to an instance type.
+- **Executor pod (Ballista):** x86_64, 8 vCPU, 64 GiB memory, plus a
+  dedicated 1000 GiB `gp3` EBS PVC mounted at `/data` for the executor's
+  shuffle work-dir (see Executor storage below).
+- **Executor pod (Spark):** x86_64, 8 vCPU, 64 GiB + 10 GiB overhead, plus a
+  dedicated `gp3` PVC via `spark-local-dir-1`.
+- **Executor pod (Spark + Comet):** the same as Spark, plus 32 GiB of
+  off-heap memory for Comet's native execution (about 106 GiB per pod in
+  total, compared with 64 GiB for a Ballista executor).
+- **Worker pod (Trino):** x86_64, 8 vCPU, 64 GiB memory (the image's
+  default JVM heap of 80 %, about 51 GiB), no spill volume. A separate
+  coordinator pod plans and schedules but does not scan data.
+- **Driver (Spark and Spark + Comet):** runs the queries through a Spark
+  benchmark harness that is not in this repository yet. See
+  [Spark](spark.md).
+- **Client pod (Ballista):** the `tpch` Rust benchmark runner from
+  [`benchmarks/`](https://github.com/apache/datafusion-ballista/tree/main/benchmarks)
+  in this repo (`cargo run --release --bin tpch -- benchmark ballista ...`),
+  which submits SQL through a Ballista `SessionContext` and collects
+  results locally.
+- **Data:** TPC-H SF1000 Parquet on S3 (`us-west-2`), ZSTD compression,
+  ~512 MiB row groups, one directory per table. `lineitem`, `orders`,
+  `customer` and `part` are Hive-style partitioned (e.g.
+  `lineitem/l_shipdate=YYYY-MM-DD/`). The `tpch` runner declares the
+  partition columns (see
+  [Hive-partitioned data](https://github.com/apache/datafusion-ballista/tree/main/benchmarks#hive-partitioned-data)),
+  so Ballista skips the partitions a filter excludes, as Spark, Comet and
+  Trino do. Results from before
+  [#2553](https://github.com/apache/datafusion-ballista/pull/2553) were
+  collected without partition columns and pruned these scans only through
+  Parquet statistics.
+
+## Executor storage
+
+Each Ballista executor pod is attached to a **fresh 1000 GiB `gp3` EBS
+volume** (generic-ephemeral PVC, `storageClassName: gp3`), mounted at
+`/data`, and the executor is launched with `--work-dir /data`. All shuffle
+temp files land on this dedicated volume.
+
+Without it, `--work-dir` defaults to a random directory under `/tmp` on the
+container overlay filesystem, i.e. onto the node's root EBS volume, which
+is shared with container images, `kubelet`, and every other pod on the
+same node. On `r6i.24xlarge` that shared bandwidth becomes the binding
+constraint under sustained shuffle-write pressure — `EXPLAIN ANALYZE`
+observed Q8's `SortShuffleWriter.write_time` inflate ~5× on the second
+run of a suite compared to a fresh cluster, despite the same 313 GB of
+shuffle output and zero spilling, because Q1–Q7's dirty pages force
+synchronous flushes to EBS at cap. Attaching a dedicated PVC removes that
+contention and brings in-suite per-query times in line with the standalone
+number.
+
+Spark on the same cluster has always used this pattern via
+`spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1`.
+
+## Queries
+
+The SQLBench-H phrasing of the 22 TPC-H queries from
+[apache/datafusion-benchmarks](https://github.com/apache/datafusion-benchmarks).
+
+The Spark harness uses the same queries with the same substitution
+parameters. The only differences are in wording: it writes some dates as
+arithmetic (for example `date '1998-12-01' - interval '90' day` instead of
+`date '1998-09-02'`), and it phrases Q15 as a CTE instead of a view.
diff --git a/docs/source/contributors-guide/benchmarking/index.md b/docs/source/contributors-guide/benchmarking/index.md
new file mode 100644
index 000000000..4ccaee75d
--- /dev/null
+++ b/docs/source/contributors-guide/benchmarking/index.md
@@ -0,0 +1,107 @@
+<!---
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+
+# Benchmarking
+
+Current TPC-H **SF1000** results for Ballista, compared against vanilla
+**Spark 4.1.3** and **Spark 4.1.3 with Apache DataFusion Comet 1.1.0-rc2**,
+running on the same cluster shape. A **Trino 483** column is included for
+reference; it is not an apples-to-apples comparison (see
+[Trino configuration](trino.md)).
+
+## Versions under test
+
+| Engine   | Version                                                                                                                                                                                                                |
+| -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Ballista | [`c76278cd`](https://github.com/apache/datafusion-ballista/commit/c76278cd48583504821ac4d71ea74b44e54ed4ea) (`main`, 2026-10-07), Cargo pkg `55.0.0`, DataFusion `55.1.0`                                              |
+| Spark    | `4.1.3` (`apache/spark:4.1.3` image, vanilla, no acceleration plugin)                                                                                                                                                  |
+| Comet    | [Apache DataFusion Comet](https://github.com/apache/datafusion-comet) `1.1.0-rc2` ([`992c806a`](https://github.com/apache/datafusion-comet/commit/992c806a7e38c2e88bd018aa5774164b0850e1fa)) on the same Spark `4.1.3` |
+| Trino    | `483` (`trinodb/trino:483` image, Hive connector)                                                                                                                                                                      |
+
+## Results
+
+**Times in seconds; lower is better.** Ballista, Spark, and Spark + Comet:
+mean of 2 iterations, including the cold first iteration. Trino: mean of 3
+iterations after an untimed warm-up run (see
+[Trino configuration](trino.md)).
+
+|     Query | Ballista (s) | Spark 4.1.3 (s) | Spark 4.1.3 + Comet 1.1.0-rc2 (s) | Trino 483 (s) |
+| --------: | -----------: | --------------: | --------------------------------: | ------------: |
+|         1 |        14.07 |           71.45 |                             10.66 |          5.32 |
+|         2 |        25.81 |           37.98 |                             21.66 |         15.34 |
+|         3 |        17.22 |           30.61 |                             16.05 |         13.47 |
+|         4 |        15.31 |           23.86 |                              8.11 |          8.01 |
+|         5 |        46.88 |           57.39 |                             38.32 |         25.98 |
+|         6 |         1.59 |            1.73 |                              0.99 |          0.84 |
+|         7 |        37.26 |           25.34 |                             16.60 |         16.19 |
+|         8 |        20.13 |           58.17 |                             46.04 |         30.93 |
+|         9 |        35.94 |           74.90 |                             55.78 |         41.71 |
+|        10 |        43.36 |           35.31 |                             21.31 |         20.41 |
+|        11 |        15.01 |           31.66 |                             14.79 |          5.73 |
+|        12 |        10.49 |           13.24 |                              5.33 |          4.40 |
+|        13 |        11.05 |           22.73 |                             11.31 |         10.56 |
+|        14 |         5.54 |            7.25 |                              2.50 |          1.66 |
+|        15 |        10.58 |           22.45 |                             11.85 |          8.30 |
+|        16 |        12.78 |           24.34 |                              8.94 |          6.98 |
+|        17 |        20.74 |           81.36 |                             25.14 |         24.76 |
+|        18 |        45.63 |          134.11 |                             46.42 |         66.00 |
+|        19 |        14.59 |           13.27 |                              9.07 |          5.93 |
+|        20 |        14.46 |           16.08 |                              6.57 |          8.57 |
+|        21 |        75.74 |           98.04 |                             68.43 |         83.22 |
+|        22 |         8.25 |           16.90 |                              9.22 |          7.29 |
+| **Total** |   **502.43** |      **898.17** |                        **455.09** |    **411.62** |
+
+## Reproducing
+
+The Ballista numbers come from the `tpch` runner in this repository. The Spark,
+Spark + Comet, and Trino numbers were collected with a benchmark harness that
+is not in this repository yet. We are working on adding scripts so that anyone
+can reproduce all of these results. See the per-engine pages below for the
+commands and settings.
+
+## Details
+
+How the cluster was set up and how each engine was configured:
+
+```{toctree}
+:maxdepth: 1
+
+Environment <environment>
+Ballista <ballista>
+Spark <spark>
+Spark + Comet <comet>
+Trino <trino>
+```
+
+## Recording a new result set
+
+- Pin the **exact commit** the numbers came from, not a branch name.
+- Replace the version, environment, config, and results tables together — a
+  row that mixes numbers from different commits silently misattributes a
+  regression.
+- Prefer a single continuous suite run: a long-lived executor deep into a
+  suite is not in the same state as a freshly started one.
+- Report `FAIL` for a query that ran but did not produce an answer, and
+  `OOM` when the failure is a known memory exhaustion. See the tracker for
+  open issues found by benchmarking: [#1359][aqe],
+  [#2025][q18], [#2063][aqe-hang].
+
+[aqe]: https://github.com/apache/datafusion-ballista/issues/1359
+[q18]: https://github.com/apache/datafusion-ballista/issues/2025
+[aqe-hang]: https://github.com/apache/datafusion-ballista/issues/2063
diff --git a/docs/source/contributors-guide/benchmarking/spark.md b/docs/source/contributors-guide/benchmarking/spark.md
new file mode 100644
index 000000000..d521d7f58
--- /dev/null
+++ b/docs/source/contributors-guide/benchmarking/spark.md
@@ -0,0 +1,75 @@
+<!---
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+
+# Spark Benchmark Configuration
+
+How vanilla Spark 4.1.3 was configured for the TPC-H SF1000 results on the
+[benchmarking page](index.md). The cluster, data and queries are described in
+[Benchmark Environment](environment.md).
+
+## Spark configuration (highlights)
+
+Both Spark runs use these settings. The vanilla Spark run has no Comet
+plugin and uses the stock `SortShuffleManager`.
+
+| Key                                                        | Value                                |
+| ---------------------------------------------------------- | ------------------------------------ |
+| `spark.executor.instances`                                 | `32`                                 |
+| `spark.executor.cores`                                     | `16` (task parallelism per executor) |
+| `spark.kubernetes.executor.limit.cores` / `.request.cores` | `8` (physical vCPU)                  |
+| `spark.executor.memory`                                    | `64G`                                |
+| `spark.executor.memoryOverhead`                            | `10G`                                |
+| `spark.memory.fraction`                                    | `0.6`                                |
+| `spark.memory.storageFraction`                             | `0.2`                                |
+| `spark.sql.shuffle.partitions`                             | `512`                                |
+| `spark.sql.broadcastTimeout`                               | `900`                                |
+| `spark.serializer`                                         | `KryoSerializer`                     |
+| `spark.io.compression.codec`                               | `zstd`                               |
+
+Spark AQE is left at its Spark 4.1 defaults. Shuffle spills to a `gp3`-backed
+per-executor volume (`spark.kubernetes.executor.volumes...spark-local-dir-1`).
+
+Note that `spark.executor.cores=16` is Spark's **task parallelism** setting,
+not a CPU allocation — each executor pod is given only **8 physical vCPU**
+via `spark.kubernetes.executor.limit.cores` / `.request.cores`, so Spark
+schedules 16 concurrent tasks onto 8 physical cores (2× oversubscription).
+The matching Ballista executor runs `--vcores=8` on the same
+8 physical vCPU (1:1).
+
+## Reproducing
+
+Until the harness is added, the closest public equivalent is
+`tpcbench.py` from
+[apache/datafusion-benchmarks](https://github.com/apache/datafusion-benchmarks),
+with the settings above and stock Spark 4.1 defaults for everything else:
+
+```sh
+spark-submit \
+  --master <master> \
+  --conf spark.executor.instances=32 \
+  --conf spark.executor.cores=16 \
+  --conf spark.executor.memory=64G \
+  --conf spark.executor.memoryOverhead=10G \
+  --conf spark.sql.shuffle.partitions=512 \
+  tpcbench.py \
+    --benchmark tpch \
+    --data s3a://<bucket>/tpch/sf1000 \
+    --format parquet \
+    --iterations 2
+```
diff --git a/docs/source/contributors-guide/benchmarking/trino.md b/docs/source/contributors-guide/benchmarking/trino.md
new file mode 100644
index 000000000..3fee5eb43
--- /dev/null
+++ b/docs/source/contributors-guide/benchmarking/trino.md
@@ -0,0 +1,64 @@
+<!---
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+
+# Trino Benchmark Configuration
+
+How Trino 483 was configured for the TPC-H SF1000 results on the
+[benchmarking page](index.md). The cluster, data and queries are described in
+[Benchmark Environment](environment.md).
+
+## Differences from the other engines
+
+Trino is a useful reference point, but the setup differs from the other
+engines in ways that matter:
+
+- **Warm timings.** Each query runs once untimed, then the reported number
+  is the mean of 3 timed iterations. The other engines include the cold
+  first iteration in their mean.
+- **In-memory exchanges.** Trino streams data between stages over the
+  network and never writes shuffle files, while Ballista, Spark, and Comet
+  materialize every shuffle on disk. Spilling was disabled, so a query
+  that does not fit in memory would fail rather than slow down (none did).
+- **Partition pruning.** Tables are external Hive tables over the same
+  Parquet files with the partition columns declared, so Trino prunes
+  partitions like Spark and Comet do.
+- **No table statistics.** `ANALYZE` was not run, so the cost-based
+  optimizer had no row counts or column statistics.
+- **Queries.** The same SQL text as the Spark harness.
+
+| Key                                                              | Value                                |
+| ---------------------------------------------------------------- | ------------------------------------ |
+| Workers                                                          | `32` (`include-coordinator=false`)   |
+| `query.max-memory-per-node`                                      | `38GB`                               |
+| `memory.heap-headroom-per-node`                                  | `6GB`                                |
+| `spill-enabled`                                                  | `false`                              |
+| `hive.metastore`                                                 | `file` (stored in S3)                |
+| `hive.metastore-cache-ttl`, `hive.file-status-cache-expire-time` | `24h` (metadata cached across a run) |
+| `hive.dynamic-filtering.wait-timeout`                            | `1s`                                 |
+
+Everything else is left at Trino's defaults, with no session properties
+set. No tuning was done beyond this, and the configuration has not been
+reviewed by Trino experts.
+
+## Reproducing
+
+The Trino numbers were collected with the same harness. It creates
+external Hive tables over the Parquet files, syncs their partitions, and runs
+each query once untimed and then three times, one query at a time, through the
+Trino client, with the settings on this page.
diff --git a/docs/source/index.rst b/docs/source/index.rst
index 06382501d..aeeb46a1b 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -82,7 +82,7 @@ Table of content
    contributors-guide/code-organization
    contributors-guide/user-personas
    contributors-guide/development
-   contributors-guide/benchmarking
+   contributors-guide/benchmarking/index
    Source code <https://github.com/apache/datafusion-ballista/>
 
 .. _toc.community:


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org