You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by ag...@apache.org on 2026/07/21 12:39:35 UTC
(datafusion-ballista) branch main updated: ci(tpch): add AQE-on smoke pass with multi-partition tasks (#2120)
This is an automated email from the ASF dual-hosted git repository.
andygrove pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-ballista.git
The following commit(s) were added to refs/heads/main by this push:
new bb2fdfee2 ci(tpch): add AQE-on smoke pass with multi-partition tasks (#2120)
bb2fdfee2 is described below
commit bb2fdfee2fe42958d3ecd791dd73c25248e03949
Author: Brent Gardner <bg...@squarelabs.net>
AuthorDate: Tue Jul 21 06:39:31 2026 -0600
ci(tpch): add AQE-on smoke pass with multi-partition tasks (#2120)
* ci(tpch): add AQE-on smoke pass with multi-partition tasks
Existing CI runs at the default max_partitions_per_task=1 (one task per
input partition), which never exercises the multi-partition-task path.
Add a third suite invocation that flips it to 0 (unbounded, budget-capped
at vcores=4) alongside AQE on, so the mode we actually plan to ship gets
end-to-end verification against SF10.
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* ci(tpch): split into a 3-job matrix, one per planner config
Running three sequential run_suite invocations on a single runner had all
three passes contending for the same ~30m wall clock. Move to a matrix of
three parallel jobs (AQE off, AQE on, AQE on + multi-partition tasks),
each with its own scheduler+executor+SF10 data. Wall clock drops to
roughly the longest single suite (~14m). rust-cache shared-key: tpch-sf10
lets all three legs reuse the same build cache, so the extra CI-minutes
cost is bounded to data-gen + suite runtime.
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <no...@anthropic.com>
---
.github/workflows/tpch.yml | 66 +++++++++++++++++++++++++---------------------
1 file changed, 36 insertions(+), 30 deletions(-)
diff --git a/.github/workflows/tpch.yml b/.github/workflows/tpch.yml
index 4be3eb535..a0da579ff 100644
--- a/.github/workflows/tpch.yml
+++ b/.github/workflows/tpch.yml
@@ -52,10 +52,23 @@ on:
jobs:
tpch-sf10:
- name: TPC-H SF10 (all queries, AQE off + on)
+ name: TPC-H SF10 (${{ matrix.label }})
runs-on: ubuntu-latest
container:
image: amd64/rust
+ strategy:
+ fail-fast: false
+ matrix:
+ include:
+ - label: "AQE off"
+ planner_args: ""
+ slug: "aqe-off"
+ - label: "AQE on"
+ planner_args: "-c ballista.planner.adaptive.enabled=true"
+ slug: "aqe-on"
+ - label: "AQE on, multi-partition tasks"
+ planner_args: "-c ballista.planner.adaptive.enabled=true -c ballista.scheduler.max_partitions_per_task=0"
+ slug: "aqe-on-mpt"
steps:
- name: Install dependencies
run: |
@@ -72,6 +85,10 @@ jobs:
rust-version: stable
- uses: swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 #2.9.1
+ with:
+ # Share the build cache across all matrix legs — the compiled
+ # binaries are identical; only the per-suite CLI args differ.
+ shared-key: tpch-sf10
- name: Build Ballista binaries
run: |
@@ -151,40 +168,29 @@ jobs:
done
nc -z 127.0.0.1 50051 || { echo "executor did not start"; exit 1; }
- # Run the suite once per planner mode against the same data and
- # cluster. AQE on/off is a per-query session setting, so the scheduler
- # picks the planner per job; no need to regenerate data or restart the
- # cluster.
- run_suite() {
- local label="$1"; shift
- # q16 omitted: still unsupported (matches benchmarks/run.sh).
- for q in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 17 18 19 20 21 22; do
- echo "::group::[$label] Query $q"
- ./target/tpch-ci/tpch benchmark ballista \
- --host 127.0.0.1 --port 50050 \
- --query "$q" \
- --path "$DATA_DIR" \
- --format parquet \
- --partitions 16 \
- --iterations 1 \
- "$@"
- echo "::endgroup::"
- done
- }
-
- run_suite "AQE off" \
- --verify \
- -c datafusion.optimizer.prefer_hash_join=false
- run_suite "AQE on" \
- --verify \
- -c datafusion.optimizer.prefer_hash_join=false \
- -c ballista.planner.adaptive.enabled=true
+ # This matrix leg runs one planner configuration against the whole
+ # SF10 dataset. The other legs run in parallel jobs.
+ # q16 omitted: still unsupported (matches benchmarks/run.sh).
+ for q in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 17 18 19 20 21 22; do
+ echo "::group::[${{ matrix.label }}] Query $q"
+ ./target/tpch-ci/tpch benchmark ballista \
+ --host 127.0.0.1 --port 50050 \
+ --query "$q" \
+ --path "$DATA_DIR" \
+ --format parquet \
+ --partitions 16 \
+ --iterations 1 \
+ --verify \
+ -c datafusion.optimizer.prefer_hash_join=false \
+ ${{ matrix.planner_args }}
+ echo "::endgroup::"
+ done
- name: Upload cluster logs on failure
if: failure()
uses: actions/upload-artifact@v7
with:
- name: tpch-sf10-cluster-logs
+ name: tpch-sf10-cluster-logs-${{ matrix.slug }}
retention-days: 14
path: |
${{ runner.temp }}/scheduler.log
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org