You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/05/02 14:16:20 UTC
[PR] build(bench): rework docker-compose TPC-H stack and add manual benchmark workflow [datafusion-ballista]
andygrove opened a new pull request, #1646:
URL: https://github.com/apache/datafusion-ballista/pull/1646
# Which issue does this PR close?
<!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements -->
N/A
# Rationale for this change
The docker-compose benchmark setup had several latent bugs (broken `nc` healthchecks against `ubuntu:24.04`, `depends_on` without health gating, missing `tpch-gen.sh` referenced by `dev/integration-tests.sh`, obsolete compose `version` key, mismatched image names vs `dev/build-ballista-docker.sh`) and was not a useful benchmark target out of the box (TBL format, one iteration, default `--partitions 2`). It also wasn't reachable from CI.
This PR makes the stack runnable end-to-end again, brings the defaults to a sensible benchmark shape, and adds an on-demand GitHub workflow so anyone can run it against `main` or a PR/branch.
# What changes are included in this PR?
`docker-compose.yml`
- Drop obsolete `version: '3.3'`.
- Align `image:` tags with what `dev/build-ballista-docker.sh` produces (`apache/datafusion-ballista-{scheduler,executor,benchmarks}:latest`) so prebuilt images get reused.
- Replace broken `nc -z` healthchecks with `bash </dev/tcp/127.0.0.1/PORT>` (no extra packages required).
- Switch `depends_on` to the long form with `condition: service_healthy`; remove the now-unneeded `--scheduler-connect-timeout-seconds 15`.
- Executor: `--concurrent-tasks 8`, `--memory-pool-size 8GB`, `--work-dir /work` (tmpfs), `cpus: '8'` and `memory: 12G` per replica, 2 replicas.
- Drop the misleading `50051:50051` publish on `ballista-client` (it just sleeps).
- Mount `benchmarks/queries` and `benchmarks/run.sh` into the client read-only for fast iteration.
- Consistent `restart: "no"` for benchmark runs.
`benchmarks/tpch-gen.sh` (new)
- Generates partitioned Parquet via [`tpchgen-cli`](https://crates.io/crates/tpchgen-cli) (auto-installed via `cargo install` if missing).
- Defaults: `SCALE_FACTOR=10`, `PARTITIONS=16`, `OUTPUT_DIR=benchmarks/data`. All env-overridable.
`benchmarks/run.sh`
- `--format parquet --partitions 16 --iterations 3`, env-overridable.
- Drop the legacy `--format tbl` + `--expected /data` correctness branch (CI's `verify-benchmark-results` job already covers correctness).
`dev/integration-tests.sh`
- Use `docker compose` v2.
- `trap`-based teardown so `down` always runs.
- Wait for healthy services with `--wait`.
- Forward `SCALE_FACTOR` / `PARTITIONS` / `ITERATIONS` to gen + run.
`.github/workflows/benchmark.yml` (new)
- `workflow_dispatch` only — never runs automatically on PRs.
- Inputs: `ref` (default `main`), `scale_factor` (10), `partitions` (16), `iterations` (3).
- Uses ASF-funded runs-on.com large runner (`cpu=32`, `family=m8a+m7a+c8a`, `image=ubuntu24-full-x64`, `tag=ballista`) when `repository_owner == 'apache'`; falls back to `ubuntu-latest` on forks so the workflow can still validate end-to-end pipeline changes.
- Dumps compose logs on failure; 120-min timeout.
# Are there any user-facing changes?
No public API changes. Benchmark/dev-tooling only. Behaviour changes for users running `dev/integration-tests.sh` or `docker compose up`:
- Generated data is now Parquet under `benchmarks/data` (was TBL).
- Default benchmark shape is SF10, 16 partitions, 3 iterations (was effectively SF1, 2 partitions, 1 iteration).
- Each executor replica is bound to 8 vCPU / 12 GB / 8 GB memory pool — meaningful only on hosts with at least 16 vCPU available to Docker.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] build(bench): rework docker-compose TPC-H stack [datafusion-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #1646:
URL: https://github.com/apache/datafusion-ballista/pull/1646
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] build(bench): rework docker-compose TPC-H stack and add manual benchmark workflow [datafusion-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #1646:
URL: https://github.com/apache/datafusion-ballista/pull/1646#discussion_r3176746890
##########
.github/workflows/benchmark.yml:
##########
@@ -0,0 +1,81 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: TPC-H Benchmark (manual)
+
+on:
+ workflow_dispatch:
+ inputs:
+ ref:
+ description: 'Git ref (branch, tag, or SHA) to benchmark'
+ required: true
+ default: 'main'
+ scale_factor:
+ description: 'TPC-H scale factor'
+ required: true
+ default: '10'
+ partitions:
+ description: 'Number of partitions / target_partitions'
+ required: true
+ default: '16'
+ iterations:
+ description: 'Iterations per query'
+ required: true
+ default: '3'
+
+concurrency:
+ group: benchmark-${{ github.event.inputs.ref }}
+ cancel-in-progress: false
+
+jobs:
+ benchmark:
+ name: TPC-H SF${{ github.event.inputs.scale_factor }}, ${{ github.event.inputs.partitions }} partitions
+ # Use ASF-funded runs-on.com large runner inside apache/, fall back to
+ # ubuntu-latest on forks (slow but exercises the full pipeline).
+ # Ballista binaries are built inside the dockerized builder (see
+ # dev/build-ballista-executables.sh) so the host only needs cargo to
+ # install tpchgen-cli for data generation - both runner images ship cargo.
+ runs-on: ${{ github.repository_owner == 'apache' && format('runs-on={0},family=m8a+m7a+c8a,cpu=32,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=ballista', github.run_id) || 'ubuntu-latest' }}
+ timeout-minutes: 120
+ steps:
+ - uses: runs-on/action@742bf56072eb4845a0f94b3394673e4903c90ff0 # v2.1.0
+ if: ${{ github.repository_owner == 'apache' }}
+
+ - name: Checkout ${{ github.event.inputs.ref }}
+ uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+ with:
+ ref: ${{ github.event.inputs.ref }}
+ submodules: true
+
+ - name: Show host info
+ run: |
+ uname -a
+ cargo --version
+ docker version
+ docker compose version
+ nproc
+
+ - name: Run integration benchmark
+ env:
+ SCALE_FACTOR: ${{ github.event.inputs.scale_factor }}
+ PARTITIONS: ${{ github.event.inputs.partitions }}
+ ITERATIONS: ${{ github.event.inputs.iterations }}
+ run: ./dev/integration-tests.sh
+
+ - name: Collect compose logs on failure
+ if: failure()
+ run: docker compose logs --no-color --tail=500 || true
Review Comment:
Added a top-level `permissions: { contents: read }` block in afe449c0 — it's the only scope this workflow needs (checkout only, no pushes/comments/artifacts).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] build(bench): rework docker-compose TPC-H stack and add manual benchmark workflow [datafusion-ballista]
Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #1646:
URL: https://github.com/apache/datafusion-ballista/pull/1646#discussion_r3176744764
##########
.github/workflows/benchmark.yml:
##########
@@ -0,0 +1,81 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: TPC-H Benchmark (manual)
+
+on:
+ workflow_dispatch:
+ inputs:
+ ref:
+ description: 'Git ref (branch, tag, or SHA) to benchmark'
+ required: true
+ default: 'main'
+ scale_factor:
+ description: 'TPC-H scale factor'
+ required: true
+ default: '10'
+ partitions:
+ description: 'Number of partitions / target_partitions'
+ required: true
+ default: '16'
+ iterations:
+ description: 'Iterations per query'
+ required: true
+ default: '3'
+
+concurrency:
+ group: benchmark-${{ github.event.inputs.ref }}
+ cancel-in-progress: false
+
+jobs:
+ benchmark:
+ name: TPC-H SF${{ github.event.inputs.scale_factor }}, ${{ github.event.inputs.partitions }} partitions
+ # Use ASF-funded runs-on.com large runner inside apache/, fall back to
+ # ubuntu-latest on forks (slow but exercises the full pipeline).
+ # Ballista binaries are built inside the dockerized builder (see
+ # dev/build-ballista-executables.sh) so the host only needs cargo to
+ # install tpchgen-cli for data generation - both runner images ship cargo.
+ runs-on: ${{ github.repository_owner == 'apache' && format('runs-on={0},family=m8a+m7a+c8a,cpu=32,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=ballista', github.run_id) || 'ubuntu-latest' }}
+ timeout-minutes: 120
+ steps:
+ - uses: runs-on/action@742bf56072eb4845a0f94b3394673e4903c90ff0 # v2.1.0
+ if: ${{ github.repository_owner == 'apache' }}
+
+ - name: Checkout ${{ github.event.inputs.ref }}
+ uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+ with:
+ ref: ${{ github.event.inputs.ref }}
+ submodules: true
+
+ - name: Show host info
+ run: |
+ uname -a
+ cargo --version
+ docker version
+ docker compose version
+ nproc
+
+ - name: Run integration benchmark
+ env:
+ SCALE_FACTOR: ${{ github.event.inputs.scale_factor }}
+ PARTITIONS: ${{ github.event.inputs.partitions }}
+ ITERATIONS: ${{ github.event.inputs.iterations }}
+ run: ./dev/integration-tests.sh
+
+ - name: Collect compose logs on failure
+ if: failure()
+ run: docker compose logs --no-color --tail=500 || true
Review Comment:
## CodeQL / Workflow does not contain permissions
Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {{contents: read}}
[Show more details](https://github.com/apache/datafusion-ballista/security/code-scanning/19)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] build(bench): rework docker-compose TPC-H stack [datafusion-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #1646:
URL: https://github.com/apache/datafusion-ballista/pull/1646#issuecomment-4364375837
I can now run `./dev/integration-tests.sh` with no errors
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org