You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "adriangb (via GitHub)" <gi...@apache.org> on 2026/05/14 18:40:13 UTC
[PR] ci: integrate CodSpeed continuous benchmarking [arrow-rs]
adriangb opened a new pull request, #9975:
URL: https://github.com/apache/arrow-rs/pull/9975
## Summary
Wires the existing criterion benches in this workspace into [CodSpeed](https://codspeed.io) for continuous performance tracking. CodSpeed runs benches under CPU simulation in CI and posts per-PR comparison reports vs. the base branch's latest main run.
This PR is opt-in once activated: the PR workflow only fires when a maintainer adds a `bench:*` label, so external contributors don't blindly burn CI capacity. The main-push workflow keeps the baseline current.
The integration has been validated end-to-end on a fork ([`pydantic/arrow-rs`](https://github.com/pydantic/arrow-rs)): 3031 benchmarks captured from a single main run, PR runs produce clean comparison comments (e.g. *"Merging this PR will not alter performance — ✅ 7 untouched benchmarks, ⏩ 3024 skipped benchmarks, comparing codspeed-smoke-test (5b1320a) with main (fcbe248)"*). Public dashboard: https://codspeed.io/pydantic/arrow-rs
## Design
### Drop-in shim, no bench source changes
The `criterion` workspace dependency is renamed (via the `[package]` cargo trick) to `codspeed-criterion-compat`. This is a CodSpeed-maintained passthrough — when not running under `cargo codspeed`, it forwards to real criterion, so `cargo bench` locally is unchanged and every existing `use criterion::*` in every bench source file compiles unmodified.
```toml
# Cargo.toml (workspace)
criterion = { package = "codspeed-criterion-compat", version = "4.6", default-features = false }
```
### Sharded one job per `[[bench]]` target
Required for two reasons:
1. The full workspace produces well over 1000 individual benchmarks (criterion parameterizes heavily), which exceeds CodSpeed's [per-upload limit](https://codspeed.io/docs/features/sharded-benchmarks).
2. Even the `parquet` crate alone exceeds 1000 — per-crate sharding wasn't fine enough.
Jobs within a single workflow are auto-aggregated by CodSpeed into one unified report.
### Build once, run many
```
setup ─┐
├──→ bench (matrix, ~78 shards)
build ─┘
```
- `setup` parses every workspace member's `Cargo.toml` for `[[bench]]` entries (awk + jq), emits a JSON `{crate, bench}` array; new bench targets are picked up automatically.
- `build` runs the full-workspace `cargo codspeed build` exactly once, packs `target/codspeed/` into a tarball (tar preserves the +x bit; `actions/upload-artifact` strips it otherwise), uploads as a 1-day artifact.
- Each bench shard downloads the artifact, unpacks it, runs `cargo codspeed run -p <crate> --bench <bench>`. No per-shard rebuild — CI cost scales with N × ~2 min instead of N × full build.
### Label-gated PRs
`codspeed-pr.yml` fires on `pull_request: [labeled, synchronize, opened, reopened]` and only runs when the PR has at least one `bench:*` label:
| Label | Effect |
| --------------------------------------- | ------------------------------------- |
| `bench:all` | Every `[[bench]]` in the workspace |
| `bench:<crate>` | Every `[[bench]]` in that crate |
| `bench:<crate-a> bench:<crate-b>` | Union |
Label suffixes are validated against `^[a-z][a-z0-9_-]*$`. Authorization is implicit: only users with write access can add labels.
While the label is attached, every push to the PR re-runs the suite (`synchronize` event); re-runs cancel in-progress shards via `concurrency: cancel-in-progress: true`.
### OIDC auth
Public repo, no `CODSPEED_TOKEN` secret required — the workflow's `id-token: write` claim is what CodSpeed verifies. Workflows are repo-agnostic.
## Exclusions
Ten bench targets currently fail at runtime in this workspace — pre-existing issues in the bench targets themselves, not the integration. They're listed in an `EXCLUDED_BENCHES` env in both workflows so the remaining ~78 shards run clean. Each excluded target should be fixed (or removed) and dropped from the list one by one:
| Target | Observed failure mode |
| --- | --- |
| `arrow / merge_kernels` | panics at `arrow-data/src/transform/primitive.rs:31:43` |
| `arrow / buffer_bit_ops` | runtime error |
| `arrow / buffer_create` | runtime error |
| `arrow / sort_kernel` | runtime error |
| `arrow / string_run_builder` | runtime error |
| `arrow / primitive_run_accessor` | runtime error |
| `arrow-array / union_array` | runtime error |
| `arrow-cast / parse_date` | runtime error |
| `parquet / row_selection_cursor` | runtime error |
| `parquet-variant-compute / variant_kernels` | intermittent |
I'm happy to file separate upstream issues for each if helpful — or to drop the exclusion list entirely if maintainers prefer to investigate them all at once. The same `merge_kernels` exclusion was added by the official CodSpeed wizard's auto-generated PR (https://codspeed.io/docs/get-started/wizard), so this is consistent prior art.
## Prerequisites for activation
This PR adds the workflow files but they're inert until two repo-admin actions land:
1. **Install the [CodSpeed GitHub App](https://github.com/apps/codspeed)** on `apache/arrow-rs`. This is what posts the PR comparison comment + status check.
2. **Enroll the repository at https://codspeed.io**. OIDC is automatic for public repos — no secret token configuration required.
Once both are done, the first push to `main` will populate the baseline and PRs labeled `bench:*` will receive automated comparison comments.
## CI cost notes
- Main-push workflow: 1 build + 78 shards. Build job dominates wallclock (~10 min); shards run in parallel and download from one artifact, ~2 min each.
- PR workflow: same `build`, but only the bench shards for the labeled crates. A typical `bench:arrow-cast` run is build + 3 shards.
- Per-target bench binaries are bundled in one ~1-2 GB artifact (well under GitHub's 5 GB free-tier limit).
## Test plan
- [x] `cargo check --workspace --benches --features arrow/test_utils,arrow-schema/ffi,parquet/test_common,parquet/experimental,parquet/async,parquet/object_store` passes against this branch
- [x] End-to-end validation on `pydantic/arrow-rs`: main baseline run captured 3031 benchmarks; PR run posts comparison comment correctly; per-shard sharding stays under the 1000-benchmark limit
- [ ] After merge and CodSpeed-App install on `apache/arrow-rs`, first main run populates baseline at https://codspeed.io/apache/arrow-rs
- [ ] Create the `bench:all` and per-crate `bench:<crate>` labels in repo settings
- [ ] Add `bench:<crate>` to a real PR; confirm comparison comment + status check appear
## References
- CodSpeed docs: https://codspeed.io/docs
- Sharded benchmarks: https://codspeed.io/docs/features/sharded-benchmarks
- Compat shim source: https://github.com/CodSpeedHQ/codspeed-rust
- Prior auto-generated wizard PR on pydantic fork: https://github.com/pydantic/arrow-rs/pull/11 (single-shard; hit the >1000 limit, which this PR resolves)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ci: integrate CodSpeed continuous benchmarking [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #9975:
URL: https://github.com/apache/arrow-rs/pull/9975#issuecomment-5215662175
we can test it on main for a while; one thing to note is i think for the github app permissions we'll need infra to handle this for us? for reference see [INFRA-26030](https://issues.apache.org/jira/browse/INFRA-26030) which was opendal adding the app to the organization, but checking the comments there looks like they would have specified it only for the opendal repository so we might need to open another ticket to add arrow-rs to the permissions there?
something else i notice whilst checking opendal was it seems they actually removed the integration:
- https://github.com/apache/opendal/pull/5626
see how the codspeed project for them doesnt have any recent runs either: https://app.codspeed.io/apache/opendal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ci: integrate CodSpeed continuous benchmarking [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #9975:
URL: https://github.com/apache/arrow-rs/pull/9975#discussion_r3460276998
##########
.github/workflows/codspeed-pr.yml:
##########
@@ -0,0 +1,179 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Opt-in CodSpeed benchmarking for pull requests, gated by labels and
+# sharded one job per `[[bench]]` target in each selected crate.
+#
+# Label convention (managed manually on each PR):
+#
+# bench:all # every [[bench]] in the workspace
+# bench:<crate> # every [[bench]] in that crate
+# bench:<crate> bench:<crate> # union
+#
+# Where <crate> is a workspace member name, e.g. `bench:arrow`,
+# `bench:parquet`, `bench:arrow-cast`. `bench:all` short-circuits and
+# supersedes any per-crate labels.
+#
+# Topology mirrors codspeed.yml (setup + build run in parallel; bench
+# is a matrix that downloads the build artifact and runs one bench
+# target per shard). The `setup` job additionally filters the matrix
+# by labels.
+#
+# Authorization: only users with write access to the repo can add
+# labels, so the label is itself the authorization gate.
+#
+# Baseline: native `pull_request` event → CodSpeed compares against
+# the base branch's latest CodSpeed report automatically.
+#
+# Fork PR caveat: workflows triggered by `pull_request` from fork PRs
+# do not get an OIDC token. For benches on fork PRs, push the branch
+# to this repo and label it there.
Review Comment:
i suppose this is an important caveat considering essentially all our PRs come from forks 🤔
##########
.github/workflows/codspeed.yml:
##########
@@ -0,0 +1,163 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Continuous benchmarking on CodSpeed.
+#
+# Runs the full workspace bench suite on every push to main, sharded
+# one job per `[[bench]]` target. Sharding at this granularity is
+# required because both crate-level shards (e.g. parquet alone has 16
+# bench targets producing >1000 benchmarks) and the workspace as a
+# whole exceed CodSpeed's 1000-benchmark per-upload limit. Jobs in the
+# same workflow are auto-aggregated by CodSpeed into one report.
+# https://codspeed.io/docs/features/sharded-benchmarks
Review Comment:
I do wonder how heavy this would be to run entire benchmark suite on every commit 🤔
##########
.github/workflows/codspeed.yml:
##########
@@ -0,0 +1,163 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Continuous benchmarking on CodSpeed.
+#
+# Runs the full workspace bench suite on every push to main, sharded
+# one job per `[[bench]]` target. Sharding at this granularity is
+# required because both crate-level shards (e.g. parquet alone has 16
+# bench targets producing >1000 benchmarks) and the workspace as a
+# whole exceed CodSpeed's 1000-benchmark per-upload limit. Jobs in the
+# same workflow are auto-aggregated by CodSpeed into one report.
+# https://codspeed.io/docs/features/sharded-benchmarks
+#
+# Topology:
+#
+# setup ─┐
+# ├──→ bench (matrix, ~78 jobs)
+# build ─┘
+#
+# `setup` discovers every `[[bench]]` target via `cargo metadata` (see
+# codspeed-matrix.sh) and emits a {crate, bench} matrix. `build` does the
+# full-workspace
+# `cargo codspeed build` exactly once and uploads
+# `target/codspeed/<mode>/` as an artifact. `bench` shards download the
+# artifact and run a single bench target each via the CodSpeed action;
+# no rebuild per shard.
+#
+# Auth is via GitHub OIDC; the workflow's `id-token` claim is what
+# CodSpeed verifies, so no secret token is required.
+#
+# The `criterion` workspace dependency is renamed to
+# `codspeed-criterion-compat`, so existing `use criterion::*` benches
+# work unmodified; outside of CodSpeed the compat layer falls back to
+# standard criterion behavior.
+
+name: codspeed
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.workflow }}-${{ github.sha }}
+ cancel-in-progress: false
+
+on:
+ push:
+ branches:
+ - main
+ workflow_dispatch:
+
+permissions:
+ contents: read
+ id-token: write
+
+env:
+ CODSPEED_FEATURES: arrow/test_utils,arrow/csv,arrow/json,arrow/chrono-tz,arrow/prettyprint,arrow-schema/ffi,parquet/arrow,parquet/async,parquet/test_common,parquet/experimental,parquet/object_store
+
+jobs:
+ setup:
+ name: Generate bench matrix
+ runs-on: ubuntu-latest
+ outputs:
+ matrix: ${{ steps.gen.outputs.matrix }}
+ steps:
+ - uses: actions/checkout@v6
+
+ - name: Generate {crate, bench} matrix across the workspace
+ id: gen
+ # Discovery + the known-broken exclusion list live in the shared
+ # codspeed-matrix.sh (also used by codspeed-pr.yml) so they stay in
+ # one place. No args = every workspace crate.
+ run: |
+ matrix="$(bash .github/workflows/codspeed-matrix.sh)"
+ echo "matrix=$matrix" >> "$GITHUB_OUTPUT"
+ echo "::notice::Generated $(jq length <<<"$matrix") bench shards (one per bench target, known-broken targets excluded)"
+
+ build:
+ name: Build workspace benchmarks
+ runs-on: ubuntu-latest
+ timeout-minutes: 60
+ steps:
+ - uses: actions/checkout@v6
+ with:
+ submodules: true
+
+ - name: Install protoc
+ run: sudo apt-get update && sudo apt-get install -y protobuf-compiler
+
+ - name: Setup Rust toolchain, cache and cargo-codspeed
+ uses: moonrepo/setup-rust@v1
+ with:
+ channel: stable
+ cache-target: release
+ bins: cargo-codspeed
+
+ - name: Build benchmarks
+ # The --features list enables every feature any workspace bench
+ # target gates behind `required-features`; without these those
+ # benches silently aren't built.
+ run: cargo codspeed build --workspace --features "$CODSPEED_FEATURES"
+
+ - name: Pack bench binaries into a tarball
+ # actions/upload-artifact does not preserve Unix executable
+ # bits, so downloaded bench binaries land as 644 and
+ # `cargo codspeed run` then fails with EACCES. Tar preserves
+ # mode bits, so we wrap the directory before upload.
+ run: tar -cf codspeed-binaries.tar -C target codspeed
+
+ - name: Upload built bench binaries
+ uses: actions/upload-artifact@v4
+ with:
+ name: codspeed-binaries
+ path: codspeed-binaries.tar
+ retention-days: 1
+ if-no-files-found: error
+
+ bench:
+ needs: [setup, build]
+ name: ${{ matrix.config.crate }} / ${{ matrix.config.bench }}
+ runs-on: ubuntu-latest
+ timeout-minutes: 30
+ strategy:
+ fail-fast: false
+ matrix:
+ config: ${{ fromJson(needs.setup.outputs.matrix) }}
+ steps:
+ - uses: actions/checkout@v6
Review Comment:
minor note: we should change the versions to hashes, but can do in a followup
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ci: integrate CodSpeed continuous benchmarking [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9975:
URL: https://github.com/apache/arrow-rs/pull/9975#issuecomment-5208068681
@Jefffrey -- what would you think about merging this to test it out. I don't think we'll be able to really evaluate how helpful it is until we are actually running it regularly and see if the results make any sense
@adriangb -- what do you think about starting with running just on main every night at first, which would keep the noise down during evaluation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org