You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@magpie.apache.org by po...@apache.org on 2026/10/19 22:30:22 UTC
(magpie) branch main updated: ci(tests): batch pytest jobs and sync only each job's members (#1717)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/magpie.git
The following commit(s) were added to refs/heads/main by this push:
new acb35444 ci(tests): batch pytest jobs and sync only each job's members (#1717)
acb35444 is described below
commit acb35444ea1c2fd3d27eccd69110f43ddff968f2
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Tue Oct 20 00:30:15 2026 +0200
ci(tests): batch pytest jobs and sync only each job's members (#1717)
* ci(tests): group pytest jobs into batches when many members are picked
A full pytest run was 49 one-member jobs, and each one spent 27-75 s on
checkout, uv and a workspace sync against 2-18 s of tests; with ASF's
15-concurrent-job cap that took about 8 minutes of wall time and ~41
runner-minutes.
The member selection moves out of the workflow's inline Python into
`tools/dev/ci-pytest-batches.py`, with tests, unchanged in what it picks.
It now also groups: up to six picked members get one job each, named
after the member as before, so a typical pull request looks the same;
more are split into at most eight jobs balanced by lines of test code.
A batch job runs every member even after one fails, prints a log group
per member and a pass/fail table in the step summary, and fails at the
end if any member did. `tests-ok` is unchanged.
The uv cache suffix interpolated the matrix object, which renders as
`tests-Object`; it is now the explicit shared key it in effect was,
since every leg syncs the same `--all-packages` set.
Generated-by: Claude Opus 5
* ci(tests): sync only each pytest job's own members
Every pytest job ran `uv sync --all-packages`, installing all fifty
workspace members to test one to eight of them; that sync was 17-56 s
of each job. It also hid undeclared dependencies: a test that imports a
workspace sibling its member does not declare passed only because the
whole workspace happened to be installed.
`ci-pytest-batches.py` now emits each job's project names, and the job
syncs `--frozen --group dev` plus one `--package` per member. The uv
cache gets one slot per batch, since batches now install different sets.
Run locally, every member's tests pass with only its own package synced,
except `plugins`: its pytest also collected the generated copy of
`tools/adversarial-review` under `plugins/magpie-adversarial-review/tools/`
(byte-identical, tested at its source). The agent-guard and vetted-ops
copies were already excluded by `norecursedirs`; this one now is too.
Generated-by: Claude Opus 5
---
.github/workflows/tests.yml | 188 +++++++++++++-----------------
plugins/pyproject.toml | 2 +-
tools/dev/README.md | 1 +
tools/dev/ci-pytest-batches.py | 145 +++++++++++++++++++++++
tools/dev/tests/test_ci_pytest_batches.py | 144 +++++++++++++++++++++++
5 files changed, 373 insertions(+), 107 deletions(-)
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index f8b49456..be7e6b8b 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -45,7 +45,7 @@ jobs:
permissions:
contents: read
outputs:
- members: ${{ steps.list.outputs.members }}
+ batches: ${{ steps.list.outputs.batches }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
@@ -54,23 +54,22 @@ jobs:
# head against its merge base. Harmless on push events.
fetch-depth: 0
- id: list
- # Emit the workspace members as a JSON array of {name,path}
- # objects so the matrix below can both label jobs by name
- # AND pass the path to `uv run --directory`. The `pytest`
- # applicability filter mirrors the auto-discovery rule in
- # `tools/dev/run-workspace-check.sh`: only members with a
- # `[tool.pytest.ini_options]` section get pytest jobs, and
- # an explicit `[tool.magpie.checks] skip = [..., "pytest"]`
- # opts out.
+ # `tools/dev/ci-pytest-batches.py` picks the members and groups
+ # them into jobs; its docstring has the rules. In short:
#
- # Path filter (PRs only): a member's pytest job is emitted
- # only when that member's own Python source/test files changed
- # in the PR. On push to `main` — and when a shared build input
- # changes (root `pyproject.toml` / `uv.lock` / the
- # workspace-check helper / this workflow) — the full matrix
- # runs. If no member's tests are affected the matrix is empty
- # and the `pytest` job is skipped; `tests-ok` treats that as a
- # pass (see below).
+ # - Applicability mirrors `tools/dev/run-workspace-check.sh`: a
+ # member with `[tool.pytest.ini_options]` and no
+ # `[tool.magpie.checks] skip = [..., "pytest"]`.
+ # - Path filter (PRs only): a member is picked when its own
+ # Python source/test files changed. On push to `main`, and when
+ # a shared build input changes (root `pyproject.toml`,
+ # `uv.lock`, the helpers, this workflow), every member is.
+ # - Grouping: most of a job is setup, so up to six members get a
+ # job each (named after the member, as before); more are split
+ # into at most eight jobs balanced by test size.
+ #
+ # An empty selection skips the `pytest` job, which `tests-ok`
+ # treats as a pass (see below).
env:
EVENT: ${{ github.event_name }}
BASE_SHA: ${{ github.event.pull_request.base.sha }}
@@ -81,102 +80,44 @@ jobs:
# list thousands of paths, and putting that into the environment
# overflows the exec arg/env size limit — `python3` then dies with
# "Argument list too long" (exit 126). A file path stays small.
- changed_file="${RUNNER_TEMP}/changed-files.txt"
if [ "${EVENT}" = "pull_request" ]; then
+ changed_file="${RUNNER_TEMP}/changed-files.txt"
git diff --name-only "${BASE_SHA}...${HEAD_SHA}" > "${changed_file}"
+ batches=$(python3 tools/dev/ci-pytest-batches.py --changed-file "${changed_file}")
else
- : > "${changed_file}"
+ batches=$(python3 tools/dev/ci-pytest-batches.py)
fi
- members=$(CHANGED_FILE="${changed_file}" python3 <<'PY'
- import json
- import os
- import tomllib
-
- event = os.environ.get("EVENT", "")
- changed_file = os.environ.get("CHANGED_FILE", "")
- changed = []
- if changed_file:
- with open(changed_file, encoding="utf-8") as fh:
- changed = [line.strip() for line in fh if line.strip()]
-
- with open("pyproject.toml", "rb") as f:
- root = tomllib.load(f)
- paths = root["tool"]["uv"]["workspace"]["members"]
-
- # Shared build inputs that force the full matrix even on a PR:
- # a change to any of these can affect every member's tests.
- GLOBAL = {
- "pyproject.toml",
- "uv.lock",
- "tools/dev/run-workspace-check.sh",
- ".github/workflows/tests.yml",
- }
- run_all = event != "pull_request" or any(c in GLOBAL for c in changed)
+ echo "batches=${batches}" >> "$GITHUB_OUTPUT"
+ echo "pytest jobs:"
+ echo "${batches}" | python3 -m json.tool
- def member_changed(path: str) -> bool:
- # A member is affected when the PR touches a `.py` file
- # anywhere under it, or any file under its `tests/` dir
- # (fixtures / data a test reads).
- prefix = path + "/"
- tests_prefix = path + "/tests/"
- for c in changed:
- if c.startswith(prefix) and (
- c.endswith(".py") or c.startswith(tests_prefix)
- ):
- return True
- return False
-
- out = []
- for path in paths:
- with open(f"{path}/pyproject.toml", "rb") as f:
- data = tomllib.load(f)
- tool = data.get("tool", {})
- if "ini_options" not in tool.get("pytest", {}):
- continue
- skip = tool.get("magpie", {}).get("checks", {}).get("skip", [])
- if "pytest" in skip:
- continue
- if not run_all and not member_changed(path):
- continue
- out.append({"name": path.split("/")[-1], "path": path})
- # Sort for deterministic CI ordering.
- out.sort(key=lambda x: x["name"])
- print(json.dumps(out))
- PY
- )
- echo "members=${members}" >> "$GITHUB_OUTPUT"
- echo "Selected workspace members for pytest matrix:"
- echo "${members}" | python3 -m json.tool
-
- # Per-member pytest matrix. Each Python project under tools/ that
- # is a uv-workspace member with a `[tool.pytest.ini_options]`
- # section gets its own job. Running them as separate matrix jobs
- # surfaces per-project pass/fail in the CI checks list, which is
- # easier to triage than the bundled `workspace-pytest` line inside
- # the `prek` workflow's output. The `prek` workflow still
- # exercises pytest as a hook, so this workflow is the visible
- # signal — not the gate.
+ # The pytest matrix, one entry per job from the `members` job. A
+ # pull request touching a few projects gets one job per project, so
+ # per-project pass/fail shows in the checks list; a full run batches
+ # them, and each batch job prints a group and a summary row per
+ # member. The `prek` workflow still exercises pytest as a hook, so
+ # this workflow is the visible signal — not the gate.
pytest:
- name: "pytest (${{ matrix.project.name }})"
+ name: "pytest (${{ matrix.batch.name }})"
runs-on: ubuntu-slim
needs: members
# Guard against an empty selection. When the path filter selects no
- # members (a PR that changed no project's Python), the members output
+ # members (a PR that changed no project's Python), the batches output
# is `[]`; feeding an empty array to `matrix` makes GitHub mark this
# job `failure` ("matrix vector does not contain any values"), not
# `skipped` — which would fail `tests-ok`. Skipping the job on an
# empty list yields the `skipped` result `tests-ok` treats as a pass.
- if: ${{ needs.members.outputs.members != '[]' }}
+ if: ${{ needs.members.outputs.batches != '[]' }}
permissions:
contents: read
strategy:
fail-fast: false
- # The full matrix is ~46 jobs. ASF Infra policy caps a workflow at
- # 20 concurrent jobs and recommends 15:
- # https://infra.apache.org/github-actions-policy.html
+ # At most eight batches, but a PR can still pick six single
+ # jobs; ASF Infra policy caps a workflow at 20 concurrent jobs and
+ # recommends 15: https://infra.apache.org/github-actions-policy.html
max-parallel: 15
matrix:
- project: ${{ fromJSON(needs.members.outputs.members) }}
+ batch: ${{ fromJSON(needs.members.outputs.batches) }}
# GitHub Actions log viewer renders ANSI colour escapes; without
# an attached TTY most tools default to monochrome. `FORCE_COLOR`
# is the de-facto signal honoured by uv, ruff, mypy, and pytest's
@@ -197,19 +138,28 @@ jobs:
- uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0
with:
enable-cache: true
- # One slot per matrix leg, not merely per job: these run in parallel
- # with each other and each installs a different workspace member, so a
- # shared key both races and stores the wrong contents. See the note in
- # pre-commit.yml.
- cache-suffix: "tests-${{ matrix.project }}"
- - name: Sync workspace (installs ruff / mypy / pytest from root dev group)
+ # One slot per batch: each syncs only its own members, so a shared
+ # key would race and store the wrong contents. A batch's key is
+ # stable across runs (the split is deterministic), and a single
+ # member's key is its name.
+ cache-suffix: "tests-${{ matrix.batch.key }}"
+ - name: Sync the batch's members (and ruff / mypy / pytest from the root dev group)
# Dev-group deps live in the root pyproject (`[dependency-groups]
# dev`) and only land in the shared workspace `.venv` when an
- # explicit `--group dev` sync runs. `--all-packages` ensures
- # every workspace member is installed too (the root has
- # `package = false`, so without it members and their
- # `[project.scripts]` entry points would be skipped).
- run: uv sync --all-packages --group dev
+ # explicit `--group dev` sync runs. One `--package` per member
+ # installs that member, its `[project.scripts]` entry points and
+ # what it declares — workspace siblings included — and nothing
+ # else, so a test that imports a sibling it does not declare
+ # fails here rather than passing by accident. `--frozen`: the
+ # lock is the input, never rewritten in CI.
+ env:
+ MEMBER_PACKAGES: ${{ matrix.batch.packages }}
+ run: |
+ args=()
+ for package in ${MEMBER_PACKAGES}; do
+ args+=(--package "${package}")
+ done
+ uv sync --frozen --group dev "${args[@]}"
- name: Run pytest
# `--directory` (not `--project`) — both move uv's project
# context, but `--directory` also changes cwd, which pytest
@@ -217,7 +167,33 @@ jobs:
# `testpaths = ["tests"]` relative to its own root. No
# `--group` flag here because the sync above already
# installed the dev tools into the shared workspace venv.
- run: uv run --directory ${{ matrix.project.path }} pytest --color=yes
+ #
+ # Every member of the batch runs even after one fails; the job
+ # fails at the end, and the summary table names which member did.
+ env:
+ MEMBER_PATHS: ${{ matrix.batch.paths }}
+ run: |
+ failed=()
+ {
+ echo "| Project | pytest |"
+ echo "|---|---|"
+ } >> "${GITHUB_STEP_SUMMARY}"
+ for path in ${MEMBER_PATHS}; do
+ name="${path##*/}"
+ echo "::group::pytest (${name})"
+ if uv run --directory "${path}" pytest --color=yes; then
+ result="passed"
+ else
+ result="**failed**"
+ failed+=("${name}")
+ fi
+ echo "::endgroup::"
+ echo "| \`${name}\` | ${result} |" >> "${GITHUB_STEP_SUMMARY}"
+ done
+ if [ "${#failed[@]}" -gt 0 ]; then
+ echo "::error::pytest failed for: ${failed[*]}"
+ exit 1
+ fi
# Umbrella status check. `.asf.yaml`'s `required_status_checks`
# cannot use patterns or wildcards (GitHub's branch-protection and
diff --git a/plugins/pyproject.toml b/plugins/pyproject.toml
index eb8bbe6c..69fc2dd0 100644
--- a/plugins/pyproject.toml
+++ b/plugins/pyproject.toml
@@ -103,7 +103,7 @@ addopts = "-ra -q"
# `tools/<name>`, which is its own workspace member with its own pytest job —
# don't collect it twice.
testpaths = ["."]
-norecursedirs = ["magpie-agent-guard/tools", "magpie-vetted-ops/tools"]
+norecursedirs = ["magpie-adversarial-review/tools", "magpie-agent-guard/tools", "magpie-vetted-ops/tools"]
[dependency-groups]
# The shared toolchain (mypy, pytest, ruff) comes from `magpie-dev`
diff --git a/tools/dev/README.md b/tools/dev/README.md
index f8651d9e..2705ccfd 100644
--- a/tools/dev/README.md
+++ b/tools/dev/README.md
@@ -70,6 +70,7 @@ installable for other members to depend on it.
| [`check-workspace-members.py`](check-workspace-members.py) | Catches a new `tools/<name>/pyproject.toml` that was never added to `[tool.uv.workspace] members` — an omission that silently drops the tool from both the pre-commit hooks and the CI pytest matrix. Also verifies each member's tests actually run: both surfaces key off `[tool.pytest.ini_options]`, so a project can carry a full `tests/` directory and be executed by nothing. Reports tests-without-config, config-without-tests, and [...]
| [`generate-labeler-config.py`](generate-labeler-config.py) | Generates `.github/labeler.yml` — the path → label map the [`labeler.yml`](../../.github/workflows/labeler.yml) workflow uses to pre-apply labels to a PR and its linked issues — from the `**Capability:**` line of every `tools/<name>/README.md` (`contract:*` / `substrate:*`), the `family:` / `capability:` frontmatter of every skill (its directory and its eval suite), and the paths of the non-skill families (`family:tools`, `fa [...]
| [`run-workspace-check.sh`](run-workspace-check.sh) | Runs one static-check or test command across every workspace member, auto-discovering which members a given check applies to. The four `workspace-*` hooks call it, so adding a tool needs no edit to the pre-commit config. |
+| [`ci-pytest-batches.py`](ci-pytest-batches.py) | Picks the members the `tests` workflow runs pytest for, with the same applicability rule as `run-workspace-check.sh` plus a pull request's path filter, and groups them into jobs. Most of a pytest job is setup (checkout, uv, a workspace sync: 27–75 s against 2–18 s of tests), so up to six members get a job each, named after the member, and more are split into at most eight jobs balanced by lines of test code. Each job also gets its member [...]
| [`run-skill-script-tests.sh`](run-skill-script-tests.sh) | Runs the stdlib `unittest` suites under `plugins/*/skills/*/tests`, which cover skills' sibling scripts. Those scripts are not workspace members, so the `workspace-pytest` hook never reaches them. Runs as the `skill-script-tests` pre-commit hook. |
| [`add-license-headers.py`](add-license-headers.py) | Stamps the SPDX licence header into Markdown files that lack one. |
| [`agent-pre-commit.sh`](agent-pre-commit.sh) | Wrapper for `prek run --all-files`, for agent use. An agent running `pytest` / `ruff` / `mypy` individually still misses the rest of the CI gate (doctoc, markdownlint, typos, the checks above); this runs what CI runs. |
diff --git a/tools/dev/ci-pytest-batches.py b/tools/dev/ci-pytest-batches.py
new file mode 100644
index 00000000..d39d068b
--- /dev/null
+++ b/tools/dev/ci-pytest-batches.py
@@ -0,0 +1,145 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pick the workspace members the `tests` workflow runs, and group them into jobs.
+
+A member gets pytest when it has a `[tool.pytest.ini_options]` section and does
+not opt out with `[tool.magpie.checks] skip = [..., "pytest"]` — the same rule
+as `tools/dev/run-workspace-check.sh`. On a pull request only the members whose
+Python or `tests/` files changed are picked, unless a shared build input changed;
+on any other event every member is.
+
+Most of a pytest job is setup (checkout, uv, a workspace sync), so the picked
+members are grouped:
+
+* up to `SOLO_MAX` members: one job each, named after the member, so a typical
+ pull request keeps one check per project;
+* more: at most `MAX_BATCHES` jobs, balanced by the size of each member's tests
+ (heaviest first onto the lightest job), each running its members one by one.
+
+Prints a JSON array of `{"key", "name", "paths", "packages"}` for the workflow's
+matrix (`paths` and `packages` space-separated, in the same order; `packages` are
+the members' project names, so a job syncs only what its members declare).
+`--changed-file` names a file listing the changed paths; omit it for a full run.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+import tomllib
+from pathlib import Path
+
+SOLO_MAX = 6
+MAX_BATCHES = 8
+
+# A change to any of these can affect every member's tests.
+GLOBAL = frozenset(
+ {
+ "pyproject.toml",
+ "uv.lock",
+ "tools/dev/run-workspace-check.sh",
+ "tools/dev/ci-pytest-batches.py",
+ ".github/workflows/tests.yml",
+ }
+)
+
+
+def has_pytest(root: Path, path: str) -> bool:
+ with (root / path / "pyproject.toml").open("rb") as f:
+ tool = tomllib.load(f).get("tool", {})
+ if "ini_options" not in tool.get("pytest", {}):
+ return False
+ return "pytest" not in tool.get("magpie", {}).get("checks", {}).get("skip", [])
+
+
+def member_changed(path: str, changed: list[str]) -> bool:
+ """A `.py` file anywhere under the member, or any file under its `tests/`."""
+ prefix, tests = path + "/", path + "/tests/"
+ return any(c.startswith(prefix) and (c.endswith(".py") or c.startswith(tests)) for c in changed)
+
+
+def select(root: Path, changed: list[str] | None) -> list[str]:
+ with (root / "pyproject.toml").open("rb") as f:
+ paths = tomllib.load(f)["tool"]["uv"]["workspace"]["members"]
+ run_all = changed is None or any(c in GLOBAL for c in changed)
+ picked = [p for p in paths if has_pytest(root, p) and (run_all or member_changed(p, changed or []))]
+ return sorted(picked, key=lambda p: p.rsplit("/", 1)[-1])
+
+
+def weight(root: Path, path: str) -> int:
+ """Lines of test code: a cheap, deterministic stand-in for how long a member's tests take."""
+ tests = root / path / "tests"
+ if not tests.is_dir():
+ return 1
+ return max(1, sum(len(f.read_bytes().splitlines()) for f in sorted(tests.rglob("*.py"))))
+
+
+def package(root: Path, path: str) -> str:
+ with (root / path / "pyproject.toml").open("rb") as f:
+ return str(tomllib.load(f)["project"]["name"])
+
+
+def _name(path: str) -> str:
+ return path.rsplit("/", 1)[-1]
+
+
+def batches(paths: list[str], weights: dict[str, int]) -> list[dict[str, str]]:
+ if len(paths) <= SOLO_MAX:
+ return [{"key": _name(p), "name": _name(p), "paths": p} for p in paths]
+ count = min(MAX_BATCHES, len(paths))
+ groups: list[list[str]] = [[] for _ in range(count)]
+ loads = [0] * count
+ # Longest-processing-time first; ties broken by name so the split is stable.
+ for path in sorted(paths, key=lambda p: (-weights[p], _name(p))):
+ slot = min(range(count), key=lambda i: (loads[i], i))
+ groups[slot].append(path)
+ loads[slot] += weights[path]
+ out = []
+ for index, group in enumerate(groups, start=1):
+ names = sorted(_name(p) for p in group)
+ shown = ", ".join(names[:3]) + (f", +{len(names) - 3}" if len(names) > 3 else "")
+ out.append(
+ {"key": f"batch-{index}", "name": f"{index}/{count}: {shown}", "paths": " ".join(sorted(group))}
+ )
+ return out
+
+
+def main(argv: list[str] | None = None) -> int:
+ parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+ parser.add_argument("--root", type=Path, default=Path("."))
+ parser.add_argument("--changed-file", type=Path, help="file listing a pull request's changed paths")
+ args = parser.parse_args(argv)
+ changed = None
+ if args.changed_file:
+ changed = [
+ line.strip()
+ for line in args.changed_file.read_text(encoding="utf-8").splitlines()
+ if line.strip()
+ ]
+ paths = select(args.root, changed)
+ out = batches(paths, {p: weight(args.root, p) for p in paths})
+ for batch in out:
+ batch["packages"] = " ".join(package(args.root, p) for p in batch["paths"].split())
+ json.dump(out, sys.stdout)
+ print()
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/tools/dev/tests/test_ci_pytest_batches.py b/tools/dev/tests/test_ci_pytest_batches.py
new file mode 100644
index 00000000..7f39ce93
--- /dev/null
+++ b/tools/dev/tests/test_ci_pytest_batches.py
@@ -0,0 +1,144 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Tests for ``ci-pytest-batches.py``: which members the `tests` workflow runs, in which jobs."""
+
+from __future__ import annotations
+
+import importlib.util
+import json
+from pathlib import Path
+from types import ModuleType
+
+import pytest
+
+_SCRIPT = Path(__file__).resolve().parents[1] / "ci-pytest-batches.py"
+REPO_ROOT = Path(__file__).resolve().parents[3]
+
+
+def _load() -> ModuleType:
+ spec = importlib.util.spec_from_file_location("ci_pytest_batches", _SCRIPT)
+ assert spec and spec.loader
+ module = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(module)
+ return module
+
+
+cpb = _load()
+
+
+def _workspace(tmp_path: Path, members: dict[str, str]) -> Path:
+ """`members` maps a member path to its pyproject `[tool]` text."""
+ listed = ", ".join(f'"{p}"' for p in members)
+ (tmp_path / "pyproject.toml").write_text(f"[tool.uv.workspace]\nmembers = [{listed}]\n")
+ for path, tool in members.items():
+ (tmp_path / path / "tests").mkdir(parents=True)
+ (tmp_path / path / "pyproject.toml").write_text(
+ f'[project]\nname = "pkg-{path.rsplit("/", 1)[-1]}"\n' + tool
+ )
+ return tmp_path
+
+
+PYTEST = "[tool.pytest.ini_options]\ntestpaths = ['tests']\n"
+
+
+def test_only_members_with_pytest_and_no_opt_out_are_picked(tmp_path):
+ root = _workspace(
+ tmp_path,
+ {
+ "tools/a": PYTEST,
+ "tools/b": "[tool.ruff]\n",
+ "tools/c": PYTEST + "[tool.magpie.checks]\nskip = ['pytest']\n",
+ },
+ )
+ assert cpb.select(root, None) == ["tools/a"]
+
+
+@pytest.mark.parametrize(
+ ("changed", "picked"),
+ [
+ (["tools/a/src/x.py"], ["tools/a"]),
+ (["tools/a/tests/fixtures/data.json"], ["tools/a"]),
+ (["tools/a/README.md"], []),
+ (["uv.lock"], ["tools/a", "tools/b"]),
+ (["tools/dev/ci-pytest-batches.py"], ["tools/a", "tools/b"]),
+ ([], []),
+ ],
+)
+def test_a_pull_request_picks_only_what_it_changed(tmp_path, changed, picked):
+ root = _workspace(tmp_path, {"tools/a": PYTEST, "tools/b": PYTEST})
+ assert cpb.select(root, changed) == picked
+
+
+def test_up_to_solo_max_members_get_one_job_each_named_after_the_member():
+ paths = [f"tools/m{i}" for i in range(cpb.SOLO_MAX)]
+ out = cpb.batches(paths, dict.fromkeys(paths, 1))
+ assert [b["name"] for b in out] == [f"m{i}" for i in range(cpb.SOLO_MAX)]
+ assert [b["paths"] for b in out] == paths
+
+
+def test_more_members_are_capped_at_max_batches_and_balanced():
+ paths = [f"tools/m{i:02}" for i in range(49)]
+ weights = {p: (i % 7 + 1) * 100 for i, p in enumerate(paths)}
+ out = cpb.batches(paths, weights)
+ assert len(out) == cpb.MAX_BATCHES
+ grouped = [b["paths"].split() for b in out]
+ assert sorted(p for g in grouped for p in g) == paths
+ loads = [sum(weights[p] for p in g) for g in grouped]
+ assert max(loads) - min(loads) <= max(weights.values())
+ assert [b["key"] for b in out] == [f"batch-{i}" for i in range(1, cpb.MAX_BATCHES + 1)]
+
+
+def test_a_batch_name_lists_three_members_then_a_count():
+ paths = [f"tools/m{i:02}" for i in range(cpb.MAX_BATCHES * 5)]
+ first = cpb.batches(paths, dict.fromkeys(paths, 1))[0]
+ assert first["name"].startswith(f"1/{cpb.MAX_BATCHES}: ")
+ assert first["name"].endswith(", +2")
+
+
+def test_the_split_is_deterministic():
+ paths = [f"tools/m{i:02}" for i in range(20)]
+ weights = dict.fromkeys(paths, 5)
+ assert cpb.batches(paths, weights) == cpb.batches(list(reversed(paths)), weights)
+
+
+def test_the_cli_prints_the_matrix(tmp_path, capsys):
+ root = _workspace(tmp_path, {"tools/a": PYTEST, "tools/b": PYTEST})
+ changed = tmp_path / "changed.txt"
+ changed.write_text("tools/b/src/y.py\n")
+ assert cpb.main(["--root", str(root), "--changed-file", str(changed)]) == 0
+ assert json.loads(capsys.readouterr().out) == [
+ {"key": "b", "name": "b", "paths": "tools/b", "packages": "pkg-b"}
+ ]
+
+
+def test_packages_follow_the_paths_of_each_batch(tmp_path, capsys):
+ members = {f"tools/m{i:02}": PYTEST for i in range(cpb.SOLO_MAX + 3)}
+ root = _workspace(tmp_path, members)
+ assert cpb.main(["--root", str(root)]) == 0
+ for batch in json.loads(capsys.readouterr().out):
+ names = [p.rsplit("/", 1)[-1] for p in batch["paths"].split()]
+ assert batch["packages"].split() == [f"pkg-{n}" for n in names]
+
+
+@pytest.mark.skipif(not (REPO_ROOT / "pyproject.toml").is_file(), reason="needs the framework tree")
+def test_the_real_workspace_splits_into_the_capped_batches():
+ paths = cpb.select(REPO_ROOT, None)
+ assert len(paths) > cpb.SOLO_MAX
+ out = cpb.batches(paths, {p: cpb.weight(REPO_ROOT, p) for p in paths})
+ assert len(out) == cpb.MAX_BATCHES
+ assert sorted(p for b in out for p in b["paths"].split()) == sorted(paths)