You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "SEZ9 (via GitHub)" <gi...@apache.org> on 2026/07/25 08:39:44 UTC
[PR] [Feature][CLI] Add accuracy benchmark for the AI CLI (100 tasks, layered verdict gates, repair-loop measurement) [seatunnel]
SEZ9 opened a new pull request, #11553:
URL: https://github.com/apache/seatunnel/pull/11553
### Purpose of this pull request
Implements #11549 — adds a standalone accuracy benchmark for the AI CLI under `seatunnel-cli/benchmark/`. All new files; no product code changes.
The benchmark answers a question the project previously could not answer quantitatively: **when the CLI's prompts, connector metadata, or repair logic change, does config-generation accuracy actually improve — and by how much, for which models?**
**What's included** (19 files, ~4,000 lines):
| Component | Files | Role |
|-----------|-------|------|
| Task suite | `tasks/tier{1,2,3}_*.json` | 100 declarative tasks: 20 simple / 45 medium / 35 complex-DAG, 12 ETL categories, 10 Chinese prompts, 18 rule probes targeting known LLM failure modes |
| L1 static gate | `scoring.py` | pyhocon parse + connector metadata + per-task assertions (offline) |
| L2/L3 gates | `execution.py` | engine `--dry-run static`; real job execution on the official `apache/seatunnel` image (batch: exit code; streaming: 60s health) |
| Data environment | `docker/` | docker-compose with MySQL/PostgreSQL/Kafka/ClickHouse/Elasticsearch/MinIO + optional Doris/StarRocks profile, seeded schemas/topics |
| Runner | `runner.py` | model matrix (`models.json` or `--provider/--model/--api-key`), repair loop driven by the failing gate's real errors, `--trials` for pass^k, CLI version + commit stamping |
| Reports | `report.py` | layer funnel, repair curve (pass@1 / pass@≤k), per-tier & per-category matrices → console + summary.md/csv + results.json |
| Entry point | `run_benchmark.sh` | one command: dependency install → environment preflight → run → report |
| Tests | `tests/test_benchmark_scoring.py` | 12 tests over the L1 scorer and task-suite integrity |
**Design guarantees**:
- Gates mirror the CLI's own check → dry-run → run pipeline and degrade gracefully (missing engine/Docker disables deeper gates per task, never fails the run).
- All verdicts are deterministic (exit codes, liveness, regex assertions) — no LLM-as-judge.
- Credentials are environment variables only, never written to files (ASF compliance); `results/` and `models.json` are git-ignored.
- Reports are stamped with the CLI version + git commit, so any CLI change can be A/B measured on identical tasks.
**Track record**: runs of this benchmark found and isolated #11508 (fixed in #11509), #11510 (image-level opengauss-jdbc driver conflict — workaround documented inline and applied identically for every model), and #11511. Measured results for 7 models are documented on the website (AI CLI → Model Benchmark, #11551).
### Does this PR introduce _any_ user-facing change?
No changes to shipped behavior. Adds an optional developer tool under `seatunnel-cli/benchmark/` with its own README; two `.gitignore` entries for runtime artifacts.
### How was this patch tested?
- `tests/test_benchmark_scoring.py`: 12 tests covering the L1 scorer (structure checks, connector matching, assertion evaluation, credential-placeholder handling) and task-suite integrity (100 tasks load, unique ids, compilable regexes, service references match the compose file). Full seatunnel-cli suite: 54 passed.
- The extracted PR tree was verified self-contained: tests pass and all 100 tasks load from a clean checkout with no repo-local dependencies.
- End-to-end: the harness has executed ~1,150 task evaluations across 7 Bedrock models (L1) and 3 models through the full L1→L3 pipeline with real job execution and repair loops — the published numbers in #11551 were produced by exactly this code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][CLI] Add accuracy benchmark for the AI CLI (100 tasks, layered verdict gates, repair-loop measurement) [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo merged PR #11553:
URL: https://github.com/apache/seatunnel/pull/11553
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][CLI] Add accuracy benchmark for the AI CLI (100 tasks, layered verdict gates, repair-loop measurement) [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11553:
URL: https://github.com/apache/seatunnel/pull/11553#issuecomment-5119976863
@yzeng1618 @davidzollo Both of you called the same core defect and you were right — my previous round only *warned* about skipped gates without excluding them from the metrics. Fixed in head `8cef011d5`:
1. **Skipped gates excluded from all success metrics** (both #1): `evaluate_layers` now computes `full_gate_passed` = all requested gates *executed and* passed, exactly the split davidzollo proposed (`all_passed` survives only as repair-loop control — a skipped gate isn't repairable, but it's no longer a success either). `first_pass_round` is set solely from `full_gate_passed`, so pass@1 / pass@≤k / pass^k, per-tier/category rates, and the CSV `passed` column all inherit the strict semantics. The new regression (`test_skipped_gate_excluded_from_pass_metrics`) runs the full `run_task_with_repairs` path with a skipped L3 and asserts `first_pass_round` stays `None`, `full_gate_passed` is false, and no repair round is spent on it. README metric definitions updated to state the exclusion explicitly.
2. **Per-task fault isolation + env hygiene** (both #2): an unexpected harness exception (scoring crash, setup-file I/O, subprocess construction) now becomes one structured failed trial carrying the traceback in `generation_error` instead of aborting the model run, and `restore_env` moved into a `finally` so `AI_PROVIDER`/model variables are restored even when a run dies mid-model.
3. **`git diff --check`** (davidzollo #3): trailing whitespace removed.
Suite: 50 passed. Scope kept to the three findings — no metric-pipeline re-architecture beyond the pass-predicate change you asked for.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][CLI] Add accuracy benchmark for the AI CLI (100 tasks, layered verdict gates, repair-loop measurement) [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11553:
URL: https://github.com/apache/seatunnel/pull/11553#issuecomment-5094167828
All seven findings addressed in head `3e4299b` (rebased onto the branch's dev merge):
1. **Credential exposure**: README quick starts rewritten to provider-standard environment variables; `--api-key` remains accepted but is explicitly documented as suitable only for throwaway keys. Agreed the examples were teaching a bad habit.
2. **Wrapper defects**: `prev_arg` initialized (the `set -u` crash was deterministic on every run — good catch); the Docker prompt now runs only when stdin is a TTY, otherwise it prints the manual compose command and continues. `bash -n` plus a simulated no-TTY run verified.
3. **Repair-round semantics** — *documented rather than changed, deliberately*: this benchmark measures the CLI product, and the CLI's generation pipeline includes its internal validate/fix loop; "round 0 = the CLI's first delivered config" is the intended unit of measurement (it's what a user receives). Your underlying point stands though: that contract was implicit and the internal rounds were unobservable. Now the README states the unit explicitly, and the internal fix rounds are counted via the Orchestrator's status callback and recorded per trial as `internal_repair_rounds` in results.json and summary.csv — both timelines are fully observable without touching product code.
4. **Gate-skip accounting**: `evaluate_layers` now records `skipped_layers` + `all_gates_executed` per trial; requested-but-skipped layers are never treated as passes, the CSV carries both fields, and the console summary prints a per-model warning with the count of incomplete-coverage trials, so runs with different coverage can't be silently compared. Unit tests cover skip-vs-fail semantics.
5. **L3 semantic scope** — *acknowledged as a documented limitation*: batch exit code (+ `verify` probes where defined) and streaming health catch "doesn't run", not full output equivalence. The README now states this scope explicitly and names per-task `verify` extension as the intended path. Retrofitting deterministic semantic probes to all 96 remaining tasks is real work we'd prefer to do incrementally in follow-ups rather than inflate this PR.
6. **PostgreSQL port mapping**: confirmed — the compose host remap to 15432 broke the local-engine path for 15 tasks. The local backend now rewrites canonical `localhost:5432` endpoints to the remapped host port (mirroring the docker backend's hostname rewrite), and the README service table documents both forms.
7. **Docs/tests**: aggregation-denominator docstrings added; 4 new unit tests (`tests/test_benchmark_gates.py`) for the skip/coverage semantics. Full suite: 49 passed.
Scope note: fixes are strictly limited to the findings — no re-architecture of the metric pipeline. Items 3 and 5 are contract-documentation resolutions; if the community prefers a "raw model single-shot" mode as well, that's a clean follow-up flag (`--no-internal-repair` would need a product-side switch, which is out of scope here).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org