You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by li...@apache.org on 2026/07/31 22:48:09 UTC

(seatunnel) branch dev updated: [Docs][CLI] Add "Running the Benchmark Yourself" section to the AI CLI benchmark pages (#11605)

This is an automated email from the ASF dual-hosted git repository.

davidzollo pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/seatunnel.git


The following commit(s) were added to refs/heads/dev by this push:
     new c299232132 [Docs][CLI] Add "Running the Benchmark Yourself" section to the AI CLI benchmark pages (#11605)
c299232132 is described below

commit c299232132e358d5425a12e405e4eee99b5c2366
Author: SEZ <94...@qq.com>
AuthorDate: Sat Aug 1 06:48:02 2026 +0800

    [Docs][CLI] Add "Running the Benchmark Yourself" section to the AI CLI benchmark pages (#11605)
    
    Co-authored-by: SEZ9 <20...@users.noreply.github.com>
---
 docs/en/ai-cli/benchmark.md | 35 +++++++++++++++++++++++++++++++++++
 docs/zh/ai-cli/benchmark.md | 32 ++++++++++++++++++++++++++++++++
 2 files changed, 67 insertions(+)

diff --git a/docs/en/ai-cli/benchmark.md b/docs/en/ai-cli/benchmark.md
index f8a20fb11b..24aee2441a 100644
--- a/docs/en/ai-cli/benchmark.md
+++ b/docs/en/ai-cli/benchmark.md
@@ -77,3 +77,38 @@ These clusters are engineering targets, not permanent limits: they are being add
 ## Repair Loop: Measured Effectiveness
 
 Feeding **real engine errors** back to the repair agent recovered 47% of runtime failures (top-3 models combined). The same model repairs structured validation errors at ~2× the rate of raw Java stack traces — evidence that structured error parsing, not a smarter model, is the highest-leverage next improvement for the repair loop.
+
+## Running the Benchmark Yourself
+
+The benchmark harness ships in the main repository under
+[`seatunnel-cli/benchmark/`](https://github.com/apache/seatunnel/tree/dev/seatunnel-cli/benchmark) —
+100 declarative tasks, the layered verdict gates, the Docker data
+environment, and the report generator.
+
+```bash
+cd seatunnel-cli
+
+# Credentials via provider-standard environment variables
+export OPENAI_API_KEY=sk-...          # or ANTHROPIC_API_KEY / AWS credentials
+
+# One command: installs deps, preflights the environment, runs, reports
+./benchmark/run_benchmark.sh --provider openai --model gpt-4o
+
+# Multi-model comparison
+./benchmark/run_benchmark.sh --models benchmark/models.json
+
+# Optional deeper gates:
+#   L2 (engine dry-run)   — set SEATUNNEL_HOME to a dev-branch build
+#   L3 (real execution)   — docker compose -f benchmark/docker/docker-compose.yml up -d --wait
+```
+
+Missing infrastructure degrades gracefully: without an engine or Docker you
+get a static-gate (L1) report; trials whose requested gates could not execute
+are excluded from every pass metric and flagged in the summary, so results
+from differently-equipped machines are never silently compared.
+
+Every report is stamped with the CLI version and git commit under test —
+rerun the same model on a new CLI build to measure the impact of any prompt,
+metadata, or repair-logic change on identical tasks. See
+[`benchmark/README.md`](https://github.com/apache/seatunnel/blob/dev/seatunnel-cli/benchmark/README.md)
+for the full methodology, task-suite layout, and metric definitions.
diff --git a/docs/zh/ai-cli/benchmark.md b/docs/zh/ai-cli/benchmark.md
index 96d3b1ec41..58ad22c7e6 100644
--- a/docs/zh/ai-cli/benchmark.md
+++ b/docs/zh/ai-cli/benchmark.md
@@ -77,3 +77,35 @@ AI CLI 的准确率靠实测而非假设:专门的基准测试包含 100 个
 ## 修复循环的实测效果
 
 把**真实引擎报错**喂回修复 Agent,可挽回 47% 的运行时失败(前三名模型合计)。同一个模型修复结构化校验错误的成功率约为修复原始 Java 堆栈的 2 倍——证明修复循环下一步最有杠杆的改进是结构化错误解析,而不是更聪明的模型。
+
+## 自己运行基准测试
+
+基准测试工程随主仓库发布,位于
+[`seatunnel-cli/benchmark/`](https://github.com/apache/seatunnel/tree/dev/seatunnel-cli/benchmark)——
+包含 100 个声明式任务、分层判定门、Docker 数据环境和报告生成器。
+
+```bash
+cd seatunnel-cli
+
+# 凭证走各提供商的标准环境变量
+export OPENAI_API_KEY=sk-...          # 或 ANTHROPIC_API_KEY / AWS 凭证
+
+# 一条命令:装依赖、预检环境、运行、出报告
+./benchmark/run_benchmark.sh --provider openai --model gpt-4o
+
+# 多模型对比
+./benchmark/run_benchmark.sh --models benchmark/models.json
+
+# 可选的更深判定门:
+#   L2(引擎 dry-run)  — 设置 SEATUNNEL_HOME 指向 dev 分支构建
+#   L3(真实执行)      — docker compose -f benchmark/docker/docker-compose.yml up -d --wait
+```
+
+环境缺失时自动降级:没有引擎或 Docker 时输出静态门(L1)报告;请求了但
+无法执行的判定门对应的测试轮次会被排除在所有通过率指标之外并在摘要中
+标注,不同配置的机器之间不会被静默比较。
+
+每份报告都标记被测 CLI 的版本与 git commit——在新的 CLI 构建上重跑同一
+模型,即可在相同任务集上量化任何 prompt、元数据或修复逻辑改动的效果。
+完整方法论、任务集结构与指标定义见
+[`benchmark/README.md`](https://github.com/apache/seatunnel/blob/dev/seatunnel-cli/benchmark/README.md)。