You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@maka.apache.org by as...@apache.org on 2026/09/01 02:16:03 UTC
(maka) branch main updated: docs(eval): publish the nine-arm Terminal-Bench 2.1 leaderboard
This is an automated email from the ASF dual-hosted git repository.
Astro-Han pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/maka.git
The following commit(s) were added to refs/heads/main by this push:
new 279c2e8571 docs(eval): publish the nine-arm Terminal-Bench 2.1 leaderboard
279c2e8571 is described below
commit 279c2e8571c1ced5563e02d57814af5e6c50cacc
Author: Haoqing Wang <78...@users.noreply.github.com>
AuthorDate: Tue Sep 1 10:15:57 2026 +0800
docs(eval): publish the nine-arm Terminal-Bench 2.1 leaderboard
Publishes the final nine-harness comparison on DeepSeek V4 Flash at max
reasoning effort, with an 89-task accepted-outcome CSV. Extends the
four-arm report with OpenCode, Kimi Code, ZCode, Pi, and DeepSeek Harness.
Codex leads at 82.0% pass@1; Maka follows at 77.5% using 28% fewer tokens.
All nine harnesses pass the same 28 tasks and five fail on all nine.
The original eight arms ran as one cohort; DSH is a configuration-aligned
later run, disclosed as descriptive rather than simultaneously paired.
Generated-by: Codex
---
...rminal-bench-2.1-deepseek-v4-flash-nine-arm.csv | 90 ++++++++++
...erminal-bench-2.1-deepseek-v4-flash-nine-arm.md | 193 +++++++++++++++++++++
2 files changed, 283 insertions(+)
diff --git a/docs/eval/terminal-bench-2.1-deepseek-v4-flash-nine-arm.csv b/docs/eval/terminal-bench-2.1-deepseek-v4-flash-nine-arm.csv
new file mode 100644
index 0000000000..c9206b87b3
--- /dev/null
+++ b/docs/eval/terminal-bench-2.1-deepseek-v4-flash-nine-arm.csv
@@ -0,0 +1,90 @@
+task,codex,maka,pi,dsh,zcode,reasonix,opencode,kimi-code,claude-code
+adaptive-rejection-sampler,pass,pass,pass,fail,fail,fail,pass,fail,fail
+bn-fit-modify,fail,fail,pass,pass,pass,pass,fail,fail,pass
+break-filter-js-from-html,pass,pass,pass,pass,pass,pass,pass,pass,pass
+build-cython-ext,pass,pass,pass,pass,pass,pass,pass,pass,fail
+build-pmars,pass,pass,pass,pass,pass,pass,pass,pass,fail
+build-pov-ray,pass,pass,pass,pass,fail,pass,fail,pass,fail
+caffe-cifar-10,pass,fail,pass,fail,pass,fail,pass,fail,fail
+cancel-async-tasks,fail,pass,fail,pass,pass,pass,fail,fail,pass
+chess-best-move,fail,fail,fail,pass,pass,fail,fail,fail,fail
+circuit-fibsqrt,pass,pass,pass,pass,pass,fail,pass,pass,pass
+cobol-modernization,pass,pass,pass,pass,pass,pass,pass,pass,pass
+code-from-image,pass,pass,pass,pass,pass,pass,fail,fail,fail
+compile-compcert,pass,pass,pass,pass,pass,pass,fail,fail,pass
+configure-git-webserver,pass,pass,pass,pass,pass,fail,pass,fail,pass
+constraints-scheduling,pass,pass,pass,pass,pass,pass,pass,pass,pass
+count-dataset-tokens,pass,pass,pass,pass,pass,pass,pass,pass,pass
+crack-7z-hash,pass,pass,pass,pass,pass,pass,pass,fail,fail
+custom-memory-heap-crash,pass,pass,pass,pass,pass,pass,pass,pass,pass
+db-wal-recovery,pass,fail,pass,fail,pass,pass,pass,fail,pass
+distribution-search,pass,pass,pass,pass,pass,pass,pass,pass,pass
+dna-assembly,pass,pass,fail,fail,fail,fail,fail,fail,fail
+dna-insert,pass,fail,pass,pass,fail,fail,fail,fail,fail
+extract-elf,pass,fail,fail,fail,pass,fail,fail,fail,pass
+extract-moves-from-video,fail,fail,fail,fail,fail,fail,fail,fail,fail
+feal-differential-cryptanalysis,pass,pass,pass,pass,pass,pass,pass,pass,pass
+feal-linear-cryptanalysis,pass,pass,pass,pass,pass,fail,pass,pass,pass
+filter-js-from-html,fail,fail,fail,fail,fail,fail,fail,fail,fail
+financial-document-processor,pass,pass,pass,pass,pass,pass,pass,fail,pass
+fix-code-vulnerability,pass,pass,pass,pass,pass,pass,pass,pass,pass
+fix-git,pass,pass,pass,pass,pass,pass,pass,pass,pass
+fix-ocaml-gc,pass,pass,pass,pass,pass,pass,pass,pass,pass
+gcode-to-text,fail,fail,fail,fail,fail,fail,fail,fail,fail
+git-leak-recovery,pass,pass,pass,pass,pass,pass,pass,pass,fail
+git-multibranch,pass,pass,pass,pass,pass,pass,pass,pass,pass
+gpt2-codegolf,pass,fail,fail,fail,fail,fail,fail,fail,fail
+headless-terminal,pass,pass,pass,pass,pass,pass,pass,pass,pass
+hf-model-inference,pass,pass,pass,pass,pass,pass,pass,pass,pass
+install-windows-3.11,pass,pass,fail,fail,fail,fail,fail,fail,fail
+kv-store-grpc,pass,pass,pass,fail,fail,fail,pass,pass,fail
+large-scale-text-editing,pass,pass,pass,pass,pass,pass,pass,pass,pass
+largest-eigenval,pass,pass,fail,pass,fail,fail,pass,fail,fail
+llm-inference-batching-scheduler,pass,pass,pass,pass,pass,pass,pass,pass,pass
+log-summary-date-ranges,pass,pass,pass,pass,pass,pass,pass,pass,pass
+mailman,pass,pass,pass,fail,pass,pass,fail,pass,fail
+make-doom-for-mips,fail,fail,fail,fail,fail,fail,fail,fail,fail
+make-mips-interpreter,pass,fail,fail,pass,fail,fail,fail,fail,fail
+mcmc-sampling-stan,pass,pass,pass,pass,pass,pass,pass,fail,fail
+merge-diff-arc-agi-task,pass,pass,pass,pass,pass,pass,pass,pass,fail
+model-extraction-relu-logits,fail,pass,fail,pass,pass,pass,fail,pass,fail
+modernize-scientific-stack,pass,pass,pass,pass,pass,pass,pass,pass,pass
+mteb-leaderboard,pass,pass,fail,pass,pass,pass,pass,pass,pass
+mteb-retrieve,pass,pass,pass,pass,pass,pass,fail,fail,pass
+multi-source-data-merger,pass,pass,pass,pass,pass,pass,pass,pass,pass
+nginx-request-logging,pass,pass,pass,pass,pass,pass,pass,pass,fail
+openssl-selfsigned-cert,pass,pass,pass,pass,pass,pass,pass,pass,pass
+overfull-hbox,pass,fail,pass,pass,pass,fail,pass,fail,pass
+password-recovery,pass,pass,pass,fail,pass,fail,pass,pass,pass
+path-tracing,pass,pass,pass,pass,pass,pass,pass,fail,fail
+path-tracing-reverse,pass,pass,pass,pass,fail,fail,fail,fail,pass
+polyglot-c-py,pass,pass,pass,pass,pass,pass,pass,pass,pass
+polyglot-rust-c,pass,pass,pass,pass,pass,pass,pass,pass,pass
+portfolio-optimization,pass,pass,pass,pass,pass,pass,pass,pass,pass
+protein-assembly,fail,fail,fail,fail,fail,pass,fail,fail,pass
+prove-plus-comm,pass,pass,pass,pass,pass,pass,pass,pass,pass
+pypi-server,fail,pass,pass,fail,fail,pass,pass,pass,pass
+pytorch-model-cli,pass,pass,pass,pass,pass,pass,pass,fail,fail
+pytorch-model-recovery,pass,pass,fail,fail,pass,pass,fail,pass,pass
+qemu-alpine-ssh,fail,pass,fail,fail,fail,pass,pass,pass,pass
+qemu-startup,pass,pass,pass,fail,fail,fail,pass,pass,pass
+query-optimize,pass,pass,fail,pass,fail,pass,pass,pass,fail
+raman-fitting,fail,fail,fail,fail,pass,fail,fail,fail,fail
+regex-chess,pass,pass,pass,pass,fail,fail,fail,fail,fail
+regex-log,pass,pass,pass,pass,pass,pass,pass,pass,pass
+reshard-c4-data,pass,pass,pass,pass,pass,pass,pass,pass,pass
+rstan-to-pystan,pass,pass,pass,fail,pass,pass,fail,pass,fail
+sam-cell-seg,pass,pass,pass,pass,pass,fail,pass,fail,fail
+sanitize-git-repo,fail,pass,pass,pass,fail,pass,fail,pass,fail
+schemelike-metacircular-eval,pass,pass,pass,pass,pass,pass,pass,pass,fail
+sparql-university,pass,pass,pass,pass,pass,pass,pass,pass,pass
+sqlite-db-truncate,pass,pass,pass,pass,pass,pass,pass,pass,pass
+sqlite-with-gcov,pass,pass,pass,pass,pass,pass,pass,pass,fail
+torch-pipeline-parallelism,fail,fail,fail,fail,fail,fail,fail,fail,fail
+torch-tensor-parallelism,pass,fail,pass,pass,fail,fail,fail,pass,pass
+train-fasttext,fail,fail,fail,pass,fail,fail,fail,fail,fail
+tune-mjcf,pass,pass,fail,pass,fail,pass,fail,fail,fail
+video-processing,fail,fail,fail,fail,fail,fail,pass,fail,fail
+vulnerable-secret,pass,pass,pass,pass,pass,pass,pass,pass,pass
+winning-avg-corewars,pass,pass,pass,pass,pass,pass,fail,pass,pass
+write-compressor,pass,fail,pass,fail,pass,pass,pass,pass,fail
diff --git a/docs/eval/terminal-bench-2.1-deepseek-v4-flash-nine-arm.md b/docs/eval/terminal-bench-2.1-deepseek-v4-flash-nine-arm.md
new file mode 100644
index 0000000000..3116fa5be9
--- /dev/null
+++ b/docs/eval/terminal-bench-2.1-deepseek-v4-flash-nine-arm.md
@@ -0,0 +1,193 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Terminal-Bench 2.1 — DeepSeek V4 Flash nine-harness comparison
+
+This report records the final selected outcomes for nine coding-agent harnesses around the same
+DeepSeek V4 Flash model on all 89 Terminal-Bench 2.1 tasks. It extends the earlier four-arm report
+in [#2208](https://github.com/maka-agent/maka-agent/pull/2208) with OpenCode, Kimi Code, ZCode, Pi,
+and DeepSeek Harness (DSH).
+
+**Benchmark:** Terminal-Bench 2.1, revision
+`d49e28f1e4ddd13d289e85a5f312a66750951932`
+
+**Model:** `deepseek-v4-flash`
+
+**Reasoning effort:** `max`
+
+**Metric:** end-to-end pass@1 by the official task verifier
+
+**Per-task outcomes:**
+[`terminal-bench-2.1-deepseek-v4-flash-nine-arm.csv`](./terminal-bench-2.1-deepseek-v4-flash-nine-arm.csv)
+
+## TL;DR
+
+- **Codex leads at 73/89 (82.0%)**, followed by Maka at 69/89 (77.5%), Pi at 66/89
+ (74.2%), and DSH at 65/89 (73.0%).
+- All nine harnesses pass the same 28 tasks. Five tasks fail on every harness:
+ `extract-moves-from-video`, `filter-js-from-html`, `gcode-to-text`,
+ `make-doom-for-mips`, and `torch-pipeline-parallelism`.
+- Pi records the lowest observed cost per pass at $0.019739. DSH and Maka are nearly tied on this
+ measure at $0.026230 and $0.026307.
+- The original eight arms ran as one cohort. DSH ran later under the same benchmark, model,
+ reasoning, tool-egress, permission, verifier, and task-native timeout contracts, but not in the
+ same simultaneous cohort. The nine-way ranking is therefore descriptive rather than a paired
+ causal comparison.
+
+## Leaderboard
+
+The reporting projection treats the token, cache, and cost values recorded by the selected attempts
+as authoritative and does not extrapolate unobserved provider usage.
+
+| Rank | Harness | Passed | Pass@1 | Tokens | Average/task | Cache rate | Cost |
+| ---: | --- | ---: | ---: | ---: | ---: | ---: | ---: |
+| 1 | Codex | 73/89 | **82.0%** | 259.62M | 2.92M | 98.6% | $2.105 |
+| 2 | Maka | 69/89 | **77.5%** | 185.71M | 2.09M | 98.8% | $1.815 |
+| 3 | Pi | 66/89 | **74.2%** | 131.40M | 1.48M | 98.5% | $1.303 |
+| 4 | DSH | 65/89 | **73.0%** | 182.84M | 2.05M | 98.6% | $1.705 |
+| 5 | ZCode | 63/89 | **70.8%** | 223.46M | 2.51M | 98.9% | $2.287 |
+| 6 | Reasonix | 60/89 | **67.4%** | 216.17M | 2.43M | 99.0% | $1.935 |
+| 7 | OpenCode | 58/89 | **65.2%** | 166.44M | 1.87M | 98.6% | $1.745 |
+| 8 | Kimi Code | 53/89 | **59.6%** | 193.75M | 2.18M | 98.2% | $2.162 |
+| 9 | Claude Code | 49/89 | **55.1%** | 247.60M | 2.78M | 98.9% | $2.633 |
+
+The 26.9-point spread between Codex and Claude Code is 24 tasks. Every task is worth approximately
+1.12 percentage points on this fixed suite.
+
+## Outcome agreement
+
+| Agreement class | Tasks |
+| --- | ---: |
+| All nine pass | 28 |
+| All nine fail | 5 |
+| Mixed outcome | 56 |
+
+The per-task CSV is the authoritative projection for these counts. It contains exactly 89 unique
+task ids and reproduces every harness total in the leaderboard.
+
+## Outcome-normalized economics
+
+Cost per pass includes the recorded cost of both successful and failed selected cells.
+
+| Harness | Recorded cost | Passed | Cost/pass |
+| --- | ---: | ---: | ---: |
+| Pi | $1.303 | 66 | **$0.019739** |
+| DSH | $1.705 | 65 | **$0.026230** |
+| Maka | $1.815 | 69 | **$0.026307** |
+| Codex | $2.105 | 73 | **$0.028832** |
+| OpenCode | $1.745 | 58 | **$0.030084** |
+| Reasonix | $1.935 | 60 | **$0.032252** |
+| ZCode | $2.287 | 63 | **$0.036303** |
+| Kimi Code | $2.162 | 53 | **$0.040798** |
+| Claude Code | $2.633 | 49 | **$0.053741** |
+
+These are descriptive point estimates from one selected attempt per cell. They are not billing
+invoices or statistical cost-equivalence claims.
+
+## Frozen execution contract
+
+| Dimension | Value |
+| --- | --- |
+| Benchmark | Terminal-Bench 2.1; 89 explicit tasks at revision `d49e28f1e4ddd13d289e85a5f312a66750951932` |
+| Model | `deepseek-v4-flash` on every arm |
+| Reasoning | `max`; provider thinking enabled |
+| Repetitions | 1 |
+| Verifier | Official task-native Terminal-Bench verifier |
+| Deadline | Each task's native agent timeout ×1 |
+| Max steps | 100,000 |
+| Web tools | Removed from the provider-visible tool surface |
+| Shell networking | Enabled, with benchmark-contamination egress filtering |
+| Permissions | Non-interactive benchmark execution |
+| Selection | Valid scored attempts; authorized infrastructure replacements and recorded metric recovery |
+
+The harnesses retain their native prompts, tool schemas, context management, process lifecycle, and
+execution loops. The result compares complete harness systems around one model; it is not a
+single-component ablation.
+
+## Outcome accounting and recovery
+
+The eight-arm result was assembled from the full cohort plus bounded infrastructure, usage, and
+trajectory recovery runs. The final eight-arm trajectory manifest records 712/712 cells with a
+physical trajectory artifact and no missing selected cell.
+
+DSH initially recorded 61/89. Real-machine inspection identified three Eval-specific defects:
+
+1. A five-minute DSH Bash timeout interrupted package installation and left `dpkg` unusable for
+ verification.
+2. Package installation inherited an interactive `tzdata` configuration path instead of the
+ benchmark's non-interactive execution contract.
+3. DSH and the Eval relay terminated task-created background services before the shared-environment
+ verifier could reach them.
+
+Commit `28ebe0949` deferred Bash lifetime to the benchmark deadline, made package installation
+non-interactive, and preserved background descendants only after successful DSH exit. Seven
+affected cells were rerun. Four changed from fail to pass:
+
+- `hf-model-inference`
+- `merge-diff-arc-agi-task`
+- `polyglot-c-py`
+- `regex-log`
+
+Three remained genuine verifier failures:
+
+- `dna-assembly`
+- `extract-moves-from-video`
+- `torch-pipeline-parallelism`
+
+The repaired DSH result is 65/89.
+
+## Caveats
+
+- This is one repetition over one fixed suite.
+- DSH was a configuration-aligned follow-up run, not a ninth arm started simultaneously with the
+ original eight-arm cohort.
+- Some persisted trajectories hit the Eval recording limit. This affects replay completeness, not
+ the official verifier score used here.
+- The report uses the recorded token, cache, and cost aggregates directly, without replacing them
+ with projections.
+- Recovery admissions can sample a different model trajectory. The CSV freezes the selected final
+ outcomes used by this report.
+
+## Artifact pointers
+
+| Artifact | Path |
+| --- | --- |
+| Original eight-arm run | `/mnt/deepswe/eval-runs/deepseek-v4-flash-eight-arm-full-24-903dd1b6-v1` |
+| Eight-arm trajectory overlay | `/mnt/deepswe/eval-runs/deepseek-v4-flash-final-trajectory-overlay-8a1ab4635.json` |
+| DSH full run | `/mnt/deepswe/eval-runs/deepseek-v4-flash-dsh-eightarm-comparable-full-89-v1` |
+| DSH service repair | `/mnt/deepswe/eval-runs/deepseek-v4-flash-dsh-service-canary-v3-74c98a6ac` |
+| DSH six-cell repair | `/mnt/deepswe/eval-runs/deepseek-v4-flash-dsh-verifier-fix-retry-6-v4-28ebe0949` |
+| DSH Eval bundle | `/mnt/deepswe/maka-eval-dsh-verifier-fix-74c98a6ac` |
+| DSH toolchain | `/mnt/deepswe/toolchains/deepseek-harness-0.1.0-rc.6-bullseye-v3` |
+
+The DSH toolchain fingerprint is
+`sha256:a0882b448718ddfb7b64e33a12369c92b0064baf8388fe08a8ff64fe3dd98896`.
+
+## Integrity and verification
+
+- CSV rows: 89
+- Unique task ids: 89
+- Recomputed pass totals: 73, 69, 66, 65, 63, 60, 58, 53, 49
+- All-nine passes: 28
+- All-nine failures: 5
+- CSV SHA-256:
+ `c27c3bcbfc3ebe8e21cc250dc409f02f49ae055032eaf2f19fd6349986e94e6a`
+- DSH repair code: `28ebe0949`
+- DSH repair verification: Eval Node 36/36; Python relay/policy/artifact 33/33; lint,
+ format, and Eval typecheck passed
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org