You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@maka.apache.org by "liuxiaocs7 (via GitHub)" <gi...@apache.org> on 2026/08/26 16:28:20 UTC
[PR] refactor(cli): ship one Eval runtime in the npm CLI package [maka]
liuxiaocs7 opened a new pull request, #3946:
URL: https://github.com/apache/maka/pull/3946
## Problem
`copyEvalMirror()` in `scripts/release-cli-package.mjs` copied the entire staged `node_modules/@maka/eval` tree into `packages/eval` inside the published CLI package, so every install carried the Eval runtime twice and the two copies could diverge between releases.
The mirror is not pure redundancy. Eval containers bind-mount the CLI package root read-only at `/opt/maka-agent` (`packages/cli/src/eval-bundle-path.ts` sets `MAKA_EVAL_MAKA_BUNDLE_PATH` to the package root; the experiment mount resolves in `packages/eval/src/harness-executor.ts`), so `/opt/maka-agent/packages/eval/...` resolves to the mirror. Only a small container-facing subset is actually read from it; the host Eval runtime is read once from `node_modules/@maka/eval`.
## Change
Ship one Eval runtime. `node_modules/@maka/eval` stays the complete runtime; the mirror now carries only the container entry points and the assets those entry points resolve.
### Exact entry points / assets the mirror keeps
**Container entry points** (launched inside eval cells — an experiment's `shimPath` / `args[0]`):
- `dist/harbor-external-subject.js`
- `dist/harbor-maka-subject.js`
**Their transitive `dist/` import closure** — the package is emitted per-module by `tsc`, so each entry point pulls its siblings at container runtime. The closure is walked programmatically from the entry points at release time (not hard-coded), so a new `import` cannot silently break the container. Current closure adds:
- `dist/metering-checkpoint.js`, `dist/toolchain-verification.js`, `dist/provider-admission.js`, `dist/provider-metering.js`, `dist/provider-web-tool-surface.js`, `dist/relay-result-frame.js`, `dist/maka-artifacts.js`, `dist/maka-runtime-policy.js`
**Harbor data / egress assets the entry points resolve:**
- `harbor/deepseek-codex-models.json` (read in-container by `harbor-external-subject.js`)
- `harbor/deepseek-harness-profile/package.json`, `harbor/deepseek-harness-profile/cordis.yml`, `harbor/deepseek-harness-profile/cordis.patch.yml`
- `harbor/docker-compose-egress-proxy.yaml` (host reads via the bundle path)
- `harbor/egress-proxy/network-policy` (host reads via the bundle path)
That is the full kept set: **16 files (10 `dist` + 6 harbor), down from the entire runtime tree.**
### Dropped from the mirror (all remain in `node_modules/@maka/eval` via `releaseFiles`)
- The host-only `dist/*.js` modules (`index`, `cli`, `harness-executor`, `harness-environment`, `external-subject`, `result`, `runner`, `spec`, …) — run on the host from `node_modules`, never loaded in-container.
- `harbor/relay_agent.py`, `harbor/run_trial.py`, `harbor/eval_framework.py` — read on the host from `node_modules/@maka/eval/harbor` via `BUNDLED_HARNESS_RELAY_ROOT`, not through the mount.
- `harbor/egress-proxy/Dockerfile`, `harbor/egress-proxy/entrypoint.sh`, `harbor/egress_filter.py` — CI image-build inputs only; the compose file references the image by tag with no `build:` stanza, so nothing resolves them from the mirror at eval runtime.
## Proof no experiment configuration resolves a mirrored path outside that list
`assertEvalMirrorResolvesEveryReference()` runs during the release build: it scans every `packages/eval/experiments/*.json` and every shipped mirror `*.js` for `packages/eval/...` references (with or without the `/opt/maka-agent` prefix) and fails the build if any resolves to a path the release did not copy. A regression test (`scripts/release-cli-file-policy.test.mjs`) asserts the same over the real experiment configs without needing a build.
Verified against the real Eval source import graph (`tsc` preserves import specifiers): the walk yields exactly the 10-file `dist` closure above, and no mirror-path literal in those modules resolves outside the kept set — the deepseek data references all map to shipped assets.
## Guards moved to match where the runtime resolves them
The `relay_agent.py` / `run_trial.py` existence guards move from the mirror to `node_modules/@maka/eval/harbor/...` (in `release-cli-package.mjs`, `smoke-release-cli-package.mjs`, `release-cli-eval-package.mjs`), because that is where the runtime actually reads them.
`maka eval` remains the only public Eval CLI. Behavior, standalone packaging, and every container path the experiments depend on are unchanged.
Closes #3933
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(cli): ship one Eval runtime in the npm CLI package [maka]
Posted by "liuxiaocs7 (via GitHub)" <gi...@apache.org>.
liuxiaocs7 commented on code in PR #3946:
URL: https://github.com/apache/maka/pull/3946#discussion_r3889628644
##########
scripts/release-cli-package.mjs:
##########
@@ -520,13 +523,8 @@ function pruneNodePtyBuildInputs(destination) {
}
function copyEvalMirror() {
- const stagedEval = join(stageRoot, 'node_modules/@maka/eval');
- const mirror = join(stageRoot, 'packages/eval');
- cpSync(stagedEval, mirror, {
- recursive: true,
- preserveTimestamps: true,
- filter: (path) =>
- path === stagedEval || !relative(stagedEval, path).split(sep).includes('node_modules'),
+ stageEvalMirror(join(stageRoot, 'node_modules/@maka/eval'), join(stageRoot, 'packages/eval'), {
Review Comment:
Done in 78e56a7 — the mirror is gone, one physical runtime remains.
- Repointed every container-facing path from `/opt/maka-agent/packages/eval/...` to `/opt/maka-agent/node_modules/@maka/eval/...`: the two container entry points (an experiment's `args[0]` / `shimPath`), the codex model catalog and the deepseek-harness profile the external subject reads, and the egress compose / network-policy resolved from the bundle. `eval-bundle-path.ts` keys its installed-CLI guard on `node_modules/@maka/eval`.
- Deleted `copyEvalMirror` / `stageEvalMirror`, the acorn import-closure and path analyzer, the mirror inventory and the packed-tarball reconciliation, the `acorn` devDependency, and the mirror synchronization tests.
- The release packs only `node_modules/@maka/eval` (no `packages/eval` copy); `validateStaging` and the installed smoke test assert the container entry point, `relay_agent.py`, the egress compose, and the network-policy ship under `node_modules/@maka/eval`.
No AST walker, no second inventory, no split-string proof gap — there's nothing left to under-approximate. Thanks for pushing on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(cli): ship one Eval runtime in the npm CLI package [maka]
Posted by "liuxiaocs7 (via GitHub)" <gi...@apache.org>.
liuxiaocs7 commented on code in PR #3946:
URL: https://github.com/apache/maka/pull/3946#discussion_r3889537617
##########
scripts/release-cli-package.mjs:
##########
@@ -520,13 +523,8 @@ function pruneNodePtyBuildInputs(destination) {
}
function copyEvalMirror() {
- const stagedEval = join(stageRoot, 'node_modules/@maka/eval');
- const mirror = join(stageRoot, 'packages/eval');
- cpSync(stagedEval, mirror, {
- recursive: true,
- preserveTimestamps: true,
- filter: (path) =>
- path === stagedEval || !relative(stagedEval, path).split(sep).includes('node_modules'),
+ stageEvalMirror(join(stageRoot, 'node_modules/@maka/eval'), join(stageRoot, 'packages/eval'), {
Review Comment:
Agreed — I'll take this all the way to a single physical runtime in this PR rather than shipping a validated partial mirror.
I traced the consumers of `/opt/maka-agent/packages/eval/...` and they're all internal and checked-in: the four `terminal-bench-2.1-*` experiment configs, plus two source constants (`external-subject.ts`'s `BUNDLED_EXTERNAL_WRAPPER` and the harbor asset paths in `harbor-external-subject.ts`). No deployed or third-party consumer of that path exists, so there's no external compatibility obligation to preserve it.
And `node_modules/@maka/eval` already ships everything the container reads — `releaseFiles` includes the whole `dist/` (both `harbor-external-subject.js` and `harbor-maka-subject.js`) and the harbor assets (`deepseek-codex-models.json`, `deepseek-harness-profile`, the egress files). So the mirror is pure redundancy.
Plan for this PR:
- Repoint the four experiment configs and the two source constants to `/opt/maka-agent/node_modules/@maka/eval/...`.
- Delete `stageEvalMirror` and the mirror staging in the release script, the `acorn` dependency, the AST closure/path analyzer, and the mirror synchronization tests.
- The runtime then ships as one physical copy under `node_modules/@maka/eval` (via `releaseFiles`), covered by the existing release-file checks.
That also closes the proof gap you flagged (a path split across string expressions): with no partial mirror to validate, there's nothing to under-approximate. Will push the update shortly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(cli): ship one Eval runtime in the npm CLI package [maka]
Posted by "Astro-Han (via GitHub)" <gi...@apache.org>.
Astro-Han commented on code in PR #3946:
URL: https://github.com/apache/maka/pull/3946#discussion_r3886266541
##########
scripts/release-cli-package.mjs:
##########
@@ -520,13 +523,8 @@ function pruneNodePtyBuildInputs(destination) {
}
function copyEvalMirror() {
- const stagedEval = join(stageRoot, 'node_modules/@maka/eval');
- const mirror = join(stageRoot, 'packages/eval');
- cpSync(stagedEval, mirror, {
- recursive: true,
- preserveTimestamps: true,
- filter: (path) =>
- path === stagedEval || !relative(stagedEval, path).split(sep).includes('node_modules'),
+ stageEvalMirror(join(stageRoot, 'node_modules/@maka/eval'), join(stageRoot, 'packages/eval'), {
Review Comment:
[P2] Remove the mirror instead of proving a partial mirror
The package root is already mounted at `/opt/maka-agent`, and the canonical Eval runtime is already present at `node_modules/@maka/eval`. Could we update the checked-in experiment configurations and internal path constants to use `/opt/maka-agent/node_modules/@maka/eval/...` directly, then delete the mirror, `stageEvalMirror`, the Acorn dependency, the custom closure/path analyzer, and the associated synchronization tests?
The current approach reduces tarball size but retains two physical representations and adds an AST walker, path scanning, multiple inventories, and more than 700 lines of policy tests. It also remains incomplete as a proof: a path split across string expressions is accepted without the referenced asset being shipped.
If `/opt/maka-agent/packages/eval/...` has a deployed or third-party compatibility consumer, that obligation should be identified explicitly. Otherwise preserving this internal path appears to be the source of the added entropy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(cli): ship one Eval runtime in the npm CLI package [maka]
Posted by "Astro-Han (via GitHub)" <gi...@apache.org>.
Astro-Han merged PR #3946:
URL: https://github.com/apache/maka/pull/3946
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org