You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@maka.apache.org by "amaldevcm (via GitHub)" <gi...@apache.org> on 2026/08/30 17:09:24 UTC
[PR] docs(eval): restructure README with navigation and a troubleshooting section [maka]
amaldevcm opened a new pull request, #4281:
URL: https://github.com/apache/maka/pull/4281
## Summary
`packages/eval/README.md` already existed and was technically accurate, but had zero section headers — one H1 title followed by 27 unbroken paragraphs, with no way to jump to "how do I run an eval" or "what do I do when it fails."
This restructures the file in place:
- Adds a table of contents and 8 section headers over the existing prose (Architecture, Experiment spec format, Running an evaluation, Extending, Relay protocol, Docker harness profiles, Egress audit and security model, Troubleshooting)
- No existing technical content was altered — only reorganized. Every relocated paragraph was diffed against the original to confirm it survived verbatim.
- Adds a spec-format field table sourced from `ExperimentSpec` (`experiment.ts`) and `parseExperimentSpec` (`spec.ts`)
- Adds a new **Extending: custom executors and subjects** section — the `ExperimentExecutor`/`SubjectAdapter` interfaces and the `EvalCliDependencies` override seam (`runner.ts`, `cli.ts`) had no prior documentation at all
- Adds a new **Troubleshooting** section as a symptom → cause → fix table, built from the actual error strings in `install-preflight.ts` and the egress-audit failure modes, not paraphrased
- One-line clarification that `maka eval` is a subcommand of the main `maka` CLI, not a separate binary, matching how `ARCHITECTURE.md` already describes it
Fixes #3895
## Verification
- Diffed the restructured file against the original to confirm every relocated paragraph survived verbatim (`git diff` shows only moves, not content changes)
- Cross-checked every new claim (interfaces, error strings, field list) against the current source: `runner.ts`, `cli.ts`, `spec.ts`, `experiment.ts`, `install-preflight.ts`
- Cross-checked the `maka eval` CLI-wiring claim against `packages/cli/src/cli-core.ts`, and for consistency against the root `README.md` and `ARCHITECTURE.md`, both before and after rebasing onto latest `main`
- `npx biome format` does not process markdown in this repo (confirmed — no formatter to run against this file)
- Not done: did not execute `maka eval run` end-to-end (would need a real spec + provider credentials); verified the command's wiring in source instead
## AI use
Select exactly one:
- [ ] No generative tool made a substantive contribution
- [x] Generative tooling made a substantive contribution
Tool(s) and scope: Claude (Sonnet 5) — investigated the issue, explored the eval package source (dispatched as two parallel research passes covering architecture/CLI and security/egress), designed the restructuring plan, and wrote the changes. `Generated-by: Claude` trailer is on the commit.
## Checklist
- [x] Tests cover the change and fail without it — N/A, documentation-only change
- [x] Lint, format, typecheck and the affected suites pass locally — no code changed; markdown isn't covered by this repo's formatter/typecheck
Does this PR entail a change in behavior?
- [ ] Yes — described under Summary above
- [x] No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] docs(eval): restructure README with navigation and a troubleshooting section [maka]
Posted by "amaldevcm (via GitHub)" <gi...@apache.org>.
amaldevcm commented on code in PR #4281:
URL: https://github.com/apache/maka/pull/4281#discussion_r3890222675
##########
packages/eval/README.md:
##########
@@ -39,18 +81,55 @@ Use `--cell <cell-id>` to replace one failed or indeterminate cell. The attempt
Before starting a trial, the public CLI validates the selected executor's machine paths, bundled relay files, pinned Harbor or Pier Python distribution, and Docker daemon availability for Docker environments. A missing or mismatched prerequisite is reported with the configured environment-variable name and expected framework version; the CLI does not start a trial or install external software. Subject-specific toolchain verification remains part of subject preparation and also completes before any trial starts.
-The built-in Harbor and Pier executors use one relay Agent. The framework prepares the task environment, the relay invokes exactly one Eval subject from `Agent.run()`, and the framework runs its native verifier and finalizer. Harbor and Pier use separate, explicitly versioned Python environments because their Agent and task contracts differ.
+## Extending: custom executors and subjects
+
+An executor decides how one attempt runs; a subject decides how one cell's agent is invoked. Both are pluggable interfaces defined in [`runner.ts:74-130`](src/runner.ts):
+
+```ts
+export interface SubjectAdapter {
+ readonly kind: ExperimentCell['subject']['kind'];
+ validate?(cell: ExperimentCell): void;
+ prepare?(input: { readonly spec: ExperimentSpec; readonly cells: readonly ExperimentCell[] }): Promise<void>;
+ canReuse?(input: { readonly cell: ExperimentCell; readonly attempt: CellAttempt }): boolean;
+ execute(input: { readonly cell: ExperimentCell; readonly context: SubjectExecutionContext }): Promise<SubjectExecutionResult>;
+}
+
+export interface ExperimentExecutor {
+ readonly kind: string;
+ validate?(cell: ExperimentCell): void;
+ runAttempt(
+ input: { readonly cell: ExperimentCell; readonly subjectCredentialNames: readonly string[]; readonly signal?: AbortSignal },
+ operation: (attempt: { readonly context: SubjectExecutionContext; verify(): Promise<ExecutorVerification> }) => Promise<EvalResult>,
+ ): Promise<ExecutorAttemptOutcome>;
+}
+```
-Maka subjects ask the Runtime Host client to run one owned execution in a dedicated Host root. Session, Turn, Goal and continuation semantics remain inside Runtime Host. External subjects declare a command and arguments, and may add non-secret environment values, target-to-source bindings for declared credentials, and an explicit result contract. Omitted credential bindings use declared names unchanged. The generic `exit-code` contract discards unstructured stdout and records null usage and cost. The structured `protocol-v1` contract is restricted to the bundled external wrapper so the shared relay can separate a bounded result frame from Harbor/Pier's merged process output; cohort-specific wrappers do not gain Runtime authority.
+The built-in Harbor and Pier executors are two instantiations of one generic implementation (`createHarborExecutor`/`createPierExecutor`, both delegating to `createHarnessExecutor(framework, config, specPath)` in [`harness-executor.ts:108-121`](src/harness-executor.ts)), parameterized by `HarnessFramework`. The built-in subjects are `createMakaSubjectAdapter()` ([`maka-subject.ts`](src/maka-subject.ts)) and `createExternalSubjectAdapter()` ([`external-subject.ts`](src/external-subject.ts)).
-The result kernel contains only score, normalized usage, attributable cost, duration, status, and artifacts. Specs carry every semantic setting; environment variables are reserved for credentials and machine-local paths.
+To swap in a custom executor or subject set, pass overrides to `runMakaEvalCli` ([`cli.ts:36-42`](src/cli.ts)) rather than the CLI's default wiring:
-The checked-in Terminal-Bench 2.1 four-arm cohort is `experiments/terminal-bench-2.1-deepseek-v4-flash-four-arm.json`. It freezes provider endpoints, framework version, container paths and read-only mount policy. Set each declared machine-path environment variable to its trusted prepared directory, and set the declared API-key credentials. Machine-local paths select artifacts; they do not alter experiment semantics and are not presented as a cryptographic identity scheme.
+```ts
+export interface EvalCliDependencies {
+ readonly loadExecutor: (spec: ExperimentSpec, specPath: string) => ExperimentExecutor;
+ readonly subjects: readonly SubjectAdapter[];
+ readonly signal?: AbortSignal;
+ readonly stdout: (text: string) => void;
+ readonly stderr: (text: string) => void;
+}
+```
+
+Any object implementing `runAttempt`/`validate` (executor) or `execute`/`validate`/`prepare`/`canReuse` (subject) can be supplied this way — `prepare`, `canReuse`, and `validate` are all optional.
+
+## Relay protocol
+
+Harbor/Pier subprocess stdout is merged Docker stdout/stderr that can interleave, so a structured result cannot simply be parsed from the tail of the output. `writeRelayResult` ([`relay-result-frame.ts:24,34-43`](src/relay-result-frame.ts)) emits one bounded (under 2 KB, below Linux `PIPE_BUF`) line — `MAKA-EVAL-RESULT-V1 <token> <bytes> <sha256> <base64url payload>` — so the executor can extract a verified, size-capped result frame out of otherwise-unstructured merged output. `takeRelayResultToken` consumes a one-time environment-variable token (`MAKA_EVAL_RESULT_TOKEN`) so the frame can be authenticated per-attempt.
Review Comment:
You're right, and no additional transport-level cap beyond PIPE_BUF — the base64url expansion alone puts the full line around 2.8 KiB. Fixed to say the payload is capped at 2 KiB and the complete line stays below Linux PIPE_BUF, without implying the whole line is under 2 KB. Pushed in 383cda907.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] docs(eval): restructure README with navigation and a troubleshooting section [maka]
Posted by "Astro-Han (via GitHub)" <gi...@apache.org>.
Astro-Han commented on code in PR #4281:
URL: https://github.com/apache/maka/pull/4281#discussion_r3890144599
##########
packages/eval/README.md:
##########
@@ -39,18 +81,55 @@ Use `--cell <cell-id>` to replace one failed or indeterminate cell. The attempt
Before starting a trial, the public CLI validates the selected executor's machine paths, bundled relay files, pinned Harbor or Pier Python distribution, and Docker daemon availability for Docker environments. A missing or mismatched prerequisite is reported with the configured environment-variable name and expected framework version; the CLI does not start a trial or install external software. Subject-specific toolchain verification remains part of subject preparation and also completes before any trial starts.
-The built-in Harbor and Pier executors use one relay Agent. The framework prepares the task environment, the relay invokes exactly one Eval subject from `Agent.run()`, and the framework runs its native verifier and finalizer. Harbor and Pier use separate, explicitly versioned Python environments because their Agent and task contracts differ.
+## Extending: custom executors and subjects
+
+An executor decides how one attempt runs; a subject decides how one cell's agent is invoked. Both are pluggable interfaces defined in [`runner.ts:74-130`](src/runner.ts):
+
+```ts
+export interface SubjectAdapter {
+ readonly kind: ExperimentCell['subject']['kind'];
+ validate?(cell: ExperimentCell): void;
+ prepare?(input: { readonly spec: ExperimentSpec; readonly cells: readonly ExperimentCell[] }): Promise<void>;
+ canReuse?(input: { readonly cell: ExperimentCell; readonly attempt: CellAttempt }): boolean;
+ execute(input: { readonly cell: ExperimentCell; readonly context: SubjectExecutionContext }): Promise<SubjectExecutionResult>;
+}
+
+export interface ExperimentExecutor {
+ readonly kind: string;
+ validate?(cell: ExperimentCell): void;
+ runAttempt(
+ input: { readonly cell: ExperimentCell; readonly subjectCredentialNames: readonly string[]; readonly signal?: AbortSignal },
+ operation: (attempt: { readonly context: SubjectExecutionContext; verify(): Promise<ExecutorVerification> }) => Promise<EvalResult>,
+ ): Promise<ExecutorAttemptOutcome>;
+}
+```
-Maka subjects ask the Runtime Host client to run one owned execution in a dedicated Host root. Session, Turn, Goal and continuation semantics remain inside Runtime Host. External subjects declare a command and arguments, and may add non-secret environment values, target-to-source bindings for declared credentials, and an explicit result contract. Omitted credential bindings use declared names unchanged. The generic `exit-code` contract discards unstructured stdout and records null usage and cost. The structured `protocol-v1` contract is restricted to the bundled external wrapper so the shared relay can separate a bounded result frame from Harbor/Pier's merged process output; cohort-specific wrappers do not gain Runtime authority.
+The built-in Harbor and Pier executors are two instantiations of one generic implementation (`createHarborExecutor`/`createPierExecutor`, both delegating to `createHarnessExecutor(framework, config, specPath)` in [`harness-executor.ts:108-121`](src/harness-executor.ts)), parameterized by `HarnessFramework`. The built-in subjects are `createMakaSubjectAdapter()` ([`maka-subject.ts`](src/maka-subject.ts)) and `createExternalSubjectAdapter()` ([`external-subject.ts`](src/external-subject.ts)).
-The result kernel contains only score, normalized usage, attributable cost, duration, status, and artifacts. Specs carry every semantic setting; environment variables are reserved for credentials and machine-local paths.
+To swap in a custom executor or subject set, pass overrides to `runMakaEvalCli` ([`cli.ts:36-42`](src/cli.ts)) rather than the CLI's default wiring:
-The checked-in Terminal-Bench 2.1 four-arm cohort is `experiments/terminal-bench-2.1-deepseek-v4-flash-four-arm.json`. It freezes provider endpoints, framework version, container paths and read-only mount policy. Set each declared machine-path environment variable to its trusted prepared directory, and set the declared API-key credentials. Machine-local paths select artifacts; they do not alter experiment semantics and are not presented as a cryptographic identity scheme.
+```ts
+export interface EvalCliDependencies {
+ readonly loadExecutor: (spec: ExperimentSpec, specPath: string) => ExperimentExecutor;
+ readonly subjects: readonly SubjectAdapter[];
+ readonly signal?: AbortSignal;
+ readonly stdout: (text: string) => void;
+ readonly stderr: (text: string) => void;
+}
+```
+
+Any object implementing `runAttempt`/`validate` (executor) or `execute`/`validate`/`prepare`/`canReuse` (subject) can be supplied this way — `prepare`, `canReuse`, and `validate` are all optional.
+
+## Relay protocol
+
+Harbor/Pier subprocess stdout is merged Docker stdout/stderr that can interleave, so a structured result cannot simply be parsed from the tail of the output. `writeRelayResult` ([`relay-result-frame.ts:24,34-43`](src/relay-result-frame.ts)) emits one bounded (under 2 KB, below Linux `PIPE_BUF`) line — `MAKA-EVAL-RESULT-V1 <token> <bytes> <sha256> <base64url payload>` — so the executor can extract a verified, size-capped result frame out of otherwise-unstructured merged output. `takeRelayResultToken` consumes a one-time environment-variable token (`MAKA_EVAL_RESULT_TOKEN`) so the frame can be authenticated per-attempt.
Review Comment:
[P3] (category ① — normal documentation path)
Thanks for documenting this invariant. `RESULT_PAYLOAD_LIMIT_BYTES` caps the JSON payload at 2 KiB, but the complete emitted line also includes the token, length, 64-byte SHA-256 digest, prefix, and base64url expansion, so the full frame can be roughly 2.8 KiB. It still stays below Linux `PIPE_BUF`; could this say “the payload is capped at 2 KiB and the complete line stays below Linux `PIPE_BUF`” instead? Please push back if I am overlooking an additional transport-level cap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] docs(eval): restructure README with navigation and a troubleshooting section [maka]
Posted by "amaldevcm (via GitHub)" <gi...@apache.org>.
amaldevcm commented on code in PR #4281:
URL: https://github.com/apache/maka/pull/4281#discussion_r3890222675
##########
packages/eval/README.md:
##########
@@ -39,18 +81,55 @@ Use `--cell <cell-id>` to replace one failed or indeterminate cell. The attempt
Before starting a trial, the public CLI validates the selected executor's machine paths, bundled relay files, pinned Harbor or Pier Python distribution, and Docker daemon availability for Docker environments. A missing or mismatched prerequisite is reported with the configured environment-variable name and expected framework version; the CLI does not start a trial or install external software. Subject-specific toolchain verification remains part of subject preparation and also completes before any trial starts.
-The built-in Harbor and Pier executors use one relay Agent. The framework prepares the task environment, the relay invokes exactly one Eval subject from `Agent.run()`, and the framework runs its native verifier and finalizer. Harbor and Pier use separate, explicitly versioned Python environments because their Agent and task contracts differ.
+## Extending: custom executors and subjects
+
+An executor decides how one attempt runs; a subject decides how one cell's agent is invoked. Both are pluggable interfaces defined in [`runner.ts:74-130`](src/runner.ts):
+
+```ts
+export interface SubjectAdapter {
+ readonly kind: ExperimentCell['subject']['kind'];
+ validate?(cell: ExperimentCell): void;
+ prepare?(input: { readonly spec: ExperimentSpec; readonly cells: readonly ExperimentCell[] }): Promise<void>;
+ canReuse?(input: { readonly cell: ExperimentCell; readonly attempt: CellAttempt }): boolean;
+ execute(input: { readonly cell: ExperimentCell; readonly context: SubjectExecutionContext }): Promise<SubjectExecutionResult>;
+}
+
+export interface ExperimentExecutor {
+ readonly kind: string;
+ validate?(cell: ExperimentCell): void;
+ runAttempt(
+ input: { readonly cell: ExperimentCell; readonly subjectCredentialNames: readonly string[]; readonly signal?: AbortSignal },
+ operation: (attempt: { readonly context: SubjectExecutionContext; verify(): Promise<ExecutorVerification> }) => Promise<EvalResult>,
+ ): Promise<ExecutorAttemptOutcome>;
+}
+```
-Maka subjects ask the Runtime Host client to run one owned execution in a dedicated Host root. Session, Turn, Goal and continuation semantics remain inside Runtime Host. External subjects declare a command and arguments, and may add non-secret environment values, target-to-source bindings for declared credentials, and an explicit result contract. Omitted credential bindings use declared names unchanged. The generic `exit-code` contract discards unstructured stdout and records null usage and cost. The structured `protocol-v1` contract is restricted to the bundled external wrapper so the shared relay can separate a bounded result frame from Harbor/Pier's merged process output; cohort-specific wrappers do not gain Runtime authority.
+The built-in Harbor and Pier executors are two instantiations of one generic implementation (`createHarborExecutor`/`createPierExecutor`, both delegating to `createHarnessExecutor(framework, config, specPath)` in [`harness-executor.ts:108-121`](src/harness-executor.ts)), parameterized by `HarnessFramework`. The built-in subjects are `createMakaSubjectAdapter()` ([`maka-subject.ts`](src/maka-subject.ts)) and `createExternalSubjectAdapter()` ([`external-subject.ts`](src/external-subject.ts)).
-The result kernel contains only score, normalized usage, attributable cost, duration, status, and artifacts. Specs carry every semantic setting; environment variables are reserved for credentials and machine-local paths.
+To swap in a custom executor or subject set, pass overrides to `runMakaEvalCli` ([`cli.ts:36-42`](src/cli.ts)) rather than the CLI's default wiring:
-The checked-in Terminal-Bench 2.1 four-arm cohort is `experiments/terminal-bench-2.1-deepseek-v4-flash-four-arm.json`. It freezes provider endpoints, framework version, container paths and read-only mount policy. Set each declared machine-path environment variable to its trusted prepared directory, and set the declared API-key credentials. Machine-local paths select artifacts; they do not alter experiment semantics and are not presented as a cryptographic identity scheme.
+```ts
+export interface EvalCliDependencies {
+ readonly loadExecutor: (spec: ExperimentSpec, specPath: string) => ExperimentExecutor;
+ readonly subjects: readonly SubjectAdapter[];
+ readonly signal?: AbortSignal;
+ readonly stdout: (text: string) => void;
+ readonly stderr: (text: string) => void;
+}
+```
+
+Any object implementing `runAttempt`/`validate` (executor) or `execute`/`validate`/`prepare`/`canReuse` (subject) can be supplied this way — `prepare`, `canReuse`, and `validate` are all optional.
+
+## Relay protocol
+
+Harbor/Pier subprocess stdout is merged Docker stdout/stderr that can interleave, so a structured result cannot simply be parsed from the tail of the output. `writeRelayResult` ([`relay-result-frame.ts:24,34-43`](src/relay-result-frame.ts)) emits one bounded (under 2 KB, below Linux `PIPE_BUF`) line — `MAKA-EVAL-RESULT-V1 <token> <bytes> <sha256> <base64url payload>` — so the executor can extract a verified, size-capped result frame out of otherwise-unstructured merged output. `takeRelayResultToken` consumes a one-time environment-variable token (`MAKA_EVAL_RESULT_TOKEN`) so the frame can be authenticated per-attempt.
Review Comment:
You're right, and no additional transport-level cap beyond PIPE_BUF. The base64url expansion alone puts the full line around 2.8 KiB. Fixed to say the payload is capped at 2 KiB and the complete line stays below Linux PIPE_BUF, without implying the whole line is under 2 KB. Pushed in 383cda907.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] docs(eval): restructure README with navigation and a troubleshooting section [maka]
Posted by "Astro-Han (via GitHub)" <gi...@apache.org>.
Astro-Han merged PR #4281:
URL: https://github.com/apache/maka/pull/4281
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org