You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@maka.apache.org by as...@apache.org on 2026/09/15 04:46:15 UTC
(maka) branch main updated: fix(eval): expose product Bash through the headless tool profile (#5322)
This is an automated email from the ASF dual-hosted git repository.
Astro-Han pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/maka.git
The following commit(s) were added to refs/heads/main by this push:
new 72cd8b1f53 fix(eval): expose product Bash through the headless tool profile (#5322)
72cd8b1f53 is described below
commit 72cd8b1f532872ef1bcca875a7a7cae4e7b1448e
Author: AstroHan <le...@outlook.com>
AuthorDate: Tue Sep 15 12:46:10 2026 +0800
fix(eval): expose product Bash through the headless tool profile (#5322)
fix(runtime-host): compose Bash without a boundary declaration under Full access
The composer already knows the session permission mode (it derives
fullAccess for plan tools and the plan prompt) but built Bash and
request_sandbox_boundary unconditionally. Under Full access the boundary
is bypass and cannot be widened: the runtime already ignores the
declaration at call time, so the advertised schema and the sentence
"Enforced by the current session sandbox boundary." contradicted what
the runtime enforces. The nested required_boundary object is also the
schema that #4861 reproduced as whitespace-only tool-argument streams,
which is what the Eval profile's foreground-only copy used to avoid.
Decide it in the one place that assembles the surface: buildDefaultHostTools
passes declareSandboxBoundary = !fullAccess to buildBuiltinTools and omits
request_sandbox_boundary under Full access. buildBuiltinTools forwards the
existing buildManagedBashTool switch and gives executor Bash the same one
instead of a hard-coded declaration. The schema stays strict: a
declaration a model still sends under Full access is refused with the
field list, like any other unknown field. An unknown mode is not Full
access, so the startup catalog and child composition keep declaring.
Refs #4859
fix(eval): expose product Bash through the headless tool profile
The headless-coding-v1 profile kept its own copy of the Bash schema and
description and swapped it over the product tool at projection time. The
copy drifted from the one Bash builder in @maka/runtime: it never told
the model the foreground default (120000 ms) or that timeout_ms raises
it to 600000 ms, and it removed run_in_background, so a subject could
neither raise the limit knowingly nor move a long command out of the
foreground. The TB4 scored-run audit shows 27 of 1789 Bash calls killed
at the silent default and 79 commands declaring more time under
`timeout N` than the tool would grant.
Delete the copy instead of patching it. The profile admits the product
Bash unchanged plus StopBackgroundTask and WriteStdin, which the product
Bash description refers to. The profile name stays: only Eval hosts
persist it, every attempt runs under a fresh per-execution root that no
later binary reopens, and reproducibility of frozen runs rests on the
pinned Maka commit.
Generated-by: Claude Code
---
packages/eval/README.md | 7 +--
.../__tests__/execution-model-composition.test.ts | 46 +++++++++++++++--
.../hosted-execution-tool-profile.test.ts | 14 ++---
.../src/__tests__/interactive-run-composer.test.ts | 59 ++++++++++++++++++++++
.../src/server/hosted-execution-tool-profile.ts | 21 ++------
.../src/server/interactive-run-composer.ts | 13 +++--
.../runtime/src/__tests__/builtin-tools.test.ts | 27 ++++++++++
packages/runtime/src/builtin-tools.ts | 43 +++++++++++-----
8 files changed, 176 insertions(+), 54 deletions(-)
diff --git a/packages/eval/README.md b/packages/eval/README.md
index ee2876a542..d0a5e9433b 100644
--- a/packages/eval/README.md
+++ b/packages/eval/README.md
@@ -189,9 +189,10 @@ A subject that exhausts the framework timeout is reported as `subject_failed` wi
Maka benchmark subjects freeze a versioned Session profile. `headless-coding-v1` is persisted in
the Session header, so later turns and backend rebuilds retain the same contract. It fixes the
system prompt, disables product identity/personalization/skills/workspace-memory prompt fragments,
-admits only `Bash`, `Read`, `Write`, `Edit`, `Glob`, `Grep`, and `apply_patch` as tool candidates,
-and exposes a foreground-only Bash schema without `run_in_background` or `pty`. Provider-specific
-routing remains authoritative: DeepSeek Responses exposes `apply_patch` instead of `Write` and
+and admits only `Bash`, `StopBackgroundTask`, `WriteStdin`, `Read`, `Write`, `Edit`, `Glob`,
+`Grep`, and `apply_patch` as tool candidates. Bash is the product tool unchanged; because the
+subject runs with Full access, it carries no sandbox boundary declaration and the widening tool
+is absent, as in any Full access root session. Provider-specific routing remains authoritative: DeepSeek Responses exposes `apply_patch` instead of `Write` and
`Edit`, and `Read` also accepts Session-scoped Maka tool-result paths. A real
`hosted.execution.start` regression test pins SHA-256 hashes for the first main provider request's
developer prompt and complete tool schema.
diff --git a/packages/runtime-host/src/__tests__/execution-model-composition.test.ts b/packages/runtime-host/src/__tests__/execution-model-composition.test.ts
index 39823f8b8d..3446e38a8d 100644
--- a/packages/runtime-host/src/__tests__/execution-model-composition.test.ts
+++ b/packages/runtime-host/src/__tests__/execution-model-composition.test.ts
@@ -28,6 +28,10 @@ import { join } from 'node:path';
import { promisify } from 'node:util';
import { test } from 'node:test';
import { z } from 'zod';
+import {
+ DEFAULT_BASH_TIMEOUT_MS,
+ MAX_FOREGROUND_BASH_TIMEOUT_MS,
+} from '@maka/runtime/shell-run-contract';
import {
clientCapabilityConnectionIdentity,
clientCapabilityCoordinatorTestAdmission,
@@ -151,8 +155,7 @@ const MAX_IMPLEMENTATION_CHILD_REQUESTS =
const HEADLESS_CODING_V1_PROMPT_HASH =
'sha256:b2773282ac4755dc8d8a663eafdec68c3fa6f5680ec8557d261b5f723672b467';
const HEADLESS_CODING_V1_TOOLS_HASH =
- // Unified Read pages and Grep completeness share the hosted tool profile.
- 'sha256:fb7f539090471695ec1d8ca31555d083d8c3c0e0c40d5dcf655b14c103b10c22';
+ 'sha256:5cd4bc0df70d35f251065a3163fbaf1c54763b73838406514a21bccfb4552f2e';
const execFileAsync = promisify(execFile);
test('backend creation resolves a bound Session by immutable Connection identity', async () => {
let observedRef: unknown;
@@ -2549,11 +2552,27 @@ test('hosted execution freezes the headless coding provider wire contract', asyn
'Glob',
'Grep',
'Read',
+ 'StopBackgroundTask',
'Write',
+ 'WriteStdin',
]);
const bash = (tools as Array<Record<string, unknown>>).find((tool) => tool.name === 'Bash');
assert.ok(bash);
- assert.doesNotMatch(JSON.stringify(bash), /run_in_background|pty/u);
+ // The Eval session runs with Full access: the product Bash, minus the
+ // boundary declaration that Full access has nothing to enforce.
+ assert.deepEqual(
+ Object.keys((bash.parameters as { properties: Record<string, unknown> }).properties),
+ ['command', 'timeout_ms', 'run_in_background', 'pty'],
+ );
+ assert.match(
+ String(bash.description),
+ new RegExp(
+ `timeout ${DEFAULT_BASH_TIMEOUT_MS}ms, maximum ${MAX_FOREGROUND_BASH_TIMEOUT_MS}ms`,
+ 'u',
+ ),
+ );
+ assert.doesNotMatch(String(bash.description), /sandbox boundary/u);
+ assert.equal(responsesToolNames(request?.body).includes('request_sandbox_boundary'), false);
const stores = await openInteractiveExecutionStoresForWrite(owner.lease);
assert.equal(
@@ -4739,7 +4758,14 @@ test('the headless coding profile freezes the Eval prompt and tool ceiling', asy
},
} as unknown as HostMemoryCoordinator,
sessionTodo: {} as SessionTodoToolStore,
- builtinTools: {},
+ builtinTools: {
+ shellRuns: {
+ runForegroundBash: () => Promise.reject(new Error('not used')),
+ runBackgroundBash: () => Promise.reject(new Error('not used')),
+ },
+ backgroundTasks: { stopBackgroundTask: () => Promise.reject(new Error('not used')) },
+ ptyControls: { writeStdin: () => Promise.reject(new Error('not used')) },
+ },
toolProfile: 'headless-coding-v1',
parentAgentTools: buildParentAgentTools(),
scheduledTaskTool: {
@@ -4752,7 +4778,17 @@ test('the headless coding profile freezes the Eval prompt and tool ceiling', asy
assert.deepEqual(
composition.tools.map(({ name }) => name),
- ['Bash', 'Read', 'Write', 'Edit', 'Glob', 'Grep', 'apply_patch'],
+ [
+ 'Bash',
+ 'StopBackgroundTask',
+ 'WriteStdin',
+ 'Read',
+ 'Write',
+ 'Edit',
+ 'Glob',
+ 'Grep',
+ 'apply_patch',
+ ],
);
assert.equal(composition.toolAvailability, undefined);
assert.equal(
diff --git a/packages/runtime-host/src/__tests__/hosted-execution-tool-profile.test.ts b/packages/runtime-host/src/__tests__/hosted-execution-tool-profile.test.ts
index 4b1b812d4e..e2ede9da6d 100644
--- a/packages/runtime-host/src/__tests__/hosted-execution-tool-profile.test.ts
+++ b/packages/runtime-host/src/__tests__/hosted-execution-tool-profile.test.ts
@@ -55,11 +55,13 @@ test('hosted execution tool profiles are durable Session creation inputs', () =>
);
});
-test('the headless coding profile freezes prompt, tools, memory, and foreground Bash', async () => {
+test('the headless coding profile freezes prompt, tools, and memory and passes product Bash through', () => {
const profile = hostedExecutionRunProfile('headless-coding-v1');
assert.ok(profile);
assert.deepEqual(profile.toolNames, [
'Bash',
+ 'StopBackgroundTask',
+ 'WriteStdin',
'Read',
'Write',
'Edit',
@@ -114,15 +116,7 @@ test('the headless coding profile freezes prompt, tools, memory, and foreground
profileTools.map(({ name }) => name),
profile.toolNames,
);
- const bash = profileTools[0];
- assert.ok(bash);
- const schema = bash.parameters as z.ZodType;
- assert.equal((await schema.safeParseAsync({ command: 'true' })).success, true);
- assert.equal(
- (await schema.safeParseAsync({ command: 'true', run_in_background: true })).success,
- false,
- );
- assert.equal((await schema.safeParseAsync({ command: 'true', pty: true })).success, false);
+ assert.equal(profileTools[0], original);
});
test('the WorkHub coordination profile has conversational authority but zero tools', () => {
diff --git a/packages/runtime-host/src/__tests__/interactive-run-composer.test.ts b/packages/runtime-host/src/__tests__/interactive-run-composer.test.ts
index cc815f77c9..1892c9be39 100644
--- a/packages/runtime-host/src/__tests__/interactive-run-composer.test.ts
+++ b/packages/runtime-host/src/__tests__/interactive-run-composer.test.ts
@@ -19,8 +19,10 @@
import assert from 'node:assert/strict';
import { test } from 'node:test';
+import { emptyPlanSessionState, type PlanStore } from '@maka/core/plan';
import { createDefaultRuntimePolicy } from '@maka/core/runtime-policy';
import type { SessionTodoToolStore } from '@maka/runtime/session-todo-tools';
+import { z } from 'zod';
import type { MakaTool } from '@maka/runtime/tool-runtime';
import { createInteractiveRunComposer } from '../server/interactive-run-composer.js';
import type { HostMemoryCoordinator } from '../server/memory-coordinator.js';
@@ -114,8 +116,54 @@ test('scoped Tool resolution receives the complete stable Host binding', () => {
);
});
+test('Full access composes Bash without a boundary declaration and without the widening tool', () => {
+ const bashKeys = (permissionMode: 'bypass' | 'ask' | undefined) => {
+ const composer = createFixtureComposer({
+ builtinTools: unusedManagedShellBuiltinTools(),
+ ...(permissionMode
+ ? {
+ plan: {
+ store: {} as PlanStore,
+ state: emptyPlanSessionState('session'),
+ mode: 'agent' as const,
+ permissionMode,
+ },
+ }
+ : {}),
+ });
+ const tools = composer.resolveTools?.() ?? [];
+ const bash = tools.find(({ name }) => name === 'Bash');
+ assert.ok(bash);
+ return {
+ keys: Object.keys(z.toJSONSchema(bash.parameters as z.ZodTypeAny).properties ?? {}),
+ widening: tools.some(({ name }) => name === 'request_sandbox_boundary'),
+ enforced: bash.description.includes('Enforced by the current session sandbox boundary.'),
+ };
+ };
+ assert.deepEqual(bashKeys('bypass'), {
+ keys: ['command', 'timeout_ms', 'run_in_background', 'pty'],
+ widening: false,
+ enforced: false,
+ });
+ for (const mode of ['ask', undefined] as const) {
+ assert.deepEqual(bashKeys(mode), {
+ keys: [
+ 'command',
+ 'timeout_ms',
+ 'run_in_background',
+ 'pty',
+ 'boundary_intent',
+ 'required_boundary',
+ ],
+ widening: true,
+ enforced: true,
+ });
+ }
+});
+
test('an explicit tool profile remains an exact ceiling over scoped Tool additions', () => {
const composer = createFixtureComposer({
+ builtinTools: unusedManagedShellBuiltinTools(),
toolProfile: 'headless-coding-v1',
resolveAdditionalTools: () => [tool('Read'), tool('plugin_only')],
});
@@ -256,6 +304,17 @@ test('WorkHub v2 binds control, tasks, attachment reading and user questions whi
);
});
+function unusedManagedShellBuiltinTools(): Parameters<
+ typeof createInteractiveRunComposer
+>[0]['builtinTools'] {
+ const unused = () => Promise.reject(new Error('not used'));
+ return {
+ shellRuns: { runForegroundBash: unused, runBackgroundBash: unused },
+ backgroundTasks: { stopBackgroundTask: unused },
+ ptyControls: { writeStdin: unused },
+ };
+}
+
function createFixtureComposer(
overrides: Partial<Parameters<typeof createInteractiveRunComposer>[0]> = {},
) {
diff --git a/packages/runtime-host/src/server/hosted-execution-tool-profile.ts b/packages/runtime-host/src/server/hosted-execution-tool-profile.ts
index c8f6649324..cc3967cd0f 100644
--- a/packages/runtime-host/src/server/hosted-execution-tool-profile.ts
+++ b/packages/runtime-host/src/server/hosted-execution-tool-profile.ts
@@ -21,11 +21,12 @@ import type { SessionToolProfile } from '@maka/core/session';
import type { WorkHubRoutingDecision } from '@maka/core/workhub-routing';
import { parseAttachmentResourceRef } from '@maka/core/attachments';
import type { MakaTool } from '@maka/runtime/tool-runtime';
-import { z } from 'zod';
import { readParameters, resolveReadInput } from '@maka/runtime/read-page';
const HEADLESS_CODING_V1_TOOL_NAMES = [
'Bash',
+ 'StopBackgroundTask',
+ 'WriteStdin',
'Read',
'Write',
'Edit',
@@ -41,16 +42,6 @@ const HEADLESS_CODING_V1_SYSTEM_PROMPT = [
'Stop when the task is complete.',
].join('\n');
-const HEADLESS_CODING_V1_BASH_DESCRIPTION =
- 'Run a foreground shell command in the session cwd. Use Bash for inspection, builds, tests, and task-local generation. Background execution and PTY sessions are unavailable in this profile.';
-
-const HEADLESS_CODING_V1_BASH_PARAMETERS = z
- .object({
- command: z.string().describe('The shell command to execute'),
- timeout_ms: z.number().int().positive().max(600_000).optional(),
- })
- .strict();
-
const WORKHUB_COORDINATION_V1_SYSTEM_PROMPT = [
'You are the conversational coordinator for WorkHub.',
'Answer ordinary questions directly and help the user clarify intent.',
@@ -162,12 +153,6 @@ export function projectHostedExecutionTools(
impl: (input, context) =>
tool.impl(WORKHUB_ATTACHMENT_READ_PARAMETERS.parse(input), context),
}
- : tool.name === 'Bash'
- ? {
- ...tool,
- description: HEADLESS_CODING_V1_BASH_DESCRIPTION,
- parameters: HEADLESS_CODING_V1_BASH_PARAMETERS,
- }
- : tool,
+ : tool,
);
}
diff --git a/packages/runtime-host/src/server/interactive-run-composer.ts b/packages/runtime-host/src/server/interactive-run-composer.ts
index 92bb45f797..8de2f851ca 100644
--- a/packages/runtime-host/src/server/interactive-run-composer.ts
+++ b/packages/runtime-host/src/server/interactive-run-composer.ts
@@ -548,9 +548,14 @@ function buildDefaultHostTools(
plan?: InteractiveRunComposerInput['plan'],
deepResearchTools: readonly MakaTool[] = [],
): MakaTool[] {
- const builtins = builtinOptions ? buildBuiltinTools(builtinOptions) : [];
+ // Full access has no boundary to widen, so neither the Bash declaration nor
+ // the widening tool is offered. An unknown mode is not Full access.
+ const fullAccess = plan?.permissionMode === 'bypass';
+ const builtins = builtinOptions
+ ? buildBuiltinTools({ ...builtinOptions, declareSandboxBoundary: !fullAccess })
+ : [];
const question = buildAskUserQuestionTool();
- const sandboxBoundary = buildRequestSandboxBoundaryTool();
+ const sandboxBoundary = fullAccess ? undefined : buildRequestSandboxBoundaryTool();
const todoTools = buildSessionTodoTools(sessionTodo);
const activeExecution = plan ? activePlanExecution(plan.state) : undefined;
const interruptedExecution = plan
@@ -570,7 +575,7 @@ function buildDefaultHostTools(
...builtins.map((tool) => tool.name),
...hostTools.map((tool) => tool.name),
question.name,
- sandboxBoundary.name,
+ ...(sandboxBoundary ? [sandboxBoundary.name] : []),
'Skill',
'SkillSearch',
...todoTools.map((tool) => tool.name),
@@ -586,7 +591,7 @@ function buildDefaultHostTools(
...builtins,
...hostTools,
question,
- sandboxBoundary,
+ ...(sandboxBoundary ? [sandboxBoundary] : []),
buildSkillAgentToolFromInventory(inventoryFor, skillHost, { shadowTracker }),
buildSkillSearchAgentToolFromInventory(inventoryFor, skillHost, { shadowTracker }),
...todoTools,
diff --git a/packages/runtime/src/__tests__/builtin-tools.test.ts b/packages/runtime/src/__tests__/builtin-tools.test.ts
index b550881aef..1c919748cf 100644
--- a/packages/runtime/src/__tests__/builtin-tools.test.ts
+++ b/packages/runtime/src/__tests__/builtin-tools.test.ts
@@ -349,6 +349,33 @@ describe('builtin Bash streaming output', () => {
);
});
+ test('Bash drops its boundary declaration when the session has no boundary to widen', () => {
+ const shellRuns = {
+ runForegroundBash: () => Promise.reject(new Error('not used')),
+ runBackgroundBash: () => Promise.reject(new Error('not used')),
+ };
+ for (const options of [
+ { shellRuns, declareSandboxBoundary: false },
+ {
+ sandboxManager: availableLinuxManager(),
+ sandboxPlatform: 'linux' as const,
+ declareSandboxBoundary: false,
+ },
+ ]) {
+ const bash = buildBuiltinTools(options).find((tool) => tool.name === 'Bash');
+ if (!bash) throw new Error('Bash tool missing');
+ const parameters = bash.parameters as z.ZodTypeAny;
+ const keys = Object.keys(z.toJSONSchema(parameters).properties ?? {});
+ assert.equal(keys.includes('boundary_intent'), false);
+ assert.equal(keys.includes('required_boundary'), false);
+ assert.doesNotMatch(bash.description, /sandbox boundary/u);
+ assert.equal(
+ parameters.safeParse({ command: 'echo', boundary_intent: 'expand' }).success,
+ false,
+ );
+ }
+ });
+
test('Bash schema exposes explicit background execution and boundary declarations', () => {
const bash = buildBuiltinTools({
shellRuns: {
diff --git a/packages/runtime/src/builtin-tools.ts b/packages/runtime/src/builtin-tools.ts
index 2a4b04d541..e2df229a5f 100644
--- a/packages/runtime/src/builtin-tools.ts
+++ b/packages/runtime/src/builtin-tools.ts
@@ -188,6 +188,12 @@ export interface BuildBuiltinToolsOptions {
shellEnvironment?: Readonly<Record<string, string>>;
permissionProfile?: PermissionProfile;
sandboxManager?: SandboxManager;
+ /**
+ * Whether Bash advertises `boundary_intent` / `required_boundary`. False for
+ * a session whose boundary cannot be widened (Full access); a declaration no
+ * host enforces is noise in the model's tool selection. Defaults to true.
+ */
+ declareSandboxBoundary?: boolean;
/** Sandboxed worker used for all local filesystem tools. */
filesystemWorker?: Pick<FilesystemWorkerClient, 'execute'>;
/** Test/embedding override. Production callers use the current process platform. */
@@ -216,6 +222,7 @@ export function buildBuiltinTools(options: BuildBuiltinToolsOptions = {}): MakaT
buildManagedBashTool(options.shellRuns, {
executionFacts,
shell,
+ declareSandboxBoundary: options.declareSandboxBoundary !== false,
...(options.sandboxManager
? {
transformCommand: ({ command, pty, requiredBoundary, ctx }) => {
@@ -254,6 +261,7 @@ export function buildBuiltinTools(options: BuildBuiltinToolsOptions = {}): MakaT
buildExecutorBashTool(executor, shell, {
...(options.permissionProfile ? { permissionProfile: options.permissionProfile } : {}),
...(options.sandboxManager ? { sandboxManager: options.sandboxManager } : {}),
+ declareSandboxBoundary: options.declareSandboxBoundary !== false,
sandboxPlatform,
}),
];
@@ -616,6 +624,7 @@ interface ExecutorBashSandboxOptions {
permissionProfile?: PermissionProfile;
sandboxManager?: SandboxManager;
sandboxPlatform: SandboxPlatform;
+ declareSandboxBoundary?: boolean;
}
function buildExecutorBashTool(
@@ -623,25 +632,31 @@ function buildExecutorBashTool(
shell: TurnShellPlan,
sandboxOptions: ExecutorBashSandboxOptions,
): MakaTool {
+ const declareSandboxBoundary = sandboxOptions.declareSandboxBoundary !== false;
+ const executorBashFields = {
+ command: z.string().describe('The shell command to execute'),
+ timeout_ms: z.number().int().positive().max(600_000).optional(),
+ };
return {
name: 'Bash',
activityKind: 'command',
description:
withTurnShellGuidance('Run a shell command in the session cwd.', shell) +
- ' Enforced by the current session sandbox boundary.',
- parameters: preprocessBashBoundaryDeclaration(
- z
- .object({
- command: z.string().describe('The shell command to execute'),
- timeout_ms: z.number().int().positive().max(600_000).optional(),
- boundary_intent: bashBoundaryIntentSchema,
- required_boundary: sandboxBoundaryExpansionSchema
- .optional()
- .describe(BASH_REQUIRED_BOUNDARY_DESCRIPTION),
- })
- .strict()
- .superRefine(refineBashBoundaryDeclaration),
- ),
+ (declareSandboxBoundary ? ' Enforced by the current session sandbox boundary.' : ''),
+ parameters: declareSandboxBoundary
+ ? preprocessBashBoundaryDeclaration(
+ z
+ .object({
+ ...executorBashFields,
+ boundary_intent: bashBoundaryIntentSchema,
+ required_boundary: sandboxBoundaryExpansionSchema
+ .optional()
+ .describe(BASH_REQUIRED_BOUNDARY_DESCRIPTION),
+ })
+ .strict()
+ .superRefine(refineBashBoundaryDeclaration),
+ )
+ : z.object(executorBashFields).strict(),
toModelOutput: ({ output }) => bashToolResultToModelOutput(output),
executionFacts: executor.facts,
impl: async (input, ctx) => {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org