You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@maka.apache.org by zh...@apache.org on 2026/09/24 13:04:38 UTC
(maka) 01/01: feat(runtime): project unknown outcomes on explicit turns
This is an automated email from the ASF dual-hosted git repository.
me2seeks pushed a commit to branch feat/ts-manual-turn-after-unknown
in repository https://gitbox.apache.org/repos/asf/maka.git
commit b2f88b60d985b3aaf028d167c6f489c54ef27a05
Author: me2seeks <me...@gmail.com>
AuthorDate: Thu Sep 24 20:24:08 2026 +0800
feat(runtime): project unknown outcomes on explicit turns
---
docs/architecture/runtime-resume-architecture.md | 34 ++-
.../runtime-resume-architecture.zh-CN.md | 23 +-
.../cli/src/__tests__/activation-command.test.ts | 5 +-
packages/cli/src/__tests__/pi-tui-runner.test.ts | 29 +++
.../src/__tests__/runtime-host-run-command.test.ts | 72 ++++++
.../__tests__/runtime-host-session-driver.test.ts | 51 ++++
packages/cli/src/activation-command.ts | 6 +-
packages/cli/src/runtime-host-run-command.ts | 28 ++-
packages/cli/src/runtime-host-session-driver.ts | 24 +-
packages/core/src/__tests__/runtime-event.test.ts | 1 +
packages/core/src/backend-types.ts | 7 +
packages/core/src/turn-origin.ts | 18 ++
.../runtime/src/__tests__/ai-sdk-backend.test.ts | 277 ++++++++++++++++++++-
.../src/__tests__/recovery-resolver.test.ts | 241 +++---------------
.../runtime/src/__tests__/session-manager.test.ts | 87 +++----
packages/runtime/src/ai-sdk-turn.ts | 131 +++++++---
packages/runtime/src/model-history.ts | 168 +++++++++++++
packages/runtime/src/recovery-resolver.ts | 103 --------
packages/runtime/src/runtime-kernel.ts | 16 ++
packages/runtime/src/session-manager.ts | 104 +++-----
20 files changed, 935 insertions(+), 490 deletions(-)
diff --git a/docs/architecture/runtime-resume-architecture.md b/docs/architecture/runtime-resume-architecture.md
index bd8ed98415..678a93efe9 100644
--- a/docs/architecture/runtime-resume-architecture.md
+++ b/docs/architecture/runtime-resume-architecture.md
@@ -493,6 +493,31 @@ CLI/TUI `/resume` uses the same `SessionManager` plan/execute seam. Startup
recovery can reconstruct an already admitted continuation through that seam,
but it does not automatically select an ordinary failed or cancelled Run.
+### A missing tool result is projected only for a fresh user turn
+
+An uncertain tool side effect is not a resumable boundary. If recovery finds a
+durable T1 dispatch without a committed T2 result, it seals the old invocation
+as `outcome_unknown`; it does not invent a durable tool response and does not
+retry the tool. The old Run remains stopped.
+
+When the user later sends a new explicit message, that message starts a new
+Turn. Only provider requests in that fresh Turn receive a temporary history
+projection: the old tool call is paired with an `outcome_unknown` response and
+a system notice, then the new user message follows. The temporary response and
+notice are not written to the RuntimeEvent ledger or transcript. For example, after
+`Bash("touch marker.txt")` was dispatched but its result was not committed, a
+new request such as “check whether `marker.txt` exists” lets the model inspect
+the current state before deciding what to do; Maka does not decide whether the
+write happened and does not retry it automatically.
+
+This projection must not lose the T1 call. If an existing checkpoint or the
+current context budget would hide it, Runtime falls back to the full effective
+history for that request. This is a deliberate fail-closed trade-off: a very
+large unresolved history may exceed the provider's context limit, in which case
+the provider's real error is surfaced rather than dropping the uncertainty or
+claiming success. Compaction that preserves this uncertainty while reducing
+the rest of that history is not implemented here.
+
### Current parked-reason boundary
Runtime Host projects Runtime planner rejection reasons into the closed
@@ -833,14 +858,19 @@ Start with production-shaped red tests, then land core contract, storage constra
| Setting | Purpose | Rollback meaning |
|---|---|---|
-| unset | Enable explicit Desktop and CLI/TUI resume; keep model-driven WorkHub resume disabled | Default product behavior |
-| `MAKA_RUNTIME_SAFE_BOUNDARY_RESUME=1` | Also enable model-driven WorkHub resume | Preserves the previous full opt-in behavior |
+| unset | Enable explicit Desktop and CLI/TUI resume; keep automated WorkHub and `maka activate` resume disabled | Default product behavior |
+| `MAKA_RUNTIME_SAFE_BOUNDARY_RESUME=1` | Also enable automated WorkHub and `maka activate` resume | Preserves the previous full opt-in behavior |
| `MAKA_RUNTIME_SAFE_BOUNDARY_RESUME=0` | Disable explicit and model-driven resume planning | May park reconstruction; does not delete durable facts |
Unknown non-empty values fail closed like `0`. Every enabled entry point still
uses the same authoritative planner; the policy only controls whether a new
resume attempt may reach it.
+With automated `maka activate` resume enabled, a ready continuation is used
+instead of submitting that activation's new stimulus. If no continuation is
+ready, the stimulus is submitted normally. Callers that need every activation
+stimulus processed should not opt into this legacy resume behavior.
+
RuntimeEvent migration is unconditional on the first write. Downgrading to a
reader that does not understand the new schema requires explicit, verified
export. Migration failure must preserve legacy JSONL. A newer database schema
diff --git a/docs/architecture/runtime-resume-architecture.zh-CN.md b/docs/architecture/runtime-resume-architecture.zh-CN.md
index 6044cf3bbf..c9df002fab 100644
--- a/docs/architecture/runtime-resume-architecture.zh-CN.md
+++ b/docs/architecture/runtime-resume-architecture.zh-CN.md
@@ -500,6 +500,23 @@ sequenceDiagram
CLI/TUI 的 `/resume` 走同一个 `SessionManager` plan/execute seam。启动恢复也会通过这条
seam 重建已经 admission 的 continuation，但不会自动选择普通的 failed 或 cancelled Run。
+### 工具已派发但没有已提交结果：只在新用户 Turn 中临时投影
+
+工具副作用结果未知时，旧 Run 不能安全续跑。如果恢复时发现 T1 派发事实已经写入，但没有已提交的
+T2 结果，Runtime 会把旧 invocation 封存为 `outcome_unknown`；它不会伪造一条 durable tool response，
+也不会重试该工具。旧 Run 仍然是停止状态。
+
+之后用户显式发送新消息时，Runtime 会开启一个新的 Turn。只有这个新 Turn 中发给模型的请求会看到临时的
+历史投影：旧工具调用后面暂时附上一条 `outcome_unknown` 响应和 system 提醒，再接上新用户消息。这条临时响应
+和提醒不会写入 RuntimeEvent ledger 或 transcript。例如，`Bash("touch marker.txt")` 已派发、但结果来不及提交；
+用户新消息可以说“检查 `marker.txt` 是否存在”。模型可以先检查当前状态，再决定下一步；Maka 不会替模型
+判断文件是否写入，也不会自动重试命令。
+
+这个投影不能丢掉 T1 调用。如果已有 checkpoint 或当前 context budget 会把它隐藏，Runtime 会为这次请求
+退回完整的 effective history。这是有意的 fail-closed 取舍：未解决的历史很大时，可能超过 provider 的上下文
+限制；此时返回 provider 的真实错误，而不是删掉不确定事实或谎称工具成功。当前实现还不能在压缩其余历史的同时
+保留这份不确定性。
+
### 当前的 parked 原因边界
Runtime Host 负责把 Runtime planner 的 rejection reasons 投影成封闭的
@@ -864,13 +881,15 @@ RuntimeEvent 迁移不再由开关控制；首次写入必然迁移。当前恢
| 设置 | 作用 | 回滚含义 |
|---|---|---|
-| 未设置 | 开启 Desktop 与 CLI/TUI 的显式 resume；保持 WorkHub 模型驱动 resume 关闭 | 默认产品行为 |
-| `MAKA_RUNTIME_SAFE_BOUNDARY_RESUME=1` | 额外开启 WorkHub 模型驱动 resume | 保留此前完整 opt-in 行为 |
+| 未设置 | 开启 Desktop 与 CLI/TUI 的显式 resume；保持 WorkHub 和 `maka activate` 自动 resume 关闭 | 默认产品行为 |
+| `MAKA_RUNTIME_SAFE_BOUNDARY_RESUME=1` | 额外开启 WorkHub 和 `maka activate` 自动 resume | 保留此前完整 opt-in 行为 |
| `MAKA_RUNTIME_SAFE_BOUNDARY_RESUME=0` | 关闭显式和模型驱动 resume planning | 可能 park reconstruction，但不会删除 durable facts |
未知的非空值与 `0` 一样 fail closed。所有已开启的入口仍使用同一个权威 planner；策略只决定
新的 resume 尝试能否到达 planner。
+启用 `maka activate` 自动 resume 后，若有可续跑的 continuation，会续跑旧 Turn，而不提交本次 activation 的新 stimulus；若没有可续跑项，则正常提交 stimulus。需要确保每次 activation 的新内容都被处理时，不应启用这项旧式自动续跑行为。
+
真正降级到不理解新 schema 的旧版本前，必须显式 export 并验证。Migration 失败不能删除 legacy JSONL；数据库版本比当前程序新时必须 fail closed。
未来 checkpoint 和 recovery capability 也应遵守同一原则：一旦在 T1 或 accepted boundary 前选定 durable mode，不能在执行中静默 fallback 到较弱协议。降级必须产生明确、可审计的 mode mismatch 或 park reason。
diff --git a/packages/cli/src/__tests__/activation-command.test.ts b/packages/cli/src/__tests__/activation-command.test.ts
index 7de4efd1ab..97908e98d7 100644
--- a/packages/cli/src/__tests__/activation-command.test.ts
+++ b/packages/cli/src/__tests__/activation-command.test.ts
@@ -553,9 +553,10 @@ describe('maka activate JSONL protocol', () => {
}
});
- test('keeps a fresh activation stimulus instead of implicitly resuming an existing session', async () => {
+ test('preserves a new activation stimulus and origin instead of implicitly resuming an existing session', async () => {
let created = false;
let resumed = false;
+ let sentOrigin: unknown;
let sentText: string | undefined;
let requestedConnection: string | undefined;
const lines: string[] = [];
@@ -583,6 +584,7 @@ describe('maka activate JSONL protocol', () => {
resumed = true;
},
sendMessage: async function* (_runtime, _sessionId, input) {
+ sentOrigin = input.origin;
sentText = input.text;
yield* completedEvents();
},
@@ -594,6 +596,7 @@ describe('maka activate JSONL protocol', () => {
assert.equal(result, 0);
assert.equal(created, false);
assert.equal(resumed, false);
+ assert.deepEqual(sentOrigin, { kind: 'cloud_activation', activationId: 'activation-1' });
assert.equal(sentText, 'Inspect the workspace');
assert.equal(requestedConnection, 'local');
assert.equal(JSON.parse(lines.at(-1)!).makaSessionId, 'maka-session-1');
diff --git a/packages/cli/src/__tests__/pi-tui-runner.test.ts b/packages/cli/src/__tests__/pi-tui-runner.test.ts
index a0030a58c5..c6c553e3f0 100644
--- a/packages/cli/src/__tests__/pi-tui-runner.test.ts
+++ b/packages/cli/src/__tests__/pi-tui-runner.test.ts
@@ -9095,6 +9095,35 @@ Slug openai-work<cursor>
await run;
});
+ test('/resume starts the latest safe-boundary continuation for the selected session', async () => {
+ const terminal = new FakeTerminal();
+ const driver = new SlashCommandDriver();
+ const run = runMakaPiTui({
+ title: 'Maka',
+ driver,
+ cwd: '/repo',
+ model: 'm',
+ connectionSlug: 'c',
+ permissionMode: 'bypass',
+ terminal,
+ });
+
+ terminal.input('/session');
+ terminal.input('\r');
+ await waitFor(() => plainTerminalOutput(terminal.output()).includes('Resume Session'));
+ terminal.input('\r');
+ await waitFor(() => driver.sessionIds.length === 1);
+
+ terminal.input('/resume');
+ terminal.input('\r');
+ await waitFor(() => plainTerminalOutput(terminal.output()).includes('resumed safely'));
+
+ assert.equal(driver.resumeCalls, 1);
+ terminal.input('/exit');
+ terminal.input('\r');
+ await run;
+ });
+
test('/resume with no interrupted run explains there is nothing to resume', async () => {
const terminal = new FakeTerminal();
const driver = new SlashCommandDriver();
diff --git a/packages/cli/src/__tests__/runtime-host-run-command.test.ts b/packages/cli/src/__tests__/runtime-host-run-command.test.ts
index 7e69be8465..5fd9180acb 100644
--- a/packages/cli/src/__tests__/runtime-host-run-command.test.ts
+++ b/packages/cli/src/__tests__/runtime-host-run-command.test.ts
@@ -622,6 +622,56 @@ describe('Runtime Host maka run adapter', () => {
assert.equal(durable.failure?.class, 'aborted');
});
+ test('observes a resumed Host Turn through the same outcome path as a fresh Turn', async () => {
+ const observed: MakaRunOutcome[] = [];
+ const fixture = runFixture({ observed, resumeReady: true });
+ const session = await fixture.context.runtime.createSession({
+ cwd: '/workspace',
+ llmConnectionSlug: 'openai-main',
+ model: 'gpt-5',
+ permissionMode: 'ask',
+ });
+
+ assert.ok(fixture.context.runtime.resumeLatest);
+ const resumed = await fixture.context.runtime.resumeLatest(session.id);
+ assert.ok(resumed);
+ await collect(resumed);
+
+ assert.equal(observed.length, 1);
+ assert.equal(observed[0]?.outcomeId, 'run-resumed');
+ assert.equal(observed[0]?.status, 'completed');
+ assert.equal(observed[0]?.finalOutput, 'Resumed answer');
+ });
+
+ test('stops a resumed Host Turn if cancellation races its start', async () => {
+ const resumeStarted = deferred<void>();
+ const resumeGate = deferred<void>();
+ const fixture = runFixture({
+ resumeReady: true,
+ resumeGate: resumeGate.promise,
+ onResumeStarted: () => resumeStarted.resolve(),
+ });
+ const session = await fixture.context.runtime.createSession({
+ cwd: '/workspace',
+ llmConnectionSlug: 'openai-main',
+ model: 'gpt-5',
+ permissionMode: 'ask',
+ });
+ assert.ok(fixture.context.runtime.resumeLatest);
+ const resumed = await fixture.context.runtime.resumeLatest(session.id);
+ assert.ok(resumed);
+ const draining = collect(resumed);
+ await resumeStarted.promise;
+
+ await fixture.context.runtime.stopSession(session.id);
+ resumeGate.resolve();
+ await draining;
+
+ assert.deepEqual(fixture.exactTurnStops, [
+ { sessionId: 'session-created', turnId: 'turn-resumed', runId: 'run-resumed' },
+ ]);
+ });
+
test('classifies live and durable step-cap failures equally', async () => {
const live = await observeFixtureOutcome({
turnEvents: completionEvents('turn-1', 'step_limit'),
@@ -1012,6 +1062,9 @@ function runFixture(input: {
prepareGate?: Promise<void>;
onPrepareStarted?: () => void;
turnEvents?: AsyncIterable<SessionEvent>;
+ resumeReady?: boolean;
+ resumeGate?: Promise<void>;
+ onResumeStarted?: () => void;
pendingInteractions?: InteractionPendingSnapshot[];
pendingAfterTurnStarts?: boolean;
graphProjectionRace?: boolean;
@@ -1130,6 +1183,16 @@ function runFixture(input: {
: events,
};
},
+ resumeLatestTurn: async () => {
+ input.onResumeStarted?.();
+ await input.resumeGate;
+ return {
+ sessionId: switches.at(-1) ?? 'session-created',
+ turnId: 'turn-resumed',
+ runId: 'run-resumed',
+ events: eventsFor('turn-resumed', 'Resumed answer'),
+ };
+ },
respondToSandboxBoundary: async (response: { requestId: string; decision: 'deny' }) => {
sandboxResponses.push(response);
},
@@ -1153,6 +1216,15 @@ function runFixture(input: {
const connection = {
hostEpoch: 'host-1',
request: async (operation: string, requestInput: Record<string, unknown>) => {
+ if (operation === 'turn.resume.query' && input.resumeReady) {
+ return {
+ sessionId: requestInput.sessionId,
+ disposition: 'ready',
+ sourceRunId: 'run-source',
+ sourceTurnId: 'turn-source',
+ sourceRuntimeEventHighWater: 3,
+ };
+ }
if (operation === 'session.execution_boundary.query') {
return { kind: 'managed', access: 'writable', revision: 0 };
}
diff --git a/packages/cli/src/__tests__/runtime-host-session-driver.test.ts b/packages/cli/src/__tests__/runtime-host-session-driver.test.ts
index 937e4b66be..a5e9f9d095 100644
--- a/packages/cli/src/__tests__/runtime-host-session-driver.test.ts
+++ b/packages/cli/src/__tests__/runtime-host-session-driver.test.ts
@@ -2493,6 +2493,45 @@ describe('Runtime Host Maka Session driver', () => {
assert.equal((await nextEvent(turn.events)).text, 'Recovered');
});
+ test('returns the admitted Turn identity when starting a safe-boundary resume', async () => {
+ const subscription = new FakeSubscription(
+ continuitySnapshot({ rootTurn: null }),
+ Promise.resolve([]),
+ );
+ const connection = new FakeConnection([subscription]);
+ const driver = createRuntimeHostMakaSessionDriver({
+ connection: connection.value,
+ cwd: '/tmp',
+ llmConnectionId: 'connection-1',
+ llmConnectionSlug: 'openai-main',
+ model: 'gpt-5',
+ newId: () => 'turn-resume',
+ });
+ await driver.switchSession('session-1');
+
+ const turn = await driver.resumeLatestTurn({
+ sessionId: 'session-1',
+ disposition: 'ready',
+ sourceRunId: 'run-source',
+ sourceTurnId: 'turn-source',
+ sourceRuntimeEventHighWater: 3,
+ });
+
+ assert.deepEqual(
+ { sessionId: turn.sessionId, turnId: turn.turnId, runId: turn.runId },
+ { sessionId: 'session-1', turnId: 'turn-resume', runId: 'run-resumed' },
+ );
+ assert.deepEqual(connection.requests.at(-1), {
+ operation: 'turn.resume.start',
+ input: {
+ sessionId: 'session-1',
+ turnId: 'turn-resume',
+ sourceRunId: 'run-source',
+ sourceRuntimeEventHighWater: 3,
+ },
+ });
+ });
+
test('starts explicit Skills through the Host command and preserves its typed feedback', async () => {
const subscription = new FakeSubscription(
continuitySnapshot({ rootTurn: null }),
@@ -2988,6 +3027,18 @@ class FakeConnection {
if (operation === 'turn.stop') {
return {} as OperationOutput<K>;
}
+ if (operation === 'turn.resume.start') {
+ const resumeInput = input as OperationInput<'turn.resume.start'>;
+ return {
+ kind: 'started',
+ turn: {
+ sessionId: resumeInput.sessionId,
+ turnId: resumeInput.turnId,
+ runId: 'run-resumed',
+ status: 'running',
+ },
+ } as OperationOutput<K>;
+ }
const turnInput = input as {
sessionId?: string;
turnId?: string;
diff --git a/packages/cli/src/activation-command.ts b/packages/cli/src/activation-command.ts
index 689f9ad2e5..97fc28091d 100644
--- a/packages/cli/src/activation-command.ts
+++ b/packages/cli/src/activation-command.ts
@@ -511,7 +511,11 @@ export async function runMakaActivationCli(
const stream = await activationStream(
context.runtime,
session.id,
- { turnId: deps.newId(), text },
+ {
+ turnId: deps.newId(),
+ text,
+ origin: { kind: 'cloud_activation', activationId: request.activationId },
+ },
existing !== undefined && deps.automatedResumeEnabled(),
);
const drain = (async () => {
diff --git a/packages/cli/src/runtime-host-run-command.ts b/packages/cli/src/runtime-host-run-command.ts
index 7277411455..412120fd88 100644
--- a/packages/cli/src/runtime-host-run-command.ts
+++ b/packages/cli/src/runtime-host-run-command.ts
@@ -32,7 +32,11 @@ import {
type RuntimeHostProfile,
} from '@maka/runtime-host/client';
import { runtimeHostProfileUsesHostWorkspace } from '@maka/runtime-host/profile-kind';
-import type { InteractionPendingSnapshot, SessionCatalogItem } from '@maka/runtime-host/protocol';
+import type {
+ InteractionPendingSnapshot,
+ SessionCatalogItem,
+ TurnResumePlan,
+} from '@maka/runtime-host/protocol';
import {
runMakaTextCliCore,
type MakaRunContext,
@@ -348,7 +352,27 @@ class RuntimeHostRunRuntime implements MakaRunRuntime {
async resumeLatest(sessionId: string): Promise<AsyncIterable<SessionEvent> | null> {
await this.#attach(sessionId);
const plan = await this.#connection.request('turn.resume.query', { sessionId });
- return plan.disposition === 'ready' ? this.#driver.resumeLatest() : null;
+ return plan.disposition === 'ready' ? this.#resumeAndObserve(plan) : null;
+ }
+
+ async *#resumeAndObserve(
+ plan: Extract<TurnResumePlan, { disposition: 'ready' }>,
+ ): AsyncIterable<SessionEvent> {
+ const turn = await this.#driver.resumeLatestTurn(plan);
+ if (!turn.runId) throw new Error('Runtime Host did not return a Run identity');
+ const activeTurn: ActiveRuntimeHostTurn = {
+ sessionId: turn.sessionId,
+ turnId: turn.turnId,
+ runId: turn.runId,
+ outcome: new TurnOutcomeClassifier(turn.runId),
+ };
+ this.#activeTurn = activeTurn;
+ try {
+ if (this.#stopRequested) await this.#stopTurn(activeTurn);
+ yield* this.#observeTurn(turn, activeTurn);
+ } finally {
+ if (this.#activeTurn === activeTurn) this.#activeTurn = undefined;
+ }
}
async stopSession(sessionId: string): Promise<void> {
diff --git a/packages/cli/src/runtime-host-session-driver.ts b/packages/cli/src/runtime-host-session-driver.ts
index 0a53e498fb..c480c3a28b 100644
--- a/packages/cli/src/runtime-host-session-driver.ts
+++ b/packages/cli/src/runtime-host-session-driver.ts
@@ -76,6 +76,7 @@ import {
type GoalProjection,
type SessionContinuitySnapshot,
type SessionDomainChangedFrame,
+ type TurnResumePlan,
type TurnResumeParkReason,
} from '@maka/runtime-host/protocol';
import { RuntimeHostSessionChannel } from './runtime-host-session-channel.js';
@@ -173,6 +174,9 @@ export interface RuntimeHostMakaSessionDriver extends MakaSessionDriver {
readMessages(): Promise<StoredMessage[]>;
getWorkspaceTarget(): WorkspaceTarget | undefined;
resumeLatest(): AsyncIterable<SessionEvent>;
+ resumeLatestTurn(
+ plan: Extract<TurnResumePlan, { disposition: 'ready' }>,
+ ): Promise<MakaPreparedSessionTurn>;
subscribePendingInteractions(listener: (pending: InteractionPendingSnapshot) => void): () => void;
subscribeStartedTurns(listener: (turn: MakaAttachedSessionTurn) => void): () => void;
subscribeResolvedInteractions(
@@ -496,6 +500,17 @@ class RuntimeHostMakaSessionDriverImpl implements RuntimeHostMakaSessionDriver {
if (plan.disposition !== 'ready') {
throw new SafeBoundaryResumeParkedError(plan.reason);
}
+ const turn = await this.resumeLatestTurn(plan);
+ yield* turn.events;
+ }
+
+ async resumeLatestTurn(
+ plan: Extract<TurnResumePlan, { disposition: 'ready' }>,
+ ): Promise<MakaPreparedSessionTurn> {
+ const sessionId = this.#requireSession('resume');
+ if (plan.sessionId !== sessionId) {
+ throw new Error('Runtime Host resume plan changed Session identity');
+ }
const channel = await this.#ensureChannel(sessionId);
const turnId = this.#newId();
this.#claimedTurnIds.add(turnId);
@@ -508,13 +523,18 @@ class RuntimeHostMakaSessionDriverImpl implements RuntimeHostMakaSessionDriver {
sourceRuntimeEventHighWater: plan.sourceRuntimeEventHighWater,
});
if (result.kind !== 'started') {
- channel.failTurn(turnId, new SafeBoundaryResumeParkedError(result.plan.reason));
+ throw new SafeBoundaryResumeParkedError(result.plan.reason);
}
+ return {
+ sessionId: result.turn.sessionId,
+ turnId: result.turn.turnId,
+ runId: result.turn.runId,
+ events,
+ };
} catch (error) {
channel.failTurn(turnId, error);
throw error;
}
- yield* events;
}
submitMessage(
diff --git a/packages/core/src/__tests__/runtime-event.test.ts b/packages/core/src/__tests__/runtime-event.test.ts
index a0d6b8b023..48c8e6c9f8 100644
--- a/packages/core/src/__tests__/runtime-event.test.ts
+++ b/packages/core/src/__tests__/runtime-event.test.ts
@@ -149,6 +149,7 @@ test('decodes a released provider dropping note that nothing writes any more', (
test('shares one decoder across all TurnOrigin variants', () => {
const origins = [
{ kind: 'scheduled_task', scheduledTaskId: 'task-1' },
+ { kind: 'cloud_activation', activationId: 'activation-1' },
{ kind: 'goal', goalId: 'goal-1' },
{ kind: 'agent_graph', graphId: 'graph-1', wakeId: 'wake-1', attemptId: 'attempt-1' },
] as const;
diff --git a/packages/core/src/backend-types.ts b/packages/core/src/backend-types.ts
index ae933a1a41..1c757cb530 100644
--- a/packages/core/src/backend-types.ts
+++ b/packages/core/src/backend-types.ts
@@ -92,6 +92,13 @@ export interface BackendSendInput {
* Optional prior RuntimeEvent ledger for model-history projection.
*/
runtimeContext?: RuntimeEvent[];
+ /**
+ * Trusted marker for an explicit client-authored fresh turn. When prior
+ * history contains a sealed dispatched tool with no result, the backend may
+ * project that uncertainty into this request only. Continuations and hosted
+ * automation must never set this.
+ */
+ allowPriorUnknownToolOutcomes?: boolean;
/**
* The invocations `runtimeContext` came from, used only to verify
* provider-owned replay against the current model route. RuntimeEvents stay
diff --git a/packages/core/src/turn-origin.ts b/packages/core/src/turn-origin.ts
index c6bbae7662..96844772da 100644
--- a/packages/core/src/turn-origin.ts
+++ b/packages/core/src/turn-origin.ts
@@ -29,8 +29,14 @@ export interface WorkHubResultOrigin {
targetTurnId: string;
}
+export interface CloudActivationOrigin {
+ kind: 'cloud_activation';
+ activationId: string;
+}
+
export type TurnOrigin =
| WorkHubResultOrigin
+ | CloudActivationOrigin
| { kind: 'scheduled_task'; scheduledTaskId: string }
| { kind: 'legacy_automation'; automationId: string }
| { kind: 'goal'; goalId: string }
@@ -44,6 +50,7 @@ export type TurnOrigin =
};
type ScheduledTaskOrigin = Extract<TurnOrigin, { kind: 'scheduled_task' }>;
+type CloudActivationOriginType = Extract<TurnOrigin, { kind: 'cloud_activation' }>;
type LegacyAutomationOrigin = Extract<TurnOrigin, { kind: 'legacy_automation' }>;
type GoalOrigin = Extract<TurnOrigin, { kind: 'goal' }>;
type AgentGraphOrigin = Extract<TurnOrigin, { kind: 'agent_graph' }>;
@@ -52,6 +59,10 @@ const SCHEDULED_TASK_ORIGIN_SHAPE = defineObjectShape<ScheduledTaskOrigin>()(
['kind', 'scheduledTaskId'],
[],
);
+const CLOUD_ACTIVATION_ORIGIN_SHAPE = defineObjectShape<CloudActivationOriginType>()(
+ ['kind', 'activationId'],
+ [],
+);
const LEGACY_AUTOMATION_ORIGIN_SHAPE = defineObjectShape<LegacyAutomationOrigin>()(
['kind', 'automationId'],
[],
@@ -91,6 +102,13 @@ export function decodeTurnOrigin(value: unknown): TurnOrigin | undefined {
) {
return { kind: 'scheduled_task', scheduledTaskId: value.scheduledTaskId };
}
+ if (
+ hasExactShape(value, CLOUD_ACTIVATION_ORIGIN_SHAPE) &&
+ value.kind === 'cloud_activation' &&
+ typeof value.activationId === 'string'
+ ) {
+ return { kind: 'cloud_activation', activationId: value.activationId };
+ }
if (
hasExactShape(value, LEGACY_AUTOMATION_ORIGIN_SHAPE) &&
(value.kind === 'automation' || value.kind === 'legacy_automation') &&
diff --git a/packages/runtime/src/__tests__/ai-sdk-backend.test.ts b/packages/runtime/src/__tests__/ai-sdk-backend.test.ts
index 1948c14265..12a5bd336c 100644
--- a/packages/runtime/src/__tests__/ai-sdk-backend.test.ts
+++ b/packages/runtime/src/__tests__/ai-sdk-backend.test.ts
@@ -64,6 +64,7 @@ import type { DurableSessionEventSink, MakaTool, ToolRuntime } from '../tool-run
import { TOOL_SEARCH_NAME } from '../tool-availability.js';
import { buildNativeWebSearchTool } from '../native-web-search-tool.js';
import { canonicalizeToolSet } from '../request-shape.js';
+import { canonicalToolArgsHash } from '@maka/core/tool-args-identity';
import {
ARCHIVED_TOOL_RESULT_PLACEHOLDER_KIND,
ARCHIVED_TOOL_RESULT_REWRITE_VERSION,
@@ -103,7 +104,7 @@ import { deferred, waitFor as pollFor } from '@maka/core/test-only/async-primiti
import { Context } from '../plugin-kernel.js';
import { MakaCompositionLoader } from '../plugin-composition-loader.js';
import { PluginToolService } from '../plugin-tool-service.js';
-import { testInvocationOpening } from './invocation-fixture.js';
+import { testInvocationOpening, testInvocationRecord } from './invocation-fixture.js';
for (const terminal of ['gateway', 'eof', 'other'] as const) {
test(`recovers ${terminal} SSE with one failed attempt and no repeated tool effects`, async () => {
@@ -15126,7 +15127,14 @@ describe('AiSdkBackend steering durability and identity', () => {
const steeringBackend = (
model: MockLanguageModelV4,
options: Partial<
- Pick<AiSdkBackendInput, 'supportsVision' | 'readAttachmentBytes' | 'loadTurnRuntimeEvents'>
+ Pick<
+ AiSdkBackendInput,
+ | 'supportsVision'
+ | 'readAttachmentBytes'
+ | 'loadTurnRuntimeEvents'
+ | 'loadHistoryCompactCheckpoint'
+ | 'contextBudget'
+ >
> = {},
): AiSdkBackend =>
createTestAiSdkBackend({
@@ -15380,6 +15388,271 @@ describe('AiSdkBackend steering durability and identity', () => {
);
});
+ test('a fresh client message projects prior unknown tool outcomes only into the model request', async () => {
+ const args = { command: 'touch marker.txt' };
+ const priorIdentity = {
+ invocationId: 'prior-invocation',
+ runId: 'prior-run',
+ sessionId: 'session-1',
+ turnId: 'prior-turn',
+ ts: 1,
+ partial: false,
+ } as const;
+ const earlierIdentity = {
+ invocationId: 'earlier-invocation',
+ runId: 'earlier-run',
+ sessionId: 'session-1',
+ turnId: 'earlier-turn',
+ ts: 1,
+ partial: false,
+ } as const;
+ const priorEvents: RuntimeEvent[] = [
+ {
+ ...earlierIdentity,
+ id: 'earlier-protocol',
+ role: 'system',
+ author: 'system',
+ actions: { runtimeProtocol: { toolBoundary: 't1_after_preflight_v1' } },
+ },
+ {
+ ...earlierIdentity,
+ id: 'earlier-user',
+ role: 'user',
+ author: 'user',
+ content: { kind: 'text', text: 'the first turn completed normally' },
+ },
+ {
+ ...priorIdentity,
+ id: 'prior-protocol',
+ role: 'system',
+ author: 'system',
+ actions: { runtimeProtocol: { toolBoundary: 't1_after_preflight_v1' } },
+ },
+ {
+ ...priorIdentity,
+ id: 'prior-user',
+ role: 'user',
+ author: 'user',
+ content: { kind: 'text', text: 'make a marker file' },
+ },
+ {
+ ...priorIdentity,
+ id: 'prior-call',
+ role: 'model',
+ author: 'agent',
+ origin: 'provider',
+ modelVisibility: 'visible',
+ content: { kind: 'function_call', id: 'provider-call-1', name: 'Bash', args },
+ refs: { operationId: 'prior-operation', toolCallId: 'provider-call-1' },
+ },
+ {
+ ...priorIdentity,
+ id: 'prior-dispatch',
+ role: 'system',
+ author: 'system',
+ actions: {
+ toolDispatch: {
+ protocol: 't1_after_preflight_v1',
+ operationId: 'prior-operation',
+ providerToolCallId: 'provider-call-1',
+ toolName: 'Bash',
+ canonicalArgsHash: canonicalToolArgsHash('Bash', args),
+ recoveryMode: 'never_auto_retry',
+ },
+ },
+ refs: { operationId: 'prior-operation', toolCallId: 'provider-call-1' },
+ },
+ {
+ ...priorIdentity,
+ id: 'hidden-nested-call',
+ role: 'model',
+ author: 'agent',
+ origin: 'code_mode',
+ modelVisibility: 'hidden',
+ content: {
+ kind: 'function_call',
+ id: 'hidden-nested-call-1',
+ name: 'SecretNestedTool',
+ args: { secret: 'hidden-operation-argument' },
+ },
+ refs: {
+ operationId: 'hidden-nested-operation',
+ toolCallId: 'hidden-nested-call-1',
+ parentOperationId: 'prior-operation',
+ parentToolCallId: 'provider-call-1',
+ },
+ },
+ {
+ ...priorIdentity,
+ id: 'hidden-nested-dispatch',
+ role: 'system',
+ author: 'system',
+ origin: 'code_mode',
+ modelVisibility: 'hidden',
+ actions: {
+ toolDispatch: {
+ protocol: 't1_after_preflight_v1',
+ operationId: 'hidden-nested-operation',
+ providerToolCallId: 'hidden-nested-call-1',
+ toolName: 'SecretNestedTool',
+ canonicalArgsHash: canonicalToolArgsHash('SecretNestedTool', {
+ secret: 'hidden-operation-argument',
+ }),
+ recoveryMode: 'never_auto_retry',
+ },
+ },
+ refs: {
+ operationId: 'hidden-nested-operation',
+ toolCallId: 'hidden-nested-call-1',
+ parentOperationId: 'prior-operation',
+ parentToolCallId: 'provider-call-1',
+ },
+ },
+ ];
+ const earlierInvocation = testInvocationRecord({
+ sessionId: 'session-1',
+ invocationId: 'earlier-invocation',
+ runId: 'earlier-run',
+ turnId: 'earlier-turn',
+ outcome: 'completed',
+ });
+ const priorInvocation = testInvocationRecord({
+ sessionId: 'session-1',
+ invocationId: 'prior-invocation',
+ runId: 'prior-run',
+ turnId: 'prior-turn',
+ outcome: 'failed',
+ failureClass: 'outcome_unknown',
+ });
+ const model = textCompletionModel('I will inspect the current state first.');
+ const priorCheckpoint = buildHistoryCompactCheckpoint({
+ sessionId: 'session-1',
+ coveredRuntimeEvents: [priorEvents[1]!],
+ summary: structuredSummary('older settled history was compacted'),
+ });
+ const durable = durableTurnHarness('turn-resume', 'check whether the marker exists', {
+ runId: 'fresh-run',
+ });
+ const backend = steeringBackend(model, {
+ loadTurnRuntimeEvents: durable.loadTurnRuntimeEvents,
+ loadHistoryCompactCheckpoint: () => priorCheckpoint,
+ contextBudget: buildDefaultContextBudgetPolicy(),
+ });
+
+ await drainDurably(
+ backend.send(
+ durable.input({
+ runtimeContext: priorEvents,
+ runtimeContextInvocations: [earlierInvocation, priorInvocation],
+ allowPriorUnknownToolOutcomes: true,
+ }),
+ ),
+ durable,
+ );
+
+ const prompt = JSON.stringify(model.doStreamCalls[0]);
+ assert.match(prompt, /provider-call-1/);
+ assert.match(prompt, /outcome_unknown/);
+ assert.match(prompt, /may or may not have happened/);
+ assert.match(prompt, /check whether the marker exists/);
+ assert.doesNotMatch(
+ prompt,
+ /SecretNestedTool|hidden-nested-operation|hidden-operation-argument/,
+ );
+ assert.match(prompt, /older settled history was compacted/);
+ assert.equal(
+ durable.ledger.some(
+ (event) =>
+ event.content?.kind === 'function_response' && event.content.id === 'provider-call-1',
+ ),
+ false,
+ 'request-only unknown results must not be written into the new turn ledger',
+ );
+
+ const checkpointCoveringUnknownCall = buildHistoryCompactCheckpoint({
+ sessionId: 'session-1',
+ coveredRuntimeEvents: priorEvents.slice(0, 6),
+ summary: structuredSummary('checkpoint omitted the unresolved provider call'),
+ });
+ const checkpointed = durableTurnHarness(
+ 'turn-checkpointed-unknown',
+ 'inspect the marker before deciding what to do',
+ { runId: 'checkpointed-run' },
+ );
+ const checkpointedModel = textCompletionModel('I will inspect the marker first.');
+ const checkpointedBackend = steeringBackend(checkpointedModel, {
+ loadTurnRuntimeEvents: checkpointed.loadTurnRuntimeEvents,
+ loadHistoryCompactCheckpoint: () => checkpointCoveringUnknownCall,
+ contextBudget: buildDefaultContextBudgetPolicy(),
+ });
+ await drainDurably(
+ checkpointedBackend.send(
+ checkpointed.input({
+ runtimeContext: priorEvents,
+ runtimeContextInvocations: [earlierInvocation, priorInvocation],
+ allowPriorUnknownToolOutcomes: true,
+ }),
+ ),
+ checkpointed,
+ );
+ const checkpointedPrompt = JSON.stringify(checkpointedModel.doStreamCalls[0]);
+ assert.match(checkpointedPrompt, /provider-call-1/);
+ assert.match(checkpointedPrompt, /outcome_unknown/);
+ assert.match(checkpointedPrompt, /may or may not have happened/);
+ assert.doesNotMatch(checkpointedPrompt, /checkpoint omitted the unresolved provider call/);
+
+ const inconsistentModel = textCompletionModel('must not be sent');
+ const inconsistent = durableTurnHarness('turn-inconsistent', 'new message', {
+ runId: 'inconsistent-run',
+ });
+ const inconsistentBackend = steeringBackend(inconsistentModel, {
+ loadTurnRuntimeEvents: inconsistent.loadTurnRuntimeEvents,
+ });
+ await assert.rejects(
+ drainDurably(
+ inconsistentBackend.send(
+ inconsistent.input({
+ runtimeContext: priorEvents,
+ runtimeContextInvocations: [
+ earlierInvocation,
+ {
+ ...priorInvocation,
+ terminalEvent: {
+ ...priorInvocation.terminalEvent!,
+ status: 'completed',
+ },
+ },
+ ],
+ allowPriorUnknownToolOutcomes: true,
+ }),
+ ),
+ inconsistent,
+ ),
+ /prior unknown tool outcome has no sealed invocation/,
+ );
+ assert.equal(inconsistentModel.doStreamCalls.length, 0);
+
+ const unprivilegedModel = textCompletionModel('No recovery context was admitted.');
+ const unprivileged = durableTurnHarness('turn-unprivileged', 'ordinary message', {
+ runId: 'unprivileged-run',
+ });
+ const unprivilegedBackend = steeringBackend(unprivilegedModel, {
+ loadTurnRuntimeEvents: unprivileged.loadTurnRuntimeEvents,
+ });
+ await drainDurably(
+ unprivilegedBackend.send(
+ unprivileged.input({
+ runtimeContext: priorEvents,
+ runtimeContextInvocations: [earlierInvocation, priorInvocation],
+ }),
+ ),
+ unprivileged,
+ );
+ const unprivilegedPrompt = JSON.stringify(unprivilegedModel.doStreamCalls[0]);
+ assert.doesNotMatch(unprivilegedPrompt, /outcome_unknown/);
+ assert.doesNotMatch(unprivilegedPrompt, /provider-call-1/);
+ });
+
test('persists canonical steering content and materializes attachments for the model', async () => {
const model = textCompletionModel('done');
const pngBytes = new Uint8Array([137, 80, 78, 71]);
diff --git a/packages/runtime/src/__tests__/recovery-resolver.test.ts b/packages/runtime/src/__tests__/recovery-resolver.test.ts
index b85b5bb6d7..3b027d7d73 100644
--- a/packages/runtime/src/__tests__/recovery-resolver.test.ts
+++ b/packages/runtime/src/__tests__/recovery-resolver.test.ts
@@ -18,17 +18,10 @@
*/
import assert from 'node:assert/strict';
-import { mkdtemp, rm } from 'node:fs/promises';
-import { tmpdir } from 'node:os';
-import { join } from 'node:path';
import { describe, it } from 'node:test';
import { canonicalToolArgsHash } from '@maka/core/tool-args-identity';
import type { RuntimeEvent } from '@maka/core/runtime-event';
-import { createSqliteRuntimeStore } from '@maka/storage/sqlite-runtime-store';
-import {
- buildInterruptedToolOutcomeCommits,
- resolveRuntimeRecovery,
-} from '../recovery-resolver.js';
+import { resolveRuntimeRecovery } from '../recovery-resolver.js';
describe('RecoveryResolver', () => {
it('proves a new-protocol call without dispatch was never dispatched', () => {
@@ -72,7 +65,7 @@ describe('RecoveryResolver', () => {
assert.equal(resolution.requiresReconciliation, true);
});
- it('settles every dispatched operation, including hidden nested calls', () => {
+ it('keeps dispatched operations unresolved instead of inventing durable tool results', () => {
const outerCall = event({
id: 'outer-call',
role: 'model',
@@ -115,219 +108,43 @@ describe('RecoveryResolver', () => {
parentToolCallId: 'exec-1',
});
- const commits = buildInterruptedToolOutcomeCommits(
- [initialEvent('t1_after_preflight_v1'), outerCall, outerDispatch, nestedCall, nestedDispatch],
- 50,
- 'code_mode',
- );
+ const events = [
+ initialEvent('t1_after_preflight_v1'),
+ outerCall,
+ outerDispatch,
+ nestedCall,
+ nestedDispatch,
+ ];
+ const resolution = resolveRuntimeRecovery(events);
- assert.equal(commits.length, 2);
- const outerCommit = commits.find((commit) => commit.operationId === 'outer-op');
- const nestedCommit = commits.find((commit) => commit.operationId === 'nested-op');
- assert.ok(outerCommit);
- assert.ok(nestedCommit);
- assert.equal(outerCommit.runtimeEvent.content?.kind, 'function_response');
- assert.equal(
- outerCommit.runtimeEvent.content?.kind === 'function_response'
- ? outerCommit.runtimeEvent.content.isError
- : false,
- true,
- );
assert.deepEqual(
- outerCommit.runtimeEvent.content?.kind === 'function_response'
- ? outerCommit.runtimeEvent.content.result
- : undefined,
- {
- kind: 'json',
- value: {
- kind: 'code_mode',
- status: 'interrupted',
- message:
- 'Code Mode execution was interrupted by runtime recovery. Its side effects may or may not have occurred; do not retry it immediately. Inspect the current state first.',
- uncertainOutcome: { code: 'outcome_unknown', retrySafe: false },
- },
- },
- );
- assert.deepEqual(
- outerCommit.runtimeEvent.content?.kind === 'function_response'
- ? outerCommit.runtimeEvent.content.modelProjection
- : undefined,
- {
- version: 1,
- kind: 'json',
- value: {
- kind: 'json',
- value: {
- kind: 'code_mode',
- status: 'interrupted',
- message:
- 'Code Mode execution was interrupted by runtime recovery. Its side effects may or may not have occurred; do not retry it immediately. Inspect the current state first.',
- uncertainOutcome: { code: 'outcome_unknown', retrySafe: false },
- },
- },
- isError: true,
- },
- );
- assert.equal(nestedCommit.runtimeEvent.modelVisibility, 'hidden');
- assert.equal(
- nestedCommit.runtimeEvent.content?.kind === 'function_response'
- ? nestedCommit.runtimeEvent.content.isError
- : false,
- true,
- );
- assert.equal(
- nestedCommit.runtimeEvent.content?.kind === 'function_response' &&
- typeof nestedCommit.runtimeEvent.content.result === 'object' &&
- nestedCommit.runtimeEvent.content.result !== null &&
- 'uncertainOutcome' in nestedCommit.runtimeEvent.content.result,
- true,
- );
- });
-
- it('commits a projected recovery outcome through the projection-aware SQLite T2 gate', async () => {
- const root = await mkdtemp(join(tmpdir(), 'maka-recovery-projection-'));
- const store = createSqliteRuntimeStore(join(root, 'runtime.sqlite'));
- try {
- const call = event({
- id: 'outer-call',
- role: 'model',
- author: 'agent',
- origin: 'provider',
- modelVisibility: 'visible',
- content: { kind: 'function_call', id: 'exec-1', name: 'exec', args: { code: 'work()' } },
- refs: { operationId: 'outer-op', toolCallId: 'exec-1' },
- });
- const dispatch = dispatchFor({
- id: 'outer-dispatch',
- operationId: 'outer-op',
- toolCallId: 'exec-1',
- toolName: 'exec',
- args: { code: 'work()' },
- resultProjectionVersion: 1,
- });
- await store.commitToolPrepared({
- operationId: 'outer-op',
- journalEventId: 'outer-op_prepared',
- runtimeEvent: call,
- dispatchRuntimeEvent: dispatch,
- providerToolCallId: 'exec-1',
- toolName: 'exec',
- canonicalArgsHash: canonicalToolArgsHash('exec', { code: 'work()' }),
- recoveryMode: 'never_auto_retry',
- committedAt: 10,
- });
-
- const [commit] = buildInterruptedToolOutcomeCommits(
- await store.readImmutableRuntimeEvents('session-1', 'run-1'),
- 50,
- 'code_mode',
- );
- assert.ok(commit);
- await store.commitToolOutcome(commit);
-
- assert.equal((await store.readToolOperation('outer-op'))?.currentState, 'outcome_committed');
- const response = (await store.readImmutableRuntimeEvents('session-1', 'run-1')).at(-1);
- assert.equal(response?.content?.kind, 'function_response');
- assert.equal(
- response?.content?.kind === 'function_response'
- ? response.content.modelProjection?.version
- : undefined,
- 1,
- );
- } finally {
- store.close();
- await rm(root, { recursive: true, force: true });
- }
- });
-
- it('marks generic interrupted tools as outcome-unknown and does not retry them', () => {
- const commits = buildInterruptedToolOutcomeCommits(
+ resolution.decisions.map(({ toolCallId, status, reason }) => ({
+ toolCallId,
+ status,
+ reason,
+ })),
[
- initialEvent('t1_after_preflight_v1'),
- event({
- id: 'direct-exec-call',
- role: 'model',
- author: 'agent',
- origin: 'provider',
- modelVisibility: 'visible',
- content: { kind: 'function_call', id: 'exec-1', name: 'exec', args: {} },
- refs: { operationId: 'direct-exec-op', toolCallId: 'exec-1' },
- }),
- dispatchFor({
- id: 'direct-exec-dispatch',
- operationId: 'direct-exec-op',
- toolCallId: 'exec-1',
- toolName: 'exec',
- args: {},
- }),
+ { toolCallId: 'exec-1', status: 'indeterminate', reason: 'dispatch_without_response' },
+ { toolCallId: 'nested-1', status: 'indeterminate', reason: 'dispatch_without_response' },
],
- 50,
- 'direct',
);
-
- assert.equal(commits.length, 1);
- assert.equal(commits[0]?.operationId, 'direct-exec-op');
assert.equal(
- commits[0]?.runtimeEvent.content?.kind === 'function_response' &&
- typeof commits[0].runtimeEvent.content.result === 'object' &&
- commits[0].runtimeEvent.content.result !== null &&
- 'uncertainOutcome' in commits[0].runtimeEvent.content.result,
- true,
- );
- });
-
- it('settles a dispatched user question as interrupted without claiming a side effect', () => {
- const args = { questions: [{ question: 'Continue?', options: ['Yes', 'No'] }] };
- const call = event({
- id: 'question-call',
- role: 'model',
- author: 'agent',
- content: {
- kind: 'function_call',
- id: 'question-1',
- name: 'AskUserQuestion',
- args,
- },
- });
- const dispatch = dispatchFor({
- id: 'question-dispatch',
- operationId: 'question-op',
- toolCallId: 'question-1',
- toolName: 'AskUserQuestion',
- args,
- });
-
- const [commit] = buildInterruptedToolOutcomeCommits(
- [initialEvent('t1_after_preflight_v1'), call, dispatch],
- 50,
- 'direct',
- );
- assert.ok(commit);
- assert.equal(commit.runtimeEvent.content?.kind, 'function_response');
- assert.deepEqual(
- commit.runtimeEvent.content?.kind === 'function_response'
- ? commit.runtimeEvent.content.result
- : undefined,
- {
- kind: 'text',
- text: 'This question was interrupted by runtime recovery before an answer was submitted.',
- },
+ events.some((candidate) => candidate.content?.kind === 'function_response'),
+ false,
);
});
- it('does not append an interrupted result after the invocation terminal event', () => {
- const commits = buildInterruptedToolOutcomeCommits(
- [
- initialEvent('t1_after_preflight_v1'),
- functionCallEvent(),
- toolDispatchEvent(),
- event({ id: 'terminal-1', status: 'aborted', actions: { endInvocation: true } }),
- ],
- 50,
- 'direct',
- );
+ it('a terminal invocation does not turn a dispatched operation into a settled tool result', () => {
+ const resolution = resolveRuntimeRecovery([
+ initialEvent('t1_after_preflight_v1'),
+ functionCallEvent(),
+ toolDispatchEvent(),
+ event({ id: 'terminal-1', status: 'failed', actions: { endInvocation: true } }),
+ ]);
- assert.deepEqual(commits, []);
+ assert.equal(resolution.decisions[0]?.status, 'indeterminate');
+ assert.equal(resolution.decisions[0]?.reason, 'dispatch_without_response');
+ assert.equal(resolution.requiresReconciliation, true);
});
it('treats a matching response without dispatch as a completed pre-T1 result', () => {
diff --git a/packages/runtime/src/__tests__/session-manager.test.ts b/packages/runtime/src/__tests__/session-manager.test.ts
index 05c743c36c..997cf7076b 100644
--- a/packages/runtime/src/__tests__/session-manager.test.ts
+++ b/packages/runtime/src/__tests__/session-manager.test.ts
@@ -65,7 +65,7 @@ import { RUNTIME_CONTINUATION_AUTHORITY_V1 } from '@maka/core/runtime-event-stor
import { deriveTurnRecords } from '@maka/core/session';
import { isTerminalRuntimeEvent } from '@maka/core/runtime-event';
import { canonicalToolArgsHash } from '@maka/core/tool-args-identity';
-import { buildInterruptedToolOutcomeCommits } from '../recovery-resolver.js';
+import { resolveRuntimeRecovery } from '../recovery-resolver.js';
import { buildImmutableRuntimePrefix, decodeContinuationClaim } from '@maka/core/runtime-boundary';
import type {
CreateSandboxBoundaryRequest,
@@ -141,7 +141,6 @@ import { RuntimeReadModel, RuntimeReadModelError } from '../runtime-read-model.j
import type { AgentBackend } from '@maka/core/backend-types';
import type { MakaTool } from '../tool-runtime.js';
import type { ShellRunProcessManager } from '../shell-run-manager.js';
-import type { RuntimeCommitSink } from '../runtime-commit-sink.js';
import {
buildHistoryCompactCheckpoint,
type HistoryCompactCheckpoint,
@@ -6414,6 +6413,16 @@ describe('SessionManager permission mode updates', () => {
['turn-1-final', 'turn-1-complete'],
);
assert.strictEqual(backend?.sendInputs[0]?.toolMode, 'code_mode');
+ assert.strictEqual(backend?.sendInputs[0]?.allowPriorUnknownToolOutcomes, true);
+
+ await collectSessionEvents(
+ manager.sendMessage(session.id, {
+ turnId: 'turn-2',
+ text: 'automated activation',
+ origin: { kind: 'cloud_activation', activationId: 'activation-2' },
+ }),
+ );
+ assert.strictEqual(backend?.sendInputs[1]?.allowPriorUnknownToolOutcomes, undefined);
const [run] = await runtimeEventStore.listSessionInvocations(session.id);
if (!run) throw new Error('the run opened no invocation');
@@ -12814,38 +12823,20 @@ describe('SessionManager permission mode updates', () => {
assert.strictEqual((await turnOf(activeDone.id, 'active-turn'))?.status, 'completed');
});
- test('startup recovery settles arbitrary dispatched tools before terminalizing the run', async () => {
+ test('startup recovery seals dispatched tools as unknown without inventing durable results', async () => {
const store = new MemorySessionStore();
const runStore = new MemoryAgentRunStore();
const backends = new BackendRegistry();
backends.register('ai-sdk', (ctx) => new TestBackend(ctx));
- let sessionId = '';
- let outcomeCommitFailuresRemaining = 3;
- let outcomeCommitAttempts = 0;
- const runtimeCommitSink: RuntimeCommitSink = {
- commitToolPrepared: async () => {
- throw new Error('not used during recovery');
- },
- commitToolOutcome: async (input) => {
- outcomeCommitAttempts += 1;
- if (outcomeCommitFailuresRemaining > 0) {
- outcomeCommitFailuresRemaining -= 1;
- throw new Error('transient outcome commit failure');
- }
- await runStore.appendRuntimeEvent(sessionId, 'run-1', input.runtimeEvent);
- return { created: true, runtimeEventSeq: 4 };
- },
- };
const manager = new SessionManager({
store,
runStore,
- runtimeEventStore: Object.assign(runStore, runtimeCommitSink),
+ runtimeEventStore: runStore,
backends,
newId: nextId(),
now: nextNow(12_825),
});
const session = await manager.createSession(makeInput({ status: 'running' }));
- sessionId = session.id;
await seedRunningTurn(store, session.id, 'turn-1');
await seedRun(
runStore,
@@ -12913,48 +12904,28 @@ describe('SessionManager permission mode updates', () => {
}),
);
- const preRecoveryEvents = await runStore.readRuntimeEvents(session.id, 'run-1');
- assert.equal(
- buildInterruptedToolOutcomeCommits(preRecoveryEvents, 12_825, 'code_mode').length,
- 1,
- );
await manager.recoverInterruptedSessions();
- assert.equal((await readInvocation(runStore, session.id, 'run-1')).terminalEvent, undefined);
- assert.equal(outcomeCommitAttempts, 2);
+ const runtimeEvents = await runStore.readRuntimeEvents(session.id, 'run-1');
+ const recovery = resolveRuntimeRecovery(runtimeEvents);
assert.equal(
- (await runStore.readRuntimeEvents(session.id, 'run-1')).some(
- (event) => event.content?.kind === 'function_response',
- ),
+ runtimeEvents.some((event) => event.content?.kind === 'function_response'),
false,
);
-
- await manager.recoverInterruptedSessions();
-
- assert.equal(outcomeCommitAttempts, 4);
- const runtimeEvents = await runStore.readRuntimeEvents(session.id, 'run-1');
- const response = runtimeEvents.find(
- (event) => event.content?.kind === 'function_response' && event.content.id === 'exec-1',
- );
- assert.equal(response?.content?.kind, 'function_response');
- assert.equal(response?.content?.kind === 'function_response' && response.content.isError, true);
assert.deepEqual(
- response?.content?.kind === 'function_response' ? response.content.result : undefined,
- {
- kind: 'text',
- text: 'Tool Bash was interrupted before its result was committed. Its side effects may or may not have occurred. Do not retry it immediately; inspect the current state first.',
- uncertainOutcome: { code: 'outcome_unknown', retrySafe: false },
- },
- );
- const responseIndex = runtimeEvents.findIndex(
- (event) => event.content?.kind === 'function_response',
- );
- const terminalIndex = runtimeEvents.findIndex(isTerminalRuntimeEvent);
- assert.ok(responseIndex >= 0 && terminalIndex > responseIndex);
- assert.equal(
- runtimeInvocationOutcome(await readInvocation(runStore, session.id, 'run-1')),
- 'failed',
- );
+ recovery.decisions.map(({ toolCallId, status, reason }) => ({
+ toolCallId,
+ status,
+ reason,
+ })),
+ [{ toolCallId: 'exec-1', status: 'indeterminate', reason: 'dispatch_without_response' }],
+ );
+ const terminalEvent = runtimeEvents.find(isTerminalRuntimeEvent);
+ assert.equal(terminalEvent?.status, 'failed');
+ const invocation = await readInvocation(runStore, session.id, 'run-1');
+ assert.ok(invocation.terminalEvent);
+ assert.equal(runtimeInvocationFailureClass(invocation), 'outcome_unknown');
+ assert.equal(runtimeInvocationOutcome(invocation), 'failed');
});
test('startup recovery does not leave stale permission waits stuck', async () => {
diff --git a/packages/runtime/src/ai-sdk-turn.ts b/packages/runtime/src/ai-sdk-turn.ts
index cb2f3cca8b..541f76cb47 100644
--- a/packages/runtime/src/ai-sdk-turn.ts
+++ b/packages/runtime/src/ai-sdk-turn.ts
@@ -130,15 +130,19 @@ import {
} from './sandbox-boundary-tool.js';
import {
buildRuntimeEventModelReplayPlan,
+ appendPriorUnknownToolResponses,
buildSteeringEnvelope,
collectToolActivityTurnIds,
compatibleProviderReasoningReplayEventIds,
formatTextWithInlineRefs,
+ inspectPriorUnknownToolOutcomes,
steeringMessagesMissingFromBase,
steeringModelMessage,
+ type PriorUnknownToolOutcomeProjection,
type RuntimeEventModelReplayPlan,
type RuntimeEventReplayFallbackGate,
} from './model-history.js';
+import type { ContextBudgetPolicy } from './context-budget.js';
import {
toolSchemaCharsForDiagnostics,
requestCompositionToolSchemas,
@@ -855,7 +859,20 @@ export class AiSdkTurn {
const toolRuntime = this.toolRuntime;
const turnAbortController = this.abortController;
- const midTurnState = this.deps.compaction.buildMidTurnCapacityCompactState(input);
+ const priorEvents = (input.runtimeContext ?? []).filter(
+ (event) => input.continuation !== undefined || event.turnId !== input.turnId,
+ );
+ const priorUnknownProjection: PriorUnknownToolOutcomeProjection =
+ input.allowPriorUnknownToolOutcomes && !input.continuation
+ ? inspectPriorUnknownToolOutcomes(priorEvents, input.runtimeContextInvocations)
+ : { kind: 'none' };
+ if (priorUnknownProjection.kind === 'blocked') {
+ throw new Error(`Cannot use prior tool history: ${priorUnknownProjection.reason}`);
+ }
+ const preservingUnknownHistory = priorUnknownProjection.kind === 'projected';
+ const midTurnState = preservingUnknownHistory
+ ? undefined
+ : this.deps.compaction.buildMidTurnCapacityCompactState(input);
const queue = new AsyncEventQueue<SessionEvent>();
const codeModeExecTool = this.createCodeModeExecTool(queue);
@@ -1131,7 +1148,7 @@ export class AiSdkTurn {
let systemPrompt: string | undefined;
// --- Build messages from RuntimeEvent history and its compatibility projection. ---
- const priorReplayResult = await this.buildPriorMessages(input);
+ const priorReplayResult = await this.buildPriorMessages(input, priorUnknownProjection);
if (this.aborted) {
queue.push({
type: 'abort',
@@ -1283,6 +1300,10 @@ export class AiSdkTurn {
};
const loadDurableTurnProjection = async (): Promise<ModelMessage[]> => {
const turnEvents = await loadDurableTurnEvents();
+ // Prior unknown call/result pairs are outside this current-turn event
+ // set and are preserved separately in priorReplay. Continue pruning
+ // new tool results from this turn so one unknown predecessor does not
+ // make an unrelated large result permanently unprunable.
const pruned = await this.deps.compaction.pruneToolResults(turnEvents, turnId);
if (pruned.stats) {
contextBudgetForTelemetry = addToolResultPruneStats(
@@ -1468,6 +1489,9 @@ export class AiSdkTurn {
}
const requestSystemPromptBase = joinPromptFragments([
systemPrompt,
+ priorUnknownProjection.kind === 'projected'
+ ? priorUnknownProjection.systemNotice
+ : undefined,
finalChildSummaryStep ? CHILD_STEP_BUDGET_FINALIZATION_PROMPT : undefined,
toolRuntime.hasSandboxBoundaryDenial() ? SANDBOX_BOUNDARY_DENIED_FOR_TURN : undefined,
sandboxBoundaryFinalizationStep ? SANDBOX_BOUNDARY_FINALIZATION_PROMPT : undefined,
@@ -1948,6 +1972,7 @@ export class AiSdkTurn {
// nothing left to grant it, so the error is terminal.
const stepBudgetRemains = maxSteps === undefined || runtimeSteps < maxSteps;
const recovered =
+ !preservingUnknownHistory &&
stepBudgetRemains &&
failure.kind === 'context_overflow' &&
providerAttempt < MAX_PROVIDER_ATTEMPTS_PER_STEP &&
@@ -2678,7 +2703,10 @@ export class AiSdkTurn {
}
/** Materialize canonical RuntimeEvent history into ai-sdk's message format. */
- private async buildPriorMessages(input: BackendSendInput): Promise<PriorReplayResult> {
+ private async buildPriorMessages(
+ input: BackendSendInput,
+ priorUnknownProjection: PriorUnknownToolOutcomeProjection,
+ ): Promise<PriorReplayResult> {
if (!input.runtimeContext) {
return {
status: 'ready',
@@ -2696,40 +2724,81 @@ export class AiSdkTurn {
// the durable projection-transition reducer (#4283). Replay, budgeting and
// compaction share one input, so no RuntimeEvent replay path can resurrect
// content a committed transition removed.
- const preparedContextBudget = await this.deps.compaction.prepareContextBudgetPolicy(
- rawPriorRuntimeContext,
- input.turnId,
- );
- const priorRuntimeContext = preparedContextBudget.events;
+ const preservingUnknownHistory = priorUnknownProjection.kind === 'projected';
+ let priorRuntimeContext: RuntimeEvent[];
+ let runtimeContext: RuntimeEvent[];
+ let contextBudget: ContextBudgetPolicy | undefined;
+ let contextBudgetDiagnostic: ContextBudgetDiagnostic | undefined;
+ let projectedHistoryCompactCheckpoint: HistoryCompactCheckpoint | undefined;
+ if (priorUnknownProjection.kind === 'projected') {
+ const preparedContextBudget = await this.deps.compaction.prepareContextBudgetPolicy(
+ rawPriorRuntimeContext,
+ input.turnId,
+ );
+ priorRuntimeContext = preparedContextBudget.events;
+ contextBudget = preparedContextBudget.policy;
+ const budgeted = applyRuntimeEventContextBudget(rawPriorRuntimeContext, contextBudget);
+ const budgetedEvents = budgeted?.events ?? rawPriorRuntimeContext;
+ const preservesUnknownCalls = priorUnknownProjection.outcomes.every((outcome) =>
+ budgetedEvents.some((event) => event.id === outcome.callEventId),
+ );
+ // Existing checkpoints and durable result pruning are safe when they
+ // leave every unknown call available to pair with its temporary result.
+ // Otherwise replay the full effective ledger: a summary that erases T1
+ // would make the request-local T2 orphaned or hide the uncertainty.
+ const selectedEvents = preservesUnknownCalls ? budgetedEvents : rawPriorRuntimeContext;
+ const effectiveEvents = await this.deps.compaction.foldEffectiveModelHistory(
+ selectedEvents,
+ preparedContextBudget.projectionSnapshot,
+ );
+ runtimeContext = appendPriorUnknownToolResponses(effectiveEvents, priorUnknownProjection);
+ if (preservesUnknownCalls) {
+ contextBudgetDiagnostic = budgeted?.diagnostic;
+ projectedHistoryCompactCheckpoint = budgeted?.historyCompactCheckpoint;
+ }
+ if (preparedContextBudget.diagnosticPatch) {
+ contextBudgetDiagnostic = mergeContextBudgetDiagnostic(
+ contextBudgetDiagnostic ??
+ buildContextBudgetDiagnosticShell(priorRuntimeContext, effectiveEvents, contextBudget),
+ preparedContextBudget.diagnosticPatch,
+ );
+ }
+ } else {
+ const preparedContextBudget = await this.deps.compaction.prepareContextBudgetPolicy(
+ rawPriorRuntimeContext,
+ input.turnId,
+ );
+ priorRuntimeContext = preparedContextBudget.events;
+ contextBudget = preparedContextBudget.policy;
+ // Match the durable checkpoint against the RAW ledger prefix: every
+ // creation path (standalone compactHistory and the mid-turn state) pins
+ // its coverage digest on raw events, so matching the folded view here
+ // lets any durable projection transition inside the covered prefix orphan
+ // the checkpoint and silently fail open into a full-history replay
+ // (#4842). The projected [block, tail] is then folded through the
+ // transition reducer before it becomes messages, so a committed
+ // transition still cannot resurrect content for the model (#4283).
+ const budgeted = applyRuntimeEventContextBudget(rawPriorRuntimeContext, contextBudget);
+ runtimeContext = await this.deps.compaction.foldEffectiveModelHistory(
+ budgeted?.events ?? rawPriorRuntimeContext,
+ preparedContextBudget.projectionSnapshot,
+ );
+ contextBudgetDiagnostic = budgeted?.diagnostic;
+ projectedHistoryCompactCheckpoint = budgeted?.historyCompactCheckpoint;
+ if (preparedContextBudget.diagnosticPatch) {
+ contextBudgetDiagnostic = mergeContextBudgetDiagnostic(
+ contextBudgetDiagnostic ??
+ buildContextBudgetDiagnosticShell(priorRuntimeContext, runtimeContext, contextBudget),
+ preparedContextBudget.diagnosticPatch,
+ );
+ }
+ }
const providerReasoningReplayEventIds = compatibleProviderReasoningReplayEventIds(
priorRuntimeContext,
input.runtimeContextInvocations,
this.deps.backend.providerStateIdentity,
this.deps.backend.modelId,
);
- let contextBudget = preparedContextBudget.policy;
- // Match the durable checkpoint against the RAW ledger prefix: every
- // creation path (standalone compactHistory and the mid-turn state) pins
- // its coverage digest on raw events, so matching the folded view here
- // lets any durable projection transition inside the covered prefix orphan
- // the checkpoint and silently fail open into a full-history replay
- // (#4842). The projected [block, tail] is then folded through the
- // transition reducer before it becomes messages, so a committed
- // transition still cannot resurrect content for the model (#4283).
- const budgeted = applyRuntimeEventContextBudget(rawPriorRuntimeContext, contextBudget);
- let runtimeContext = await this.deps.compaction.foldEffectiveModelHistory(
- budgeted?.events ?? rawPriorRuntimeContext,
- preparedContextBudget.projectionSnapshot,
- );
- let contextBudgetDiagnostic = budgeted?.diagnostic;
- let projectedHistoryCompactCheckpoint = budgeted?.historyCompactCheckpoint;
- if (preparedContextBudget.diagnosticPatch) {
- contextBudgetDiagnostic = mergeContextBudgetDiagnostic(
- contextBudgetDiagnostic ??
- buildContextBudgetDiagnosticShell(priorRuntimeContext, runtimeContext, contextBudget),
- preparedContextBudget.diagnosticPatch,
- );
- }
// No pre-turn estimate gate: the turn's first request is judged by the
// request-projection hook from the previous request's real usage, and by
diff --git a/packages/runtime/src/model-history.ts b/packages/runtime/src/model-history.ts
index 476424e66c..f70feb3196 100644
--- a/packages/runtime/src/model-history.ts
+++ b/packages/runtime/src/model-history.ts
@@ -66,6 +66,7 @@ import type {
UserContent,
UserModelMessage,
} from './model-protocol.js';
+import { runtimeInvocationFailureClass } from './runtime-event-read-model.js';
import {
decodeEffectiveToolResultProjection,
durableProjectionToToolResultOutput,
@@ -74,9 +75,176 @@ import {
import { MATERIALIZED_IMAGE_TOKENS } from '@maka/core/attachments';
import { estimateTokens, stableJsonLength, turnKey } from './context-budget-helpers.js';
import type { DurableToolResultProjection } from '@maka/core/durable-tool-result-projection';
+import { resolveRuntimeRecovery } from './recovery-resolver.js';
export const PROVIDER_REPLAY_PROJECTION_VERSION = 2;
+export interface PriorUnknownToolOutcome {
+ readonly callEventId: string;
+ readonly toolCallId: string;
+ readonly toolName: string;
+ readonly operationId: string;
+}
+
+export type PriorUnknownToolOutcomeProjection =
+ | { readonly kind: 'none' }
+ | { readonly kind: 'blocked'; readonly reason: string }
+ | {
+ readonly kind: 'projected';
+ readonly outcomes: readonly PriorUnknownToolOutcome[];
+ readonly systemNotice: string;
+ };
+
+/**
+ * Admit only sealed T1-without-T2 facts for a manual fresh message. The
+ * synthetic tool response is request-local; it is never appended to the event
+ * ledger and must not be mistaken for evidence that the tool settled.
+ */
+export function inspectPriorUnknownToolOutcomes(
+ events: readonly RuntimeEvent[],
+ invocations: readonly RuntimeInvocationRecord[] | undefined,
+): PriorUnknownToolOutcomeProjection {
+ const eventsByInvocationId = new Map<string, RuntimeEvent[]>();
+ for (const event of events) {
+ const invocationEvents = eventsByInvocationId.get(event.invocationId) ?? [];
+ invocationEvents.push(event);
+ eventsByInvocationId.set(event.invocationId, invocationEvents);
+ }
+ const indeterminate = [];
+ for (const invocationEvents of eventsByInvocationId.values()) {
+ const recovery = resolveRuntimeRecovery(invocationEvents);
+ if (recovery.hasCorruption) {
+ return { kind: 'blocked', reason: 'prior tool history is corrupt' };
+ }
+ indeterminate.push(
+ ...recovery.decisions.filter((decision) => decision.status === 'indeterminate'),
+ );
+ }
+ if (indeterminate.length === 0) return { kind: 'none' };
+ if (
+ indeterminate.some(
+ (decision) =>
+ decision.reason !== 'dispatch_without_response' ||
+ !decision.operationId ||
+ !decision.callRuntimeEventId ||
+ !decision.dispatchRuntimeEventId,
+ )
+ ) {
+ return { kind: 'blocked', reason: 'prior tool outcome is not a sealed dispatched operation' };
+ }
+
+ const eventsById = new Map(events.map((event) => [event.id, event] as const));
+ const invocationById = new Map(
+ (invocations ?? []).map((run) => [run.invocationId, run] as const),
+ );
+ const outcomes: PriorUnknownToolOutcome[] = [];
+ for (const decision of indeterminate) {
+ const callEvent = eventsById.get(decision.callRuntimeEventId!);
+ const dispatchEvent = eventsById.get(decision.dispatchRuntimeEventId!);
+ const call = callEvent?.content;
+ const invocation = callEvent ? invocationById.get(callEvent.invocationId) : undefined;
+ const dispatch = dispatchEvent?.actions?.toolDispatch;
+ if (
+ !callEvent ||
+ call?.kind !== 'function_call' ||
+ !dispatch ||
+ dispatchEvent.invocationId !== callEvent.invocationId ||
+ dispatch.operationId !== decision.operationId ||
+ dispatch.providerToolCallId !== call.id ||
+ dispatch.toolName !== call.name ||
+ !decision.operationId ||
+ !invocation?.terminalEvent ||
+ invocation.sessionId !== callEvent.sessionId ||
+ invocation.runId !== callEvent.runId ||
+ invocation.turnId !== callEvent.turnId ||
+ invocation.terminalEvent.sessionId !== callEvent.sessionId ||
+ invocation.terminalEvent.runId !== callEvent.runId ||
+ invocation.terminalEvent.turnId !== callEvent.turnId ||
+ invocation.terminalEvent.invocationId !== callEvent.invocationId ||
+ !isTerminalRuntimeEvent(invocation.terminalEvent) ||
+ invocation.terminalEvent.status !== 'failed' ||
+ runtimeInvocationFailureClass(invocation) !== 'outcome_unknown'
+ ) {
+ return { kind: 'blocked', reason: 'prior unknown tool outcome has no sealed invocation' };
+ }
+ // Hidden nested operations are an implementation detail of their parent
+ // tool. Validate their durable boundary above, but never surface their
+ // names or operation identities in model-visible request context.
+ if (callEvent.modelVisibility === 'hidden') continue;
+ outcomes.push({
+ callEventId: callEvent.id,
+ toolCallId: call.id,
+ toolName: call.name,
+ operationId: decision.operationId,
+ });
+ }
+ if (outcomes.length === 0) return { kind: 'none' };
+
+ const rows = outcomes
+ .slice(0, 64)
+ .map((outcome) => `- ${outcome.toolName} (operation ${outcome.operationId})`);
+ if (outcomes.length > 64) rows.push(`- and ${outcomes.length - 64} more unknown operations`);
+ return {
+ kind: 'projected',
+ outcomes,
+ systemNotice: [
+ 'A prior execution was interrupted after these tools were dispatched. No durable tool results were recorded; their side effects may or may not have happened.',
+ 'The outcome_unknown tool responses in history are temporary context, not proof of success or failure. Inspect current state before repeating any action.',
+ ...rows,
+ ].join('\n'),
+ };
+}
+
+/** Insert request-only tool responses beside their calls so provider history stays well-formed. */
+export function appendPriorUnknownToolResponses(
+ events: readonly RuntimeEvent[],
+ projection: Extract<PriorUnknownToolOutcomeProjection, { kind: 'projected' }>,
+): RuntimeEvent[] {
+ const byCallEventId = new Map(
+ projection.outcomes.map((outcome) => [outcome.callEventId, outcome] as const),
+ );
+ const projected: RuntimeEvent[] = [];
+ for (const event of events) {
+ projected.push(event);
+ const outcome = byCallEventId.get(event.id);
+ const call = event.content;
+ if (!outcome || call?.kind !== 'function_call') continue;
+ projected.push({
+ id: `model-projection:outcome-unknown:${event.id}`,
+ invocationId: event.invocationId,
+ runId: event.runId,
+ sessionId: event.sessionId,
+ turnId: event.turnId,
+ ts: event.ts,
+ partial: false,
+ role: 'tool',
+ author: 'tool',
+ origin: event.origin ?? 'provider',
+ modelVisibility: event.modelVisibility ?? 'visible',
+ content: {
+ kind: 'function_response',
+ id: call.id,
+ name: call.name,
+ result: {
+ kind: 'text',
+ text: `outcome_unknown: ${call.name} was dispatched, but no durable result was recorded. Its effect may have happened. Inspect current state before deciding whether to repeat it.`,
+ uncertainOutcome: { code: 'outcome_unknown', retrySafe: false },
+ },
+ isError: true,
+ },
+ refs: {
+ operationId: outcome.operationId,
+ toolCallId: call.id,
+ ...(event.refs?.parentToolCallId ? { parentToolCallId: event.refs.parentToolCallId } : {}),
+ ...(event.refs?.parentOperationId
+ ? { parentOperationId: event.refs.parentOperationId }
+ : {}),
+ },
+ });
+ }
+ return projected;
+}
+
/**
* Resolve the RuntimeEvents whose provider-owned reasoning may cross the
* current provider boundary.
diff --git a/packages/runtime/src/recovery-resolver.ts b/packages/runtime/src/recovery-resolver.ts
index a989c4ee87..23b8c4ea3d 100644
--- a/packages/runtime/src/recovery-resolver.ts
+++ b/packages/runtime/src/recovery-resolver.ts
@@ -19,7 +19,6 @@
import {
TOOL_BOUNDARY_PROTOCOL_V1,
- isTerminalRuntimeEvent,
type RuntimeEvent,
type ToolBoundaryProtocol,
} from '@maka/core/runtime-event';
@@ -29,9 +28,6 @@ import {
type ToolLedgerScanOperation,
} from '@maka/core/tool-ledger-scanner';
import { interpretScannedToolRecovery } from '@maka/core/tool-recovery-bundle';
-import type { ToolOutcomeCommit } from './runtime-commit-sink.js';
-import type { ToolMode } from '@maka/core/tool-mode';
-import { compatibilityToolResultProjection } from './durable-tool-result-projection.js';
export type ToolRecoveryDecisionStatus =
| 'completed'
@@ -125,105 +121,6 @@ export function resolveRuntimeRecovery(events: readonly RuntimeEvent[]): Runtime
};
}
-export function buildInterruptedToolOutcomeCommits(
- events: readonly RuntimeEvent[],
- now: number,
- toolMode: ToolMode,
-): ToolOutcomeCommit[] {
- const eventsById = new Map(events.map((event) => [event.id, event] as const));
- const terminalInvocations = new Set(
- events.filter(isTerminalRuntimeEvent).map((event) => event.invocationId),
- );
- const recovery = resolveRuntimeRecovery(events);
- if (recovery.hasCorruption) return [];
- return recovery.decisions.flatMap((decision) => {
- if (
- decision.status !== 'indeterminate' ||
- decision.reason !== 'dispatch_without_response' ||
- !decision.operationId ||
- !decision.callRuntimeEventId
- ) {
- return [];
- }
- const callEvent = eventsById.get(decision.callRuntimeEventId);
- const call = callEvent?.content;
- if (!callEvent || call?.kind !== 'function_call') {
- return [];
- }
- // A sealed invocation cannot accept new RuntimeEvents. Legacy sessions can
- // still contain this gap; their projection is repaired from the terminal
- // fact instead of appending a result after the immutable tail.
- if (terminalInvocations.has(callEvent.invocationId)) return [];
-
- const codeModeExec =
- toolMode === 'code_mode' &&
- call.name === 'exec' &&
- callEvent.origin !== 'code_mode' &&
- callEvent.modelVisibility !== 'hidden';
- const interaction = call.name === 'AskUserQuestion';
- const result = codeModeExec
- ? {
- kind: 'json' as const,
- value: {
- kind: 'code_mode' as const,
- status: 'interrupted' as const,
- message:
- 'Code Mode execution was interrupted by runtime recovery. Its side effects may or may not have occurred; do not retry it immediately. Inspect the current state first.',
- uncertainOutcome: { code: 'outcome_unknown' as const, retrySafe: false as const },
- },
- }
- : {
- kind: 'text' as const,
- text: interaction
- ? 'This question was interrupted by runtime recovery before an answer was submitted.'
- : `Tool ${call.name} was interrupted before its result was committed. Its side effects may or may not have occurred. Do not retry it immediately; inspect the current state first.`,
- ...(!interaction
- ? { uncertainOutcome: { code: 'outcome_unknown' as const, retrySafe: false as const } }
- : {}),
- };
- const responseContent = {
- kind: 'function_response' as const,
- id: call.id,
- name: call.name,
- result,
- isError: true as const,
- };
- const modelProjection = compatibilityToolResultProjection(responseContent, callEvent.sessionId);
- const runtimeEvent: RuntimeEvent = {
- id: `${decision.operationId}_response`,
- invocationId: callEvent.invocationId,
- runId: callEvent.runId,
- sessionId: callEvent.sessionId,
- turnId: callEvent.turnId,
- ts: now,
- partial: false,
- role: 'tool',
- author: 'tool',
- origin: callEvent.origin ?? 'provider',
- modelVisibility: callEvent.modelVisibility ?? 'visible',
- content: { ...responseContent, ...(modelProjection ? { modelProjection } : {}) },
- refs: {
- operationId: decision.operationId,
- toolCallId: call.id,
- ...(callEvent.refs?.parentToolCallId
- ? { parentToolCallId: callEvent.refs.parentToolCallId }
- : {}),
- ...(callEvent.refs?.parentOperationId
- ? { parentOperationId: callEvent.refs.parentOperationId }
- : {}),
- },
- };
- return [
- {
- operationId: decision.operationId,
- journalEventId: `${decision.operationId}_outcome`,
- runtimeEvent,
- committedAt: now,
- },
- ];
- });
-}
-
function decisionFromOperation(
operation: ToolLedgerScanOperation,
toolBoundaryProtocol: ToolBoundaryProtocol | undefined,
diff --git a/packages/runtime/src/runtime-kernel.ts b/packages/runtime/src/runtime-kernel.ts
index a5de4ddedf..c926f5af44 100644
--- a/packages/runtime/src/runtime-kernel.ts
+++ b/packages/runtime/src/runtime-kernel.ts
@@ -724,6 +724,7 @@ export class RuntimeKernel implements RuntimeKernelLike {
this.attachExecutionClaim(execution, run);
yield* this.runAgentTurn(sessionId, run, execution, {
steering: true,
+ allowPriorUnknownToolOutcomes: isExplicitClientMessage(input),
onRunStarted: options.onRunStarted,
initialHeader: header,
});
@@ -1325,6 +1326,7 @@ export class RuntimeKernel implements RuntimeKernelLike {
execution: PendingExecutionClaim,
options: {
steering?: boolean;
+ allowPriorUnknownToolOutcomes?: boolean;
onRunStarted?: (runId: string, initialHeader: SessionHeader) => void | Promise<void>;
initialHeader?: SessionHeader;
prepareBackendActivation?: () => Promise<void>;
@@ -1376,6 +1378,7 @@ export class RuntimeKernel implements RuntimeKernelLike {
invocationId: begin.initialRuntimeEvent.invocationId,
runId: run.runId,
...begin.backendInput,
+ ...(options.allowPriorUnknownToolOutcomes ? { allowPriorUnknownToolOutcomes: true } : {}),
headAnchorRuntimeEvent: begin.initialRuntimeEvent,
handoffBoundary: (signal, remainingSteps) =>
run.reachHandoffBoundary(signal, remainingSteps),
@@ -3539,6 +3542,19 @@ function assertNoRemovedChildAgentRunLineage(input: UserMessageInput): void {
}
}
+function isExplicitClientMessage(input: UserMessageInput): boolean {
+ return (
+ input.origin === undefined &&
+ input.agentId === undefined &&
+ input.agentName === undefined &&
+ input.parentSessionId === undefined &&
+ input.parentTurnId === undefined &&
+ input.retriedFromTurnId === undefined &&
+ input.regeneratedFromTurnId === undefined &&
+ input.branchOfTurnId === undefined
+ );
+}
+
async function interactionResumeAllowed(
interactionRun: RuntimeInteractionRunBinding | undefined,
event: SessionEvent,
diff --git a/packages/runtime/src/session-manager.ts b/packages/runtime/src/session-manager.ts
index 40b020fc83..4dd371e31f 100644
--- a/packages/runtime/src/session-manager.ts
+++ b/packages/runtime/src/session-manager.ts
@@ -187,13 +187,12 @@ import type { HistoryCompactCheckpoint } from './history-compact-checkpoint.js';
import type { ModelProjectionTransition } from '@maka/core/model-projection-transition';
import type { LoadedModelProjectionTransitions } from './model-projection-transition-ledger.js';
import type { RuntimeContinuationFailpoint } from './agent-run.js';
-import type { RuntimeCommitResult, RuntimeCommitSink } from './runtime-commit-sink.js';
import {
attributeSandboxBoundaryRestartClosure,
classifyAgentRunRecovery,
type AgentRunRecoveryDecision,
} from './agent-run-recovery.js';
-import { buildInterruptedToolOutcomeCommits, resolveRuntimeRecovery } from './recovery-resolver.js';
+import { resolveRuntimeRecovery } from './recovery-resolver.js';
import {
isRuntimeHostedRootAuthority,
RuntimeMessageAuthorityInvariantError,
@@ -259,16 +258,6 @@ function runtimeContinuationAuthority(
: undefined;
}
-function runtimeCommitSinkFromEventStore(
- store: RuntimeEventStore | undefined,
-): RuntimeCommitSink | undefined {
- const candidate = store as Partial<RuntimeCommitSink> | undefined;
- return typeof candidate?.commitToolPrepared === 'function' &&
- typeof candidate.commitToolOutcome === 'function'
- ? (candidate as RuntimeCommitSink)
- : undefined;
-}
-
export type StopSessionInput =
| {
source?: 'stop_button' | 'graph_supervisor';
@@ -814,7 +803,6 @@ interface SessionManagerBaseDeps {
planStore?: PlanStore;
runStore?: AgentRunStore;
runtimeEventStore?: RuntimeEventStore;
- runtimeCommitSink?: RuntimeCommitSink;
/** Host capability; RuntimeKernel gates it by the selected backend. */
toolBoundaryProtocol?: ToolBoundaryProtocol;
backends: BackendRegistry;
@@ -908,7 +896,6 @@ export class SessionManager {
private readonly runtimeKernel: RuntimeKernelLike;
private readonly runtimeLedgerRepair?: RuntimeLedgerRepair;
private readonly preparedTranscriptLedgers = new Set<string>();
- private readonly runtimeCommitSink?: RuntimeCommitSink;
private readonly activeHostedLinkedChildSessions = new Set<string>();
private readonly childSessionSpawns = new Map<
string,
@@ -927,8 +914,6 @@ export class SessionManager {
if (deps.publishChildWorkspacePatch && !deps.listArtifactsForTurn) {
throw new Error('Child workspace patch publication requires Artifact turn listing');
}
- this.runtimeCommitSink =
- deps.runtimeCommitSink ?? runtimeCommitSinkFromEventStore(deps.runtimeEventStore);
if (deps.runStore && deps.runtimeEventStore) {
this.runtimeLedgerRepair = new RuntimeLedgerRepair({
runtimeEventStore: deps.runtimeEventStore,
@@ -4855,51 +4840,24 @@ export class SessionManager {
// T1 states; generic app-restart repair must never write into them.
continue;
}
- if (this.runtimeCommitSink) {
- const interruptedOutcomes = buildInterruptedToolOutcomeCommits(
- inspected.runtimeEvents,
- this.deps.now(),
- run.opening.configuration.toolMode,
- );
- let outcomeCommitFailed = false;
- for (const outcome of interruptedOutcomes) {
- const committed = await commitInterruptedOutcomeWithRetry(policy, () =>
- this.runtimeCommitSink!.commitToolOutcome(outcome),
- );
- if (!committed) {
- // Keep the run non-terminal so a later recovery pass can retry the
- // missing outcome before any terminal repair seals the ledger.
- outcomeCommitFailed = true;
- } else {
- recovered ||= committed.created;
- }
- }
- if (outcomeCommitFailed) {
- continue;
- }
- if (interruptedOutcomes.length > 0) {
- inspected = await inspectAgentRunReadModel(
- recoveryRunStore,
- this.deps.runtimeEventStore,
- {
- sessionId,
- runId: run.runId,
- invocation: run,
- includeModelReplay: false,
- includeProjection: false,
- },
- );
- }
- }
+ let unknownDispatchedTools: { toolCallId: string; toolName?: string }[] = [];
if (!inspected.runtimeEvents.some(isTerminalRuntimeEvent)) {
const toolRecovery = resolveRuntimeRecovery(inspected.runtimeEvents);
+ const indeterminate = toolRecovery.decisions.filter(
+ (decision) => decision.status === 'indeterminate',
+ );
+ unknownDispatchedTools = indeterminate
+ .filter((decision) => decision.reason === 'dispatch_without_response')
+ .map(({ toolCallId, toolName }) => ({
+ toolCallId,
+ ...(toolName ? { toolName } : {}),
+ }));
if (
toolRecovery.hasCorruption ||
- toolRecovery.decisions.some((decision) => decision.status === 'indeterminate')
+ (indeterminate.length > 0 && unknownDispatchedTools.length !== indeterminate.length)
) {
- // Never seal an invocation while a dispatched operation still lacks
- // a result or an explicit recovery decision. A later recovery pass
- // may retry the durable settlement; corruption stays fail-closed.
+ // Only a structurally valid T1-without-T2 boundary can be sealed as
+ // unknown. Legacy gaps and corrupt recovery facts still fail closed.
continue;
}
}
@@ -4915,10 +4873,25 @@ export class SessionManager {
const runtimeDecision = this.classifyRuntimeEventRecovery(inspected);
const classified = runtimeDecision ?? classifyAgentRunRecovery(run, inspected.events);
if (!classified) continue;
- const decision =
+ let decision =
classified.status === 'failed'
? attributeSandboxBoundaryRestartClosure(classified, await readBoundaryClosures())
: classified;
+ if (unknownDispatchedTools.length > 0) {
+ decision = {
+ ...decision,
+ status: 'failed',
+ failureClass: 'outcome_unknown',
+ diagnostic: {
+ ...decision.diagnostic,
+ recoveryReason: 'outcome_unknown',
+ unresolvedToolCalls: unknownDispatchedTools.map(({ toolCallId, toolName }) => ({
+ toolCallId,
+ ...(toolName ? { toolName } : {}),
+ })),
+ },
+ };
+ }
if (await this.applyAgentRunRecovery(sessionId, decision, inspected, policy)) {
recovered = true;
}
@@ -5030,8 +5003,6 @@ function groupContinuationClaimsBySession(
return grouped;
}
-const MAX_BEST_EFFORT_OUTCOME_COMMIT_ATTEMPTS = 2;
-
function listSessionsForRecovery(
store: SessionStore,
policy: RecoveryPolicy,
@@ -5052,21 +5023,6 @@ async function recoverOr<T>(
}
}
-async function commitInterruptedOutcomeWithRetry(
- policy: RecoveryPolicy,
- operation: () => Promise<RuntimeCommitResult>,
-): Promise<RuntimeCommitResult | undefined> {
- const attempts = policy.kind === 'strict' ? 1 : MAX_BEST_EFFORT_OUTCOME_COMMIT_ATTEMPTS;
- for (let attempt = 0; attempt < attempts; attempt += 1) {
- try {
- return await operation();
- } catch (error) {
- if (policy.kind === 'strict') throw error;
- }
- }
- return undefined;
-}
-
function continuationRepairEventId(
kind: 'start' | 'terminal' | 'run-terminal',
claimId: string,
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org