You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@maka.apache.org by "Astro-Han (via GitHub)" <gi...@apache.org> on 2026/09/05 22:08:24 UTC
[PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Astro-Han opened a new pull request, #4879:
URL: https://github.com/apache/maka/pull/4879
Closes #4791.
## The problem
Every execution fact was written twice: once as a `RuntimeEvent`, once as a `session_messages` row. Two authorities for one fact means every writer has to keep them in step, every reader has to choose one, and a crash between the two writes leaves a Session that disagrees with itself.
## What this does
Makes the `RuntimeEvent` ledger the only durable transcript authority and deletes the second write. `session_messages` survives in exactly two roles: input to the one-way importer that converts a pre-ledger transcript on first read, and the WorkHub Coordination Session's own store, which is out of scope for this issue.
The six commits are the staging of one cutover — notes onto the ledger, a whole and resumable conversion, the read model, then the cut — and are meant to land together. Losing a legacy tool card in conversion is acceptable; losing a Session or a conversation is not, and the importer is a total function over every legacy row type.
## The cut, site by site
- **`markMessagesHandedOff`** no longer projects admitted Messages into transcript rows. It validates the admission and retires it. The proof those rows carried already lives in the agent-run admission's `sourceMessages` and in the RuntimeEvent steering proof.
- **Catalog projection** (`lastMessagePreview`, `lastMessageAt`, `connectionLocked`) used to fall out of a transcript insert. `AgentRun` now commits it explicitly through `commitMessageCatalogProjection`: fail-closed for a user message, because that write also takes the Session's one-way connection lock, and fail-open for the assistant preview, which costs a stale sidebar line at worst.
- **The read marker** no longer needs an ordered index of visible transcript rows. `lastReadMessageId` has no client consumer, so `hasUnread` is the only decision left, and it clears when the client has caught up with the ledger's newest visible message — read off a bounded tail of the last run rather than a whole-Session scan.
- **Startup recovery** writes a crashed Turn's admitted prompt into the invocation that had already opened for it. A Root folded from several queued Messages carries no single admitted Message identity, so the prompt is durable under a derived `${runId}-admitted-prompt`; recovering the same crash twice writes the same event and the store dedupes it. A sealed Run takes nothing — it is immutable, and a Run that reached its terminal fact has a prompt the crash did not eat.
- **WorkHub target linkage** (#4699) enumerated a delegated Message's identity from three lifecycle tables, one of which was the transcript row this PR stops writing. Its handed-off arm now reads `core_root_source_message_proofs` — the Root admission that consumed the Message, in the same database and as durable as the Session.
Deleted with their last caller: `markSessionReadThroughMessage`, `SessionReadMarkerMessageNotFoundError`, `readMessagesForRecovery` (byte-identical to `readMessages`), `listForRecovery`'s separate query, the transcript-ordering privates in the SQLite store, and `buildTurnStateMessage` with its lineage types.
## Migration
`transcriptLedgerVersion` distinguishes the three states: absent means pre-ledger and converted on read, `0` means an imported transcript staged for conversion, `1` means ledger-authoritative. Event ids are derived from the run and the position within it, so an interrupted import is resumable — re-running it writes the same events and the store dedupes them.
## Ablations kept out
- An `existing.some(role !== 'system')` guard in recovery layered on top of the terminal-event check: the terminal check alone is exact, so the extra read was removed.
- Removing the singular `appendMessage`: pure test churn with no production gain, so it stayed.
## Verification
Net **-2545** lines. On this base, all five workspace suites pass with zero failures: storage 1092, runtime-host 1709, runtime 3129, core 821, cli 805.
Not covered by automated tests and worth a human pass before merge: opening a Session created by an older build and confirming its whole history renders, and the desktop WorkHub view after a delegated Message has been consumed by a Turn.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "Astro-Han (via GitHub)" <gi...@apache.org>.
Astro-Han commented on code in PR #4879:
URL: https://github.com/apache/maka/pull/4879#discussion_r3951356935
##########
packages/core/src/session.ts:
##########
@@ -1259,7 +1270,6 @@ const TOKEN_USAGE_MESSAGE_SHAPE = defineObjectShape<TokenUsageMessage>()(
const TURN_STATE_MESSAGE_SHAPE = defineObjectShape<TurnStateMessage>()(
['type', 'id', 'turnId', 'ts', 'status'],
[
Review Comment:
Fixed in `3a4371a74`. Your reproduction is the decisive part — the control Session is what rules out "something else broke", and I reproduced the same three cases before changing anything.
**The diagnosis I accept, and the one I do not.** You are right that the field's deletion is correct and that the gap is a persisted-shape change made as if it were a type change. I did not fix it where you pointed, because fixing it there would have left the same trap armed three more times.
**Fixed at the shape helper, not at this call site.** `defineObjectShape` now takes a third list, `retired`: keys older writers persisted that the type no longer has. `hasExactShape` accepts them, `pickShape` strips them before the value is returned, and the decode branch for `turn_state` now returns `pickShape(...)`. So what a shape *emits* may shrink freely while what it *accepts* only grows — which is the rule the exact-shape design needed and never stated.
**The reason for the owner-level fix is that the same obligation already had three hand-rolled copies.** `session.ts` carried `RETIRED_SUBAGENT_RUNTIME_KEYS` with its own filter function; `usage-record-schema.ts` carried a second decode shape for the context budget and a third for the last-request anchor, each a widened duplicate of the emit shape. All three said the same thing in three different ways, and none of them made the next removal safe. They are gone: two deleted outright, the context-budget pair collapsed into one `defineObjectShape` call with the retired list as its third argument.
**Ablation.** Removing `retired` and reverting to a widened `allowed` set still passes the new test, because a widened `allowed` also decodes the old rows — but it re-emits the key, which is the second authority this PR set out to delete. Keeping `retired` is what makes accept and emit different sets on purpose.
Test: `session-retired-fields.test.ts` — a stored `turn_state` with the field `true`, with it `false`, and without it all decode to the same current record, and a key no released writer produced is still rejected.
**On the epoch:** agreed, and I did not use it for this. The bump stands for the wire change; the disk change is what `retired` answers.
<details><summary>简体中文</summary>
已在 `3a4371a74` 修复。你的复现是决定性的那一半——**对照 Session** 正是排除「是别的东西坏了」的东西；我在动手之前把那三种情形都独立复现了一遍。
**我接受的诊断，和我不接受的那部分。** 你说「删除本身是对的」「缺口在于把一次持久化形状变更当成类型变更来做」——都对。但我**没有修在你指的位置**，因为修在那里会让同一个陷阱在另外三处继续张着。
**修在形状定义处，不是修在这个调用点。** `defineObjectShape` 现在接受第三个列表 `retired`：旧写入方持久化过、而当前类型已不再拥有的键。`hasExactShape` 接受它们，`pickShape` 在返回值之前把它们剥掉，`turn_state` 的解码分支现在返回 `pickShape(...)`。于是——**一个形状「发出」的可以自由收缩，而它「接受」的只能增长**。这正是 exact-shape 这个设计一直需要、却从未写明的规则。
**做成 owner 级修复的理由是：同一条义务此前已有三份手写拷贝。** `session.ts` 里有 `RETIRED_SUBAGENT_RUNTIME_KEYS` 和它自己的过滤函数；`usage-record-schema.ts` 里有 context budget 的第二份解码形状、以及 last-request anchor 的第三份，每一份都是 emit 形状的加宽副本。三处说的是同一件事、说法各不相同，而且**没有一处能让下一次删除变得安全**。它们没了：两份直接删除，context budget 那一对合并成一次 `defineObjectShape` 调用、把 retired 列表作为第三参数传入。
**消融实验。** 去掉 `retired`、改回加宽 `allowed`，新测试**照样通过**——因为加宽的 `allowed` 也能解码旧行。但它会把这个键**重新发出去**，而那正是本 PR 要删掉的第二权威。保留 `retired`，才让「接受」与「发出」成为两个刻意不同的集合。
测试：`session-retired-fields.test.ts`——存储的 `turn_state` 带该字段为 `true`、为 `false`、以及不带该字段，三者都解码为同一条当前记录；而任何已发布写入方都不会产出的键，仍然被拒绝。
**关于 epoch：** 同意，我没有拿它来顶这一条。抬升对应的是协议线上的变更；磁盘上的那一半，是 `retired` 回答的。
</details>
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
packages/runtime-host/src/server/session-transcript-reader.ts:
##########
@@ -137,12 +196,449 @@ export interface SessionTranscriptReader {
sessionId: string,
request: SessionTranscriptMessageLookupRequest,
): Promise<readonly StoredMessage[]>;
+ readDurableTurnContributions(
+ sessionId: string,
+ throughSequence: number | null,
+ position: number,
+ maxContributions: number,
+ ): Promise<SessionTurnContributionPage>;
+ readDurableTurnLandmarks(
+ sessionId: string,
+ maxLandmarks: number,
+ ): Promise<SessionTurnLandmarkSnapshot>;
readActiveOverlay(
sessionId: string,
rootTurn: TurnSnapshot | null,
): Promise<readonly StoredMessage[]>;
}
+/**
+ * Pages seek immutable Session event ordinals before decoding payloads. One
+ * Turn is projected at a time, so a page costs one Turn rather than the
+ * Session. The low sequence bits distinguish the few rows one event emits.
+ *
+ * What an event becomes is asked only of the read model. Storage selects Turns
+ * by ordinal and hands over their events; it never classifies one.
+ */
+function createDurableLedgerTranscriptReader(input: {
+ stores: ExecutionStoresWriter<'interactive'>;
+ canonicalPermissionOutcomes: CanonicalPermissionOutcomeReader;
+}) {
+ const store = input.stores.runtimeEventStore;
+ const highWater = async (sessionId: string): Promise<number | null> => {
+ const ordinal = await store.readTranscriptHighWater(sessionId);
+ return ordinal === null ? null : ordinal * EVENT_SEQUENCE_STRIDE + EVENT_SEQUENCE_STRIDE - 1;
+ };
+
+ /** One Turn's rows, each at the sequence its own event sits at. */
+ const projectTurn = async (
+ turn: RuntimeTranscriptInvocation,
+ ): Promise<{ sequence: number; message: StoredMessage }[]> => {
+ const events = turn.events.map((entry) => entry.event);
+ const projected = projectRuntimeEventsToStoredMessages(events, {
+ invocations: [turn.invocation],
+ canonicalPermissionOutcomes: await readCanonicalPermissionOutcomes(
+ events,
+ input.canonicalPermissionOutcomes,
+ ),
+ });
+ if (projected.diagnostics.some(isHardRuntimeEventReadModelDiagnostic)) {
+ throw new Error('Durable RuntimeEvent transcript projection is incomplete');
+ }
+ const ordinals = new Map(turn.events.map((entry) => [entry.event.id, entry.ordinal]));
+ const emitted = new Map<number, number>();
+ return projected.messages.map((message, index) => {
+ const ordinal = ordinals.get(projected.sourceEventIds[index]!);
+ if (ordinal === undefined) {
+ throw new Error('Durable transcript message has no source RuntimeEvent');
+ }
+ const offset = emitted.get(ordinal) ?? 0;
+ if (offset >= EVENT_SEQUENCE_STRIDE) {
+ throw new Error('RuntimeEvent exceeds its transcript sequence stride');
+ }
+ emitted.set(ordinal, offset + 1);
+ return { sequence: ordinal * EVENT_SEQUENCE_STRIDE + offset, message };
+ });
+ };
+
+ const readTurns = async (
+ sessionId: string,
+ request: { direction: 'older' | 'newer'; throughOrdinal: number; position: number },
+ ): Promise<RuntimeTranscriptInvocation[]> =>
+ store.readTranscriptInvocations(sessionId, {
+ ...request,
+ limit: TRANSCRIPT_TURN_SCAN_LIMIT,
+ maxEvents: DURABLE_TRANSCRIPT_TURN_MAX_EVENTS,
+ maxBytes: DURABLE_TRANSCRIPT_TURN_MAX_BYTES,
+ });
+
+ const scan = async function* (
+ sessionId: string,
+ request: {
+ direction: 'older' | 'newer';
+ throughSequence?: number | null;
+ position?: number;
+ /** Stops the walk after this many Turns, for a read that may find nothing. */
+ maxTurns?: number;
+ },
+ ): AsyncGenerator<{ sequence: number; message: StoredMessage }> {
+ const throughSequence =
+ request.throughSequence === undefined ? await highWater(sessionId) : request.throughSequence;
+ if (throughSequence === null) return;
+ const position = request.position ?? (request.direction === 'older' ? throughSequence : 0);
+ const throughOrdinal = ordinalOf(throughSequence);
+ let ordinal = ordinalOf(position);
+ let walked = 0;
+ while (ordinal >= 0 && ordinal <= throughOrdinal) {
+ const turns = await readTurns(sessionId, {
+ direction: request.direction,
+ throughOrdinal,
+ position: ordinal,
+ });
+ if (turns.length === 0) return;
+ for (const turn of turns) {
+ if (request.maxTurns !== undefined && walked >= request.maxTurns) return;
+ walked += 1;
+ const records = (await projectTurn(turn)).filter(
+ ({ sequence }) =>
+ sequence <= throughSequence &&
+ (request.direction === 'older' ? sequence <= position : sequence >= position),
+ );
+ if (request.direction === 'older') records.reverse();
+ yield* records;
+ }
+ const edge = turns.at(-1)!;
+ ordinal = request.direction === 'older' ? edge.firstOrdinal - 1 : edge.lastOrdinal + 1;
Review Comment:
Fixed in `e2f304fce`. Your location is the one I acted on: selection was sound, resumption was not, and the "correct in one sweep, short across pages" contrast is what proves it.
**The invariant that was missing, stated once.** A page resumes from one record's `sequence` and discards everything on the far side of it. That only works if what the walk yields is **monotone in `sequence`** — a scalar cursor can only express a position in a total order. Emitting Turn A's rows, then Turn B's rows that sit inside A's range, then A's later rows is not monotone, so the cursor cannot name where the page stopped without excluding rows it has not yet returned. Nothing was ever wrong with the queries; the walk was handing a total-order cursor a non-total-order stream.
**The fix restores monotonicity instead of adding a second cursor.** The scan now drains overlapping Turns as one cluster: it keeps reading the next Turn by ordinal while that Turn's range overlaps the accumulated range, projects the whole cluster, sorts it by `sequence` in the walk's direction, and only then yields. The cursor moves past the cluster, not past a Turn. Nesting depth falls out of this — a three-level nest is one cluster.
**This also closes #3949121492**, which is the forward face of the same defect: stepping to `lastOrdinal + 1` jumped over a contained Turn. Both directions now step by the cluster's edge, so a nested Turn is never a row the walk skipped.
**Ablation.** I tried the cheaper repair first — resume from the next unread Turn in the spine, per the earlier comment. It fixes the skip but not this: the rows still come out interleaved, so the page boundary still has to exclude some of them. Ordering is the obligation; Turn selection was only where the symptom showed.
Test: `pages a nested Turn the same way a single sweep reads it` — the nested shape, both directions, one message per page following `result.next.position`, asserted equal to a single 64-message sweep. It fails on the previous head in the `older` direction with exactly the rows you reported.
**On the attribution:** agreed and matched — the rewrite commit is not the cause, the resumption logic predates it in this PR, and the forward shape it fixed stays fixed. The `settledInline` observation is also right and I have left that part alone.
<details><summary>简体中文</summary>
已在 `e2f304fce` 修复。你的定位就是我下手的地方：**选择是成立的，续读不是**；而「一次扫完对、分页则短」这个对比正是它的证明。
**缺的那条不变量，只说一遍。** 一页从某条记录的 `sequence` 续读，并丢弃它另一侧的一切。**这只有在遍历产出的东西对 `sequence` 单调时才成立**——一个标量游标只能表达全序中的一个位置。先发 Turn A 的行、再发落在 A 区间内部的 B 的行、然后又发 A 更靠后的行，这不是单调的；于是游标无法在不排除「尚未返回的行」的前提下指出这一页停在哪。查询从来没有错；错在**遍历把一个非全序的流交给了一个全序游标**。
**修法是恢复单调，而不是再加一个游标。** 扫描现在把相互重叠的 Turn 作为**一个簇**排空：只要下一个 Turn 的区间与已累积区间重叠就继续读，把整簇一起投影，按遍历方向对 `sequence` 排序，然后才产出。**游标越过的是簇，不是单个 Turn。** 嵌套深度由此自动落地——三层嵌套就是一个簇。
**这同时关掉 #3949121492**，它是同一缺陷的前向面：步进到 `lastOrdinal + 1` 会跨过被包含的 Turn。现在两个方向都按簇的边界步进，嵌套的 Turn 永远不会是「被跨过的行」。
**消融实验。** 我先试了更省的那个修法——按前一条评论的建议，从 spine 中下一个未读 Turn 续读。它能修掉跨过，但修不掉这一条：**行仍然是交错出来的**，页边界因此仍然要排除其中一部分。**排序才是义务；Turn 选择只是症状显形的地方。**
测试：`pages a nested Turn the same way a single sweep reads it`——嵌套形状，两个方向，每页一条并跟随 `result.next.position`，断言与一次 64 条的扫描相等。在上一个 head 上，它在 `older` 方向以你报告的那批行失败。
**关于归属：** 同意，也核对过——重写那个 commit 不是成因，续读逻辑在本 PR 里更早引入，它修好的前向形状仍然是好的。`settledInline` 那段观察也是对的，我没有动它。
</details>
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener commented on PR #4879:
URL: https://github.com/apache/maka/pull/4879#issuecomment-5558371673
**Follow-up: the WorkHub linkage lane, posted separately as promised rather than amending the earlier review.** Exact head `a6fca96a`. **No finding — the substitution holds, including for Sessions that delegated before this PR.**
This was the second item the description flags as needing a human pass, and it is a cross-PR question: #4699's target linkage enumerates a delegated Message's identity from three lifecycle tables, and one of those arms read the transcript row this PR stops writing.
**The concern was the upgrade case, and it is closed.** A Session that completed a delegation before this lands would have the old transcript row and, if the proofs table were new, no replacement — so its linkage would quietly disappear. That is not the situation: `core_root_source_message_proofs` is not created by this PR, and neither the table nor its `INSERT` is new. Both have been in `agent-run-store.ts` since `4153915b8` (#1682), through `1caea265c` (#1994) and `ce6534d65` (#2445). **Historical Root admissions already wrote their proofs, so the replacement arm finds exactly the delegations the old arm found.**
**The identities line up because the proof is the same id.** For each `source` of `admission.sourceMessages` the store inserts `(sessionId, source.messageId, turnId)` — the delegated Message's own id, which is the `whm_` value the query's `GLOB 'whm_*'` and `length(message_id) = 52` predicates already selected on. The predicates are unchanged; only the table they read moved.
**It is also a better record than the one it replaces.** The three arms now map cleanly onto the three states a delegated Message can be in — pending in `message_admissions`, admitted into a Turn, cancelled in `cancelled_message_admissions` — and the middle arm reads the admission that *consumed* the Message rather than a transcript row that happened to be written alongside it. The old arm depended on a side effect of the second write this PR removes; the new one depends on the record whose purpose is to say the Message was consumed. **The dependency got more direct, not more fragile.**
Two limits worth stating rather than leaving implied:
- This was verified by reading the query, the insert site and that file's history — **not by running a probe against a real workspace with a pre-PR delegation.** The reasoning is that the proofs rows must already exist because their writer predates the change by many releases; a probe would make it a measurement instead of an inference.
- One shape would diverge: a `whm_` Message written into the transcript but never admitted and never cancelled would have been picked up by the old arm and is not picked up by the new one. **I did not find a producer for that state** — a delegated Message with no lifecycle record at all — so I am recording it as a shape I could not reach rather than as a risk.
With this, all four lanes on this PR are reported. The [P1] on the recovery path stands as the one thing to fix.
<details><summary>简体中文</summary>
**补充:WorkHub 联动这条车道,按先前承诺另发一条,而不是修改已发出的评审。** exact head `a6fca96a`。**无 finding —— 这次替换是成立的,包括对本 PR 之前就发生过委派的 Session。**
这是描述中点名需要人工过一遍的第二项,而且是个跨 PR 的问题:#4699 的目标联动从**三张生命周期表**枚举被委派 Message 的身份,**其中一条臂读的正是本 PR 停止写入的 transcript 行。**
**我担心的是升级情形,而它已经排除。** 一个在本 PR 落地之前完成过委派的 Session,会有旧的 transcript 行;若 proofs 表是新建的、没有替代记录,它的联动就会**静默消失**。事实并非如此:`core_root_source_message_proofs` **不是本 PR 创建的**,表和它的 `INSERT` 都不是新的 —— 两者自 `4153915b8`(#1682)起就在 `agent-run-store.ts` 里,并经过 `1caea265c`(#1994)与 `ce6534d65`(#2445)。**历史上的 Root admission 早已写下自己的 proof,所以替代臂找到的正是旧臂找到的那些委派。**
**身份能对上,是因为 proof 存的就是同一个 id。** 对 `admission.sourceMessages` 中的每个 `source`,store 插入 `(sessionId, source.messageId, turnId)` —— **就是被委派 Message 自身的 id**,也正是查询里 `GLOB 'whm_*'` 与 `length(message_id) = 52` 一直在筛选的那个 `whm_` 值。**谓词没有变,变的只是它读哪张表。**
**而且它比被取代的那条记录更合适。** 三条臂现在干净地对应一条被委派 Message 可能处于的三种状态 —— 在 `message_admissions` 中待处理、已被纳入某个 Turn、在 `cancelled_message_admissions` 中被取消 —— **而中间那条臂读的是「消费了该 Message 的那次 admission」,不再是碰巧与之一同写下的 transcript 行。** 旧臂依赖的是本 PR 所移除的第二次写入的**副作用**;新臂依赖的是**其存在意义就是「该 Message 已被消费」的那条记录**。**依赖变得更直接,而不是更脆弱。**
有两处限制,与其留作暗示不如明说:
- 这是通过**阅读**查询、插入点与该文件的历史核实的 —— **不是对一个含有 PR 之前委派的真实工作区跑探针跑出来的。** 推理依据是:proofs 行必然已经存在,因为它的写入方比本次改动早了许多个发布;**一次探针会把它从推断变成实测。**
- 有一种形状会分歧:一条被写入 transcript、却**既未被纳入、也未被取消**的 `whm_` Message,旧臂会捡到而新臂不会。**我没有找到能产出该状态的路径** —— 一条完全没有生命周期记录的被委派 Message —— 所以我把它记为**我未能到达的形状**,而不是一项风险。
至此本 PR 的四条车道全部报完。**恢复路径上的那条 [P1] 仍是唯一需要修的东西。**
</details>
---
> **Automated review notice:** This comment was posted by an automated review agent operated by jackwener. It is not an independent human review and does not replace one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "Astro-Han (via GitHub)" <gi...@apache.org>.
Astro-Han commented on code in PR #4879:
URL: https://github.com/apache/maka/pull/4879#discussion_r3947718421
##########
packages/runtime/src/runtime-ledger-repair.ts:
##########
@@ -35,127 +34,188 @@ import type { SessionHeader } from '@maka/core/session';
import type { StoredMessage, TurnRecord } from '@maka/core/session';
import { backfillRuntimeEventsFromStoredMessages } from './runtime-event-backfill.js';
import type { RuntimeEventBackfillOutcome } from './runtime-event-backfill.js';
-import { projectRuntimeEventUserMessage } from './runtime-event-read-model.js';
export interface RuntimeLedgerRepairDeps {
runtimeEventStore: RuntimeEventStore;
- readMessages(sessionId: string): Promise<StoredMessage[]>;
- appendMessage(sessionId: string, message: StoredMessage): Promise<void>;
- newId: () => string;
- now: () => number;
-}
-
-interface RuntimeEventTranscriptProjectionDeps {
- readMessages(sessionId: string): Promise<StoredMessage[]>;
- appendMessage(sessionId: string, message: StoredMessage): Promise<void>;
+ /**
+ * One forward page of the legacy transcript this converter reads; nothing
+ * writes back to it. It is read a page at a time because a Session cannot
+ * serve its first transcript page until this finishes, and a Session's
+ * history is not a bound.
+ */
+ readMessagesAfter(
+ sessionId: string,
+ request: { afterSequence?: number; maxMessages: number; maxStoredBytes: number },
+ ): Promise<{
+ records: readonly { sequence: number; message: StoredMessage }[];
+ highWaterSequence: number | null;
+ }>;
}
-export async function materializeRuntimeEventTranscriptProjection(
- deps: RuntimeEventTranscriptProjectionDeps,
- sessionId: string,
- event: RuntimeEvent,
- knownMessageIds?: Set<string>,
-): Promise<boolean> {
- const message = steeringMessageFromRuntimeEvent(event);
- if (!message) return false;
- const messageIds =
- knownMessageIds ?? new Set((await deps.readMessages(sessionId)).map((item) => item.id));
- if (messageIds.has(message.id)) return false;
- await deps.appendMessage(sessionId, message);
- messageIds.add(message.id);
- return true;
-}
+/** How much of a legacy transcript one conversion page holds. */
+const TRANSCRIPT_CONVERSION_PAGE_MAX_MESSAGES = 256;
+const TRANSCRIPT_CONVERSION_PAGE_MAX_BYTES = 4 * 1024 * 1024;
export class RuntimeLedgerRepair {
private readonly queues = new Map<string, Promise<void>>();
constructor(private readonly deps: RuntimeLedgerRepairDeps) {}
/**
- * Give an imported transcript a runtime spine: one invocation per turn, opened
- * by its own opening fact and closed by its own terminal event.
+ * Give a transcript a runtime spine: one invocation per turn, opened by its
+ * own opening fact and closed by its own terminal event.
*
- * The transcript is the only evidence there is, so a turn it cannot close is
- * refused rather than imported half-formed. Re-running is a no-op: a turn
- * whose invocation already exists is left exactly as it is.
+ * Every event id is derived from the run it belongs to and its position in
+ * that run, so importing the same transcript twice writes the same events and
+ * the store dedupes them. That is what makes an interrupted import resumable:
+ * a turn is skipped once its invocation has ended, and re-derived until then.
*/
async materializeTranscriptLedger(header: SessionHeader): Promise<void> {
const sessionId = header.id;
- return this.withRepairQueue(sessionId, 'transcript-runs', async () => {
- const messages = await this.deps.readMessages(sessionId);
- const ledgerMessages = messages.filter(
- (message) => message.type !== 'user' || message.steeringEventId === undefined,
- );
- const openedTurnIds = new Set(
- (await this.listInlineInvocations(sessionId)).map((invocation) => invocation.turnId),
- );
- const messagesByTurn = groupMessagesByTurn(ledgerMessages);
- const turns = deriveTurnRecords(ledgerMessages).filter((turn) =>
- (messagesByTurn.get(turn.turnId) ?? []).some((message) => message.type === 'user'),
- );
- if (turns.length === 0) return;
-
- const firstOpenedAt = Math.max(0, header.createdAt - turns.length);
+ return this.withRepairQueue(sessionId, async () => {
+ // A turn the ledger already owns is not converted again. Its own run is
+ // the authority even when it never ended — a crashed turn is settled by
+ // recovery on that run, and a second, transcript-derived invocation for
+ // the same turn would make the Session read as two. The one exception is
+ // this converter's own run: an interrupted import re-derives it, and the
+ // deterministic ids let the store dedupe what already landed.
+ const { ownedTurnIds, startedRunIds } = await this.readLedgerOwnership(sessionId);
- for (const [index, turn] of turns.entries()) {
- if (openedTurnIds.has(turn.turnId)) continue;
- const turnMessages = messagesByTurn.get(turn.turnId) ?? [];
+ for await (const scanned of this.readTurnsInPages(sessionId)) {
+ const turnMessages = scanned.messages;
+ // A turn whose only user row was steering is not a turn of its own: the
+ // steering was said into a Turn some durable Root already owns, so
+ // converting it would stand a second, synthetic run beside that one.
+ if (!turnMessages.some((message) => message.type === 'user')) continue;
+ const [turn] = deriveTurnRecords(turnMessages);
+ if (!turn) continue;
+ if (ownedTurnIds.has(turn.turnId)) continue;
Review Comment:
Fixed in `957c26923`. You were right about the mechanism and about the reachability, and my earlier "unreachable" verdict was wrong for the reason you name: I checked what this build's send path can write, not what a released one already left behind.
**Fixed where the order is decided, not at the converter.** Conversion now ends by renumbering the Session's ordinals in the order its invocations opened. `openedAt` was already computed at `:97` to put imported openings ahead of the Session's own runs, and — as your comment says — no reader consulted it. That was the actual defect in the diff: a second, dead expression of the order the reader was getting from somewhere else. The renumber is what makes the reader agree, so the value stops being decorative rather than being joined by another mechanism.
**Why renumbering is not a rewrite of positions someone holds.** Ordinals are read only on this build, only through `ensureTranscriptLedgerForRead`, which converts first; and a Session at transcript ledger version 0 cannot start a Turn while the conversion holds its queue. Nothing has read these numbers yet, so this is a recomputation.
**On your suggested standard — refuse the class of history explicitly.** I did not take it, and I want to say why rather than quietly diverge. Refusing turns a silent misordering into a Session nobody can open, for a population whose data is entirely intact. Explicit refusal is the right answer when the compatibility boundary cannot represent the history; here it can, and the order it needed was already written down.
**Also rejected: enforcing this inside `appendRuntimeEvent`** so no caller has to remember the call. The only available comparison is the openings' wall clock, so a backwards clock step would reorder a *live* Session under readers holding its cursors. The explicit, conversion-only call has no such trigger.
Test: `converts an imported turn ahead of a run the Session already sent` — an imported turn, a native run appended after it the way a released build's send would, then conversion. Ablation without the renumber fails with `'the newer question'` ordered ahead of `'the older question'`.
<details><summary>简体中文</summary>
已在 `957c26923` 修复。机制和可达性你都是对的；我先前判「不可达」错在你指出的那点上——我检查的是**本构建**的 send 路径能写出什么，而不是**已发布版本**已经留下了什么。
**修在顺序的决定处，不是修转换器。** 转换结束时，按 invocation 打开的顺序重排该 Session 的 ordinal。`:97` 的 `openedAt` 本来就是为「让导入的 opening 排在该 Session 自有 run 之前」而算的，而正如你所说，**没有任何读者用它**。那才是这份 diff 里真正的缺陷：同一个顺序有了第二份、死的表达。重排让 reader 与它一致，于是这个值不再是装饰——而不是再叠一套机制上去。
**为什么重排不是改写别人持有的位置。** ordinal 只在本构建上被读，且必经 `ensureTranscriptLedgerForRead`（先转换）；transcript ledger version 0 的 Session 在转换持有队列期间开不了 Turn。这些数字还没有被任何人读过，所以这是**重算**。
**关于你给的标准——显式拒绝该类历史。** 我没有采用，想说明理由而不是默默偏离：拒绝是把静默错序换成「一个谁都打不开的 Session」，而这批数据是完好的。当兼容边界**无法表达**某类历史时，显式拒绝是对的；这里它能表达，而且需要的顺序早就写下来了。
**同样被否掉的：把规则放进 `appendRuntimeEvent`**，让调用方无需记得。可用的判据只有 opening 的 wall clock，于是一次时钟回拨就会重排一个**活跃** Session，把读者手上的游标挪掉。显式的、只在转换时调用的做法没有这个触发条件。
测试：`converts an imported turn ahead of a run the Session already sent`——导入一个 turn，再按已发布版本 send 的方式追加一个 native run，然后转换。去掉重排的消融实验失败于 `'the newer question'` 排在 `'the older question'` 之前。
</details>
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "Astro-Han (via GitHub)" <gi...@apache.org>.
Astro-Han commented on code in PR #4879:
URL: https://github.com/apache/maka/pull/4879#discussion_r3944873797
##########
packages/runtime-host/src/server/session-transcript-reader.ts:
##########
@@ -47,29 +59,67 @@ const ACTIVE_TRANSCRIPT_SCAN_BATCH_MAX_BYTES = 256 * 1024;
export function createSessionTranscriptReader(input: {
stores: ExecutionStoresWriter<'interactive'>;
canonicalPermissionOutcomes: CanonicalPermissionOutcomeReader;
+ /**
+ * Converts a Session whose transcript predates the ledger, before this reader
+ * looks for invocations that only the conversion can create. Omitted only by
+ * tests that seed the ledger themselves.
+ */
+ ensureTranscriptLedger?: (sessionId: string) => Promise<void>;
}): SessionTranscriptReader {
+ const durable = createDurableLedgerTranscriptReader(input);
+ const prepared = async (sessionId: string): Promise<void> => {
+ await input.ensureTranscriptLedger?.(sessionId);
+ };
return {
- readDurableHighWater: (sessionId) =>
- input.stores.sessionStore.readTranscriptHighWaterSnapshot(sessionId),
- readDurablePage: (sessionId, request) =>
- input.stores.sessionStore.readTranscriptPageSnapshot(sessionId, request),
- readDurableRecords: (sessionId, request) =>
- input.stores.sessionStore.readTranscriptRecordsSnapshot(sessionId, request),
- readDurableMessagesById: (sessionId, request) =>
- input.stores.sessionStore.readTranscriptMessagesSnapshot(sessionId, request),
+ readDurableHighWater: async (sessionId) => {
+ await prepared(sessionId);
+ return durable.readHighWater(sessionId);
+ },
+ readDurablePage: async (sessionId, request) => {
+ await prepared(sessionId);
+ return durable.readPage(sessionId, request);
Review Comment:
Confirmed, and it is broader than the one kind you named — but I am accepting the loss rather than fixing it, so here is the whole reasoning.
Verified your chain in code: released `sendMessage` calls `ensureTranscriptLedger` and sets version 1; released `runtime-kernel.ts:1104` and `ai-sdk-turn.ts:2504` write the note with `store.appendMessage` / `backend.appendMessage` and no RuntimeEvent; this head skips conversion at version 1. Scanning the released `ai-sdk-turn.ts`, **seven** of the eight retained kinds are written transcript-only: `context_provider_dropping`, `context_window_overrun`, `context_reported_window_exceeded`, `context_window_suggestion`, `context_overflow_after_compaction`, `context_compaction_failed_open`, `context_compacted`. The retired kinds (`abort`, `session_resume`) are unaffected — their facts are owned by the terminal.
Why not repaired: these notes belong to turns a **real** run already sealed, and `assertRunNotSealed` is an interface obligation of every store, not a detail of one. Re-running the converter cannot reach them; putting them back would need a migration path that bypasses the seal, i.e. an exception on the guard that makes a run's ending single and final. That is a bigger change to the transcript authority than the thing it repairs.
Deriving them at read time instead only works for two of the seven. `context_compacted` / `context_compaction_failed_open` are recoverable in principle — `contextCompactionOutcome` is in the compaction terminal's `stateDelta`, and `contextBudget.compactionDecisions` is in the token-usage event, both released identically. But `context-budget.ts:218` records that a fold reaches the user as **one note per send**, and `mergeCompactionDecisionDiagnostics` accumulates, so every later token-usage event in the turn carries the same decision. The writer gets "once" from a per-turn flag; a reader would have to get it from bounded pages, and two token-usage events of one turn can land on different pages. Writing the fact as its own event is the page-stable shape. The other five depend on the *then*-declared window, the provider-reported window, and a cross-session crossing anchor — mutable config, not ledger data.
So: hidden from the model, no effect on context or execution, bounded to Sessions a released build already touched, legacy rows intact. Recorded as the accepted cost of the cutover, with a comment at the version gate in `session-manager.ts` saying plainly that version 1 means a conversion ran, not that nothing is left in `session_messages` — so the next person does not build on it as proof.
Leaving this thread open, since I did not change the behaviour you reported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "Astro-Han (via GitHub)" <gi...@apache.org>.
Astro-Han commented on code in PR #4879:
URL: https://github.com/apache/maka/pull/4879#discussion_r3947720327
##########
packages/storage/src/runtime-transcript-query.ts:
##########
@@ -0,0 +1,278 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import type { DatabaseSync } from 'node:sqlite';
+import { decodeRuntimeEvent, type RuntimeEvent } from '@maka/core/runtime-event';
+import type { RuntimeInvocationRecord } from '@maka/core/runtime-invocation';
+
+/** SQL counterpart of isTerminalRuntimeEvent; shared with the ledger store. */
+export const TERMINAL_RUNTIME_EVENT_SQL = `(
+ json_extract(payload_json, '$.actions.endInvocation') = 1
+ OR json_extract(payload_json, '$.status') IN ('completed', 'failed', 'aborted', 'cancelled')
+)`;
+
+/**
+ * One invocation's events, in ledger order, carrying the Session ordinal each
+ * one sits at.
+ *
+ * The transcript rows of a Turn come from projecting these together: what a
+ * RuntimeEvent becomes is decided by the read model alone, so nothing here
+ * classifies an event or decides whether it produces a row.
+ */
+export interface RuntimeTranscriptInvocation {
+ readonly invocation: RuntimeInvocationRecord;
+ readonly firstOrdinal: number;
+ readonly lastOrdinal: number;
+ readonly events: readonly { readonly ordinal: number; readonly event: RuntimeEvent }[];
+}
+
+/** An invocation start, with the prompt event a landmark is labelled by. */
+export interface RuntimeTranscriptLandmark {
+ readonly invocation: RuntimeInvocationRecord;
+ readonly firstOrdinal: number;
+ readonly prompt?: { readonly ordinal: number; readonly event: RuntimeEvent };
+}
+
+export interface RuntimeTranscriptInvocationRequest {
+ readonly direction: 'older' | 'newer';
+ readonly throughOrdinal: number;
+ /** Ordinal the walk starts from, inclusive, in `direction`. */
+ readonly position: number;
+ readonly limit: number;
+ /** Refused rather than truncated: half a Turn projects to a wrong transcript. */
+ readonly maxEvents: number;
+ readonly maxBytes: number;
+}
+
+export interface RuntimeTranscriptQueries {
+ readTranscriptHighWater(sessionId: string): Promise<number | null>;
+ readTranscriptInvocations(
+ sessionId: string,
+ request: RuntimeTranscriptInvocationRequest,
+ ): Promise<RuntimeTranscriptInvocation[]>;
+ readTranscriptLandmarks(
+ sessionId: string,
+ throughOrdinal: number,
+ limit: number,
+ ): Promise<RuntimeTranscriptLandmark[]>;
+}
+
+export class RuntimeTranscriptOversizedTurnError extends Error {
+ readonly name = 'RuntimeTranscriptOversizedTurnError';
+}
+
+const joins = `
+ FROM runtime_session_event_ordinals o
+ JOIN runtime_events e ON e.event_id = o.event_id
+ LEFT JOIN runtime_events opened ON opened.invocation_id = e.invocation_id AND opened.event_kind = 'invocation_opened'
+ LEFT JOIN runtime_legacy_invocation_openings legacy ON legacy.invocation_id = e.invocation_id`;
+const opening = `COALESCE(json_extract(opened.payload_json, '$.content'), legacy.opening_json)`;
+/**
+ * A Turn the Session transcript shows: one this Session ran itself rather than
+ * on behalf of a subagent, and one that has already ended.
+ *
+ * This is a fact about the invocation, not about any row it produces — which
+ * rows it produces is the read model's question, and is not asked here.
+ */
+const settledInline = `
+ ${opening} IS NOT NULL
+ AND (json_extract(${opening}, '$.lineage.parentRunId') IS NULL
+ OR (json_extract(${opening}, '$.source.kind') = 'continuation'
+ AND json_extract(${opening}, '$.lineage.agentId') IS NULL))
+ AND EXISTS (
+ SELECT 1 FROM runtime_events ended
+ JOIN runtime_session_event_ordinals ending ON ending.event_id = ended.event_id
+ WHERE ended.invocation_id = e.invocation_id
+ AND ${TERMINAL_RUNTIME_EVENT_SQL.replaceAll('payload_json', 'ended.payload_json')}
+ AND ending.ordinal <= :throughOrdinal
+ )`;
+
+type InvocationRow = { invocation_id: string; run_id: string; first: number; last: number };
+
+/** Selects invocations by Session ordinal. Payloads are decoded, never classified. */
+export class RuntimeTranscriptQuery {
+ constructor(
+ private readonly db: DatabaseSync,
+ private readonly invocation: (sessionId: string, runId: string) => RuntimeInvocationRecord,
+ ) {}
+
+ highWater(sessionId: string): number | null {
+ const row = this.db
+ .prepare(`
+ SELECT MAX(o.ordinal) AS ordinal ${joins}
+ WHERE o.session_id = :sessionId AND o.ordinal <= :throughOrdinal AND ${settledInline}
+ `)
+ .get({ sessionId, throughOrdinal: Number.MAX_SAFE_INTEGER }) as { ordinal?: unknown };
+ return typeof row.ordinal === 'number' ? row.ordinal : null;
+ }
+
+ invocations(
+ sessionId: string,
+ request: RuntimeTranscriptInvocationRequest,
+ ): RuntimeTranscriptInvocation[] {
+ assertOrdinal(request.throughOrdinal);
+ assertOrdinal(request.position);
+ if (request.direction !== 'older' && request.direction !== 'newer') {
+ throw new Error('Invalid transcript direction');
+ }
+ // An invocation is selected by where its own events sit, so a walk that
+ // starts inside a Turn still finds that Turn and can serve its rows.
+ const rows = this.db
+ .prepare(`
+ SELECT e.invocation_id, e.run_id, MIN(o.ordinal) AS first, MAX(o.ordinal) AS last ${joins}
+ WHERE o.session_id = :sessionId AND o.ordinal <= :throughOrdinal AND ${settledInline}
+ GROUP BY e.invocation_id
+ HAVING ${request.direction === 'older' ? 'first <= :position' : 'last >= :position'}
+ ORDER BY first ${request.direction === 'older' ? 'DESC' : 'ASC'}
+ LIMIT :limit
+ `)
+ .all({
+ sessionId,
+ throughOrdinal: request.throughOrdinal,
+ position: request.position,
+ limit: request.limit,
+ }) as InvocationRow[];
+ return rows.map((row) => ({
+ invocation: this.invocation(sessionId, row.run_id),
+ firstOrdinal: row.first,
+ lastOrdinal: row.last,
+ events: this.events(row.invocation_id, request),
+ }));
+ }
+
+ landmarks(sessionId: string, throughOrdinal: number, limit: number): RuntimeTranscriptLandmark[] {
+ assertOrdinal(throughOrdinal);
+ if (limit < 1) return [];
+ // Evenly spaced Turn starts, chosen before any payload is read.
+ const rows = this.db
+ .prepare(`
+ WITH candidates AS (
+ SELECT e.invocation_id, e.run_id, o.ordinal,
+ ROW_NUMBER() OVER (ORDER BY o.ordinal) - 1 AS rank, COUNT(*) OVER () AS total
+ ${joins} WHERE o.session_id = :sessionId AND o.ordinal <= :throughOrdinal
+ AND e.event_seq = 1 AND ${settledInline}
+ ), samples(n) AS (
+ SELECT 0 UNION ALL SELECT n + 1 FROM samples WHERE n + 1 < :limit
+ )
+ SELECT DISTINCT invocation_id, run_id, ordinal FROM candidates
+ JOIN samples ON rank = CASE WHEN :limit = 1 THEN total - 1
+ ELSE CAST(n * (total - 1) / (:limit - 1) AS INTEGER) END
+ ORDER BY ordinal
+ `)
+ .all({ sessionId, throughOrdinal, limit }) as Array<{
+ invocation_id: string;
+ run_id: string;
+ ordinal: number;
+ }>;
+ return rows.map((row) => {
+ // The prompt is the Turn's first user text event, which is what the read
+ // model projects a user message from. Only that one event is loaded: a
+ // landmark is a label, and projecting whole Turns to build a scrollbar
+ // would read most of the Session.
+ const prompt = this.db
+ .prepare(`
+ SELECT o.ordinal, e.event_id FROM runtime_events e
+ JOIN runtime_session_event_ordinals o ON o.event_id = e.event_id
+ WHERE e.invocation_id = ? AND o.ordinal <= ?
+ AND e.event_kind = 'text' AND json_extract(e.payload_json, '$.role') = 'user'
+ ORDER BY e.event_seq LIMIT 1
+ `)
+ .get(row.invocation_id, throughOrdinal) as
+ | { ordinal: number; event_id: string }
+ | undefined;
+ return {
+ invocation: this.invocation(sessionId, row.run_id),
+ firstOrdinal: row.ordinal,
+ ...(prompt
+ ? { prompt: { ordinal: prompt.ordinal, event: this.event(prompt.event_id) } }
+ : {}),
+ };
+ });
+ }
+
+ private events(
+ invocationId: string,
+ limits: { maxEvents: number; maxBytes: number },
+ ): RuntimeTranscriptInvocation['events'] {
+ // Walked row by row: the limits cap what one Turn may pull into memory, so
+ // a check after `.all()` has already paid the cost it was meant to refuse.
+ const cursor = this.db
+ .prepare(`
+ SELECT o.ordinal, e.event_id, e.session_id, e.invocation_id, e.run_id, e.turn_id, e.payload_json
+ FROM runtime_events e JOIN runtime_session_event_ordinals o ON o.event_id = e.event_id
+ WHERE e.invocation_id = ? ORDER BY e.event_seq
+ `)
+ .iterate(invocationId) as Iterable<StoredEventRow & { ordinal: number }>;
+ const events: Array<RuntimeTranscriptInvocation['events'][number]> = [];
+ let bytes = 0;
+ for (const row of cursor) {
+ if (events.length === limits.maxEvents) {
+ throw new RuntimeTranscriptOversizedTurnError(
+ `Turn ${invocationId} holds more RuntimeEvents than a transcript page may read`,
+ );
+ }
+ bytes += Buffer.byteLength(row.payload_json);
Review Comment:
Accepted as accurate, and left as it is. The residual is real: the driver hands back a decoded `payload_json` and `Buffer.byteLength` measures it afterwards, so one row over budget is paid before it is refused.
Closing it costs more than it saves. `octet_length(payload_json)` cannot ride along on the same row — the column comes back with it — so refusing before the string crosses into JS needs a second pass over the Turn to read lengths alone. That is one extra scan on every transcript page read, to bound a residual your own note already bounds: `tool-output.ts:35` caps retained output at 50 KB, so it is one row of that order, not an arbitrary one. Paying a per-read scan for that is a net increase, and this PR is already answering for read amplification.
Recorded in the fix commit rather than dropped, so the next person to touch this guard sees the trade instead of rediscovering it.
<details><summary>简体中文</summary>
结论属实，接受，但不改。残留是真的：驱动交回的是已解码的 `payload_json`，`Buffer.byteLength` 在其后测量，所以超预算的那一条仍是先付后拒。
收掉它的代价大于收益。`octet_length(payload_json)` 无法搭同一行返回（该列本身就跟着回来了），所以要在字符串跨进 JS 之前拒绝，就得对该 Turn 再跑一趟只取长度的扫描——**每次读 transcript 分页都多一次扫描**，换来的是约束一个你自己已经界定过的残留：`tool-object` 上游 `tool-output.ts:35` 把保留输出截到 50 KB，所以那是一条该量级的记录，不是任意大的一条。为此付出每次读一次全扫描是净增成本，而这个 PR 本来就在为读放大作答。
已写进修复的 commit message 而不是悄悄放下，下一个动这段防护的人能直接看到这个权衡，不用重新发现一遍。
</details>
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
packages/storage/src/runtime-transcript-query.ts:
##########
@@ -0,0 +1,278 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import type { DatabaseSync } from 'node:sqlite';
+import { decodeRuntimeEvent, type RuntimeEvent } from '@maka/core/runtime-event';
+import type { RuntimeInvocationRecord } from '@maka/core/runtime-invocation';
+
+/** SQL counterpart of isTerminalRuntimeEvent; shared with the ledger store. */
+export const TERMINAL_RUNTIME_EVENT_SQL = `(
+ json_extract(payload_json, '$.actions.endInvocation') = 1
+ OR json_extract(payload_json, '$.status') IN ('completed', 'failed', 'aborted', 'cancelled')
+)`;
+
+/**
+ * One invocation's events, in ledger order, carrying the Session ordinal each
+ * one sits at.
+ *
+ * The transcript rows of a Turn come from projecting these together: what a
+ * RuntimeEvent becomes is decided by the read model alone, so nothing here
+ * classifies an event or decides whether it produces a row.
+ */
+export interface RuntimeTranscriptInvocation {
+ readonly invocation: RuntimeInvocationRecord;
+ readonly firstOrdinal: number;
+ readonly lastOrdinal: number;
+ readonly events: readonly { readonly ordinal: number; readonly event: RuntimeEvent }[];
+}
+
+/** An invocation start, with the prompt event a landmark is labelled by. */
+export interface RuntimeTranscriptLandmark {
+ readonly invocation: RuntimeInvocationRecord;
+ readonly firstOrdinal: number;
+ readonly prompt?: { readonly ordinal: number; readonly event: RuntimeEvent };
+}
+
+export interface RuntimeTranscriptInvocationRequest {
+ readonly direction: 'older' | 'newer';
+ readonly throughOrdinal: number;
+ /** Ordinal the walk starts from, inclusive, in `direction`. */
+ readonly position: number;
+ readonly limit: number;
+ /** Refused rather than truncated: half a Turn projects to a wrong transcript. */
+ readonly maxEvents: number;
+ readonly maxBytes: number;
+}
+
+export interface RuntimeTranscriptQueries {
+ readTranscriptHighWater(sessionId: string): Promise<number | null>;
+ readTranscriptInvocations(
+ sessionId: string,
+ request: RuntimeTranscriptInvocationRequest,
+ ): Promise<RuntimeTranscriptInvocation[]>;
+ readTranscriptLandmarks(
+ sessionId: string,
+ throughOrdinal: number,
+ limit: number,
+ ): Promise<RuntimeTranscriptLandmark[]>;
+}
+
+export class RuntimeTranscriptOversizedTurnError extends Error {
+ readonly name = 'RuntimeTranscriptOversizedTurnError';
+}
+
+const joins = `
+ FROM runtime_session_event_ordinals o
+ JOIN runtime_events e ON e.event_id = o.event_id
+ LEFT JOIN runtime_events opened ON opened.invocation_id = e.invocation_id AND opened.event_kind = 'invocation_opened'
+ LEFT JOIN runtime_legacy_invocation_openings legacy ON legacy.invocation_id = e.invocation_id`;
+const opening = `COALESCE(json_extract(opened.payload_json, '$.content'), legacy.opening_json)`;
+/**
+ * A Turn the Session transcript shows: one this Session ran itself rather than
+ * on behalf of a subagent, and one that has already ended.
+ *
+ * This is a fact about the invocation, not about any row it produces — which
+ * rows it produces is the read model's question, and is not asked here.
+ */
+const settledInline = `
+ ${opening} IS NOT NULL
+ AND (json_extract(${opening}, '$.lineage.parentRunId') IS NULL
+ OR (json_extract(${opening}, '$.source.kind') = 'continuation'
+ AND json_extract(${opening}, '$.lineage.agentId') IS NULL))
+ AND EXISTS (
+ SELECT 1 FROM runtime_events ended
+ JOIN runtime_session_event_ordinals ending ON ending.event_id = ended.event_id
+ WHERE ended.invocation_id = e.invocation_id
+ AND ${TERMINAL_RUNTIME_EVENT_SQL.replaceAll('payload_json', 'ended.payload_json')}
+ AND ending.ordinal <= :throughOrdinal
+ )`;
+
+type InvocationRow = { invocation_id: string; run_id: string; first: number; last: number };
+
+/** Selects invocations by Session ordinal. Payloads are decoded, never classified. */
+export class RuntimeTranscriptQuery {
+ constructor(
+ private readonly db: DatabaseSync,
+ private readonly invocation: (sessionId: string, runId: string) => RuntimeInvocationRecord,
+ ) {}
+
+ highWater(sessionId: string): number | null {
+ const row = this.db
+ .prepare(`
+ SELECT MAX(o.ordinal) AS ordinal ${joins}
+ WHERE o.session_id = :sessionId AND o.ordinal <= :throughOrdinal AND ${settledInline}
+ `)
+ .get({ sessionId, throughOrdinal: Number.MAX_SAFE_INTEGER }) as { ordinal?: unknown };
+ return typeof row.ordinal === 'number' ? row.ordinal : null;
+ }
+
+ invocations(
+ sessionId: string,
+ request: RuntimeTranscriptInvocationRequest,
+ ): RuntimeTranscriptInvocation[] {
+ assertOrdinal(request.throughOrdinal);
+ assertOrdinal(request.position);
+ if (request.direction !== 'older' && request.direction !== 'newer') {
+ throw new Error('Invalid transcript direction');
+ }
+ // An invocation is selected by where its own events sit, so a walk that
+ // starts inside a Turn still finds that Turn and can serve its rows.
+ const rows = this.db
+ .prepare(`
+ SELECT e.invocation_id, e.run_id, MIN(o.ordinal) AS first, MAX(o.ordinal) AS last ${joins}
+ WHERE o.session_id = :sessionId AND o.ordinal <= :throughOrdinal AND ${settledInline}
+ GROUP BY e.invocation_id
+ HAVING ${request.direction === 'older' ? 'first <= :position' : 'last >= :position'}
+ ORDER BY first ${request.direction === 'older' ? 'DESC' : 'ASC'}
+ LIMIT :limit
+ `)
+ .all({
+ sessionId,
+ throughOrdinal: request.throughOrdinal,
+ position: request.position,
+ limit: request.limit,
+ }) as InvocationRow[];
+ return rows.map((row) => ({
+ invocation: this.invocation(sessionId, row.run_id),
+ firstOrdinal: row.first,
+ lastOrdinal: row.last,
+ events: this.events(row.invocation_id, request),
+ }));
+ }
+
+ landmarks(sessionId: string, throughOrdinal: number, limit: number): RuntimeTranscriptLandmark[] {
+ assertOrdinal(throughOrdinal);
+ if (limit < 1) return [];
+ // Evenly spaced Turn starts, chosen before any payload is read.
+ const rows = this.db
+ .prepare(`
+ WITH candidates AS (
+ SELECT e.invocation_id, e.run_id, o.ordinal,
+ ROW_NUMBER() OVER (ORDER BY o.ordinal) - 1 AS rank, COUNT(*) OVER () AS total
+ ${joins} WHERE o.session_id = :sessionId AND o.ordinal <= :throughOrdinal
+ AND e.event_seq = 1 AND ${settledInline}
+ ), samples(n) AS (
+ SELECT 0 UNION ALL SELECT n + 1 FROM samples WHERE n + 1 < :limit
+ )
+ SELECT DISTINCT invocation_id, run_id, ordinal FROM candidates
+ JOIN samples ON rank = CASE WHEN :limit = 1 THEN total - 1
+ ELSE CAST(n * (total - 1) / (:limit - 1) AS INTEGER) END
+ ORDER BY ordinal
+ `)
+ .all({ sessionId, throughOrdinal, limit }) as Array<{
+ invocation_id: string;
+ run_id: string;
+ ordinal: number;
+ }>;
+ return rows.map((row) => {
+ // The prompt is the Turn's first user text event, which is what the read
+ // model projects a user message from. Only that one event is loaded: a
+ // landmark is a label, and projecting whole Turns to build a scrollbar
+ // would read most of the Session.
+ const prompt = this.db
+ .prepare(`
+ SELECT o.ordinal, e.event_id FROM runtime_events e
+ JOIN runtime_session_event_ordinals o ON o.event_id = e.event_id
+ WHERE e.invocation_id = ? AND o.ordinal <= ?
+ AND e.event_kind = 'text' AND json_extract(e.payload_json, '$.role') = 'user'
+ ORDER BY e.event_seq LIMIT 1
+ `)
+ .get(row.invocation_id, throughOrdinal) as
+ | { ordinal: number; event_id: string }
+ | undefined;
+ return {
+ invocation: this.invocation(sessionId, row.run_id),
+ firstOrdinal: row.ordinal,
+ ...(prompt
+ ? { prompt: { ordinal: prompt.ordinal, event: this.event(prompt.event_id) } }
+ : {}),
+ };
+ });
+ }
+
+ private events(
+ invocationId: string,
+ limits: { maxEvents: number; maxBytes: number },
+ ): RuntimeTranscriptInvocation['events'] {
+ // Walked row by row: the limits cap what one Turn may pull into memory, so
+ // a check after `.all()` has already paid the cost it was meant to refuse.
+ const cursor = this.db
+ .prepare(`
+ SELECT o.ordinal, e.event_id, e.session_id, e.invocation_id, e.run_id, e.turn_id, e.payload_json
+ FROM runtime_events e JOIN runtime_session_event_ordinals o ON o.event_id = e.event_id
+ WHERE e.invocation_id = ? ORDER BY e.event_seq
+ `)
+ .iterate(invocationId) as Iterable<StoredEventRow & { ordinal: number }>;
+ const events: Array<RuntimeTranscriptInvocation['events'][number]> = [];
+ let bytes = 0;
+ for (const row of cursor) {
+ if (events.length === limits.maxEvents) {
+ throw new RuntimeTranscriptOversizedTurnError(
+ `Turn ${invocationId} holds more RuntimeEvents than a transcript page may read`,
+ );
+ }
+ bytes += Buffer.byteLength(row.payload_json);
Review Comment:
Accepted as accurate, and left as it is. The residual is real: the driver hands back a decoded `payload_json` and `Buffer.byteLength` measures it afterwards, so one row over budget is paid before it is refused.
Closing it costs more than it saves. `octet_length(payload_json)` cannot ride along on the same row — the column comes back with it — so refusing before the string crosses into JS needs a second pass over the Turn to read lengths alone. That is one extra scan on every transcript page read, to bound a residual your own note already bounds: `tool-output.ts:35` caps retained output at 50 KB, so it is one row of that order, not an arbitrary one. Paying a per-read scan for that is a net increase, and this PR is already answering for read amplification.
Recorded in the fix commit rather than dropped, so the next person to touch this guard sees the trade instead of rediscovering it.
<details><summary>简体中文</summary>
结论属实，接受，但不改。残留是真的：驱动交回的是已解码的 `payload_json`，`Buffer.byteLength` 在其后测量，所以超预算的那一条仍是先付后拒。
收掉它的代价大于收益。`octet_length(payload_json)` 无法搭同一行返回（该列本身就跟着回来了），所以要在字符串跨进 JS 之前拒绝，就得对该 Turn 再跑一趟只取长度的扫描——**每次读 transcript 分页都多一次扫描**，换来的是约束一个你自己已经界定过的残留：上游 `tool-output.ts:35` 把保留输出截到 50 KB，所以那是一条该量级的记录，不是任意大的一条。为此付出每次读一次全扫描是净增成本，而这个 PR 本来就在为读放大作答。
已写进修复的 commit message 而不是悄悄放下，下一个动这段防护的人能直接看到这个权衡，不用重新发现一遍。
</details>
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "Astro-Han (via GitHub)" <gi...@apache.org>.
Astro-Han commented on PR #4879:
URL: https://github.com/apache/maka/pull/4879#issuecomment-5565060187
@likun666661 Thanks — I traced all four. Two are fixed, two are not, with reasons below. Pushed as `86f1d7608`.
**3. Bounds checked after full materialization — fixed**
Confirmed. `RuntimeTranscriptQuery#events` called `.all()` to pull the whole invocation into memory, then checked `rows.length > maxEvents`, then accumulated bytes over rows that were already loaded. The limits exist to cap what one Turn may pull into memory, so the check ran after that memory had already been paid — which is your 20,973,944 bytes against a 16,777,216-byte budget. It now walks a cursor and refuses on the row that crosses either limit.
Your second point is confirmed and fixed with it: the budget counted `payload_json.length`, i.e. UTF-16 code units, not stored bytes. A CJK payload therefore admitted three times the size the budget asked for. It counts real bytes now.
Both are the letter of #4876's Bounded Read rule — *"the budget applies before rows are materialized or decoded"* and *"measure once, carry the measurement"*.
Regression test `bounds a transcript Turn by the bytes it stores, not by its JSON string length`: a 4,000-character CJK Turn against a 6,000-byte budget, which the old code answered with `Missing expected rejection`.
**4. Catalog projection backfill condition incomplete — fixed**
Confirmed, and slightly wider than you wrote: it is not only the `connectionLocked: false` case. In *any* state where the ledger holds the prompt but the catalog does not, recovery makes zero calls.
- `agent-run.ts` cleared `initialRuntimeEventPending` after `recordRuntimeEvents` but before `commitMessageProjection`, so `finalize()` would not retry;
- `hosted-execution-recovery.ts`'s `recordAdmittedUserMessage` was gated behind `!verifyUserMessage(...)`.
What is lost is the Session-list preview line and the one-way `connectionLocked` latch. Losing the latch means a Session that has already run a Turn can still rebind its LLM connection. This window is one this PR introduces: on main the transcript row and the latch were a single `appendMessage` transaction; now the ledger append and the catalog commit are two writes.
The fix: the flag clears only after the projection lands (covering the in-process throw), and recovery commits the projection whether or not the ledger already holds the event (covering the crash). The projection is idempotent, the latch is one-way, and the early return on `run.terminalEvent` keeps it away from Turns that already have their own assistant preview.
Regression test `startup recovery commits the catalog facts a crashed Turn wrote no projection for`, which the old code answered with `connectionLocked` actual false / expected true.
**1. Mixed-history conversion ordering — not changed; mechanism is right, the state is unreachable**
Your mechanism is correct, and I confirmed both halves in the code: ordinals are `COALESCE(MAX(ordinal),0)+1`, and `openedAt` does not participate in transcript ordering. But the premise — one Session holding legacy rows for an early turn *and* ledger events for a later turn — is not producible on a supported path:
- `session-manager.ts:1978-1985` — a non-`agentId` `sendMessage` runs `ensureTranscriptLedger` *inside* `admitTurn`, so the conversion precedes any ledger append for that Session;
- both `agentId` call sites (`:2779`, `:3127`) target a child Session created in that same flow, and `session-store.ts:1293` creates at `transcriptLedgerVersion: 1`, so a legacy Session never receives an `agentId` send;
- while `transcriptLedgerVersion === 0` (import staging), `host-session-availability.ts:66/110` reports the Session unavailable and no Turn can start.
Separately, `openedAt` is not dead: its consumer is `ORDER BY opened_at` at `sqlite-runtime-store.ts:747`, the invocation inventory. The comment's "ahead of the Session's own runs" is about that ordering, not the transcript's.
Your repro writes the state into both stores directly. If you can show a path that reaches it through the API, I will fix it immediately — until then I am treating it as unreachable.
**2. Paging work grows with whole-session history — confirmed, tracked in #4876**
The SQL does `GROUP BY e.invocation_id` over every ordinal row within the watermark before `HAVING`/`ORDER BY`/`LIMIT`, and `runtime_session_event_ordinals` carries only the PK `(session_id, ordinal)`, so the temp b-tree cannot be planned away. Against main's sequence-indexed paging this is a genuine O(page) → O(session) regression, and I accept your 2.6 / 25 / 77 ms.
Not fixed in this PR: removing the scan requires persisting a per-invocation ordinal range — a new table, a schema migration, and consistency maintenance on the append path. That is its own change, and folding it into a PR already at -5657 lines makes both harder to review. The consequence is bounded (latency; no wrong state, nothing persisted incorrectly).
This is #4876's *"a sessionId is not a bound — one session also grows without limit"*, so I am tracking it there rather than opening a parallel issue.
**Verification**: `@maka/storage` 1116 / `@maka/runtime` 3256 / `@maka/runtime-host` 1743, all green. You noted you had not re-run tests on the new head — `86f1d7608` is current, and a second look at the shape of fixes 3 and 4 would be welcome.
<details>
<summary>中文</summary>
四条都查了，两条修了两条没修。已推到 `86f1d7608`。
**3. 上限在全量物化之后才检查 —— 已修。** `.all()` 先把整个 invocation 读进内存，再判 `rows.length > maxEvents`，再在已加载的行上累加字节；护栏的目的是限制单个 Turn 拉进内存的量，检查点却在这份内存已经付掉之后，就是你测到的 20,973,944 对 16,777,216。改成游标逐行走，越界那行就抛。第二点一并修了：预算用的是 `payload_json.length`（UTF-16 code unit）而非存储字节数，CJK payload 会放到 3 倍。这两点正是 #4876 的 Bounded Read 原文：*"the budget applies before rows are materialized or decoded"* 和 *"measure once, carry the measurement"*。
**4. catalog 投影回填条件不完整 —— 已修，且比你写的更宽：** 不只 `connectionLocked: false`，而是 ledger 有 prompt、catalog 没有的任何状态，恢复侧一次调用都不发。`agent-run.ts` 在投影之前就清了 `initialRuntimeEventPending`，`finalize()` 不会重试；`hosted-execution-recovery.ts` 又被 `!verifyUserMessage(...)` 挡住。丢的是 Session 列表预览行和 `connectionLocked` 单向闩 —— 闩丢了意味着已经跑过 Turn 的 Session 还能重新绑定 LLM connection。这个窗口是本 PR 引入的：main 上 transcript 行和闩在同一个 `appendMessage` 事务里，现在是两次写。改法是标志推迟到投影落地后再清（覆盖进程内抛异常），恢复侧无论 ledger 有没有该事件都提交投影（覆盖崩溃）；投影幂等、闩单向、`run.terminalEvent` 提前返回保证不动已有 assistant 预览的 Turn。
**1. 混合历史转换顺序 —— 没改，机制对但状态不可达。** ordinal 是 `MAX+1`、`openedAt` 不参与 transcript 排序，这两点我确认了。但"同一 Session 里 legacy 行属于早期 turn、ledger 事件属于后期 turn"这个前提在受支持路径上产生不出来：非 `agentId` 的 `sendMessage` 把 `ensureTranscriptLedger` 放进 `admitTurn`（`session-manager.ts:1978-1985`）；两个 `agentId` 调用点目标都是本流程新建、`transcriptLedgerVersion: 1` 的 child session；staging（version 0）期间 Session 判不可用发不出 Turn。另外 `openedAt` 不是没用，它的消费者是 `sqlite-runtime-store.ts:747` 的 invocation 清单排序，注释说的是那个。你的复现是直接往两个 store 写状态构造的；能给出一条走 API 到达该状态的路径我马上改。
**2. 分页工作量随整段历史增长 —— 确认，挂 #4876 跟。** SQL 确实在 watermark 内全部 ordinal 行上 `GROUP BY` 后再 `HAVING`/`ORDER BY`/`LIMIT`，索引只有 PK `(session_id, ordinal)`，temp b-tree 消不掉；相对 main 的 sequence 索引分页是 O(page) → O(session) 的实质退化，2.6/25/77 ms 我认。没在本 PR 修：要消掉扫描得持久化每个 invocation 的 ordinal 区间——新表、migration、append 路径一致性维护，是独立一次改动，塞进已经 -5657 行的 PR 只会让两边都更难 review。后果有界（延迟，无错误状态、无持久化影响）。这正是 #4876 的 *"a sessionId is not a bound — one session also grows without limit"*，所以在那边跟，不另开 issue。
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener commented on code in PR #4879:
URL: https://github.com/apache/maka/pull/4879#discussion_r3947084297
##########
packages/runtime/src/runtime-ledger-repair.ts:
##########
@@ -35,127 +34,188 @@ import type { SessionHeader } from '@maka/core/session';
import type { StoredMessage, TurnRecord } from '@maka/core/session';
import { backfillRuntimeEventsFromStoredMessages } from './runtime-event-backfill.js';
import type { RuntimeEventBackfillOutcome } from './runtime-event-backfill.js';
-import { projectRuntimeEventUserMessage } from './runtime-event-read-model.js';
export interface RuntimeLedgerRepairDeps {
runtimeEventStore: RuntimeEventStore;
- readMessages(sessionId: string): Promise<StoredMessage[]>;
- appendMessage(sessionId: string, message: StoredMessage): Promise<void>;
- newId: () => string;
- now: () => number;
-}
-
-interface RuntimeEventTranscriptProjectionDeps {
- readMessages(sessionId: string): Promise<StoredMessage[]>;
- appendMessage(sessionId: string, message: StoredMessage): Promise<void>;
+ /**
+ * One forward page of the legacy transcript this converter reads; nothing
+ * writes back to it. It is read a page at a time because a Session cannot
+ * serve its first transcript page until this finishes, and a Session's
+ * history is not a bound.
+ */
+ readMessagesAfter(
+ sessionId: string,
+ request: { afterSequence?: number; maxMessages: number; maxStoredBytes: number },
+ ): Promise<{
+ records: readonly { sequence: number; message: StoredMessage }[];
+ highWaterSequence: number | null;
+ }>;
}
-export async function materializeRuntimeEventTranscriptProjection(
- deps: RuntimeEventTranscriptProjectionDeps,
- sessionId: string,
- event: RuntimeEvent,
- knownMessageIds?: Set<string>,
-): Promise<boolean> {
- const message = steeringMessageFromRuntimeEvent(event);
- if (!message) return false;
- const messageIds =
- knownMessageIds ?? new Set((await deps.readMessages(sessionId)).map((item) => item.id));
- if (messageIds.has(message.id)) return false;
- await deps.appendMessage(sessionId, message);
- messageIds.add(message.id);
- return true;
-}
+/** How much of a legacy transcript one conversion page holds. */
+const TRANSCRIPT_CONVERSION_PAGE_MAX_MESSAGES = 256;
+const TRANSCRIPT_CONVERSION_PAGE_MAX_BYTES = 4 * 1024 * 1024;
export class RuntimeLedgerRepair {
private readonly queues = new Map<string, Promise<void>>();
constructor(private readonly deps: RuntimeLedgerRepairDeps) {}
/**
- * Give an imported transcript a runtime spine: one invocation per turn, opened
- * by its own opening fact and closed by its own terminal event.
+ * Give a transcript a runtime spine: one invocation per turn, opened by its
+ * own opening fact and closed by its own terminal event.
*
- * The transcript is the only evidence there is, so a turn it cannot close is
- * refused rather than imported half-formed. Re-running is a no-op: a turn
- * whose invocation already exists is left exactly as it is.
+ * Every event id is derived from the run it belongs to and its position in
+ * that run, so importing the same transcript twice writes the same events and
+ * the store dedupes them. That is what makes an interrupted import resumable:
+ * a turn is skipped once its invocation has ended, and re-derived until then.
*/
async materializeTranscriptLedger(header: SessionHeader): Promise<void> {
const sessionId = header.id;
- return this.withRepairQueue(sessionId, 'transcript-runs', async () => {
- const messages = await this.deps.readMessages(sessionId);
- const ledgerMessages = messages.filter(
- (message) => message.type !== 'user' || message.steeringEventId === undefined,
- );
- const openedTurnIds = new Set(
- (await this.listInlineInvocations(sessionId)).map((invocation) => invocation.turnId),
- );
- const messagesByTurn = groupMessagesByTurn(ledgerMessages);
- const turns = deriveTurnRecords(ledgerMessages).filter((turn) =>
- (messagesByTurn.get(turn.turnId) ?? []).some((message) => message.type === 'user'),
- );
- if (turns.length === 0) return;
-
- const firstOpenedAt = Math.max(0, header.createdAt - turns.length);
+ return this.withRepairQueue(sessionId, async () => {
+ // A turn the ledger already owns is not converted again. Its own run is
+ // the authority even when it never ended — a crashed turn is settled by
+ // recovery on that run, and a second, transcript-derived invocation for
+ // the same turn would make the Session read as two. The one exception is
+ // this converter's own run: an interrupted import re-derives it, and the
+ // deterministic ids let the store dedupe what already landed.
+ const { ownedTurnIds, startedRunIds } = await this.readLedgerOwnership(sessionId);
- for (const [index, turn] of turns.entries()) {
- if (openedTurnIds.has(turn.turnId)) continue;
- const turnMessages = messagesByTurn.get(turn.turnId) ?? [];
+ for await (const scanned of this.readTurnsInPages(sessionId)) {
+ const turnMessages = scanned.messages;
+ // A turn whose only user row was steering is not a turn of its own: the
+ // steering was said into a Turn some durable Root already owns, so
+ // converting it would stand a second, synthetic run beside that one.
+ if (!turnMessages.some((message) => message.type === 'user')) continue;
+ const [turn] = deriveTurnRecords(turnMessages);
+ if (!turn) continue;
+ if (ownedTurnIds.has(turn.turnId)) continue;
Review Comment:
[P1] A Session that already holds both legacy rows and native runs comes back in the wrong order after conversion.
This skip is what makes the state possible to observe, and it is also the clearest evidence the state is expected: it exists so a turn a native run already owns is not converted a second time. **If a Session could never hold both kinds, this branch would be unreachable.**
**The mechanism is not in dispute.** Converted events are appended at `MAX(ordinal) + 1` (`sqlite-runtime-store.ts:3821-3837`), the durable reader orders by ordinal (`runtime-transcript-query.ts:138-142`, `session-transcript-reader.ts:209-223`), and `openedAt` — computed just below at `:97` to keep imported openings ahead of the Session's own runs — does not participate in that ordering. So an earlier legacy turn converted after a later native turn lands behind it.
**What was open was reachability, and it now has a counter-example built entirely from real implementations** — no direct store, SQL, or header writes:
1. a v0.1.8 legacy fixture (`normal-session.jsonl`) loaded as-is;
2. a real `sendMessage` **on v0.1.8** — real backend, real SQLite, provider stubbed — which lands six native events **while the legacy rows stay in the transcript**, because that release's `sendMessage` has no transcript-ledger conversion gate;
3. upgrade to this head and run the migration;
4. reading back in ascending sequence puts the older turn after the newer one, reproduced stably across five pages in each direction.
**This is why the three guards do not cover it.** `ensureTranscriptLedger` inside `admitTurn`, both `agentId` call sites targeting freshly created version-1 children, and a staging Session being unable to send are all true — and all stated on *this build's* send path. They prevent a mixed Session from being **created from here on**. The Session in the repro was not created from here: it was created by a released build that had no such gate. A guard on new operations cannot retire a database an earlier release already wrote.
**What this is not:** not data loss. Every event and row survives; the ordering of a mixed history on read is what breaks. Severity comes from that being silent — the conversion reports success and the transcript simply reads out of order.
**The standard for what to do here is one from this same review thread**, and it applies unchanged: if a class of history is out of scope, the compatibility boundary should refuse it explicitly rather than convert successfully and reorder silently.
<details><summary>简体中文</summary>
**[P1] 一个同时持有 legacy 行与 native run 的 Session,转换后读回顺序是错的。**
这条跳过语句正是让该状态可被观察的地方,**同时也是「该状态本就在预期之内」的最清楚的证据**:它的存在,是为了让一个已被 native run 拥有的 turn 不被二次转换。**如果一个 Session 永远不可能同时持有两类,这条分支就是不可达的。**
**机制本身没有争议。** 转换出的事件按 `MAX(ordinal) + 1` 追加(`sqlite-runtime-store.ts:3821-3837`),持久读路径按 ordinal 排序(`runtime-transcript-query.ts:138-142`、`session-transcript-reader.ts:209-223`),而紧邻下方 `:97` 计算的 `openedAt`(用意是让导入的 opening 排在该 Session 自有 run 之前)**并不参与那个排序**。于是一个更早的 legacy turn 若在更晚的 native turn 之后被转换,就会落到它后面。
**此前开着的是可达性,而现在它有了一个全部由真实实现构成的反例** —— 没有直接写 store、SQL 或 header:
1. 原样载入一份 v0.1.8 的 legacy fixture(`normal-session.jsonl`);
2. **在 v0.1.8 上**跑一次真实 `sendMessage`(真 backend、真 SQLite,provider 用替身),落下六条 native 事件,**而 legacy 行原样留在 transcript 里** —— 因为**那个发布版本的 `sendMessage` 没有 transcript-ledger 转换门**;
3. 升级到本 head,执行迁移;
4. 按 sequence 升序读回时,较早的 turn 排在了较晚的之后;双向各五页稳定复现。
**这就是那三条护栏为什么覆盖不到它。** `ensureTranscriptLedger` 在 `admitTurn` 内、两个 `agentId` 调用点都指向新建的 version 1 child、staging 期的 Session 发不出 Turn —— 三条都成立,**而且都陈述在「本构建」的 send 路径上**。它们阻止的是**从此以后**再造出混合 Session。**复现里那个 Session 不是从此以后造的:它是由一个没有这道门的已发布版本造的。针对新操作的护栏,退不掉一个更早的发布版本已经写下的数据库。**
**这条不是什么:不是数据丢失。** 每一条事件与行都还在,坏掉的是**混合历史在读回时的顺序**。它的严重性来自**静默** —— 转换报告成功,而 transcript 只是读出来是乱的。
**该怎么处理,标准就在这条评审线程里,而且原样适用:如果某类历史不在支持范围内,兼容边界应当明确拒绝它,而不是成功转换后静默错序。**
</details>
---
> **Automated review notice:** This comment was posted by an automated review agent operated by jackwener. It is not an independent human review and does not replace one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "Astro-Han (via GitHub)" <gi...@apache.org>.
Astro-Han commented on code in PR #4879:
URL: https://github.com/apache/maka/pull/4879#discussion_r3945815790
##########
packages/runtime/src/runtime-ledger-repair.ts:
##########
@@ -35,113 +34,173 @@ import type { SessionHeader } from '@maka/core/session';
import type { StoredMessage, TurnRecord } from '@maka/core/session';
import { backfillRuntimeEventsFromStoredMessages } from './runtime-event-backfill.js';
import type { RuntimeEventBackfillOutcome } from './runtime-event-backfill.js';
-import { projectRuntimeEventUserMessage } from './runtime-event-read-model.js';
export interface RuntimeLedgerRepairDeps {
runtimeEventStore: RuntimeEventStore;
- readMessages(sessionId: string): Promise<StoredMessage[]>;
- appendMessage(sessionId: string, message: StoredMessage): Promise<void>;
- newId: () => string;
- now: () => number;
-}
-
-interface RuntimeEventTranscriptProjectionDeps {
- readMessages(sessionId: string): Promise<StoredMessage[]>;
- appendMessage(sessionId: string, message: StoredMessage): Promise<void>;
+ /**
+ * One forward page of the legacy transcript this converter reads; nothing
+ * writes back to it. It is read a page at a time because a Session cannot
+ * serve its first transcript page until this finishes, and a Session's
+ * history is not a bound.
+ */
+ readMessagesAfter(
+ sessionId: string,
+ request: { afterSequence?: number; maxMessages: number; maxStoredBytes: number },
+ ): Promise<{
+ records: readonly { sequence: number; message: StoredMessage }[];
+ highWaterSequence: number | null;
+ }>;
}
-export async function materializeRuntimeEventTranscriptProjection(
- deps: RuntimeEventTranscriptProjectionDeps,
- sessionId: string,
- event: RuntimeEvent,
- knownMessageIds?: Set<string>,
-): Promise<boolean> {
- const message = steeringMessageFromRuntimeEvent(event);
- if (!message) return false;
- const messageIds =
- knownMessageIds ?? new Set((await deps.readMessages(sessionId)).map((item) => item.id));
- if (messageIds.has(message.id)) return false;
- await deps.appendMessage(sessionId, message);
- messageIds.add(message.id);
- return true;
-}
+/** How much of a legacy transcript one conversion page holds. */
+const TRANSCRIPT_CONVERSION_PAGE_MAX_MESSAGES = 256;
+const TRANSCRIPT_CONVERSION_PAGE_MAX_BYTES = 4 * 1024 * 1024;
export class RuntimeLedgerRepair {
private readonly queues = new Map<string, Promise<void>>();
constructor(private readonly deps: RuntimeLedgerRepairDeps) {}
/**
- * Give an imported transcript a runtime spine: one invocation per turn, opened
- * by its own opening fact and closed by its own terminal event.
+ * Give a transcript a runtime spine: one invocation per turn, opened by its
+ * own opening fact and closed by its own terminal event.
*
- * The transcript is the only evidence there is, so a turn it cannot close is
- * refused rather than imported half-formed. Re-running is a no-op: a turn
- * whose invocation already exists is left exactly as it is.
+ * Every event id is derived from the run it belongs to and its position in
+ * that run, so importing the same transcript twice writes the same events and
+ * the store dedupes them. That is what makes an interrupted import resumable:
+ * a turn is skipped once its invocation has ended, and re-derived until then.
*/
async materializeTranscriptLedger(header: SessionHeader): Promise<void> {
const sessionId = header.id;
- return this.withRepairQueue(sessionId, 'transcript-runs', async () => {
- const messages = await this.deps.readMessages(sessionId);
- const ledgerMessages = messages.filter(
- (message) => message.type !== 'user' || message.steeringEventId === undefined,
+ return this.withRepairQueue(sessionId, async () => {
+ // A turn the ledger already owns is not converted again. Its own run is
+ // the authority even when it never ended — a crashed turn is settled by
+ // recovery on that run, and a second, transcript-derived invocation for
+ // the same turn would make the Session read as two. The one exception is
+ // this converter's own run: an interrupted import re-derives it, and the
+ // deterministic ids let the store dedupe what already landed.
+ const inlineInvocations = await this.listInlineInvocations(sessionId);
+ const ownedTurnIds = new Set(
+ inlineInvocations
+ .filter(
+ (invocation) =>
+ invocation.terminalEvent ||
+ invocation.runId !== transcriptRunId(sessionId, invocation.turnId),
+ )
+ .map((invocation) => invocation.turnId),
);
- const openedTurnIds = new Set(
- (await this.listInlineInvocations(sessionId)).map((invocation) => invocation.turnId),
- );
- const messagesByTurn = groupMessagesByTurn(ledgerMessages);
- const turns = deriveTurnRecords(ledgerMessages).filter((turn) =>
- (messagesByTurn.get(turn.turnId) ?? []).some((message) => message.type === 'user'),
- );
- if (turns.length === 0) return;
-
- const firstOpenedAt = Math.max(0, header.createdAt - turns.length);
+ const startedRunIds = new Set(inlineInvocations.map((invocation) => invocation.runId));
- for (const [index, turn] of turns.entries()) {
- if (openedTurnIds.has(turn.turnId)) continue;
- const turnMessages = messagesByTurn.get(turn.turnId) ?? [];
+ for await (const scanned of this.readTurnsInPages(sessionId)) {
+ const turnMessages = scanned.messages;
+ // A turn whose only user row was steering is not a turn of its own: the
+ // steering was said into a Turn some durable Root already owns, so
+ // converting it would stand a second, synthetic run beside that one.
+ if (!turnMessages.some((message) => message.type === 'user')) continue;
+ const [turn] = deriveTurnRecords(turnMessages);
+ if (!turn) continue;
+ if (ownedTurnIds.has(turn.turnId)) continue;
const runId = transcriptRunId(sessionId, turn.turnId);
- const openedAt = firstOpenedAt + index;
+ // Ordered by where the turn starts in the transcript rather than by its
+ // index among all turns: a paged conversion never holds that count, and
+ // both keep every imported opening ahead of the Session's own runs.
+ const openedAt = Math.max(
+ 0,
+ header.createdAt - 1 - (scanned.highWater - scanned.firstSequence),
+ );
const run = { sessionId, runId, turnId: turn.turnId, invocationId: runId };
+ // A build before the ids were derived converted under random ones, so
+ // an interrupted run of its can hold events this build cannot rederive.
+ const started = startedRunIds.has(runId)
+ ? await this.deps.runtimeEventStore.readRuntimeEvents(sessionId, runId)
+ : [];
+ const undeducible = started.filter((event) => !isDerivedTranscriptEventId(runId, event.id));
+ // Its opening is the one such event that can be adopted: the run needs
+ // exactly one, `runtime_events_one_opening_per_invocation` refuses a
+ // second, and which id it landed under changes nothing a reader sees.
+ const adoptedOpening =
+ undeducible.length === 1 && undeducible[0]?.content?.kind === 'invocation_opened';
+ if (undeducible.length > 0 && !adoptedOpening) {
+ // Its converted messages cannot be adopted the same way: rederiving
+ // them would stand a second, deterministic copy of each beside the
+ // one already there, and a Session that disagrees with itself is the
+ // failure this ledger exists to remove. The conversion can neither be
+ // finished nor withdrawn, so it is sealed as the unfinished thing it
+ // is — the legacy rows stay, and no one reads this turn as converted.
+ await this.deps.runtimeEventStore.appendRuntimeEvent(
+ sessionId,
+ runId,
+ abandonedTranscriptTerminalEvent({ run, openedAt }),
+ );
+ continue;
Review Comment:
Confirmed, and the fix is not the seal — the seal was built on a wrong premise of mine.
Your reproduction is right, and so is the counterfactual you describe. Two facts I asserted in the comment on that branch are false:
1. **"The conversion can neither be finished nor withdrawn."** It can be finished. The run holds no terminal event, so it is not sealed and still accepts appends — `assertRunNotSealed` only closes a run that reached its ending.
2. **"Its already-converted messages are not adoptable."** They are. Every backfilled event names the legacy row it came from in `refs.storedMessageId`. That is a durable identity for "this row is already converted", and it does not depend on the id the event landed under.
So the branch resumes on row identity rather than sealing. The converter derives the whole turn exactly as a fresh conversion would — same ids a fresh conversion would mint — and then appends only the events whose source row does not already have that many on the run. A row half-converted by a crash between two of its own events keeps the rest; a re-run appends nothing; the opening is skipped when the run already has one, because `runtime_events_one_opening_per_invocation` admits exactly one and which id it landed under changes nothing a reader sees.
`isDerivedTranscriptEventId` and `abandonedTranscriptTerminalEvent` are deleted with the branch they existed for.
The test you correctly called insufficient now asserts the whole turn: `finishes a released conversion that had already converted messages` requires outcome `completed` and content `[invocation_opened, text, text]` — the answer has to survive, not just the question.
Fixed in `69e110eb0`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "Astro-Han (via GitHub)" <gi...@apache.org>.
Astro-Han commented on code in PR #4879:
URL: https://github.com/apache/maka/pull/4879#discussion_r3944874329
##########
packages/runtime/src/runtime-ledger-repair.ts:
##########
@@ -98,64 +89,42 @@ export class RuntimeLedgerRepair {
const firstOpenedAt = Math.max(0, header.createdAt - turns.length);
for (const [index, turn] of turns.entries()) {
- if (openedTurnIds.has(turn.turnId)) continue;
+ if (ownedTurnIds.has(turn.turnId)) continue;
const turnMessages = messagesByTurn.get(turn.turnId) ?? [];
const runId = transcriptRunId(sessionId, turn.turnId);
const openedAt = firstOpenedAt + index;
const run = { sessionId, runId, turnId: turn.turnId, invocationId: runId };
const events = [
- transcriptOpeningEvent({ header, run, openedAt, newId: this.deps.newId }),
+ transcriptOpeningEvent({ header, run, openedAt }),
Review Comment:
Fixed in `6f1528e0f`, and graded higher than you did — a Session whose durable reads throw on every attempt, with its rows intact and unreachable, is not bounded, and it compounds with the unbounded conversion in the neighbouring thread: the slower the conversion, the likelier the interruption that produces this.
The opening is adopted rather than rewritten: a run needs exactly one, `runtime_events_one_opening_per_invocation` refuses a second, and which id it landed under changes nothing a reader sees.
Its already-converted messages are not adoptable the same way, which is the case you flagged as not covered by an opening-only fix. Rederiving them would stand a second, deterministic copy of each beside the one already there, and a Session that disagrees with itself is the failure this ledger exists to remove. Deleting the prefix and reconverting would give the better end state, but it needs a delete path into the transcript authority — more surface and more risk than the shape it repairs, and it discards facts as you asked me not to. So a run holding such messages is sealed as the unfinished conversion it is, using the existing `missing_terminal_event` vocabulary: no duplicate, no hard error, legacy rows untouched, and nothing claiming that turn converted whole.
Two tests, both failing on the old behaviour: `resumes a conversion a released build opened under a random event id` reproduces your `UNIQUE constraint failed: runtime_events.invocation_id` when the adoption is removed, and `seals a released conversion that had already converted messages` asserts one user text rather than two.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener commented on code in PR #4879:
URL: https://github.com/apache/maka/pull/4879#discussion_r3945706535
##########
packages/runtime/src/runtime-ledger-repair.ts:
##########
@@ -35,113 +34,173 @@ import type { SessionHeader } from '@maka/core/session';
import type { StoredMessage, TurnRecord } from '@maka/core/session';
import { backfillRuntimeEventsFromStoredMessages } from './runtime-event-backfill.js';
import type { RuntimeEventBackfillOutcome } from './runtime-event-backfill.js';
-import { projectRuntimeEventUserMessage } from './runtime-event-read-model.js';
export interface RuntimeLedgerRepairDeps {
runtimeEventStore: RuntimeEventStore;
- readMessages(sessionId: string): Promise<StoredMessage[]>;
- appendMessage(sessionId: string, message: StoredMessage): Promise<void>;
- newId: () => string;
- now: () => number;
-}
-
-interface RuntimeEventTranscriptProjectionDeps {
- readMessages(sessionId: string): Promise<StoredMessage[]>;
- appendMessage(sessionId: string, message: StoredMessage): Promise<void>;
+ /**
+ * One forward page of the legacy transcript this converter reads; nothing
+ * writes back to it. It is read a page at a time because a Session cannot
+ * serve its first transcript page until this finishes, and a Session's
+ * history is not a bound.
+ */
+ readMessagesAfter(
+ sessionId: string,
+ request: { afterSequence?: number; maxMessages: number; maxStoredBytes: number },
+ ): Promise<{
+ records: readonly { sequence: number; message: StoredMessage }[];
+ highWaterSequence: number | null;
+ }>;
}
-export async function materializeRuntimeEventTranscriptProjection(
- deps: RuntimeEventTranscriptProjectionDeps,
- sessionId: string,
- event: RuntimeEvent,
- knownMessageIds?: Set<string>,
-): Promise<boolean> {
- const message = steeringMessageFromRuntimeEvent(event);
- if (!message) return false;
- const messageIds =
- knownMessageIds ?? new Set((await deps.readMessages(sessionId)).map((item) => item.id));
- if (messageIds.has(message.id)) return false;
- await deps.appendMessage(sessionId, message);
- messageIds.add(message.id);
- return true;
-}
+/** How much of a legacy transcript one conversion page holds. */
+const TRANSCRIPT_CONVERSION_PAGE_MAX_MESSAGES = 256;
+const TRANSCRIPT_CONVERSION_PAGE_MAX_BYTES = 4 * 1024 * 1024;
export class RuntimeLedgerRepair {
private readonly queues = new Map<string, Promise<void>>();
constructor(private readonly deps: RuntimeLedgerRepairDeps) {}
/**
- * Give an imported transcript a runtime spine: one invocation per turn, opened
- * by its own opening fact and closed by its own terminal event.
+ * Give a transcript a runtime spine: one invocation per turn, opened by its
+ * own opening fact and closed by its own terminal event.
*
- * The transcript is the only evidence there is, so a turn it cannot close is
- * refused rather than imported half-formed. Re-running is a no-op: a turn
- * whose invocation already exists is left exactly as it is.
+ * Every event id is derived from the run it belongs to and its position in
+ * that run, so importing the same transcript twice writes the same events and
+ * the store dedupes them. That is what makes an interrupted import resumable:
+ * a turn is skipped once its invocation has ended, and re-derived until then.
*/
async materializeTranscriptLedger(header: SessionHeader): Promise<void> {
const sessionId = header.id;
- return this.withRepairQueue(sessionId, 'transcript-runs', async () => {
- const messages = await this.deps.readMessages(sessionId);
- const ledgerMessages = messages.filter(
- (message) => message.type !== 'user' || message.steeringEventId === undefined,
+ return this.withRepairQueue(sessionId, async () => {
+ // A turn the ledger already owns is not converted again. Its own run is
+ // the authority even when it never ended — a crashed turn is settled by
+ // recovery on that run, and a second, transcript-derived invocation for
+ // the same turn would make the Session read as two. The one exception is
+ // this converter's own run: an interrupted import re-derives it, and the
+ // deterministic ids let the store dedupe what already landed.
+ const inlineInvocations = await this.listInlineInvocations(sessionId);
+ const ownedTurnIds = new Set(
+ inlineInvocations
+ .filter(
+ (invocation) =>
+ invocation.terminalEvent ||
+ invocation.runId !== transcriptRunId(sessionId, invocation.turnId),
+ )
+ .map((invocation) => invocation.turnId),
);
- const openedTurnIds = new Set(
- (await this.listInlineInvocations(sessionId)).map((invocation) => invocation.turnId),
- );
- const messagesByTurn = groupMessagesByTurn(ledgerMessages);
- const turns = deriveTurnRecords(ledgerMessages).filter((turn) =>
- (messagesByTurn.get(turn.turnId) ?? []).some((message) => message.type === 'user'),
- );
- if (turns.length === 0) return;
-
- const firstOpenedAt = Math.max(0, header.createdAt - turns.length);
+ const startedRunIds = new Set(inlineInvocations.map((invocation) => invocation.runId));
- for (const [index, turn] of turns.entries()) {
- if (openedTurnIds.has(turn.turnId)) continue;
- const turnMessages = messagesByTurn.get(turn.turnId) ?? [];
+ for await (const scanned of this.readTurnsInPages(sessionId)) {
+ const turnMessages = scanned.messages;
+ // A turn whose only user row was steering is not a turn of its own: the
+ // steering was said into a Turn some durable Root already owns, so
+ // converting it would stand a second, synthetic run beside that one.
+ if (!turnMessages.some((message) => message.type === 'user')) continue;
+ const [turn] = deriveTurnRecords(turnMessages);
+ if (!turn) continue;
+ if (ownedTurnIds.has(turn.turnId)) continue;
const runId = transcriptRunId(sessionId, turn.turnId);
- const openedAt = firstOpenedAt + index;
+ // Ordered by where the turn starts in the transcript rather than by its
+ // index among all turns: a paged conversion never holds that count, and
+ // both keep every imported opening ahead of the Session's own runs.
+ const openedAt = Math.max(
+ 0,
+ header.createdAt - 1 - (scanned.highWater - scanned.firstSequence),
+ );
const run = { sessionId, runId, turnId: turn.turnId, invocationId: runId };
+ // A build before the ids were derived converted under random ones, so
+ // an interrupted run of its can hold events this build cannot rederive.
+ const started = startedRunIds.has(runId)
+ ? await this.deps.runtimeEventStore.readRuntimeEvents(sessionId, runId)
+ : [];
+ const undeducible = started.filter((event) => !isDerivedTranscriptEventId(runId, event.id));
+ // Its opening is the one such event that can be adopted: the run needs
+ // exactly one, `runtime_events_one_opening_per_invocation` refuses a
+ // second, and which id it landed under changes nothing a reader sees.
+ const adoptedOpening =
+ undeducible.length === 1 && undeducible[0]?.content?.kind === 'invocation_opened';
+ if (undeducible.length > 0 && !adoptedOpening) {
+ // Its converted messages cannot be adopted the same way: rederiving
+ // them would stand a second, deterministic copy of each beside the
+ // one already there, and a Session that disagrees with itself is the
+ // failure this ledger exists to remove. The conversion can neither be
+ // finished nor withdrawn, so it is sealed as the unfinished thing it
+ // is — the legacy rows stay, and no one reads this turn as converted.
+ await this.deps.runtimeEventStore.appendRuntimeEvent(
+ sessionId,
+ runId,
+ abandonedTranscriptTerminalEvent({ run, openedAt }),
+ );
+ continue;
+ }
const events = [
- transcriptOpeningEvent({ header, run, openedAt, newId: this.deps.newId }),
+ ...(adoptedOpening ? [] : [transcriptOpeningEvent({ header, run, openedAt })]),
...backfillRuntimeEventsFromStoredMessages({
run,
outcome: transcriptOutcome(turn, turnMessages, openedAt),
messages: turnMessages,
- modelHistory: 'conversation_text',
- newId: this.deps.newId,
- now: this.deps.now,
+ // Another runtime's tool calls belong to its protocol, not to the
+ // provider this Session will talk to next, so a foreign transcript
+ // converts as the conversation it is. Maka's own history converts
+ // whole: its tool calls are the ones it would replay.
+ modelHistory: header.externalOrigin ? 'conversation_text' : 'full',
+ newId: transcriptEventIds(runId),
+ // The payload must be as repeatable as its id: SQLite dedupes
+ // complete events, including the backfill provenance timestamps.
+ now: () => openedAt,
}).events,
];
- if (!events.some(isTerminalRuntimeEvent)) {
- throw new Error(`Imported transcript Run ${runId} has no terminal RuntimeEvent`);
- }
for (const event of events) {
await this.deps.runtimeEventStore.appendRuntimeEvent(sessionId, runId, event);
}
}
});
}
- async repairSteeringMessagesOnce(sessionId: string): Promise<number> {
- return this.withRepairQueue(sessionId, 'steering-transcript', async () => {
- const messages = await this.deps.readMessages(sessionId);
- const messageIds = new Set(messages.map((message) => message.id));
- const inlineRunIds = new Set(
- (await this.listInlineInvocations(sessionId)).map((invocation) => invocation.runId),
+ /**
+ * The Session's legacy rows, one turn at a time, read a page at a time.
+ *
+ * A turn is only complete once a row of another turn follows it, so the rows
+ * of the page's last turn are carried into the next page rather than
+ * converted early. Peak memory is therefore one page plus one turn — the same
+ * bound the transcript reader keeps, and not the Session's whole history.
+ */
+ private async *readTurnsInPages(
+ sessionId: string,
+ ): AsyncGenerator<{ messages: StoredMessage[]; firstSequence: number; highWater: number }> {
+ let carried: { messages: StoredMessage[]; firstSequence: number } | undefined;
+ let afterSequence: number | undefined;
+ while (true) {
Review Comment:
**[P2] An interrupted conversion still rescans the whole transcript from sequence zero — paging made each page cheap, not the work resumable.**
`afterSequence` is a local initialised to `undefined` (`runtime-ledger-repair.ts:172-175`), and `highWaterSequence` is used only for ordering (`:180`, `:200`, `:107-110`). **It is never persisted as a cursor.** The only durable marker is still written after full success, at `session-manager.ts:4286-4289`.
Measured on the production path: a 200-turn / 600-row legacy transcript, interrupted after the 100th converted user event, then a fresh repair instance. The first attempt requested pages `[null, 255]`; the retry requested `[null, 255, 511, 599]` — **it reread the transcript from the beginning.**
Deterministic event ids do their job: the retry writes no duplicates. **But idempotency is not resumability.** Repeated interruption keeps paying full-history I/O, plus the unbounded retry inventory noted separately, and the Session stays unreadable for as long as that keeps failing.
**The existing interruption test cannot detect it**: `runtime-ledger-repair.test.ts:406-501` proves event idempotency and runs within a single page, so no second page request is ever observed.
A durable per-Session cursor committed with each page would make the retry cost proportional to what is left rather than to the whole Session, and would give the property a guard that fails when it regresses.
<details><summary>简体中文</summary>
**[P2] 被中断的转换仍会从 sequence 0 重扫整份 transcript —— 分页让每一页便宜了,并没有让这项工作可续跑。**
`afterSequence` 是初始化为 `undefined` 的局部变量(`runtime-ledger-repair.ts:172-175`),而 `highWaterSequence` 只用于排序(`:180`、`:200`、`:107-110`),**从未被持久化为游标**。唯一的持久标记仍然是在**整体成功之后**才写,位于 `session-manager.ts:4286-4289`。
**在生产路径上实测:** 一份 200 turn / 600 行的旧 transcript,在第 100 个已转换 user 事件之后中断,然后新建一个 repair 实例。第一次请求的页是 `[null, 255]`;**重试请求的是 `[null, 255, 511, 599]` —— 它从头把 transcript 重读了一遍。**
确定性事件 id 起了它该起的作用:重试不会写出重复事件。**但幂等不等于可续跑。** 反复中断就会反复付出整份历史的 I/O,外加另一条单独指出的无界重试清单,**而只要这件事持续失败,该 Session 就一直不可读。**
**现有的中断测试检测不到它**:`runtime-ledger-repair.test.ts:406-501` 证明的是事件幂等,且整个跑在**单页之内**,所以永远观察不到第二次翻页请求。
**每页提交时持久化一个按 Session 的游标**,会让重试代价与「剩下多少」成正比而不是与整个 Session 成正比,**并且让这条性质有一个在回退时会失败的守卫。**
</details>
---
> **Automated review notice:** This comment was posted by an automated review agent operated by jackwener. It is not an independent human review and does not replace one.
##########
packages/runtime/src/runtime-ledger-repair.ts:
##########
@@ -35,113 +34,173 @@ import type { SessionHeader } from '@maka/core/session';
import type { StoredMessage, TurnRecord } from '@maka/core/session';
import { backfillRuntimeEventsFromStoredMessages } from './runtime-event-backfill.js';
import type { RuntimeEventBackfillOutcome } from './runtime-event-backfill.js';
-import { projectRuntimeEventUserMessage } from './runtime-event-read-model.js';
export interface RuntimeLedgerRepairDeps {
runtimeEventStore: RuntimeEventStore;
- readMessages(sessionId: string): Promise<StoredMessage[]>;
- appendMessage(sessionId: string, message: StoredMessage): Promise<void>;
- newId: () => string;
- now: () => number;
-}
-
-interface RuntimeEventTranscriptProjectionDeps {
- readMessages(sessionId: string): Promise<StoredMessage[]>;
- appendMessage(sessionId: string, message: StoredMessage): Promise<void>;
+ /**
+ * One forward page of the legacy transcript this converter reads; nothing
+ * writes back to it. It is read a page at a time because a Session cannot
+ * serve its first transcript page until this finishes, and a Session's
+ * history is not a bound.
+ */
+ readMessagesAfter(
+ sessionId: string,
+ request: { afterSequence?: number; maxMessages: number; maxStoredBytes: number },
+ ): Promise<{
+ records: readonly { sequence: number; message: StoredMessage }[];
+ highWaterSequence: number | null;
+ }>;
}
-export async function materializeRuntimeEventTranscriptProjection(
- deps: RuntimeEventTranscriptProjectionDeps,
- sessionId: string,
- event: RuntimeEvent,
- knownMessageIds?: Set<string>,
-): Promise<boolean> {
- const message = steeringMessageFromRuntimeEvent(event);
- if (!message) return false;
- const messageIds =
- knownMessageIds ?? new Set((await deps.readMessages(sessionId)).map((item) => item.id));
- if (messageIds.has(message.id)) return false;
- await deps.appendMessage(sessionId, message);
- messageIds.add(message.id);
- return true;
-}
+/** How much of a legacy transcript one conversion page holds. */
+const TRANSCRIPT_CONVERSION_PAGE_MAX_MESSAGES = 256;
+const TRANSCRIPT_CONVERSION_PAGE_MAX_BYTES = 4 * 1024 * 1024;
export class RuntimeLedgerRepair {
private readonly queues = new Map<string, Promise<void>>();
constructor(private readonly deps: RuntimeLedgerRepairDeps) {}
/**
- * Give an imported transcript a runtime spine: one invocation per turn, opened
- * by its own opening fact and closed by its own terminal event.
+ * Give a transcript a runtime spine: one invocation per turn, opened by its
+ * own opening fact and closed by its own terminal event.
*
- * The transcript is the only evidence there is, so a turn it cannot close is
- * refused rather than imported half-formed. Re-running is a no-op: a turn
- * whose invocation already exists is left exactly as it is.
+ * Every event id is derived from the run it belongs to and its position in
+ * that run, so importing the same transcript twice writes the same events and
+ * the store dedupes them. That is what makes an interrupted import resumable:
+ * a turn is skipped once its invocation has ended, and re-derived until then.
*/
async materializeTranscriptLedger(header: SessionHeader): Promise<void> {
const sessionId = header.id;
- return this.withRepairQueue(sessionId, 'transcript-runs', async () => {
- const messages = await this.deps.readMessages(sessionId);
- const ledgerMessages = messages.filter(
- (message) => message.type !== 'user' || message.steeringEventId === undefined,
+ return this.withRepairQueue(sessionId, async () => {
+ // A turn the ledger already owns is not converted again. Its own run is
+ // the authority even when it never ended — a crashed turn is settled by
+ // recovery on that run, and a second, transcript-derived invocation for
+ // the same turn would make the Session read as two. The one exception is
+ // this converter's own run: an interrupted import re-derives it, and the
+ // deterministic ids let the store dedupe what already landed.
+ const inlineInvocations = await this.listInlineInvocations(sessionId);
+ const ownedTurnIds = new Set(
+ inlineInvocations
+ .filter(
+ (invocation) =>
+ invocation.terminalEvent ||
+ invocation.runId !== transcriptRunId(sessionId, invocation.turnId),
+ )
+ .map((invocation) => invocation.turnId),
);
- const openedTurnIds = new Set(
- (await this.listInlineInvocations(sessionId)).map((invocation) => invocation.turnId),
- );
- const messagesByTurn = groupMessagesByTurn(ledgerMessages);
- const turns = deriveTurnRecords(ledgerMessages).filter((turn) =>
- (messagesByTurn.get(turn.turnId) ?? []).some((message) => message.type === 'user'),
- );
- if (turns.length === 0) return;
-
- const firstOpenedAt = Math.max(0, header.createdAt - turns.length);
+ const startedRunIds = new Set(inlineInvocations.map((invocation) => invocation.runId));
- for (const [index, turn] of turns.entries()) {
- if (openedTurnIds.has(turn.turnId)) continue;
- const turnMessages = messagesByTurn.get(turn.turnId) ?? [];
+ for await (const scanned of this.readTurnsInPages(sessionId)) {
+ const turnMessages = scanned.messages;
+ // A turn whose only user row was steering is not a turn of its own: the
+ // steering was said into a Turn some durable Root already owns, so
+ // converting it would stand a second, synthetic run beside that one.
+ if (!turnMessages.some((message) => message.type === 'user')) continue;
+ const [turn] = deriveTurnRecords(turnMessages);
+ if (!turn) continue;
+ if (ownedTurnIds.has(turn.turnId)) continue;
const runId = transcriptRunId(sessionId, turn.turnId);
- const openedAt = firstOpenedAt + index;
+ // Ordered by where the turn starts in the transcript rather than by its
+ // index among all turns: a paged conversion never holds that count, and
+ // both keep every imported opening ahead of the Session's own runs.
+ const openedAt = Math.max(
+ 0,
+ header.createdAt - 1 - (scanned.highWater - scanned.firstSequence),
+ );
const run = { sessionId, runId, turnId: turn.turnId, invocationId: runId };
+ // A build before the ids were derived converted under random ones, so
+ // an interrupted run of its can hold events this build cannot rederive.
+ const started = startedRunIds.has(runId)
+ ? await this.deps.runtimeEventStore.readRuntimeEvents(sessionId, runId)
+ : [];
+ const undeducible = started.filter((event) => !isDerivedTranscriptEventId(runId, event.id));
+ // Its opening is the one such event that can be adopted: the run needs
+ // exactly one, `runtime_events_one_opening_per_invocation` refuses a
+ // second, and which id it landed under changes nothing a reader sees.
+ const adoptedOpening =
+ undeducible.length === 1 && undeducible[0]?.content?.kind === 'invocation_opened';
+ if (undeducible.length > 0 && !adoptedOpening) {
+ // Its converted messages cannot be adopted the same way: rederiving
+ // them would stand a second, deterministic copy of each beside the
+ // one already there, and a Session that disagrees with itself is the
+ // failure this ledger exists to remove. The conversion can neither be
+ // finished nor withdrawn, so it is sealed as the unfinished thing it
+ // is — the legacy rows stay, and no one reads this turn as converted.
+ await this.deps.runtimeEventStore.appendRuntimeEvent(
+ sessionId,
+ runId,
+ abandonedTranscriptTerminalEvent({ run, openedAt }),
+ );
+ continue;
+ }
const events = [
- transcriptOpeningEvent({ header, run, openedAt, newId: this.deps.newId }),
+ ...(adoptedOpening ? [] : [transcriptOpeningEvent({ header, run, openedAt })]),
...backfillRuntimeEventsFromStoredMessages({
run,
outcome: transcriptOutcome(turn, turnMessages, openedAt),
messages: turnMessages,
- modelHistory: 'conversation_text',
- newId: this.deps.newId,
- now: this.deps.now,
+ // Another runtime's tool calls belong to its protocol, not to the
+ // provider this Session will talk to next, so a foreign transcript
+ // converts as the conversation it is. Maka's own history converts
+ // whole: its tool calls are the ones it would replay.
+ modelHistory: header.externalOrigin ? 'conversation_text' : 'full',
+ newId: transcriptEventIds(runId),
+ // The payload must be as repeatable as its id: SQLite dedupes
+ // complete events, including the backfill provenance timestamps.
+ now: () => openedAt,
}).events,
];
- if (!events.some(isTerminalRuntimeEvent)) {
- throw new Error(`Imported transcript Run ${runId} has no terminal RuntimeEvent`);
- }
for (const event of events) {
await this.deps.runtimeEventStore.appendRuntimeEvent(sessionId, runId, event);
}
}
});
}
- async repairSteeringMessagesOnce(sessionId: string): Promise<number> {
- return this.withRepairQueue(sessionId, 'steering-transcript', async () => {
- const messages = await this.deps.readMessages(sessionId);
- const messageIds = new Set(messages.map((message) => message.id));
- const inlineRunIds = new Set(
- (await this.listInlineInvocations(sessionId)).map((invocation) => invocation.runId),
+ /**
+ * The Session's legacy rows, one turn at a time, read a page at a time.
+ *
+ * A turn is only complete once a row of another turn follows it, so the rows
+ * of the page's last turn are carried into the next page rather than
+ * converted early. Peak memory is therefore one page plus one turn — the same
+ * bound the transcript reader keeps, and not the Session's whole history.
+ */
+ private async *readTurnsInPages(
+ sessionId: string,
+ ): AsyncGenerator<{ messages: StoredMessage[]; firstSequence: number; highWater: number }> {
+ let carried: { messages: StoredMessage[]; firstSequence: number } | undefined;
+ let afterSequence: number | undefined;
+ while (true) {
+ const page = await this.deps.readMessagesAfter(sessionId, {
+ ...(afterSequence === undefined ? {} : { afterSequence }),
+ maxMessages: TRANSCRIPT_CONVERSION_PAGE_MAX_MESSAGES,
+ maxStoredBytes: TRANSCRIPT_CONVERSION_PAGE_MAX_BYTES,
+ });
+ const highWater = page.highWaterSequence;
+ if (highWater === null) return;
+ const scanned = page.records.filter(
+ ({ message }) => message.type !== 'user' || message.steeringEventId === undefined,
);
- let repaired = 0;
- for (const event of await this.deps.runtimeEventStore.readSessionRuntimeEvents(sessionId)) {
- if (!inlineRunIds.has(event.runId)) continue;
- if (
- await materializeRuntimeEventTranscriptProjection(this.deps, sessionId, event, messageIds)
- ) {
- repaired += 1;
- }
+ const grouped = new Map<string, { messages: StoredMessage[]; firstSequence: number }>();
+ if (carried) grouped.set(turnIdOf(carried.messages[0]) ?? '', carried);
+ for (const { sequence, message } of scanned) {
+ const turnId = turnIdOf(message);
+ if (!turnId) continue;
+ const bucket = grouped.get(turnId);
Review Comment:
**[P2] Paging bounds the page, but three allocations still scale with the Session, so the first readable transcript page is not yet on a fixed memory bound.**
The forward paging is a real improvement and it closes the general case. What survives it:
- **`carried` holds every row of one turn across pages** (`runtime-ledger-repair.ts:172-203`). A legacy transcript that is a single long turn has no turn boundary to flush on, so that array grows with the whole transcript.
- **The retry path re-materialises every opening.** `:82-92` calls `listSessionInvocations` unbounded, and the SQLite implementation materialises all of them at `sqlite-runtime-store.ts:602-608`.
So a large record, one very large turn, or a partially converted long Session still evades a fixed bound **before the Session can serve its first page** — which is the same availability shape as before, reached through a narrower door rather than closed.
**The existing test cannot see this.** `runtime-ledger-repair.test.ts:930-1002` uses 200 tiny three-row turns and measures the returned `page.records`; it does not observe `carried`, the retry inventory, or peak allocation. A guard for this property needs to measure what is held, not what is returned.
<details><summary>简体中文</summary>
**[P2] 分页限住了「每页」,但仍有三处分配随 Session 增长,所以「第一个可读 transcript 页」还没有落在固定内存上界内。**
前向分页是实实在在的改进,一般情形已经被它关上了。**没被关上的是:**
- **`carried` 会跨页保留一个 turn 的全部行**(`runtime-ledger-repair.ts:172-203`)。**一份「整体就是一个长 turn」的旧 transcript 没有可供刷出的 turn 边界**,于是这个数组随整份 transcript 增长。
- **重试路径会把全部 opening 重新物化。** `:82-92` 无界调用 `listSessionInvocations`,而 SQLite 实现在 `sqlite-runtime-store.ts:602-608` 把它们全部物化。
所以**大记录、单个超大 turn、或一个转换到一半的长 Session**,在**该 Session 能返回第一页之前**仍然逃出固定上界 —— **这与原先是同一种可用性形状,只是从一扇更窄的门进来,而不是被关上。**
**现有测试看不见这一点。** `runtime-ledger-repair.test.ts:930-1002` 用的是 200 个三行小 turn,量的是返回的 `page.records`;**它不观察 `carried`、不观察重试清单、也不观察峰值分配。** 要守住这条性质,测的必须是**被持有的东西**,而不是**被返回的东西**。
</details>
---
> **Automated review notice:** This comment was posted by an automated review agent operated by jackwener. It is not an independent human review and does not replace one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "Astro-Han (via GitHub)" <gi...@apache.org>.
Astro-Han commented on code in PR #4879:
URL: https://github.com/apache/maka/pull/4879#discussion_r3945818203
##########
packages/runtime-host/src/server/session-transcript-reader.ts:
##########
@@ -47,29 +59,67 @@ const ACTIVE_TRANSCRIPT_SCAN_BATCH_MAX_BYTES = 256 * 1024;
export function createSessionTranscriptReader(input: {
stores: ExecutionStoresWriter<'interactive'>;
canonicalPermissionOutcomes: CanonicalPermissionOutcomeReader;
+ /**
+ * Converts a Session whose transcript predates the ledger, before this reader
+ * looks for invocations that only the conversion can create. Omitted only by
+ * tests that seed the ledger themselves.
+ */
+ ensureTranscriptLedger?: (sessionId: string) => Promise<void>;
}): SessionTranscriptReader {
+ const durable = createDurableLedgerTranscriptReader(input);
+ const prepared = async (sessionId: string): Promise<void> => {
+ await input.ensureTranscriptLedger?.(sessionId);
+ };
return {
- readDurableHighWater: (sessionId) =>
- input.stores.sessionStore.readTranscriptHighWaterSnapshot(sessionId),
- readDurablePage: (sessionId, request) =>
- input.stores.sessionStore.readTranscriptPageSnapshot(sessionId, request),
- readDurableRecords: (sessionId, request) =>
- input.stores.sessionStore.readTranscriptRecordsSnapshot(sessionId, request),
- readDurableMessagesById: (sessionId, request) =>
- input.stores.sessionStore.readTranscriptMessagesSnapshot(sessionId, request),
+ readDurableHighWater: async (sessionId) => {
+ await prepared(sessionId);
+ return durable.readHighWater(sessionId);
+ },
+ readDurablePage: async (sessionId, request) => {
+ await prepared(sessionId);
+ return durable.readPage(sessionId, request);
Review Comment:
Agreed, and done — the disclosure belongs where acceptance happens, not in a review reply.
The PR body now carries a section, *What this cutover permanently loses*, that states the whole chain you enumerated: `transcriptLedgerVersion: 1` means a conversion ran, not that `session_messages` is empty; a released build set the marker on the first send and kept writing turn-scoped context notes to the transcript alone; re-running the converter cannot reach them because those turns belong to runs a real execution already sealed. It names all seven kinds explicitly, says it applies to every pre-existing Session a released build touched, says it is permanent rather than deferred, and says plainly what is *not* affected — no user message, assistant message, tool call, or tool result.
I also replaced the sentence you quoted. "Losing a legacy tool card in conversion is acceptable" understated it, which was your point.
Your #4808 comparison is the right standard and I have adopted it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener commented on PR #4879:
URL: https://github.com/apache/maka/pull/4879#issuecomment-5558432258
Re-reviewed at exact head `7c2c74e5`. **Both findings are closed, and closed at the right level — the P1 fix makes the derivation one rule rather than patching one side of the mismatch.** Two narrower residuals below, neither blocking.
## [P1] Closed, and it also covers rows already on disk
`admittedPromptEventId(runId, userMessageId)` is now a single function used by the writer (`agent-run.ts:739`) and by recovery (`hosted-execution-recovery.ts:88`, `:256`). A folded Root's `null` resolves to `${runId}-admitted-prompt` on both sides, and appending that id twice against production SQLite yields **one** user event.
**The part worth calling out is not in the summary: recovery also stopped filtering the turn's user messages down to the derived id.** `verifyUserMessage` now treats a single user event on the turn as the prompt being present, judged on content and origin. That matters because it is what makes the fix reach **runs already written by an older build under `newId()`** — those are exactly the rows that would still have grown a second prompt if the fix had only aligned the two derivations going forward. The new test seeds a `randomUUID` prompt already on the ledger and asserts it stays alone.
**The mirror risk is avoided.** Where `userMessageId` is present it is returned unchanged — `'msg-queued-1'` stays `'msg-queued-1'` — and `this.input.userMessageId ?? this.input.newId()` is gone from `agent-run.ts`. A fix that rewrote single-source ids into the derived form would have reintroduced the same defect pointing the other way, at existing data.
## [P2] Closed at the seam that made it worse than a crash
`initialRuntimeEventPending` is set after `openInvocation` returns and cleared once the prompt write succeeds; `failStart` still routes into `finalize`, and `finalize` now writes the prompt **before** the terminal when the flag is set. The comment names the reason precisely — the terminal seals the run against every later append, including the one crash recovery would have used to repair the same shape. Their test refuses `run-1-admitted-prompt` once so `begin` throws, then finds both that prompt and one terminal on the ledger after `finalize`.
**Residuals, both narrower than the hole they came from:**
- The retry is `.catch(() => {})`, so if the prompt write fails a second time the run still seals empty. Refusing to finalize would be worse, so this is the right trade — it is worth knowing it exists, not worth changing.
- **A throw inside `openInvocation` never sets the flag**, so `finalize` can still open-then-seal a run with no prompt. That is a smaller window than the named one and is not a regression from this change.
## [P2] The unread tail is genuinely paged now, and the bound it replaces is not lost
`#newestVisibleMessage` pages backwards instead of reading one bounded tail, and the comment names the shape I reported: a Turn ending on tool traffic can put more hidden records at the tail than one page holds.
**I checked the termination, since replacing a bound with a loop is where this kind of fix usually overcorrects.** It stops on the first visible message or when `nextPosition === null`. So the only way to walk the whole ledger is a Session with no visible message anywhere — and such a Session has no unread badge to clear in the first place. **The cost scales with the trailing hidden stretch, not with Session length**, and it runs on a user action rather than a hot path. Trading a few extra pages in a rare shape for the removal of a permanently-stuck badge is the right direction.
## [P2] The steering-only turn is now explained rather than changed
The filter still drops a turn whose only user row was steering, and three lines of comment now say why: that steering was said into a Turn a durable Root already owns, so converting it would stand a second synthetic run beside the real one. **The reasoning holds and I am not asking for a behaviour change.** Stating it plainly, though: the filter is still silent — what changed is that a reader of the code can now find out why, not that a workspace where it fires reports anything.
## Standing
The [P1] is gone, so the objection that made this NO-GO is resolved. **The PR is `DIRTY` against `main`, so I am not attaching an approval to this head** — an approval survives later pushes here, and I would rather not have one carry across an unresolved rebase, the same way I handled the conflict on #4890. Once it rebases green I will approve without re-litigating any of the above.
Everything else from the earlier passes stands: the importer is total over the legacy row types enumerated from tagged writers, an unknown type fails closed rather than truncating history, import staging is reentrant with dedupe verified on the real store, the five deleted second-writes carry their proofs elsewhere, and the WorkHub linkage swap is safe including for Sessions that delegated before this PR.
<details><summary>简体中文</summary>
在 exact head `7c2c74e5` 上复审。**两条 finding 都已关闭,而且关在了正确的层次上 —— P1 的修法是把派生变成一条规则,而不是给不一致的其中一侧打补丁。** 下面两处残留更窄,均不阻塞。
**[P1] 已关闭,而且覆盖了已经落盘的行**
`admittedPromptEventId(runId, userMessageId)` 现在是一个函数,写入方(`agent-run.ts:739`)与恢复方(`hosted-execution-recovery.ts:88`、`:256`)共用。折叠 Root 的 `null` 在两侧都解析为 `${runId}-admitted-prompt`;在生产 SQLite 上以该 id 追加两次,得到**一条** user 事件。
**值得点出的一处不在摘要里:恢复侧同时不再把该 turn 的 user 消息过滤到派生 id。** `verifyUserMessage` 现在按内容与来源,把「该 turn 上存在一条 user 事件」即视为 prompt 已在。**这一点要紧,因为正是它让修复触及「更早的构建以 `newId()` 写下的既有 run」** —— 而那些行恰恰是「只对齐今后两侧派生」时仍会长出第二条 prompt 的那批。新测试预先在账本上放了一条 `randomUUID` 的 prompt,并断言它保持独一份。
**镜像风险被避开了。** `userMessageId` 存在时原样返回(`'msg-queued-1'` 仍是 `'msg-queued-1'`),而 `this.input.userMessageId ?? this.input.newId()` 已从 `agent-run.ts` 移除。**一个把单源 id 改写成派生形式的修法,会以相反方向、在既有数据上重新制造同一个缺陷。**
**[P2] 关在了「它比崩溃更糟」的那个接缝上**
`initialRuntimeEventPending` 在 `openInvocation` 返回后置位,在 prompt 写入成功后清除;`failStart` 仍然走向 `finalize`,而 `finalize` 现在在标志置位时**先写 prompt、再写 terminal**。注释把原因说得很准 —— terminal 会把该 run 对其后所有 append 封死,**包括崩溃恢复本会用来修同一形状的那一次**。他们的测试让 `run-1-admitted-prompt` 被拒绝一次以致 `begin` 抛出,随后在 `finalize` 之后于账本上同时找到该 prompt 与一条 terminal。
**两处残留,都比它们所出自的洞更窄:**
- 那次补写是 `.catch(() => {})`,所以 prompt 写入第二次再失败时,run 仍会被封存为空。**拒绝 finalize 会更糟,所以这是正确的取舍** —— 值得知道它存在,不值得为它改动。
- **`openInvocation` 内部抛出时标志从未被置位**,所以 `finalize` 仍可能「开了就封」而没有 prompt。这个窗口比被点名的那个更小,**也不是本次改动引入的回退。**
**[P2] 未读尾部确实改成了翻页,而它取代的那个上限并没有丢**
`#newestVisibleMessage` 改为向前翻页,而不是只读一段有界尾部,注释也点名了我报的形状:**以工具流量结尾的 Turn,尾部隐藏记录可能多于一页所能容纳。**
**我核了终止条件 —— 因为「用循环取代上限」正是这类修复容易矫枉过正的地方。** 它在遇到第一条可见消息、或 `nextPosition === null` 时停止。**所以唯一会走遍整个账本的情形,是一个从头到尾没有任何可见消息的 Session —— 而这样的 Session 本来就没有未读角标可清。** 代价与**尾部隐藏连续段**成正比,而不与 Session 长度成正比,并且它跑在一次用户操作上而非热路径。**用罕见形状下多翻几页,换掉一个会永久卡住的角标,方向是对的。**
**[P2] 只有 steering 的 turn 现在是被解释了,而不是被改变了**
该过滤仍会丢弃「唯一 user 行是 steering」的 turn,现在有三行注释说明原因:那段 steering 是说进某个已有持久 Root 所拥有的 Turn 里的,**转换它就会在真实 run 旁边立起第二个合成 run。** **这个理由成立,我不要求改变行为。** 但把话说清楚:**该过滤仍然是静默的** —— 改变的是读代码的人现在能查到原因,而不是「它触发的那个工作区会报告些什么」。
**当前立场**
[P1] 已消除,所以让本单成为 NO-GO 的那条反对意见已解决。**该 PR 相对 `main` 处于 `DIRTY`,因此我不在此 head 上附加批准** —— 在本仓库批准会跨后续 push 存活,**我不愿让一个批准跨过一次尚未解决的 rebase**,与我在 #4890 上处理冲突的方式一致。待其 rebase 转绿,我会直接批准,不再重提以上任何一条。
先前各轮的其余结论均成立:导入器对**从已发布 tag 枚举**的 legacy 行类型是全的;未知类型 fail-closed 而不是截断历史;导入暂存可重入且去重已在真实 store 上验证;被删的五处第二写点其证据确在别处;WorkHub 联动的替换是安全的,**包括对本 PR 之前就发生过委派的 Session。**
</details>
---
> **Automated review notice:** This comment was posted by an automated review agent operated by jackwener. It is not an independent human review and does not replace one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "Astro-Han (via GitHub)" <gi...@apache.org>.
Astro-Han merged PR #4879:
URL: https://github.com/apache/maka/pull/4879
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "Astro-Han (via GitHub)" <gi...@apache.org>.
Astro-Han commented on code in PR #4879:
URL: https://github.com/apache/maka/pull/4879#discussion_r3944871885
##########
packages/runtime/src/runtime-ledger-repair.ts:
##########
@@ -72,24 +47,40 @@ export class RuntimeLedgerRepair {
constructor(private readonly deps: RuntimeLedgerRepairDeps) {}
/**
- * Give an imported transcript a runtime spine: one invocation per turn, opened
- * by its own opening fact and closed by its own terminal event.
+ * Give a transcript a runtime spine: one invocation per turn, opened by its
+ * own opening fact and closed by its own terminal event.
*
- * The transcript is the only evidence there is, so a turn it cannot close is
- * refused rather than imported half-formed. Re-running is a no-op: a turn
- * whose invocation already exists is left exactly as it is.
+ * Every event id is derived from the run it belongs to and its position in
+ * that run, so importing the same transcript twice writes the same events and
+ * the store dedupes them. That is what makes an interrupted import resumable:
+ * a turn is skipped once its invocation has ended, and re-derived until then.
*/
async materializeTranscriptLedger(header: SessionHeader): Promise<void> {
const sessionId = header.id;
- return this.withRepairQueue(sessionId, 'transcript-runs', async () => {
+ return this.withRepairQueue(sessionId, async () => {
const messages = await this.deps.readMessages(sessionId);
Review Comment:
Fixed in `e482026d3`. The converter now walks the legacy rows a page at a time; a turn is only whole once a row of another turn follows it, so the last turn of a page is carried into the next rather than converted from a prefix of itself. Peak memory is one page plus one turn.
No per-Session watermark: a turn is already skipped once its invocation has a terminal, which is what made an interrupted import resumable in the first place. A second progress record would duplicate that.
One consequence worth naming — `openedAt` could no longer come from a count of every turn, since a paged conversion never holds one. It is derived from where the turn starts in the transcript instead, which keeps both properties the count gave it: imported openings still sort ahead of the Session's own runs, and turns keep transcript order. Both are asserted in `converts a legacy transcript larger than one page without reading it whole`. The high-water sequence rides along with each page, so that derivation costs no read of its own.
You were right to measure this against the PR's own acceptance: #4791 says a page must not require loading the full Session, and the base's reader paged over the same table — so this was a regression, not an inherited cost.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener commented on PR #4879:
URL: https://github.com/apache/maka/pull/4879#issuecomment-5560140022
New finding at exact head `816fa172` — **[P2]: the first read of a legacy Session converts its entire history before it can serve a single page.** This one has a producer, which is why I am raising it rather than noting it: the population it affects is exactly the population the converter exists for.
## What the code does
`materializeTranscriptLedger` opens with `const messages = await this.deps.readMessages(sessionId)` — the whole legacy transcript, in one array — then groups all of it and converts sequentially. **The version marker is written only after the full loop completes.**
The reader gates on that. Every durable page goes through `ensureTranscriptLedgerForRead` before it is served (`session-transcript-reader.ts` → `session-manager.ts`), so **the Session cannot return even its first bounded page until the whole conversion has finished.**
## Why it matters
Open a long legacy Session after upgrading — a long transcript, or one carrying large tool results — and peak memory and latency scale with the **entire** history rather than with a page or batch budget. If that exceeds what the process can do, or if it is interrupted, the next read starts the same full load again, because nothing durable records partial progress. The Session stays unavailable for as long as that keeps failing.
**Nothing is lost**: the legacy rows are intact, the deterministic event ids make a re-run idempotent, and a successful run recovers the Session. That is why this is an availability and upgrade-cost regression rather than a correctness one, and why it is [P2] rather than [P1].
**It is worth raising against this PR's own stated requirement.** The design records that upgrade time needs a measured upper bound and crash-resumable execution. The deterministic ids deliver the *idempotency* half — an interrupted import re-derives the same events and the store dedupes them — but not the bounded half: there is no bounded read, no batch, and no recorded progress, so an interruption costs the whole conversion again rather than resuming.
## Direction
Page the legacy rows or turns rather than reading them all; persist a per-Session conversion watermark with each committed batch; and let the reader expose either a completed bounded prefix or an explicit "preparing" state instead of re-reading everything before every page. That keeps the idempotency this already has and adds the bound the design asked for.
## Test gap alongside it
The existing tests cover conversion semantics and restart idempotency, and they cover them well. What I could not find is a test that drives the **production reader** through an oversized legacy transcript, or one that asserts bounded peak work and forward progress across a restart. Without that, the property above has no guard: a later change could make the conversion heavier and nothing would notice.
## Scope of this comment
This supersedes my approval at `82325adc` — that head has been replaced, and under the review standard in use a PR with an open P2 is a comment rather than an approval. Everything I previously verified still stands on this head where the files are unchanged; this is an additional path, not a reversal of those findings.
<details><summary>简体中文</summary>
在 exact head `816fa172` 上的新发现 —— **[P2]:一个旧 Session 的首次读取,会在能够返回任何一页之前先转换它的全部历史。** 这一条**有产出方**,所以我是把它当 finding 提出而不是记一句:**它影响的人群,恰好就是这个转换器为之存在的那批人。**
**代码做了什么**
`materializeTranscriptLedger` 一开头就是 `const messages = await this.deps.readMessages(sessionId)` —— **整份旧 transcript,一个数组**,然后整体分组、顺序转换。**版本标记只有在整个循环完成之后才写入。**
而读取路径以此为闸:每一个持久页在被返回之前都要过 `ensureTranscriptLedgerForRead`(`session-transcript-reader.ts` → `session-manager.ts`),**所以在整次转换结束之前,这个 Session 连第一个有界页都返回不了。**
**为什么要紧**
升级之后打开一个长的旧 Session —— 长 transcript,或带有大块工具结果的 —— **峰值内存与延迟按整份历史增长,而不是按页或批的预算。** 如果它超出进程能承受的范围,或者中途被打断,**下一次读取会重新开始同样的全量加载**,因为没有任何持久记录保存部分进度。**只要这件事持续失败,该 Session 就一直不可用。**
**没有东西丢失**:旧行完好,确定性事件 id 让重跑幂等,一次成功的运行即可恢复该 Session。**这也是它属于可用性与升级成本的回退、而不是正确性问题的原因,以及它是 [P2] 而不是 [P1] 的原因。**
**值得对照本 PR 自己写下的要求来看。** 设计里记着:升级耗时需要一个**可测量的上界**与**可从崩溃续跑**的执行。确定性 id 交付了**幂等**那一半 —— 被中断的导入会重新派生出同样的事件、由 store 去重 —— **但没有交付「有界」那一半:没有有界读取、没有分批、也没有记录进度,所以一次中断的代价是整次转换重来,而不是续跑。**
**方向**
对旧行或旧 turn 分页,而不是一次读完;**每提交一批就持久化一个按 Session 的转换水位**;并让读取方要么暴露一个**已完成的有界前缀**、要么暴露一个明确的「准备中」状态,而不是在每一页之前把所有行重读一遍。**这样既保住它已经具备的幂等,又补上设计所要求的那个上界。**
**与之相伴的测试缺口**
既有测试覆盖了转换语义与重启幂等,而且覆盖得不错。**我没有找到的是:驱动生产读取路径穿过一份超大旧 transcript 的测试,或断言「峰值工作量有界」与「跨重启有前进」的测试。** 没有它,上面那条性质就没有守卫 —— **日后某次改动让转换变重,不会有任何东西发现。**
**本条评论的范围**
它**取代我在 `82325adc` 上的批准** —— 那个 head 已被替换,而按当前采用的评审口径,**存在未解决 P2 的 PR 应当是 comment 而不是 approve**。我此前验证过的内容,在本 head 上文件未变之处依然成立;**这是一条新增路径,不是对那些结论的推翻。**
</details>
---
> **Automated review notice:** This comment was posted by an automated review agent operated by jackwener. It is not an independent human review and does not replace one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "Astro-Han (via GitHub)" <gi...@apache.org>.
Astro-Han commented on code in PR #4879:
URL: https://github.com/apache/maka/pull/4879#discussion_r3945817449
##########
packages/runtime/src/runtime-ledger-repair.ts:
##########
@@ -35,113 +34,173 @@ import type { SessionHeader } from '@maka/core/session';
import type { StoredMessage, TurnRecord } from '@maka/core/session';
import { backfillRuntimeEventsFromStoredMessages } from './runtime-event-backfill.js';
import type { RuntimeEventBackfillOutcome } from './runtime-event-backfill.js';
-import { projectRuntimeEventUserMessage } from './runtime-event-read-model.js';
export interface RuntimeLedgerRepairDeps {
runtimeEventStore: RuntimeEventStore;
- readMessages(sessionId: string): Promise<StoredMessage[]>;
- appendMessage(sessionId: string, message: StoredMessage): Promise<void>;
- newId: () => string;
- now: () => number;
-}
-
-interface RuntimeEventTranscriptProjectionDeps {
- readMessages(sessionId: string): Promise<StoredMessage[]>;
- appendMessage(sessionId: string, message: StoredMessage): Promise<void>;
+ /**
+ * One forward page of the legacy transcript this converter reads; nothing
+ * writes back to it. It is read a page at a time because a Session cannot
+ * serve its first transcript page until this finishes, and a Session's
+ * history is not a bound.
+ */
+ readMessagesAfter(
+ sessionId: string,
+ request: { afterSequence?: number; maxMessages: number; maxStoredBytes: number },
+ ): Promise<{
+ records: readonly { sequence: number; message: StoredMessage }[];
+ highWaterSequence: number | null;
+ }>;
}
-export async function materializeRuntimeEventTranscriptProjection(
- deps: RuntimeEventTranscriptProjectionDeps,
- sessionId: string,
- event: RuntimeEvent,
- knownMessageIds?: Set<string>,
-): Promise<boolean> {
- const message = steeringMessageFromRuntimeEvent(event);
- if (!message) return false;
- const messageIds =
- knownMessageIds ?? new Set((await deps.readMessages(sessionId)).map((item) => item.id));
- if (messageIds.has(message.id)) return false;
- await deps.appendMessage(sessionId, message);
- messageIds.add(message.id);
- return true;
-}
+/** How much of a legacy transcript one conversion page holds. */
+const TRANSCRIPT_CONVERSION_PAGE_MAX_MESSAGES = 256;
+const TRANSCRIPT_CONVERSION_PAGE_MAX_BYTES = 4 * 1024 * 1024;
export class RuntimeLedgerRepair {
private readonly queues = new Map<string, Promise<void>>();
constructor(private readonly deps: RuntimeLedgerRepairDeps) {}
/**
- * Give an imported transcript a runtime spine: one invocation per turn, opened
- * by its own opening fact and closed by its own terminal event.
+ * Give a transcript a runtime spine: one invocation per turn, opened by its
+ * own opening fact and closed by its own terminal event.
*
- * The transcript is the only evidence there is, so a turn it cannot close is
- * refused rather than imported half-formed. Re-running is a no-op: a turn
- * whose invocation already exists is left exactly as it is.
+ * Every event id is derived from the run it belongs to and its position in
+ * that run, so importing the same transcript twice writes the same events and
+ * the store dedupes them. That is what makes an interrupted import resumable:
+ * a turn is skipped once its invocation has ended, and re-derived until then.
*/
async materializeTranscriptLedger(header: SessionHeader): Promise<void> {
const sessionId = header.id;
- return this.withRepairQueue(sessionId, 'transcript-runs', async () => {
- const messages = await this.deps.readMessages(sessionId);
- const ledgerMessages = messages.filter(
- (message) => message.type !== 'user' || message.steeringEventId === undefined,
+ return this.withRepairQueue(sessionId, async () => {
+ // A turn the ledger already owns is not converted again. Its own run is
+ // the authority even when it never ended — a crashed turn is settled by
+ // recovery on that run, and a second, transcript-derived invocation for
+ // the same turn would make the Session read as two. The one exception is
+ // this converter's own run: an interrupted import re-derives it, and the
+ // deterministic ids let the store dedupe what already landed.
+ const inlineInvocations = await this.listInlineInvocations(sessionId);
+ const ownedTurnIds = new Set(
+ inlineInvocations
+ .filter(
+ (invocation) =>
+ invocation.terminalEvent ||
+ invocation.runId !== transcriptRunId(sessionId, invocation.turnId),
+ )
+ .map((invocation) => invocation.turnId),
);
- const openedTurnIds = new Set(
- (await this.listInlineInvocations(sessionId)).map((invocation) => invocation.turnId),
- );
- const messagesByTurn = groupMessagesByTurn(ledgerMessages);
- const turns = deriveTurnRecords(ledgerMessages).filter((turn) =>
- (messagesByTurn.get(turn.turnId) ?? []).some((message) => message.type === 'user'),
- );
- if (turns.length === 0) return;
-
- const firstOpenedAt = Math.max(0, header.createdAt - turns.length);
+ const startedRunIds = new Set(inlineInvocations.map((invocation) => invocation.runId));
- for (const [index, turn] of turns.entries()) {
- if (openedTurnIds.has(turn.turnId)) continue;
- const turnMessages = messagesByTurn.get(turn.turnId) ?? [];
+ for await (const scanned of this.readTurnsInPages(sessionId)) {
+ const turnMessages = scanned.messages;
+ // A turn whose only user row was steering is not a turn of its own: the
+ // steering was said into a Turn some durable Root already owns, so
+ // converting it would stand a second, synthetic run beside that one.
+ if (!turnMessages.some((message) => message.type === 'user')) continue;
+ const [turn] = deriveTurnRecords(turnMessages);
+ if (!turn) continue;
+ if (ownedTurnIds.has(turn.turnId)) continue;
const runId = transcriptRunId(sessionId, turn.turnId);
- const openedAt = firstOpenedAt + index;
+ // Ordered by where the turn starts in the transcript rather than by its
+ // index among all turns: a paged conversion never holds that count, and
+ // both keep every imported opening ahead of the Session's own runs.
+ const openedAt = Math.max(
+ 0,
+ header.createdAt - 1 - (scanned.highWater - scanned.firstSequence),
+ );
const run = { sessionId, runId, turnId: turn.turnId, invocationId: runId };
+ // A build before the ids were derived converted under random ones, so
+ // an interrupted run of its can hold events this build cannot rederive.
+ const started = startedRunIds.has(runId)
+ ? await this.deps.runtimeEventStore.readRuntimeEvents(sessionId, runId)
+ : [];
+ const undeducible = started.filter((event) => !isDerivedTranscriptEventId(runId, event.id));
+ // Its opening is the one such event that can be adopted: the run needs
+ // exactly one, `runtime_events_one_opening_per_invocation` refuses a
+ // second, and which id it landed under changes nothing a reader sees.
+ const adoptedOpening =
+ undeducible.length === 1 && undeducible[0]?.content?.kind === 'invocation_opened';
+ if (undeducible.length > 0 && !adoptedOpening) {
+ // Its converted messages cannot be adopted the same way: rederiving
+ // them would stand a second, deterministic copy of each beside the
+ // one already there, and a Session that disagrees with itself is the
+ // failure this ledger exists to remove. The conversion can neither be
+ // finished nor withdrawn, so it is sealed as the unfinished thing it
+ // is — the legacy rows stay, and no one reads this turn as converted.
+ await this.deps.runtimeEventStore.appendRuntimeEvent(
+ sessionId,
+ runId,
+ abandonedTranscriptTerminalEvent({ run, openedAt }),
+ );
+ continue;
+ }
const events = [
- transcriptOpeningEvent({ header, run, openedAt, newId: this.deps.newId }),
+ ...(adoptedOpening ? [] : [transcriptOpeningEvent({ header, run, openedAt })]),
...backfillRuntimeEventsFromStoredMessages({
run,
outcome: transcriptOutcome(turn, turnMessages, openedAt),
messages: turnMessages,
- modelHistory: 'conversation_text',
- newId: this.deps.newId,
- now: this.deps.now,
+ // Another runtime's tool calls belong to its protocol, not to the
+ // provider this Session will talk to next, so a foreign transcript
+ // converts as the conversation it is. Maka's own history converts
+ // whole: its tool calls are the ones it would replay.
+ modelHistory: header.externalOrigin ? 'conversation_text' : 'full',
+ newId: transcriptEventIds(runId),
+ // The payload must be as repeatable as its id: SQLite dedupes
+ // complete events, including the backfill provenance timestamps.
+ now: () => openedAt,
}).events,
];
- if (!events.some(isTerminalRuntimeEvent)) {
- throw new Error(`Imported transcript Run ${runId} has no terminal RuntimeEvent`);
- }
for (const event of events) {
await this.deps.runtimeEventStore.appendRuntimeEvent(sessionId, runId, event);
}
}
});
}
- async repairSteeringMessagesOnce(sessionId: string): Promise<number> {
- return this.withRepairQueue(sessionId, 'steering-transcript', async () => {
- const messages = await this.deps.readMessages(sessionId);
- const messageIds = new Set(messages.map((message) => message.id));
- const inlineRunIds = new Set(
- (await this.listInlineInvocations(sessionId)).map((invocation) => invocation.runId),
+ /**
+ * The Session's legacy rows, one turn at a time, read a page at a time.
+ *
+ * A turn is only complete once a row of another turn follows it, so the rows
+ * of the page's last turn are carried into the next page rather than
+ * converted early. Peak memory is therefore one page plus one turn — the same
+ * bound the transcript reader keeps, and not the Session's whole history.
+ */
+ private async *readTurnsInPages(
+ sessionId: string,
+ ): AsyncGenerator<{ messages: StoredMessage[]; firstSequence: number; highWater: number }> {
+ let carried: { messages: StoredMessage[]; firstSequence: number } | undefined;
+ let afterSequence: number | undefined;
+ while (true) {
Review Comment:
Verified, and pushed back — recorded as a deliberate ablation rather than an oversight.
Your measurement is right and your distinction between idempotency and resumability is the correct one. The retry does reread from sequence zero.
What I disagree with is the cost of the remedy. A durable per-Session cursor is new persisted state, and on this Session record it is protocol-visible: `SessionHeader` is projected to clients, so the field carries a decode allowlist entry, an epoch step, and — the part that outlives the migration — an invalidation obligation. Someone has to keep answering "when is this cursor wrong?" for a field whose only reader is a one-time conversion.
Against that: the conversion is correct and idempotent under interruption, each retry’s extra cost is a sequential read of legacy rows it skips without writing, and it runs once per Session ever. Repository policy here is to ablate what this change introduces and delete what the acceptance does not need. The acceptance is "convert every legacy transcript row"; it holds without the cursor.
So: not fixed, and written into the PR body under *Ablations kept out* so the next reader sees it was decided rather than missed. If a Session is observed crash-looping this conversion in practice, that is new evidence and the cursor becomes cheap to justify — I would take it then.
One correction to a related claim of mine from the previous round, since it touches this thread: I had said a per-Session watermark would duplicate the terminal check. That was wrong in the direction you are pointing at — the terminal check makes a *turn* skippable, it does not make a *page* skippable. The reason not to add the cursor is the one above, not that one.
##########
packages/runtime-host/src/server/session-transcript-reader.ts:
##########
@@ -34,42 +37,105 @@ import type {
SessionTranscriptPageRequest,
SessionTranscriptRecordScanPage,
SessionTranscriptRecordScanRequest,
+ SessionTranscriptStorageFragment,
SessionTranscriptStoragePage,
+ SessionTurnContribution,
+ SessionTurnContributionPage,
+ SessionTurnLandmark,
+ SessionTurnLandmarkSnapshot,
+ RuntimeTranscriptInvocation,
} from '@maka/storage/execution-stores';
+import { foldTurnContribution } from '@maka/storage/session-message-projection';
import { SESSION_TRANSCRIPT_OVERLAY_MAX_MESSAGES, type TurnSnapshot } from '../protocol/index.js';
const PERMISSION_OUTCOME_READ_CONCURRENCY = 8;
+/** One event can emit content, a permission, usage, and terminal/notice rows. */
+const EVENT_SEQUENCE_STRIDE = 8;
export const ACTIVE_TRANSCRIPT_OVERLAY_MAX_MESSAGES = SESSION_TRANSCRIPT_OVERLAY_MAX_MESSAGES;
export const ACTIVE_TRANSCRIPT_OVERLAY_MAX_BYTES = 16 * 1024 * 1024;
const ACTIVE_TRANSCRIPT_SOURCE_MAX_EVENTS = ACTIVE_TRANSCRIPT_OVERLAY_MAX_MESSAGES * 2;
const ACTIVE_TRANSCRIPT_SCAN_BATCH_MAX_BYTES = 256 * 1024;
+/**
+ * What one durable page may read of a Turn, matching the bound the live
+ * overlay already holds for a run. A Turn past it is refused rather than
+ * half-projected: a prefix of a Turn is not a smaller transcript of it.
+ */
+const DURABLE_TRANSCRIPT_TURN_MAX_EVENTS = ACTIVE_TRANSCRIPT_SOURCE_MAX_EVENTS;
+const DURABLE_TRANSCRIPT_TURN_MAX_BYTES = ACTIVE_TRANSCRIPT_OVERLAY_MAX_BYTES;
+/** Turns per storage round trip: one, so a page loads no Turn it cannot use. */
+const TRANSCRIPT_TURN_SCAN_LIMIT = 1;
+/**
+ * How far back the live-to-durable handoff looks for a message id. The ids come
+ * from assistant streams the subscriber is still watching, so they are in the
+ * newest Turn or the one it continued from; an id that is in neither is treated
+ * as absent rather than searched for down the Session.
+ */
+const TRANSCRIPT_LOOKUP_MAX_TURNS = 2;
export function createSessionTranscriptReader(input: {
stores: ExecutionStoresWriter<'interactive'>;
canonicalPermissionOutcomes: CanonicalPermissionOutcomeReader;
+ /**
+ * Converts a Session whose transcript predates the ledger, before this reader
+ * looks for invocations that only the conversion can create. Omitted only by
Review Comment:
Fixed in `09f0bf278`. You were right that my previous answer was a symptom, not a guard.
I had argued the property was already protected because restoring the second write turns eight other tests red. That is true, but it is incidental: those tests fail for their own reasons, none of them names this property, and a future change could move the failure without anyone noticing what was lost.
The new test drives an ordinary `manager.sendMessage` through `SessionManager` and asserts the absence directly:
```
test(an ordinary send leaves session_messages empty, ...)
assert.deepStrictEqual(await store.readMessages(session.id), []);
// ...and the prompt is still readable, from the ledger alone.
```
Both halves matter — an empty store with an unreadable transcript would pass the first assertion and mean the opposite of what the PR claims.
Verified it discriminates rather than passing vacuously: I re-added a `store.appendMessage` on the live send path in `session-manager.ts` and this test fails.
On the second half of your comment — a reader test that goes *through* `ensureTranscriptLedgerForRead` rather than past it: not addressed here. The reader tests seed the ledger because they are about paging bounds, and threading the launcher through them changes what they measure. `runtime-ledger-repair.test.ts` covers the converter through its own production path, but you are right that no test joins the two, and I am recording that as open rather than claiming it is covered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "Astro-Han (via GitHub)" <gi...@apache.org>.
Astro-Han commented on code in PR #4879:
URL: https://github.com/apache/maka/pull/4879#discussion_r3945816859
##########
packages/runtime/src/runtime-ledger-repair.ts:
##########
@@ -35,113 +34,173 @@ import type { SessionHeader } from '@maka/core/session';
import type { StoredMessage, TurnRecord } from '@maka/core/session';
import { backfillRuntimeEventsFromStoredMessages } from './runtime-event-backfill.js';
import type { RuntimeEventBackfillOutcome } from './runtime-event-backfill.js';
-import { projectRuntimeEventUserMessage } from './runtime-event-read-model.js';
export interface RuntimeLedgerRepairDeps {
runtimeEventStore: RuntimeEventStore;
- readMessages(sessionId: string): Promise<StoredMessage[]>;
- appendMessage(sessionId: string, message: StoredMessage): Promise<void>;
- newId: () => string;
- now: () => number;
-}
-
-interface RuntimeEventTranscriptProjectionDeps {
- readMessages(sessionId: string): Promise<StoredMessage[]>;
- appendMessage(sessionId: string, message: StoredMessage): Promise<void>;
+ /**
+ * One forward page of the legacy transcript this converter reads; nothing
+ * writes back to it. It is read a page at a time because a Session cannot
+ * serve its first transcript page until this finishes, and a Session's
+ * history is not a bound.
+ */
+ readMessagesAfter(
+ sessionId: string,
+ request: { afterSequence?: number; maxMessages: number; maxStoredBytes: number },
+ ): Promise<{
+ records: readonly { sequence: number; message: StoredMessage }[];
+ highWaterSequence: number | null;
+ }>;
}
-export async function materializeRuntimeEventTranscriptProjection(
- deps: RuntimeEventTranscriptProjectionDeps,
- sessionId: string,
- event: RuntimeEvent,
- knownMessageIds?: Set<string>,
-): Promise<boolean> {
- const message = steeringMessageFromRuntimeEvent(event);
- if (!message) return false;
- const messageIds =
- knownMessageIds ?? new Set((await deps.readMessages(sessionId)).map((item) => item.id));
- if (messageIds.has(message.id)) return false;
- await deps.appendMessage(sessionId, message);
- messageIds.add(message.id);
- return true;
-}
+/** How much of a legacy transcript one conversion page holds. */
+const TRANSCRIPT_CONVERSION_PAGE_MAX_MESSAGES = 256;
+const TRANSCRIPT_CONVERSION_PAGE_MAX_BYTES = 4 * 1024 * 1024;
export class RuntimeLedgerRepair {
private readonly queues = new Map<string, Promise<void>>();
constructor(private readonly deps: RuntimeLedgerRepairDeps) {}
/**
- * Give an imported transcript a runtime spine: one invocation per turn, opened
- * by its own opening fact and closed by its own terminal event.
+ * Give a transcript a runtime spine: one invocation per turn, opened by its
+ * own opening fact and closed by its own terminal event.
*
- * The transcript is the only evidence there is, so a turn it cannot close is
- * refused rather than imported half-formed. Re-running is a no-op: a turn
- * whose invocation already exists is left exactly as it is.
+ * Every event id is derived from the run it belongs to and its position in
+ * that run, so importing the same transcript twice writes the same events and
+ * the store dedupes them. That is what makes an interrupted import resumable:
+ * a turn is skipped once its invocation has ended, and re-derived until then.
*/
async materializeTranscriptLedger(header: SessionHeader): Promise<void> {
const sessionId = header.id;
- return this.withRepairQueue(sessionId, 'transcript-runs', async () => {
- const messages = await this.deps.readMessages(sessionId);
- const ledgerMessages = messages.filter(
- (message) => message.type !== 'user' || message.steeringEventId === undefined,
+ return this.withRepairQueue(sessionId, async () => {
+ // A turn the ledger already owns is not converted again. Its own run is
+ // the authority even when it never ended — a crashed turn is settled by
+ // recovery on that run, and a second, transcript-derived invocation for
+ // the same turn would make the Session read as two. The one exception is
+ // this converter's own run: an interrupted import re-derives it, and the
+ // deterministic ids let the store dedupe what already landed.
+ const inlineInvocations = await this.listInlineInvocations(sessionId);
+ const ownedTurnIds = new Set(
+ inlineInvocations
+ .filter(
+ (invocation) =>
+ invocation.terminalEvent ||
+ invocation.runId !== transcriptRunId(sessionId, invocation.turnId),
+ )
+ .map((invocation) => invocation.turnId),
);
- const openedTurnIds = new Set(
- (await this.listInlineInvocations(sessionId)).map((invocation) => invocation.turnId),
- );
- const messagesByTurn = groupMessagesByTurn(ledgerMessages);
- const turns = deriveTurnRecords(ledgerMessages).filter((turn) =>
- (messagesByTurn.get(turn.turnId) ?? []).some((message) => message.type === 'user'),
- );
- if (turns.length === 0) return;
-
- const firstOpenedAt = Math.max(0, header.createdAt - turns.length);
+ const startedRunIds = new Set(inlineInvocations.map((invocation) => invocation.runId));
- for (const [index, turn] of turns.entries()) {
- if (openedTurnIds.has(turn.turnId)) continue;
- const turnMessages = messagesByTurn.get(turn.turnId) ?? [];
+ for await (const scanned of this.readTurnsInPages(sessionId)) {
+ const turnMessages = scanned.messages;
+ // A turn whose only user row was steering is not a turn of its own: the
+ // steering was said into a Turn some durable Root already owns, so
+ // converting it would stand a second, synthetic run beside that one.
+ if (!turnMessages.some((message) => message.type === 'user')) continue;
+ const [turn] = deriveTurnRecords(turnMessages);
+ if (!turn) continue;
+ if (ownedTurnIds.has(turn.turnId)) continue;
const runId = transcriptRunId(sessionId, turn.turnId);
- const openedAt = firstOpenedAt + index;
+ // Ordered by where the turn starts in the transcript rather than by its
+ // index among all turns: a paged conversion never holds that count, and
+ // both keep every imported opening ahead of the Session's own runs.
+ const openedAt = Math.max(
+ 0,
+ header.createdAt - 1 - (scanned.highWater - scanned.firstSequence),
+ );
const run = { sessionId, runId, turnId: turn.turnId, invocationId: runId };
+ // A build before the ids were derived converted under random ones, so
+ // an interrupted run of its can hold events this build cannot rederive.
+ const started = startedRunIds.has(runId)
+ ? await this.deps.runtimeEventStore.readRuntimeEvents(sessionId, runId)
+ : [];
+ const undeducible = started.filter((event) => !isDerivedTranscriptEventId(runId, event.id));
+ // Its opening is the one such event that can be adopted: the run needs
+ // exactly one, `runtime_events_one_opening_per_invocation` refuses a
+ // second, and which id it landed under changes nothing a reader sees.
+ const adoptedOpening =
+ undeducible.length === 1 && undeducible[0]?.content?.kind === 'invocation_opened';
+ if (undeducible.length > 0 && !adoptedOpening) {
+ // Its converted messages cannot be adopted the same way: rederiving
+ // them would stand a second, deterministic copy of each beside the
+ // one already there, and a Session that disagrees with itself is the
+ // failure this ledger exists to remove. The conversion can neither be
+ // finished nor withdrawn, so it is sealed as the unfinished thing it
+ // is — the legacy rows stay, and no one reads this turn as converted.
+ await this.deps.runtimeEventStore.appendRuntimeEvent(
+ sessionId,
+ runId,
+ abandonedTranscriptTerminalEvent({ run, openedAt }),
+ );
+ continue;
+ }
const events = [
- transcriptOpeningEvent({ header, run, openedAt, newId: this.deps.newId }),
+ ...(adoptedOpening ? [] : [transcriptOpeningEvent({ header, run, openedAt })]),
...backfillRuntimeEventsFromStoredMessages({
run,
outcome: transcriptOutcome(turn, turnMessages, openedAt),
messages: turnMessages,
- modelHistory: 'conversation_text',
- newId: this.deps.newId,
- now: this.deps.now,
+ // Another runtime's tool calls belong to its protocol, not to the
+ // provider this Session will talk to next, so a foreign transcript
+ // converts as the conversation it is. Maka's own history converts
+ // whole: its tool calls are the ones it would replay.
+ modelHistory: header.externalOrigin ? 'conversation_text' : 'full',
+ newId: transcriptEventIds(runId),
+ // The payload must be as repeatable as its id: SQLite dedupes
+ // complete events, including the backfill provenance timestamps.
+ now: () => openedAt,
}).events,
];
- if (!events.some(isTerminalRuntimeEvent)) {
- throw new Error(`Imported transcript Run ${runId} has no terminal RuntimeEvent`);
- }
for (const event of events) {
await this.deps.runtimeEventStore.appendRuntimeEvent(sessionId, runId, event);
}
}
});
}
- async repairSteeringMessagesOnce(sessionId: string): Promise<number> {
- return this.withRepairQueue(sessionId, 'steering-transcript', async () => {
- const messages = await this.deps.readMessages(sessionId);
- const messageIds = new Set(messages.map((message) => message.id));
- const inlineRunIds = new Set(
- (await this.listInlineInvocations(sessionId)).map((invocation) => invocation.runId),
+ /**
+ * The Session's legacy rows, one turn at a time, read a page at a time.
+ *
+ * A turn is only complete once a row of another turn follows it, so the rows
+ * of the page's last turn are carried into the next page rather than
+ * converted early. Peak memory is therefore one page plus one turn — the same
+ * bound the transcript reader keeps, and not the Session's whole history.
+ */
+ private async *readTurnsInPages(
+ sessionId: string,
+ ): AsyncGenerator<{ messages: StoredMessage[]; firstSequence: number; highWater: number }> {
+ let carried: { messages: StoredMessage[]; firstSequence: number } | undefined;
+ let afterSequence: number | undefined;
+ while (true) {
+ const page = await this.deps.readMessagesAfter(sessionId, {
+ ...(afterSequence === undefined ? {} : { afterSequence }),
+ maxMessages: TRANSCRIPT_CONVERSION_PAGE_MAX_MESSAGES,
+ maxStoredBytes: TRANSCRIPT_CONVERSION_PAGE_MAX_BYTES,
+ });
+ const highWater = page.highWaterSequence;
+ if (highWater === null) return;
+ const scanned = page.records.filter(
+ ({ message }) => message.type !== 'user' || message.steeringEventId === undefined,
);
- let repaired = 0;
- for (const event of await this.deps.runtimeEventStore.readSessionRuntimeEvents(sessionId)) {
- if (!inlineRunIds.has(event.runId)) continue;
- if (
- await materializeRuntimeEventTranscriptProjection(this.deps, sessionId, event, messageIds)
- ) {
- repaired += 1;
- }
+ const grouped = new Map<string, { messages: StoredMessage[]; firstSequence: number }>();
+ if (carried) grouped.set(turnIdOf(carried.messages[0]) ?? '', carried);
+ for (const { sequence, message } of scanned) {
+ const turnId = turnIdOf(message);
+ if (!turnId) continue;
+ const bucket = grouped.get(turnId);
Review Comment:
Split verdict: one half fixed, one half pushed back.
**The retry inventory — fixed in `69e110eb0`.** You are right that nothing needed the records. The ownership answer is two sets of ids, and the paged scan outlives them, so the records were retained for the whole conversion for no reason. `listInlineInvocations` is now `readLedgerOwnership`, which folds the listing into `Set<string>` and keeps nothing else.
**`listSessionInvocations` itself — not bounded, deliberately.** Making it bounded needs a turn→invocation query the store does not have, because the run that already owns a turn has an arbitrary run id: it cannot be derived from the turn id the way the converter’s own can. Adding that query is a new store API for a read of one small metadata row per turn — the same order as the conversion the loop is about to do anyway, and orders of magnitude below the message pages the paging exists to bound. I would rather not add a store contract to shave a constant off an operation that is already O(turns).
**`carried` — pushed back, with the bound stated.** `carried` holds one turn, not the transcript. It is the same bound the durable reader already accepts: a Turn is the unit a page is projected from (`ACTIVE_TRANSCRIPT_SOURCE_MAX_EVENTS` = 8192 events, `ACTIVE_TRANSCRIPT_OVERLAY_MAX_BYTES` = 16 MiB), so a Turn that does not fit in memory is a Session this reader cannot serve at all — before and after this PR. A conversion that bounded `carried` below that would produce a ledger the reader still could not page. The right place to close that is a Turn-size bound, which is a separate concern from the converter, and I have written the accepted bound into the PR body rather than leaving it implicit.
On the test: agreed that measuring `page.records` does not observe what is held. I did not add an allocation-probing test for `carried`, because the property it would guard is the Turn bound above rather than anything the converter decides.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "Astro-Han (via GitHub)" <gi...@apache.org>.
Astro-Han commented on code in PR #4879:
URL: https://github.com/apache/maka/pull/4879#discussion_r3945819281
##########
packages/runtime/src/runtime-event-read-model.ts:
##########
@@ -228,6 +238,29 @@ export function projectRuntimeEventsToStoredMessages(
};
const messages: StoredMessage[] = [];
+ const context = options.context;
+ if (context) {
+ state.hasRetainedOutput = context.hasRetainedOutput;
+ if (context.contentOrder)
+ state.contentOrderByMessageId.set(context.messageId, [...context.contentOrder]);
+ if (context.toolName && context.toolUseId)
+ state.toolNameByUseId.set(context.toolUseId, context.toolName);
+ const requestEvent = context.permissionRequest;
+ const request = requestEvent?.actions?.permissionRequest;
+ if (request && requestEvent) {
+ state.permissionRequestById.set(request.requestId, {
+ requestId: request.requestId,
+ toolUseId: request.toolUseId,
+ toolName: request.toolName,
Review Comment:
Reversing my earlier answer on this thread: no longer deferred — fixed in `56af740a2`.
I had said the two available end states were "derive one side from the other" or "compare them on a production path", and that both were larger than this PR should absorb. There was a third I did not see: delete the SQL side of the semantics entirely and let it keep only the seek.
That is what happened. `TRANSCRIPT_MESSAGE_KEY_SQL`, `TRANSCRIPT_STORED_ID_SQL`, `TRANSCRIPT_OUTPUT_SHAPE_SQL`, `transcriptSource`, `hasShape`, `flags` and `turn` are gone, along with the `options.context` seam that fed their answers back into the projector, and five of the six expression indexes migration 17 had frozen them into. `runtime-transcript-query.ts` goes 436 → 260 lines and now answers only questions SQL is uniquely good at: the terminal predicate, the invocation joins, the ordinal seek. `projectRuntimeEventsToStoredMessages` is the single definition of what a transcript row is; it returns `sourceEventIds` so a page can attribute each message back to its event, and page sequences are `ordinal * EVENT_SEQUENCE_STRIDE + offset`.
Also gone with it: the `SessionTurnContribution` shape booleans, which were the same duplicate authority on the wire — the projector answers those questions, so the Host no longer sends five derived flags. That is epoch 123.
One correction to my earlier framing, which you had already flagged: I claimed the SQL side "seeks, it does not re-decide what a transcript row is". That was wrong. Four of those expressions did re-decide it, and being expression-indexed made the duplication worse than an ordinary one — a semantic change had to land in the projector, in SQL, and in a migration together.
The cost is stated in the PR body: a page now costs one Turn’s events rather than exactly the rows it serves, bounded by the 8192-event / 16 MiB Turn bound the active overlay already enforced. Sampling a real workspace, the largest Turn was 420 events / 843 KB.
The rework found a real bug that only it could find: `firstSequence` was 8 sequences early on every Turn-index cursor, because `MIN(ordinal)` includes `invocation_opened`, which produces no row.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener commented on code in PR #4879:
URL: https://github.com/apache/maka/pull/4879#discussion_r3950356962
##########
packages/runtime-host/src/server/session-transcript-reader.ts:
##########
@@ -137,12 +196,449 @@ export interface SessionTranscriptReader {
sessionId: string,
request: SessionTranscriptMessageLookupRequest,
): Promise<readonly StoredMessage[]>;
+ readDurableTurnContributions(
+ sessionId: string,
+ throughSequence: number | null,
+ position: number,
+ maxContributions: number,
+ ): Promise<SessionTurnContributionPage>;
+ readDurableTurnLandmarks(
+ sessionId: string,
+ maxLandmarks: number,
+ ): Promise<SessionTurnLandmarkSnapshot>;
readActiveOverlay(
sessionId: string,
rootTurn: TurnSnapshot | null,
): Promise<readonly StoredMessage[]>;
}
+/**
+ * Pages seek immutable Session event ordinals before decoding payloads. One
+ * Turn is projected at a time, so a page costs one Turn rather than the
+ * Session. The low sequence bits distinguish the few rows one event emits.
+ *
+ * What an event becomes is asked only of the read model. Storage selects Turns
+ * by ordinal and hands over their events; it never classifies one.
+ */
+function createDurableLedgerTranscriptReader(input: {
+ stores: ExecutionStoresWriter<'interactive'>;
+ canonicalPermissionOutcomes: CanonicalPermissionOutcomeReader;
+}) {
+ const store = input.stores.runtimeEventStore;
+ const highWater = async (sessionId: string): Promise<number | null> => {
+ const ordinal = await store.readTranscriptHighWater(sessionId);
+ return ordinal === null ? null : ordinal * EVENT_SEQUENCE_STRIDE + EVENT_SEQUENCE_STRIDE - 1;
+ };
+
+ /** One Turn's rows, each at the sequence its own event sits at. */
+ const projectTurn = async (
+ turn: RuntimeTranscriptInvocation,
+ ): Promise<{ sequence: number; message: StoredMessage }[]> => {
+ const events = turn.events.map((entry) => entry.event);
+ const projected = projectRuntimeEventsToStoredMessages(events, {
+ invocations: [turn.invocation],
+ canonicalPermissionOutcomes: await readCanonicalPermissionOutcomes(
+ events,
+ input.canonicalPermissionOutcomes,
+ ),
+ });
+ if (projected.diagnostics.some(isHardRuntimeEventReadModelDiagnostic)) {
+ throw new Error('Durable RuntimeEvent transcript projection is incomplete');
+ }
+ const ordinals = new Map(turn.events.map((entry) => [entry.event.id, entry.ordinal]));
+ const emitted = new Map<number, number>();
+ return projected.messages.map((message, index) => {
+ const ordinal = ordinals.get(projected.sourceEventIds[index]!);
+ if (ordinal === undefined) {
+ throw new Error('Durable transcript message has no source RuntimeEvent');
+ }
+ const offset = emitted.get(ordinal) ?? 0;
+ if (offset >= EVENT_SEQUENCE_STRIDE) {
+ throw new Error('RuntimeEvent exceeds its transcript sequence stride');
+ }
+ emitted.set(ordinal, offset + 1);
+ return { sequence: ordinal * EVENT_SEQUENCE_STRIDE + offset, message };
+ });
+ };
+
+ const readTurns = async (
+ sessionId: string,
+ request: { direction: 'older' | 'newer'; throughOrdinal: number; position: number },
+ ): Promise<RuntimeTranscriptInvocation[]> =>
+ store.readTranscriptInvocations(sessionId, {
+ ...request,
+ limit: TRANSCRIPT_TURN_SCAN_LIMIT,
+ maxEvents: DURABLE_TRANSCRIPT_TURN_MAX_EVENTS,
+ maxBytes: DURABLE_TRANSCRIPT_TURN_MAX_BYTES,
+ });
+
+ const scan = async function* (
+ sessionId: string,
+ request: {
+ direction: 'older' | 'newer';
+ throughSequence?: number | null;
+ position?: number;
+ /** Stops the walk after this many Turns, for a read that may find nothing. */
+ maxTurns?: number;
+ },
+ ): AsyncGenerator<{ sequence: number; message: StoredMessage }> {
+ const throughSequence =
+ request.throughSequence === undefined ? await highWater(sessionId) : request.throughSequence;
+ if (throughSequence === null) return;
+ const position = request.position ?? (request.direction === 'older' ? throughSequence : 0);
+ const throughOrdinal = ordinalOf(throughSequence);
+ let ordinal = ordinalOf(position);
+ let walked = 0;
+ while (ordinal >= 0 && ordinal <= throughOrdinal) {
+ const turns = await readTurns(sessionId, {
+ direction: request.direction,
+ throughOrdinal,
+ position: ordinal,
+ });
+ if (turns.length === 0) return;
+ for (const turn of turns) {
+ if (request.maxTurns !== undefined && walked >= request.maxTurns) return;
+ walked += 1;
+ const records = (await projectTurn(turn)).filter(
+ ({ sequence }) =>
+ sequence <= throughSequence &&
+ (request.direction === 'older' ? sequence <= position : sequence >= position),
+ );
+ if (request.direction === 'older') records.reverse();
+ yield* records;
+ }
+ const edge = turns.at(-1)!;
+ ordinal = request.direction === 'older' ? edge.firstOrdinal - 1 : edge.lastOrdinal + 1;
Review Comment:
[P2] Backward paging still drops the outer Turn when a page runs out inside a nested one.
Selection is not where this sits — the queries handle nesting, and a single sweep large enough to hold the Session returns both Turns in both directions. **Resumption is where it breaks.** The older walk takes the innermost Turn first by opening order, then resumes from that Turn's oldest record sequence, and the outer Turn's later rows are excluded by the `sequence <= position` filter on the next page. They are never revisited.
With the original nested shape (A opens first and ends last, B opens and ends inside it) at one message per page, the older walk returns two rows and loses both of A's. With three levels it loses A and B. At sixty-four messages per page — one sweep — both directions are complete. **That the same ledger is correct in one sweep and short across pages is what locates the defect in resumption rather than in selection.**
**Reachability is bounded by page size, not by shape.** The production pager allows 256 messages or 512 KiB, so a small nested fixture fits in one page and reads correctly. What is required is an inner Turn large enough to fill a page — then the outer Turn is lost on the page after it, at any nesting depth.
**Attribution, stated precisely because it decides whether this belongs here at all:** the commit that rewrote paging is not the cause — its selection is sound in both directions. This resumption logic came in earlier in the same PR. So it is this PR's to answer, but not this commit's failure, and the forward direction it did fix is genuinely fixed: the shape reported against the previous head now returns both Turns, at three nesting levels too.
**Also confirmed while checking this:** unfinished invocations are excluded from durable reads by both directions through the same fact — they have no ending ordinal, so the older filter's `NULL <= position` is false and the forward walk has no row to find. The deleted `settledInline` predicate is not missing; the index and the filter became the same thing, which is the part of this rewrite that removes real complexity rather than moving it.
<details><summary>简体中文</summary>
**[P2] 当一页在嵌套的内层 Turn 中用尽时,向后分页仍会丢掉外层 Turn。**
**问题不在选择** —— 查询本身能处理嵌套,而一次足以装下整个 Session 的扫描,在两个方向上都能返回两个 Turn。**问题在续读。** 向后遍历按 opening 顺序先取到最内层的 Turn,随后从该 Turn 最旧的 record sequence 续读,**而外层 Turn 更靠后的行被下一页的 `sequence <= position` 过滤排除,并且再也不会被回访。**
用原来的嵌套形状(A 最先开、最后终,B 在其中开与终),每页一条消息时,向后遍历返回两行,**A 的两行全丢**。三层嵌套时丢掉 A 和 B。而每页六十四条(即一次扫完)时,**两个方向都是齐的**。**同一份账本「一次扫完是对的、分页则是短的」,这正是把缺陷定位在续读而非选择上的依据。**
**可达性由页大小决定,而不是由形状决定。** 生产分页器允许 256 条或 512 KiB,所以小的嵌套夹具装得进一页、读起来是对的。**真正需要的条件是内层 Turn 大到能填满一页** —— 那么它之后的那一页就会丢掉外层 Turn,**任何嵌套深度都如此**。
**归属要说准,因为它决定这条该不该记在这里:** 重写分页的那个 commit **不是成因** —— 它的选择逻辑在两个方向上都成立。**这段续读逻辑是同一个 PR 里更早引入的。** 所以它归本 PR 回答,**但不是那个 commit 的失败**;而它确实修好的前向方向是真的修好了:**针对上一个 head 报告的那个形状,现在两个 Turn 都能返回,三层嵌套亦然。**
**核这条时一并确认的:** 未终结的 invocation 由**同一个事实**在两个方向上被排除在持久读之外 —— 它们没有 ending ordinal,所以向后那个过滤的 `NULL <= position` 为假,而向前的遍历根本找不到行。**被删掉的 `settledInline` 谓词并不是缺失了;索引与过滤合成了同一件事** —— 这正是本次重写中**真正削减复杂度、而非搬运复杂度**的那一部分。
</details>
---
> **Automated review notice:** This comment was posted by an automated review agent operated by jackwener. It is not an independent human review and does not replace one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener commented on code in PR #4879:
URL: https://github.com/apache/maka/pull/4879#discussion_r3944395450
##########
packages/runtime/src/runtime-ledger-repair.ts:
##########
@@ -72,24 +47,40 @@ export class RuntimeLedgerRepair {
constructor(private readonly deps: RuntimeLedgerRepairDeps) {}
/**
- * Give an imported transcript a runtime spine: one invocation per turn, opened
- * by its own opening fact and closed by its own terminal event.
+ * Give a transcript a runtime spine: one invocation per turn, opened by its
+ * own opening fact and closed by its own terminal event.
*
- * The transcript is the only evidence there is, so a turn it cannot close is
- * refused rather than imported half-formed. Re-running is a no-op: a turn
- * whose invocation already exists is left exactly as it is.
+ * Every event id is derived from the run it belongs to and its position in
+ * that run, so importing the same transcript twice writes the same events and
+ * the store dedupes them. That is what makes an interrupted import resumable:
+ * a turn is skipped once its invocation has ended, and re-derived until then.
*/
async materializeTranscriptLedger(header: SessionHeader): Promise<void> {
const sessionId = header.id;
- return this.withRepairQueue(sessionId, 'transcript-runs', async () => {
+ return this.withRepairQueue(sessionId, async () => {
const messages = await this.deps.readMessages(sessionId);
Review Comment:
**[P2] This reads the whole legacy transcript before the Session can serve a single page.**
`readMessages(sessionId)` returns the entire transcript in one array; the grouping and conversion below then run over all of it, and the version marker is written only after the loop completes. Every durable page goes through `ensureTranscriptLedgerForRead` first, so **the Session cannot return even its first bounded page until the whole conversion has finished.**
The producer is the population this converter exists for: any long legacy Session. Peak memory and latency scale with the full history rather than a page budget, and because nothing durable records partial progress, an interruption starts the same full load again on the next read — so the Session stays unavailable for as long as that keeps failing.
Nothing is lost (legacy rows are intact, deterministic ids make a re-run idempotent), so this is availability and upgrade cost rather than correctness — **[P2], not [P1]**.
Worth measuring against this PR's own requirement: the design records that upgrade needs a measured upper bound and crash-resumable execution. The deterministic ids give the *idempotent* half; **the bounded half is missing** — no bounded read, no batch, no recorded progress.
**Direction:** page the legacy rows or turns, persist a per-Session conversion watermark with each committed batch, and let the reader expose a completed bounded prefix or an explicit preparing state instead of re-reading everything before every page.
<details><summary>简体中文</summary>
**[P2] 这里在该 Session 能返回任何一页之前,先读完了整份旧 transcript。**
`readMessages(sessionId)` 一次返回整份 transcript;下面的分组与转换随之覆盖全部内容,**而版本标记只有在循环结束后才写**。每个持久页都要先过 `ensureTranscriptLedgerForRead`,**所以整次转换完成前,该 Session 连第一个有界页都返回不了。**
**产出方正是这个转换器为之存在的那批人:任何一个长的旧 Session。** 峰值内存与延迟按整份历史增长,**而且因为没有任何持久记录保存部分进度,一次中断会让下一次读取从头再来** —— 只要这件事持续失败,该 Session 就一直不可用。
没有东西丢失(旧行完好、确定性 id 让重跑幂等),所以这是可用性与升级成本问题而非正确性问题 —— **[P2] 而不是 [P1]**。
值得对照本 PR 自己的要求:设计记着升级需要**可测量的上界**与**可从崩溃续跑**。确定性 id 给了**幂等**那一半;**「有界」那一半缺失** —— 没有有界读取、没有分批、没有进度记录。
**方向:** 对旧行或旧 turn 分页;每提交一批持久化一个按 Session 的转换水位;让读取方暴露**已完成的有界前缀**或明确的「准备中」状态,而不是在每一页之前重读全部。
</details>
---
> **Automated review notice:** This comment was posted by an automated review agent operated by jackwener. It is not an independent human review and does not replace one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "Astro-Han (via GitHub)" <gi...@apache.org>.
Astro-Han commented on PR #4879:
URL: https://github.com/apache/maka/pull/4879#issuecomment-5561262183
Head `e482026d3`. Both migration findings fixed, two items deferred with reasons, one pushed back. Detail is in the inline threads; summary here.
**Fixed**
- `6f1528e0f` — a conversion a released build left part-written. Its opening is adopted (a run needs exactly one; the unique index refuses a second); a prefix that already converted messages is sealed as unfinished rather than rederived, because rederiving would stand a second copy of each message beside the one already there. Ablation reproduces the reported `UNIQUE constraint failed`.
- `e482026d3` — the converter reads the legacy rows a page at a time, carrying a page's last turn into the next so no turn converts from a prefix of itself. Peak memory is one page plus one turn. No new watermark: a terminal on the turn's invocation was already the durable progress record.
**Deferred, stated rather than silent**
- The retained context notes a released build wrote transcript-only. Seven kinds, not one. They belong to turns a real run already sealed, and the seal is an interface obligation of every store — reaching them needs an exception to the guard that makes a run's ending final. Two of the seven are derivable in principle; the "one note per send" rule they carry cannot be reconstructed across bounded pages. Hidden from the model, no effect on context or execution, legacy rows intact. A comment at the version gate now says version 1 means a conversion ran, not that `session_messages` is empty.
- The SQL seek beside the JS projection. It exists to satisfy this PR's own bounded-page acceptance; deriving one side from the other is a larger change than this should absorb now.
**Pushed back**
- "Restore the dual write and the suite stays green" — it does not. Reintroduced on the live path: runtime 3123/**8 fail**, control 3132/**0 fail**. And runtime's `SessionStore` no longer has any write method, so the probe would not compile without a cast.
runtime 3135, runtime-host 1726, storage 1110 — 0 failures.
Thanks to @M4n5ter for fixing the three findings from the blind review directly, and to @hqhq1025 — the released-build compatibility case was reachable and neither of the earlier lanes had it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "likun666661 (via GitHub)" <gi...@apache.org>.
likun666661 commented on PR #4879:
URL: https://github.com/apache/maka/pull/4879#issuecomment-5564510242
## Review: single authority is the right simplification, but the read and migration boundaries still need work
以下是本轮代码审查、定向验证，以及关于奥卡姆剃刀和最小正确实现的完整意见。
**版本与验证范围**：完整讨论及下面的动态复现基于 `0d886e9f3a8ac721ffd29859e357d49aa955a610`。发布评论前发现 HEAD 已推进到 `4cddf6f15ec15e066dab0149475458ca19600e76`，已检查两者之间的增量：该提交补回 WorkHub Coordination Session 自有记录的读取，并抽取共用分页逻辑；没有修改下面指出的 `runtime-ledger-repair.ts` 和 `runtime-transcript-query.ts` 关键路径。**未在新 HEAD 重跑测试，也不声称已完成新 HEAD 的全量复审。** 下列代码链接固定到实际审查版本。
**总体结论：认可单一事件账本和单一展示投影的方向，也认可写入侧的实质简化；但建议先修复下面三个已复现的问题，再合并。净删行数不能替代顺序、分页成本与内存边界的证明。**
### 1. [P1] 混合历史转换后，早期消息被排到后期消息之后
[转换器的 openedAt 计算](https://github.com/apache/maka/blob/0d886e9f3a8ac721ffd29859e357d49aa955a610/packages/runtime/src/runtime-ledger-repair.ts#L85-L101) 试图通过更早的时间戳将导入轮次放到已有运行之前；但新读者使用的是 Session event ordinal，而 [insertRuntimeEvent](https://github.com/apache/maka/blob/0d886e9f3a8ac721ffd29859e357d49aa955a610/packages/storage/src/sqlite-runtime-store.ts#L3821-L3837) 仍将转换出的事件追加到 `MAX(ordinal) + 1`。时间戳不能改变这个排序。
复现使用真实 SQLite Session/Runtime stores、当前转换器和当前 durable transcript reader：
1. 一个未标记的旧 Session，早期 `old-turn` 只有 transcript 中的 user、assistant、completed 状态。
2. 后期 `new-turn` 已有 RuntimeEvents，同时也有旧的 transcript 行，原历史顺序是 old → new。
3. 转换器跳过已有真实 invocation 的 new-turn，补齐 old-turn。
4. `readDurableRecords(direction: 'newer')` 返回的用户消息顺序为：
```text
sequence 16: new question
sequence 40: old question
```
这不是消息内容丢失，而是该 PR 所支持的“旧 transcript + 已有 ledger”混合历史的顺序错误。需要迁移持久化顺序，而不只是迁移内容或调整 `openedAt`。如果某类历史不在支持范围内，应在兼容边界明确拒绝，不能成功转换后静默错序。
### 2. [P2] 分页查询的实际工作量仍随整段 Session 历史增长
[RuntimeTranscriptQuery.invocations](https://github.com/apache/maka/blob/0d886e9f3a8ac721ffd29859e357d49aa955a610/packages/storage/src/runtime-transcript-query.ts#L134-L155) 先遍历水位以内的会话事件并 `GROUP BY e.invocation_id`，再通过 `HAVING` 过滤 position，最后排序并 LIMIT。`limit: 1` 限制的是输出 invocation 数，不是查询必须处理的数据量。
使用当前 schema v17 和真实 `RuntimeTranscriptQuery` 在内存 SQLite 上构造历史，每轮 12 个事件、普通文本约 1 KiB，固定只取最后一个 invocation。预热后四次读取均值约为：
| Session 轮次 | 总事件数 | 只取最后一轮的平均耗时 |
| --- | --- | --- |
| 100 | 1,200 | 2.6 ms |
| 1,000 | 12,000 | 25 ms |
| 3,000 | 36,000 | 77 ms |
这些是本机合成数据测量，不是生产延迟。重要的是增长形态与执行计划一致：
```text
SEARCH o USING PRIMARY KEY (session_id=? AND ordinal<?)
...
USE TEMP B-TREE FOR GROUP BY
USE TEMP B-TREE FOR ORDER BY
```
当前 reader 每次取一个 Turn，跨多个 Turn 的页面会重复支付这个全历史成本。因此“只解码一轮”并不等于“只读取一轮的成本”。现有分页测试统计 JS `JSON.parse` 字节数，无法覆盖 SQLite 内部的扫描、JSON 提取、分组和排序。
建议先通过位置索引定位候选 invocation，再读取该轮事件；不要为了保留单一 projector，就接受每次分页全历史聚合。测试应覆盖实际存储查询工作量，至少包含不同历史长度下相同尾页请求的查询计划和增长趋势，而不仅是响应大小或 JS 解码量。
### 3. [P2] 16 MiB / 8192 events 上限在整轮载入之后才检查
[RuntimeTranscriptQuery.events](https://github.com/apache/maka/blob/0d886e9f3a8ac721ffd29859e357d49aa955a610/packages/storage/src/runtime-transcript-query.ts#L210-L234) 先 `.all(invocationId)` 取回该 invocation 的全部 `payload_json`，然后检查行数、累计字节并抛错。
真实 SQLite 探针构造一轮 10 个约 2 MiB 文本事件，加 opening/terminal；调用设置 `maxBytes = 16 * 1024 * 1024`。在拒绝前，`.all()` 已返回：
```text
allowedBytes: 16,777,216
materializedBytesBeforeRefusal: 20,973,944
```
这证明当前实现是“载入后发现超限”，不是有界载入。更大的合法或导入轮次可能在保护检查执行前就造成大量分配。需要让预算在读取阶段生效；批量上限、单条 payload 的大小预检和累计预算都应覆盖，不能只把 `.all()` 改成一个仍会先物化任意大单条记录的循环。另外字节预算应以实际编码字节计，而不是 JS 字符串长度。
### 4. 已确认但暂不升级为权限问题的观察：catalog projection 的补写条件不完整
[recordInitialRuntimeEvent](https://github.com/apache/maka/blob/0d886e9f3a8ac721ffd29859e357d49aa955a610/packages/runtime/src/agent-run.ts#L737-L751) 在 prompt 已落账本后才提交 catalog projection，并在后者成功之前清除 `initialRuntimeEventPending`。如果在两次提交之间退出，或 catalog 提交失败，就可能出现 prompt 已存在但连接锁/预览未更新的状态。
[hosted recovery](https://github.com/apache/maka/blob/0d886e9f3a8ac721ffd29859e357d49aa955a610/packages/runtime-host/src/server/hosted-execution-recovery.ts#L178-L188) 只有在 prompt 缺失、调用 `recordAdmittedUserMessage` 时才补 catalog。以已有 prompt、`connectionLocked: false` 的状态调用真实 recovery 函数并替换依赖为测试桩，观测到 catalog projection 调用次数为 0。
因此“prompt 是否需要补写”与“必须可靠的派生状态是否已经提交”不能绑定在同一个分支。建议恢复时分别判断，幂等补齐必要状态；普通侧栏预览允许暂时陈旧即可，不必为它建立复杂恢复协议。
**影响限定**：本轮确认的是字段一致性/恢复遗漏，没有发现当前配置入口依据该字段拒绝切换的充分证据，不将其描述为已证实的权限绕过或安全漏洞。
### 5. 这个 PR 解决的问题，以及为什么主方向成立
核心问题不是重复占用磁盘，而是同一个普通执行事实同时具有两套独立持久化表示：RuntimeEvent 和 `session_messages`。运行中的展示和完成后的历史读取依赖不同表示，崩溃、重试、重连都要求它们保持等价，因此系统又需要比较、修复、补写和回退逻辑。
正确的收敛是：
```text
执行 → RuntimeEvent → 一个展示 projector → StoredMessage DTO
```
本 PR 在写入侧完成了有意义的减法：
- `AgentRun`、`AiSdkTurn`、`ToolRuntime` 不再写第二份普通 transcript。
- 活跃展示与历史读取共享执行事实来源和投影定义。
- 旧数据通过一个单向转换器进入账本，而不是永久保留冻结前缀或双读回退。
- 委派关联改读已有 admission proof，恢复改读执行事实，catalog 副作用变成显式接口。
- 删除重复的状态推断、比较/修复逻辑以及 SQL 中另一套消息语义。
`StoredMessage` 作为 DTO 不必删除；WorkHub 自己拥有的协调事实也不必为了形式统一硬塞进普通运行账本。最新提交为 Coordination Session 接回其自有记录，属于保留不同领域事实的必要读取，不应与“同一个普通执行事实的双重权威”混为一谈。该新增读取实现本身未在本轮重跑验证。
### 6. 奥卡姆剃刀：问题化简正确，不等于当前代码已经是最小正确实现
认可从“怎样维护两份记录一致”转向“为什么需要两份独立记录”。真正应该减少的是需要维护的事实定义和不变量数量，不是单纯减少文件或行数。
但以下边界不能作为复杂度被一起删掉：
- 历史先后顺序。
- 实际存储读取、解码、内存占用的上界。
- 崩溃后关键派生状态的恢复。
- 单向迁移的中断恢复与明确兼容边界。
“真实样本最大只有 420 events / 843 KB”可以帮助评估常见负载，但不能证明读取上界，也不能替代超大单轮/单条记录的测试。
PR 已披露七类旧 context note 在切换后不可见，这是一项需要明确接受的兼容代价，不是从“单一权威”自动推导出的正确性。本评论不将已披露的 note 损失重新包装成新发现，但建议保留迁移不可逆、旧版本降级不受支持以及实际损失范围的清晰说明。
### 7. 建议的最小正确实现
**保留主线：单一事件账本 + 单一展示投影 + 必要的位置索引 + 一次性幂等迁移。** 不建议推翻 PR，也不建议恢复旧 transcript fallback。
**写入**：执行组件只提交 RuntimeEvent。展示/预览属于派生结果，可重建，不反向成为执行恢复的第二套事实。
**读取**：
```text
cursor → 索引定位 invocation → 有界读取该轮事件 → 统一投影 → 页面
```
优先复用已有位置索引；如果现有结构无法直接定位，增加最薄的一层 invocation 位置记录，只回答“这轮在哪里、是否已结束”，并与事件写入在同一 Runtime DB 事务中维护。不要把消息内容、可见性、thinking 附着等展示规则重新复制进 SQL。索引必须能由账本重建，读取过程必须有实际生效的预算。
**整轮投影的前提必须写清楚**：只有明确接受单轮硬上限时，它才是足够简单的方案。如果要求过去可以打开的超大历史轮次继续可读，就需要分段投影/必要依赖定位，不能把“现在超限拒绝”说成等价行为。是否支持这种历史是产品兼容性决策，不应通过内部常量暗中决定。
**迁移**：只保留一个转换器。补齐事件之后，依据旧 transcript 的持久化位置和已有事件顺序，幂等建立统一 ordinal 顺序，再校验投影并完成切换。迁移期间阻止该 Session 的新执行和对外发布半成品顺序；完成后 ordinal 不再改变。任一步中断重跑必须得到相同结果。不要把这个要求误解为新增跨数据库事务：当前 Session metadata 与 Runtime 不是一个可直接共同提交的本地事务，需要明确提交顺序和幂等重试边界。
**恢复**：补 prompt 和补关键派生状态分别判断；prompt 已存在不意味着 catalog 提交已经完成。普通预览可以 fail-open。无需为此重建 transcript 双写或增加通用“待补写消息”系统。
**暂不增加**：完整消息物化表、永久旧数据回退、SQL/TS 双投影、通用迁移框架，以及仅为降低一次性转换重试 I/O 而新增的复杂持久化游标。转换目前会等待全历史、重试会重新扫描，这是一次性代价，是否优化应由测量决定；但不能把单轮无限保留当作已经有了内存上界。
### 8. 验证与补测建议
本轮在 `0d886e9f3` 上通过了 183 项现有定向测试：
- `runtime-ledger-repair.test`、`runtime-event-read-model.test`、`session-manager-terminal-ledger.test`：121 项。
- `session-transcript-reader.test`、`session-catalog-coordinator.test`、`workhub-message-assignment.test`：62 项。
另运行了上面的混合历史、真实 SQL 查询成本/计划、超限前物化量探针，以及使用真实 recovery 函数的依赖桩测试。没有修改业务代码。
Core/Storage/MCP 构建通过；Runtime/Runtime Host 完整构建受到本机复用依赖缺失（Slack、systeminformation）和代理库类型版本不匹配阻塞，不把这些环境错误归因于本 PR。未运行桌面 E2E、完整工作区测试或生产数据探针。
建议新增能直接区分正确/错误实现的回归测试：
1. 纯旧、纯新、混合历史，转换及中断重跑后顺序一致。
2. 不同 Session 历史长度下，同一尾页请求不进行全历史分组。
3. 超事件数、超字节数和超大单条 payload，在超过预算前停止物化。
4. prompt 提交后、catalog 提交前中断，恢复补齐必要状态但不重复 prompt。
5. 保留跨终态、重连、重启的消息身份与 cursor 测试；实际旧 Session 打开以及 WorkHub Coordination 时间线仍应做桌面验证。
最终建议：**保留写入侧的减法，修复迁移顺序，补上真正有界的索引读取，并拆开恢复中的错误条件依赖。目标是最少的、但完整成立的不变量，而不是最少的代码行。**
---
本评论由 Codex 辅助完成代码阅读和本地验证，并按维护者明确要求整理发布；不代表独立人工复审或完整 CI/E2E 通过。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "Astro-Han (via GitHub)" <gi...@apache.org>.
Astro-Han commented on PR #4879:
URL: https://github.com/apache/maka/pull/4879#issuecomment-5566768132
@likun666661 **Correction: I was wrong to call finding 1 unreachable. It stands.**
I said no supported path leaves a Session with legacy rows for an early turn and ledger events for a later one, and offered to fix it if anyone showed one. An ordinary upgrade does it:
1. On v0.1.8, import a legacy JSONL Session — `createImportedSession` writes `session_messages` only, never the ledger.
2. Still on that build, send a message. The double write of the day gives that turn ledger events, so it takes Session ordinals *first*.
3. Upgrade. Conversion assigns the imported turns `MAX(ordinal) + 1`, above the later send. The reader orders by ordinal, so the newer question renders first.
My three counter-arguments were true and beside the point: they describe what the *current* build can write, and this state is left behind by a *previous* release. I checked only the build in front of me. Verified, and I'm fixing it here rather than deferring.
**Separately, my own fix in `86f1d7608` broke the Session-list preview, fixed in `41ae92cee`.**
I argued the early return on `run.terminalEvent` kept recovery's unconditional projection away from Turns with a newer preview. That holds only for sealed Turns — steering commits its projection on a *live* run (`agent-run.ts:651`), so prompt → steering → crash → recovery replays the older prompt preview over the newer one. `updateCatalogProjectionSync` made `lastMessageAt` monotonic but left the preview an unconditional replace; it now refuses to move backwards for the same reason the timestamp does. Test `a replayed older message latches the connection without moving the preview back` fails on the old behaviour. The connection lock still latches — that's the part recovery owes.
<details>
<summary>简体中文</summary>
**更正：我判 finding 1「不可达」是错的，这条成立。**
我说过没有受支持的路径能造出「同一 Session 里旧行属早期 turn、账本事件属后期 turn」，谁给出路径我就改。一次普通升级就够：
1. 在 v0.1.8 上导入 legacy JSONL——`createImportedSession` 只写 `session_messages`，不写账本；
2. 仍在该版本正常发一条消息——当时的双写让它拿到账本事件，于是**它先占了 Session ordinal**；
3. 升级。转换给导入的旧 turn 分配 `MAX(ordinal) + 1`，落在那次 send 之上。reader 按 ordinal 排序，新提问排在旧提问前面。
我那三条反驳本身都成立但跑题：它们说的是**当前 build** 能写出什么，而这个状态是**上一个发布版本**留下的。我只检查了眼前的 build。成立，本 PR 内修，不延后。
**另外，`86f1d7608` 里我自己的修法弄坏了会话列表预览，已在 `41ae92cee` 修复。**
我当时认为 `run.terminalEvent` 的提前返回能挡住已有更新预览的 Turn。这只对已封存的 Turn 成立——steering 会在**活着的** run 上提交投影（`agent-run.ts:651`），于是「prompt → steering → 崩溃 → 恢复」会用较旧的 prompt 预览盖掉更新的那条。`updateCatalogProjectionSync` 让 `lastMessageAt` 单调，却把预览留成无条件覆盖；现在预览按与时间戳相同的理由拒绝倒退。回归测试 `a replayed older message latches the connection without moving the preview back` 在旧行为下失败。连接闩仍照常上锁——那才是恢复真正欠的。
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "hqhq1025 (via GitHub)" <gi...@apache.org>.
hqhq1025 commented on code in PR #4879:
URL: https://github.com/apache/maka/pull/4879#discussion_r3944610071
##########
packages/runtime-host/src/server/session-transcript-reader.ts:
##########
@@ -47,29 +59,67 @@ const ACTIVE_TRANSCRIPT_SCAN_BATCH_MAX_BYTES = 256 * 1024;
export function createSessionTranscriptReader(input: {
stores: ExecutionStoresWriter<'interactive'>;
canonicalPermissionOutcomes: CanonicalPermissionOutcomeReader;
+ /**
+ * Converts a Session whose transcript predates the ledger, before this reader
+ * looks for invocations that only the conversion can create. Omitted only by
+ * tests that seed the ledger themselves.
+ */
+ ensureTranscriptLedger?: (sessionId: string) => Promise<void>;
}): SessionTranscriptReader {
+ const durable = createDurableLedgerTranscriptReader(input);
+ const prepared = async (sessionId: string): Promise<void> => {
+ await input.ensureTranscriptLedger?.(sessionId);
+ };
return {
- readDurableHighWater: (sessionId) =>
- input.stores.sessionStore.readTranscriptHighWaterSnapshot(sessionId),
- readDurablePage: (sessionId, request) =>
- input.stores.sessionStore.readTranscriptPageSnapshot(sessionId, request),
- readDurableRecords: (sessionId, request) =>
- input.stores.sessionStore.readTranscriptRecordsSnapshot(sessionId, request),
- readDurableMessagesById: (sessionId, request) =>
- input.stores.sessionStore.readTranscriptMessagesSnapshot(sessionId, request),
+ readDurableHighWater: async (sessionId) => {
+ await prepared(sessionId);
+ return durable.readHighWater(sessionId);
+ },
+ readDurablePage: async (sessionId, request) => {
+ await prepared(sessionId);
+ return durable.readPage(sessionId, request);
Review Comment:
[P1] Migrate retained runtime warnings before switching existing Sessions to ledger-only reads
Released v0.2.0-dev.22.20260905 writes `context_compaction_failed_open` directly to the Session transcript in `runtime-kernel.ts:1096–1104`, without a corresponding RuntimeEvent. This is not one of the retired session-level kinds: the current `core/session.ts:1146` explicitly retains it as a user-visible runtime note. A normal send already sets `transcriptLedgerVersion: 1`, so `SessionManager.ensureTranscriptLedger` skips conversion for these existing Sessions. This new ledger-only page consequently omits their stored warning.
Reproduced by running the released SessionManager/RuntimeKernel against real SQLite stores: send a normal turn, then compact with the backend returning `failed/write_failed`. The production kernel constructs and persists the warning; the compaction invocation is terminal and the header is version 1. The old public durable reader returns that warning. After opening the same databases with this head and calling the new reader through `ensureTranscriptLedgerForRead`, the identical warning-preservation assertion fails, although the original Session row still exists. The released producer files are identical to the PR base. No interrupted write or malformed fixture is needed.
Please add an explicit upgrade path for still-supported runtime notes before treating the old version-1 marker as evidence that every visible historical fact is in RuntimeEvents. Merely filtering the intentionally retired note kinds does not address this case.
> Automated review notice: This comment was posted by an automated review agent operated by hqhq1025. It is not an independent human review and does not replace one.
##########
packages/runtime/src/runtime-ledger-repair.ts:
##########
@@ -98,64 +89,42 @@ export class RuntimeLedgerRepair {
const firstOpenedAt = Math.max(0, header.createdAt - turns.length);
for (const [index, turn] of turns.entries()) {
- if (openedTurnIds.has(turn.turnId)) continue;
+ if (ownedTurnIds.has(turn.turnId)) continue;
const turnMessages = messagesByTurn.get(turn.turnId) ?? [];
const runId = transcriptRunId(sessionId, turn.turnId);
const openedAt = firstOpenedAt + index;
const run = { sessionId, runId, turnId: turn.turnId, invocationId: runId };
const events = [
- transcriptOpeningEvent({ header, run, openedAt, newId: this.deps.newId }),
+ transcriptOpeningEvent({ header, run, openedAt }),
Review Comment:
[P2] Recognize openings persisted by the released random-ID converter before retrying
In v0.2.0-dev.22.20260905, this converter uses the same deterministic transcript run ID but generates each event ID with `newId()` and appends events individually (`runtime-ledger-repair.ts:100–120`). An interruption after its opening commits leaves a valid, nonterminal converter-owned invocation and no completed transcript-version marker. This head deliberately retries that invocation, but first appends the new `${runId}-opened` ID. It does not match the released opening's random ID, so SQLite rejects the second opening with `UNIQUE constraint failed: runtime_events.invocation_id`.
I ran the actual released converter against SQLite, interrupted immediately before its second append, then opened the Session through the current public durable reader and `SessionManager.ensureTranscriptLedgerForRead`. The old reader returns the three stored user/assistant/terminal records; the new page request throws the unique-key error, leaves only the old opening in the ledger, and fails again on retry. A fresh current-version conversion and its retry both pass.
The same preservation test changes from failing to passing when a scratch production-only counterfactual retains the already-persisted opening instead of appending it again. That establishes this ID compatibility failure independently of the separate unbounded-conversion issue; it is not a complete migration solution for longer legacy prefixes. Please handle released random-ID prefixes explicitly without discarding their immutable facts or marking a failed conversion ready.
> Automated review notice: This comment was posted by an automated review agent operated by hqhq1025. It is not an independent human review and does not replace one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "hqhq1025 (via GitHub)" <gi...@apache.org>.
hqhq1025 commented on code in PR #4879:
URL: https://github.com/apache/maka/pull/4879#discussion_r3945712578
##########
packages/runtime/src/runtime-ledger-repair.ts:
##########
@@ -35,113 +34,173 @@ import type { SessionHeader } from '@maka/core/session';
import type { StoredMessage, TurnRecord } from '@maka/core/session';
import { backfillRuntimeEventsFromStoredMessages } from './runtime-event-backfill.js';
import type { RuntimeEventBackfillOutcome } from './runtime-event-backfill.js';
-import { projectRuntimeEventUserMessage } from './runtime-event-read-model.js';
export interface RuntimeLedgerRepairDeps {
runtimeEventStore: RuntimeEventStore;
- readMessages(sessionId: string): Promise<StoredMessage[]>;
- appendMessage(sessionId: string, message: StoredMessage): Promise<void>;
- newId: () => string;
- now: () => number;
-}
-
-interface RuntimeEventTranscriptProjectionDeps {
- readMessages(sessionId: string): Promise<StoredMessage[]>;
- appendMessage(sessionId: string, message: StoredMessage): Promise<void>;
+ /**
+ * One forward page of the legacy transcript this converter reads; nothing
+ * writes back to it. It is read a page at a time because a Session cannot
+ * serve its first transcript page until this finishes, and a Session's
+ * history is not a bound.
+ */
+ readMessagesAfter(
+ sessionId: string,
+ request: { afterSequence?: number; maxMessages: number; maxStoredBytes: number },
+ ): Promise<{
+ records: readonly { sequence: number; message: StoredMessage }[];
+ highWaterSequence: number | null;
+ }>;
}
-export async function materializeRuntimeEventTranscriptProjection(
- deps: RuntimeEventTranscriptProjectionDeps,
- sessionId: string,
- event: RuntimeEvent,
- knownMessageIds?: Set<string>,
-): Promise<boolean> {
- const message = steeringMessageFromRuntimeEvent(event);
- if (!message) return false;
- const messageIds =
- knownMessageIds ?? new Set((await deps.readMessages(sessionId)).map((item) => item.id));
- if (messageIds.has(message.id)) return false;
- await deps.appendMessage(sessionId, message);
- messageIds.add(message.id);
- return true;
-}
+/** How much of a legacy transcript one conversion page holds. */
+const TRANSCRIPT_CONVERSION_PAGE_MAX_MESSAGES = 256;
+const TRANSCRIPT_CONVERSION_PAGE_MAX_BYTES = 4 * 1024 * 1024;
export class RuntimeLedgerRepair {
private readonly queues = new Map<string, Promise<void>>();
constructor(private readonly deps: RuntimeLedgerRepairDeps) {}
/**
- * Give an imported transcript a runtime spine: one invocation per turn, opened
- * by its own opening fact and closed by its own terminal event.
+ * Give a transcript a runtime spine: one invocation per turn, opened by its
+ * own opening fact and closed by its own terminal event.
*
- * The transcript is the only evidence there is, so a turn it cannot close is
- * refused rather than imported half-formed. Re-running is a no-op: a turn
- * whose invocation already exists is left exactly as it is.
+ * Every event id is derived from the run it belongs to and its position in
+ * that run, so importing the same transcript twice writes the same events and
+ * the store dedupes them. That is what makes an interrupted import resumable:
+ * a turn is skipped once its invocation has ended, and re-derived until then.
*/
async materializeTranscriptLedger(header: SessionHeader): Promise<void> {
const sessionId = header.id;
- return this.withRepairQueue(sessionId, 'transcript-runs', async () => {
- const messages = await this.deps.readMessages(sessionId);
- const ledgerMessages = messages.filter(
- (message) => message.type !== 'user' || message.steeringEventId === undefined,
+ return this.withRepairQueue(sessionId, async () => {
+ // A turn the ledger already owns is not converted again. Its own run is
+ // the authority even when it never ended — a crashed turn is settled by
+ // recovery on that run, and a second, transcript-derived invocation for
+ // the same turn would make the Session read as two. The one exception is
+ // this converter's own run: an interrupted import re-derives it, and the
+ // deterministic ids let the store dedupe what already landed.
+ const inlineInvocations = await this.listInlineInvocations(sessionId);
+ const ownedTurnIds = new Set(
+ inlineInvocations
+ .filter(
+ (invocation) =>
+ invocation.terminalEvent ||
+ invocation.runId !== transcriptRunId(sessionId, invocation.turnId),
+ )
+ .map((invocation) => invocation.turnId),
);
- const openedTurnIds = new Set(
- (await this.listInlineInvocations(sessionId)).map((invocation) => invocation.turnId),
- );
- const messagesByTurn = groupMessagesByTurn(ledgerMessages);
- const turns = deriveTurnRecords(ledgerMessages).filter((turn) =>
- (messagesByTurn.get(turn.turnId) ?? []).some((message) => message.type === 'user'),
- );
- if (turns.length === 0) return;
-
- const firstOpenedAt = Math.max(0, header.createdAt - turns.length);
+ const startedRunIds = new Set(inlineInvocations.map((invocation) => invocation.runId));
- for (const [index, turn] of turns.entries()) {
- if (openedTurnIds.has(turn.turnId)) continue;
- const turnMessages = messagesByTurn.get(turn.turnId) ?? [];
+ for await (const scanned of this.readTurnsInPages(sessionId)) {
+ const turnMessages = scanned.messages;
+ // A turn whose only user row was steering is not a turn of its own: the
+ // steering was said into a Turn some durable Root already owns, so
+ // converting it would stand a second, synthetic run beside that one.
+ if (!turnMessages.some((message) => message.type === 'user')) continue;
+ const [turn] = deriveTurnRecords(turnMessages);
+ if (!turn) continue;
+ if (ownedTurnIds.has(turn.turnId)) continue;
const runId = transcriptRunId(sessionId, turn.turnId);
- const openedAt = firstOpenedAt + index;
+ // Ordered by where the turn starts in the transcript rather than by its
+ // index among all turns: a paged conversion never holds that count, and
+ // both keep every imported opening ahead of the Session's own runs.
+ const openedAt = Math.max(
+ 0,
+ header.createdAt - 1 - (scanned.highWater - scanned.firstSequence),
+ );
const run = { sessionId, runId, turnId: turn.turnId, invocationId: runId };
+ // A build before the ids were derived converted under random ones, so
+ // an interrupted run of its can hold events this build cannot rederive.
+ const started = startedRunIds.has(runId)
+ ? await this.deps.runtimeEventStore.readRuntimeEvents(sessionId, runId)
+ : [];
+ const undeducible = started.filter((event) => !isDerivedTranscriptEventId(runId, event.id));
+ // Its opening is the one such event that can be adopted: the run needs
+ // exactly one, `runtime_events_one_opening_per_invocation` refuses a
+ // second, and which id it landed under changes nothing a reader sees.
+ const adoptedOpening =
+ undeducible.length === 1 && undeducible[0]?.content?.kind === 'invocation_opened';
+ if (undeducible.length > 0 && !adoptedOpening) {
+ // Its converted messages cannot be adopted the same way: rederiving
+ // them would stand a second, deterministic copy of each beside the
+ // one already there, and a Session that disagrees with itself is the
+ // failure this ledger exists to remove. The conversion can neither be
+ // finished nor withdrawn, so it is sealed as the unfinished thing it
+ // is — the legacy rows stay, and no one reads this turn as converted.
+ await this.deps.runtimeEventStore.appendRuntimeEvent(
+ sessionId,
+ runId,
+ abandonedTranscriptTerminalEvent({ run, openedAt }),
+ );
+ continue;
Review Comment:
[P1] Preserve the rest of a released conversion before sealing its prefix
The opening-only retry now succeeds, but this branch loses conversation text when the released converter committed any message after the opening. I ran the actual v0.2.0-dev.22.20260905 converter against SQLite and interrupted it before its third append, leaving its random-ID opening and user event. The old durable reader still returned the stored question, complete assistant answer, and completed turn state. Reading the same databases on this head through `ensureTranscriptLedgerForRead` returns only the question and a new `failed/missing_terminal_event` state; the answer disappears, and the Session is marked version 1. The original rows remain on disk but are no longer read.
This is conversation loss, beyond the seven context-note kinds discussed separately and contrary to the PR's stated conversation-preservation requirement. The new test only checks that one user text remains; it does not check that the original answer survives. The same preservation assertion passes in a narrow production-only counterfactual that adopts the old opening, recognizes the already-written user by its source message ID/content, and appends the remaining answer and terminal without deleting existing events or writing to a sealed run. That is a root-cause probe, not a complete migration patch.
Please retain or finish the remaining historical conversation before treating this Session as ledger-authoritative. Sealing the partial conversion avoids the former exception but does not make its complete legacy turn safe to stop reading.
> Automated review notice: This comment was posted by an automated review agent operated by hqhq1025. It is not an independent human review and does not replace one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "Astro-Han (via GitHub)" <gi...@apache.org>.
Astro-Han commented on code in PR #4879:
URL: https://github.com/apache/maka/pull/4879#discussion_r3944872889
##########
packages/runtime-host/src/server/session-transcript-reader.ts:
##########
@@ -47,29 +59,67 @@ const ACTIVE_TRANSCRIPT_SCAN_BATCH_MAX_BYTES = 256 * 1024;
export function createSessionTranscriptReader(input: {
stores: ExecutionStoresWriter<'interactive'>;
canonicalPermissionOutcomes: CanonicalPermissionOutcomeReader;
+ /**
+ * Converts a Session whose transcript predates the ledger, before this reader
+ * looks for invocations that only the conversion can create. Omitted only by
+ * tests that seed the ledger themselves.
+ */
+ ensureTranscriptLedger?: (sessionId: string) => Promise<void>;
}): SessionTranscriptReader {
+ const durable = createDurableLedgerTranscriptReader(input);
+ const prepared = async (sessionId: string): Promise<void> => {
Review Comment:
Pushing back on the evidence: I ran the ablation and the suite does not stay green.
I put the second write back on the live path in `runtime-kernel.ts`, at the ordinary turn start. Same file, same flags:
| | result |
|---|---|
| with the second write | runtime 3123 pass / **8 fail**; `session-manager-terminal-ledger` alone 42 tests, **7 fail** (2 of them 120s timeouts; with no `--test-timeout` it hangs forever) |
| control, probe removed | runtime 3132 / **0 fail**; that file 42 / **0 fail** |
There is a harder guard than a test, and it showed up before I could run one: runtime's `SessionStore` interface has no write method at all any more — only `readMessages`. My first probe was `this.deps.store.appendMessage(...)` and it failed to compile (`TS2339`); I had to cast through `as unknown as` to run the ablation. So reintroducing the dual write is not a change that slips in — it requires widening an interface, which is visible in review.
Two honest limits. My first two probes were unfaithful (one threw and aborted the turn, one put `.catch` after `?.()` and threw `TypeError`); the numbers above are from the third, wrapped in try/catch and not touching the deterministic clock. And I did not finish tracing *why* the remaining failures happen — some failing tests construct `AgentRun` directly while the probe sits in the kernel, so that link is not closed. What I am disputing is the specific claim "the suite stays green", not the general point that no test names the absence directly.
Your two sub-points I accept, and the first one turned out to matter: the reader tests standing in for `ensureTranscriptLedger` is exactly the seam that let hqhq1025's finding through. `compareRuntimeReadModelMessages` being test-only is the same item as the thread above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener commented on code in PR #4879:
URL: https://github.com/apache/maka/pull/4879#discussion_r3949121492
##########
packages/runtime-host/src/server/session-transcript-reader.ts:
##########
@@ -137,12 +196,449 @@ export interface SessionTranscriptReader {
sessionId: string,
request: SessionTranscriptMessageLookupRequest,
): Promise<readonly StoredMessage[]>;
+ readDurableTurnContributions(
+ sessionId: string,
+ throughSequence: number | null,
+ position: number,
+ maxContributions: number,
+ ): Promise<SessionTurnContributionPage>;
+ readDurableTurnLandmarks(
+ sessionId: string,
+ maxLandmarks: number,
+ ): Promise<SessionTurnLandmarkSnapshot>;
readActiveOverlay(
sessionId: string,
rootTurn: TurnSnapshot | null,
): Promise<readonly StoredMessage[]>;
}
+/**
+ * Pages seek immutable Session event ordinals before decoding payloads. One
+ * Turn is projected at a time, so a page costs one Turn rather than the
+ * Session. The low sequence bits distinguish the few rows one event emits.
+ *
+ * What an event becomes is asked only of the read model. Storage selects Turns
+ * by ordinal and hands over their events; it never classifies one.
+ */
+function createDurableLedgerTranscriptReader(input: {
+ stores: ExecutionStoresWriter<'interactive'>;
+ canonicalPermissionOutcomes: CanonicalPermissionOutcomeReader;
+}) {
+ const store = input.stores.runtimeEventStore;
+ const highWater = async (sessionId: string): Promise<number | null> => {
+ const ordinal = await store.readTranscriptHighWater(sessionId);
+ return ordinal === null ? null : ordinal * EVENT_SEQUENCE_STRIDE + EVENT_SEQUENCE_STRIDE - 1;
+ };
+
+ /** One Turn's rows, each at the sequence its own event sits at. */
+ const projectTurn = async (
+ turn: RuntimeTranscriptInvocation,
+ ): Promise<{ sequence: number; message: StoredMessage }[]> => {
+ const events = turn.events.map((entry) => entry.event);
+ const projected = projectRuntimeEventsToStoredMessages(events, {
+ invocations: [turn.invocation],
+ canonicalPermissionOutcomes: await readCanonicalPermissionOutcomes(
+ events,
+ input.canonicalPermissionOutcomes,
+ ),
+ });
+ if (projected.diagnostics.some(isHardRuntimeEventReadModelDiagnostic)) {
+ throw new Error('Durable RuntimeEvent transcript projection is incomplete');
+ }
+ const ordinals = new Map(turn.events.map((entry) => [entry.event.id, entry.ordinal]));
+ const emitted = new Map<number, number>();
+ return projected.messages.map((message, index) => {
+ const ordinal = ordinals.get(projected.sourceEventIds[index]!);
+ if (ordinal === undefined) {
+ throw new Error('Durable transcript message has no source RuntimeEvent');
+ }
+ const offset = emitted.get(ordinal) ?? 0;
+ if (offset >= EVENT_SEQUENCE_STRIDE) {
+ throw new Error('RuntimeEvent exceeds its transcript sequence stride');
+ }
+ emitted.set(ordinal, offset + 1);
+ return { sequence: ordinal * EVENT_SEQUENCE_STRIDE + offset, message };
+ });
+ };
+
+ const readTurns = async (
+ sessionId: string,
+ request: { direction: 'older' | 'newer'; throughOrdinal: number; position: number },
+ ): Promise<RuntimeTranscriptInvocation[]> =>
+ store.readTranscriptInvocations(sessionId, {
+ ...request,
+ limit: TRANSCRIPT_TURN_SCAN_LIMIT,
+ maxEvents: DURABLE_TRANSCRIPT_TURN_MAX_EVENTS,
+ maxBytes: DURABLE_TRANSCRIPT_TURN_MAX_BYTES,
+ });
+
+ const scan = async function* (
+ sessionId: string,
+ request: {
+ direction: 'older' | 'newer';
+ throughSequence?: number | null;
+ position?: number;
+ /** Stops the walk after this many Turns, for a read that may find nothing. */
+ maxTurns?: number;
+ },
+ ): AsyncGenerator<{ sequence: number; message: StoredMessage }> {
+ const throughSequence =
+ request.throughSequence === undefined ? await highWater(sessionId) : request.throughSequence;
+ if (throughSequence === null) return;
+ const position = request.position ?? (request.direction === 'older' ? throughSequence : 0);
+ const throughOrdinal = ordinalOf(throughSequence);
+ let ordinal = ordinalOf(position);
+ let walked = 0;
+ while (ordinal >= 0 && ordinal <= throughOrdinal) {
+ const turns = await readTurns(sessionId, {
+ direction: request.direction,
+ throughOrdinal,
+ position: ordinal,
+ });
+ if (turns.length === 0) return;
+ for (const turn of turns) {
+ if (request.maxTurns !== undefined && walked >= request.maxTurns) return;
+ walked += 1;
+ const records = (await projectTurn(turn)).filter(
+ ({ sequence }) =>
+ sequence <= throughSequence &&
+ (request.direction === 'older' ? sequence <= position : sequence >= position),
+ );
+ if (request.direction === 'older') records.reverse();
+ yield* records;
+ }
+ const edge = turns.at(-1)!;
+ ordinal = request.direction === 'older' ? edge.firstOrdinal - 1 : edge.lastOrdinal + 1;
Review Comment:
[P2] Stepping past a Turn by its ordinal edge assumes Turns do not nest — the assumption this PR's own reasoning rejects.
The scan takes one Turn per step (`TRANSCRIPT_TURN_SCAN_LIMIT = 1`) and then moves the cursor to `edge.lastOrdinal + 1` going forward, or `edge.firstOrdinal - 1` going back. That is correct only while Turn ordinal ranges are disjoint. **When one Turn's range contains another's, the step jumps over the inner Turn and its rows never appear.**
`16f70e872`'s message argues that forward paging cannot use the ordinal index because *"has an event at or after this ordinal" is a claim about an invocation's ending, and no ordinal index answers it without assuming Turns never interleave, which the runtime does not promise*. **The spine was chosen to respect interleaving; the caller then steps as if it does not happen.**
**Shape:** A opens, B opens, B terminates, A terminates — accepted by the public `appendRuntimeEvent`. It also reproduces with B as a top-level continuation (`source.kind = 'continuation'`, `parentRunId`, no `agentId`, so `isSessionInlineInvocation` holds).
- forward from 0: A is taken, the cursor moves to A's `lastOrdinal + 1`, and B — whose rows sit below that — can never be reached. The newer walk returns A's rows only.
- backward from the watermark: B comes first by opening order, the next scan resumes from B's last sequence, and A's later rows are filtered out by `sequence <= position`. The older walk returns B's rows only.
**Rows are dropped, not looped** — the walk terminates, just short. Both directions lose a whole Turn.
**Bulk selection is not where the defect is.** With `limit > 1` the two SQL paths agree and are mutual inverses, nesting included; the divergence appears only under the production limit of one.
**Not [P1], and the reason is a gap rather than an argument:** the interleaved ledger was written through the append API, not produced by driving `SessionManager`. Handoff pause ends the parent Turn first, and ordinary continuation is sequential, so the common shapes page correctly. **What is missing is a demonstration that a supported flow writes this shape — until then the exposure is bounded by that uncertainty, not by a guarantee.**
The cheapest repair keeps the spine's premise: resume from the next unread Turn in the spine rather than from an ordinal computed off the previous Turn's edge, so a nested Turn is a row the walk has not reached yet instead of one it stepped over.
<details><summary>简体中文</summary>
**[P2] 用 Turn 的 ordinal 边界来跨过它,前提是 Turn 不会嵌套 —— 而这正是本 PR 自己的论证所否定的假设。**
扫描每步取一个 Turn(`TRANSCRIPT_TURN_SCAN_LIMIT = 1`),随后把游标移到向前的 `edge.lastOrdinal + 1` 或向后的 `edge.firstOrdinal - 1`。**只有在各 Turn 的 ordinal 区间互不包含时,这才是对的。当一个 Turn 的区间包含另一个时,这一步会直接跳过内层 Turn,它的行再也不会出现。**
`16f70e872` 的提交信息论证道:向前分页不能用 ordinal 索引,因为「『在此 ordinal 或其后是否有事件』是关于一次 invocation 结束的断言,任何 ordinal 索引都无法在**不假设 Turn 从不交错**的前提下回答它,**而 runtime 并不作此保证**」。**spine 正是为了尊重交错而选的;而调用方随后却按「不会发生交错」在步进。**
**形状:** A 开、B 开、B 终、A 终 —— 公共的 `appendRuntimeEvent` 接受它。把 B 换成顶层 continuation(`source.kind = 'continuation'`、有 `parentRunId`、无 `agentId`,因而 `isSessionInlineInvocation` 成立)同样复现。
- **向前从 0 出发**:取走 A,游标移到 A 的 `lastOrdinal + 1`,而 B 的行位于其下 —— **再也到不了**。newer 只返回 A 的行。
- **向后从高水位出发**:按 opening 顺序 B 在前,下一次扫描从 B 的最后一个 sequence 继续,**A 更靠后的行被 `sequence <= position` 滤掉**。older 只返回 B 的行。
**是漏行,不是死循环** —— 遍历会结束,只是短了。**两个方向各丢一整个 Turn。**
**缺陷不在批量选择上。** `limit > 1` 时两条 SQL 路径一致且互逆,含嵌套形状;**分歧只在生产所用的 limit=1 下出现。**
**不定 [P1],理由是一个缺口而不是一个论证:** 那份交错账本是**经 append API 写出来的,不是驱动 `SessionManager` 产生的**。handoff pause 会先结束父 Turn,普通 continuation 是顺序的,所以常见形状分页正确。**缺的是「某条受支持的流程会写出这个形状」的实证 —— 在那之前,这个暴露面是被那份不确定性限制住的,而不是被某种保证限制住的。**
最省的修法保留 spine 的前提:**从 spine 中下一个未读的 Turn 续读,而不是从上一个 Turn 边界算出的 ordinal 续读** —— 这样嵌套的 Turn 就是「遍历还没走到的一行」,而不是「被跨过去的一行」。
</details>
---
> **Automated review notice:** This comment was posted by an automated review agent operated by jackwener. It is not an independent human review and does not replace one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener commented on code in PR #4879:
URL: https://github.com/apache/maka/pull/4879#discussion_r3950436460
##########
packages/core/src/session.ts:
##########
@@ -1259,7 +1270,6 @@ const TOKEN_USAGE_MESSAGE_SHAPE = defineObjectShape<TokenUsageMessage>()(
const TURN_STATE_MESSAGE_SHAPE = defineObjectShape<TurnStateMessage>()(
['type', 'id', 'turnId', 'ts', 'status'],
[
Review Comment:
[P1] Removing the key from this shape makes every released `turn_state` row that carries it undecodable.
The shape is exact on both sides: `hasExactShape` requires every present key to be in `allowed`, so a stored row with `partialOutputRetained` no longer fails to *use* the field — **it fails to decode at all**, and `decodeMessage` rejects the whole message.
**The reachable path is an ordinary upgrade, not damaged data.** The Turn-state writer emitted this field on every Turn before it was removed here; at the merged-main parent it was still written by Claude Code across all four terminal branches, by Codex across both, and by OpenCode across all four. Those rows are then persisted by `createImportedSession()` at `transcriptLedgerVersion: 0`, and the first read runs the ledger converter over exactly them.
**Reproduced old-writer to new-reader, with a control.** Three Sessions written at the merged-main parent — field `true`, field `false`, and one without the field — then the same storage root reopened at this head: all three headers read; the control returned its two messages; **both the `true` and the `false` Sessions failed `readMessages()` and `readMessagesAfter()` with `Stored Session message … is incompatible`.** The control is what separates "this field breaks decoding" from "something else broke".
**Nothing is lost on disk** — the conversation bytes are intact and a later build could read them again. What breaks is availability: after upgrading, those Sessions cannot be converted or displayed. That is the reason for [P1] rather than [P2]; the population is every Session whose Turns were written by a released build, which is the normal case rather than an edge.
**The deletion itself is right and does not need reversing.** The field had two derivations and no consumer, and removing it from `TurnRecord`, the producers, the wire, shared pages, Desktop and the UI is what makes that true. **The gap is only that a persisted-shape change was made as if it were a type change.** Accepting the legacy key in the decoder and stripping it before returning the current `TurnStateMessage` restores all three cases without putting the field back into any consumer or onto the wire.
**Worth stating for the epoch, since it might look like the answer:** the epoch bump does not cover this. It rejects an incompatible *peer*, not an incompatible *row already on this machine's disk*.
<details><summary>简体中文</summary>
**[P1] 把这个键从形状里移除,会使每一条携带它的、已发布版本写下的 `turn_state` 行无法解码。**
这个形状**两侧都是精确的**:`hasExactShape` 要求出现的每个键都在 `allowed` 里,**所以带 `partialOutputRetained` 的存储行不再是「用不上这个字段」—— 而是根本解码不了**,`decodeMessage` 会拒绝整条消息。
**可达路径是一次普通升级,不是损坏数据。** 在这里被删除之前,Turn 状态写入方**对每一个 Turn 都会写出该字段**;在合并进来的 main 父提交上,Claude Code 的四条终态分支、Codex 的两条、OpenCode 的四条**仍然在写它**。这些行随后由 `createImportedSession()` 以 `transcriptLedgerVersion: 0` 持久化,**而首次读取正是让账本转换器去扫它们。**
**已做老写入方到新读取方的复现,并且带对照。** 在合并 main 的父提交上写了三个 Session —— 字段为 `true`、字段为 `false`、以及一个不带该字段的对照 —— 然后在本 head 重开同一个存储根:三个 header 都能读;**对照返回了它的两条消息;而 `true` 与 `false` 两个 Session 的 `readMessages()` 与 `readMessagesAfter()` 都以 `Stored Session message … is incompatible` 失败。** **那个对照,正是把「是这个字段破坏了解码」与「是别的东西坏了」区分开的东西。**
**磁盘上没有东西丢失** —— 对话字节完好,换一个构建仍然读得出来。**坏掉的是可用性:升级之后,这些 Session 无法被转换、也无法显示。** 这就是它定 [P1] 而非 [P2] 的理由;**受影响的人群是「其 Turn 由任一已发布构建写下」的每一个 Session,那是常态而不是边角。**
**这次删除本身是对的,不需要回退。** 该字段有两处派生、却没有消费者,**把它从 `TurnRecord`、各产出方、协议线、共享页、Desktop 与 UI 中移除,正是让这句话成立的原因。** **缺口仅仅在于:一次「持久化形状」的变更,被当作一次「类型」变更来做了。** 在解码器里接受这个旧键、并在返回当前 `TurnStateMessage` 之前把它剥掉,就能让三种情形全部恢复,**而不必把该字段放回任何消费者、也不必放回协议线。**
**关于 epoch,值得说一句,因为它看起来像是答案:epoch 抬升覆盖不了这一条。它拒绝的是不兼容的对端,而不是一条已经躺在本机磁盘上的不兼容的行。**
</details>
---
> **Automated review notice:** This comment was posted by an automated review agent operated by jackwener. It is not an independent human review and does not replace one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener commented on code in PR #4879:
URL: https://github.com/apache/maka/pull/4879#discussion_r3947828689
##########
packages/runtime-host/src/server/session-transcript-reader.ts:
##########
@@ -137,12 +196,449 @@ export interface SessionTranscriptReader {
sessionId: string,
request: SessionTranscriptMessageLookupRequest,
): Promise<readonly StoredMessage[]>;
+ readDurableTurnContributions(
+ sessionId: string,
+ throughSequence: number | null,
+ position: number,
+ maxContributions: number,
+ ): Promise<SessionTurnContributionPage>;
+ readDurableTurnLandmarks(
+ sessionId: string,
+ maxLandmarks: number,
+ ): Promise<SessionTurnLandmarkSnapshot>;
readActiveOverlay(
sessionId: string,
rootTurn: TurnSnapshot | null,
): Promise<readonly StoredMessage[]>;
}
+/**
+ * Pages seek immutable Session event ordinals before decoding payloads. One
+ * Turn is projected at a time, so a page costs one Turn rather than the
+ * Session. The low sequence bits distinguish the few rows one event emits.
+ *
+ * What an event becomes is asked only of the read model. Storage selects Turns
+ * by ordinal and hands over their events; it never classifies one.
+ */
+function createDurableLedgerTranscriptReader(input: {
+ stores: ExecutionStoresWriter<'interactive'>;
+ canonicalPermissionOutcomes: CanonicalPermissionOutcomeReader;
+}) {
+ const store = input.stores.runtimeEventStore;
+ const highWater = async (sessionId: string): Promise<number | null> => {
+ const ordinal = await store.readTranscriptHighWater(sessionId);
+ return ordinal === null ? null : ordinal * EVENT_SEQUENCE_STRIDE + EVENT_SEQUENCE_STRIDE - 1;
+ };
+
+ /** One Turn's rows, each at the sequence its own event sits at. */
+ const projectTurn = async (
+ turn: RuntimeTranscriptInvocation,
+ ): Promise<{ sequence: number; message: StoredMessage }[]> => {
+ const events = turn.events.map((entry) => entry.event);
+ const projected = projectRuntimeEventsToStoredMessages(events, {
+ invocations: [turn.invocation],
+ canonicalPermissionOutcomes: await readCanonicalPermissionOutcomes(
+ events,
+ input.canonicalPermissionOutcomes,
+ ),
+ });
+ if (projected.diagnostics.some(isHardRuntimeEventReadModelDiagnostic)) {
+ throw new Error('Durable RuntimeEvent transcript projection is incomplete');
+ }
+ const ordinals = new Map(turn.events.map((entry) => [entry.event.id, entry.ordinal]));
+ const emitted = new Map<number, number>();
+ return projected.messages.map((message, index) => {
+ const ordinal = ordinals.get(projected.sourceEventIds[index]!);
+ if (ordinal === undefined) {
+ throw new Error('Durable transcript message has no source RuntimeEvent');
+ }
+ const offset = emitted.get(ordinal) ?? 0;
+ if (offset >= EVENT_SEQUENCE_STRIDE) {
+ throw new Error('RuntimeEvent exceeds its transcript sequence stride');
+ }
+ emitted.set(ordinal, offset + 1);
+ return { sequence: ordinal * EVENT_SEQUENCE_STRIDE + offset, message };
+ });
+ };
+
+ const readTurns = async (
+ sessionId: string,
+ request: { direction: 'older' | 'newer'; throughOrdinal: number; position: number },
+ ): Promise<RuntimeTranscriptInvocation[]> =>
+ store.readTranscriptInvocations(sessionId, {
+ ...request,
+ limit: TRANSCRIPT_TURN_SCAN_LIMIT,
+ maxEvents: DURABLE_TRANSCRIPT_TURN_MAX_EVENTS,
+ maxBytes: DURABLE_TRANSCRIPT_TURN_MAX_BYTES,
+ });
+
+ const scan = async function* (
+ sessionId: string,
+ request: {
+ direction: 'older' | 'newer';
+ throughSequence?: number | null;
+ position?: number;
+ /** Stops the walk after this many Turns, for a read that may find nothing. */
+ maxTurns?: number;
+ },
+ ): AsyncGenerator<{ sequence: number; message: StoredMessage }> {
+ const throughSequence =
+ request.throughSequence === undefined ? await highWater(sessionId) : request.throughSequence;
+ if (throughSequence === null) return;
+ const position = request.position ?? (request.direction === 'older' ? throughSequence : 0);
+ const throughOrdinal = ordinalOf(throughSequence);
+ let ordinal = ordinalOf(position);
+ let walked = 0;
+ while (ordinal >= 0 && ordinal <= throughOrdinal) {
+ const turns = await readTurns(sessionId, {
+ direction: request.direction,
+ throughOrdinal,
+ position: ordinal,
+ });
+ if (turns.length === 0) return;
+ for (const turn of turns) {
+ if (request.maxTurns !== undefined && walked >= request.maxTurns) return;
+ walked += 1;
+ const records = (await projectTurn(turn)).filter(
+ ({ sequence }) =>
+ sequence <= throughSequence &&
+ (request.direction === 'older' ? sequence <= position : sequence >= position),
+ );
+ if (request.direction === 'older') records.reverse();
+ yield* records;
+ }
+ const edge = turns.at(-1)!;
+ ordinal = request.direction === 'older' ? edge.firstOrdinal - 1 : edge.lastOrdinal + 1;
+ }
+ };
+
+ return {
+ readHighWater: highWater,
+
+ ...pagedTranscriptReads({ readHighWater: highWater, scan }),
+
+ /** One row per Turn, folded from the Turn's own projected messages. */
+ async readTurnContributions(
+ sessionId: string,
+ throughSequence: number | null,
+ position: number,
+ maxContributions: number,
+ ): Promise<SessionTurnContributionPage> {
+ const watermark = throughSequence ?? (await highWater(sessionId));
+ if (watermark === null) {
+ return { throughSequence: null, contributions: [], nextPosition: null };
+ }
+ const turns = await store.readTranscriptInvocations(sessionId, {
+ direction: 'newer',
+ throughOrdinal: ordinalOf(watermark),
+ position: ordinalOf(position),
+ limit: maxContributions + 1,
+ maxEvents: DURABLE_TRANSCRIPT_TURN_MAX_EVENTS,
+ maxBytes: DURABLE_TRANSCRIPT_TURN_MAX_BYTES,
+ });
+ const contributions: SessionTurnContribution[] = [];
+ for (const turn of turns.slice(0, maxContributions)) {
+ // Folded from the Turn's own rows, so `firstSequence` lands on its first
+ // row rather than on the opening fact, which has no row at all.
+ let contribution: SessionTurnContribution | undefined;
+ for (const { sequence, message } of await projectTurn(turn)) {
+ if (sequence < position || sequence > watermark) continue;
+ contribution = foldTurnContribution(
+ contribution,
+ turn.invocation.turnId,
+ sequence,
+ message,
+ );
+ }
+ if (contribution) contributions.push(contribution);
+ }
+ const next = turns[maxContributions];
+ return {
+ throughSequence: watermark,
+ contributions,
+ nextPosition: next ? next.firstOrdinal * EVENT_SEQUENCE_STRIDE : null,
+ };
+ },
+
+ /** Evenly spaced Turn starts, selected in SQL before loading their prompts. */
+ async readTurnLandmarks(
+ sessionId: string,
+ maxLandmarks: number,
+ ): Promise<SessionTurnLandmarkSnapshot> {
+ const throughSequence = await highWater(sessionId);
+ if (throughSequence === null) return { throughSequence: null, landmarks: [] };
+ const turns = await store.readTranscriptLandmarks(
+ sessionId,
+ ordinalOf(throughSequence),
+ maxLandmarks,
+ );
+ const landmarks: SessionTurnLandmark[] = [];
+ for (const turn of turns) {
+ if (!turn.prompt) continue;
+ const message = projectRuntimeEventUserMessage(turn.prompt.event, turn.prompt.event.id);
+ const label = (message?.displayText ?? message?.text ?? '').trim();
+ if (!label) continue;
+ landmarks.push({
+ turnId: turn.invocation.turnId,
+ sequence: turn.prompt.ordinal * EVENT_SEQUENCE_STRIDE,
+ label,
+ });
+ }
+ return { throughSequence, landmarks };
+ },
+ };
+}
+
+/** An ordered, bounded walk over one Session's transcript records. */
+interface TranscriptRecordSource {
+ readHighWater(sessionId: string): Promise<number | null>;
+ scan(
+ sessionId: string,
+ request: {
+ direction: 'older' | 'newer';
+ throughSequence?: number | null;
+ position?: number;
+ /** Stops the walk after this many Turns, for a read that may find nothing. */
+ maxTurns?: number;
+ },
+ ): AsyncGenerator<{ sequence: number; message: StoredMessage }>;
+}
+
+/**
+ * The reads that are the same whatever produces the records: a byte-bounded
+ * page, a record scan, and a lookup by message id. Each walks one source's
+ * ordered records and never asks where they came from.
+ */
+function pagedTranscriptReads(source: TranscriptRecordSource) {
+ return {
+ async readPage(
+ sessionId: string,
+ request: SessionTranscriptPageRequest,
+ ): Promise<SessionTranscriptStoragePage> {
+ const throughSequence =
+ request.throughSequence === undefined
+ ? await source.readHighWater(sessionId)
+ : request.throughSequence;
+ if (throughSequence === null) {
+ return { throughSequence: null, fragments: [], rawBytes: 0, next: null };
+ }
+ const fragments: SessionTranscriptStorageFragment[] = [];
+ let rawBytes = 0;
+ let next: SessionTranscriptStoragePage['next'] = null;
+ let truncated = false;
+ for await (const record of source.scan(sessionId, { ...request, throughSequence })) {
+ if (fragments.length >= request.maxMessages || rawBytes >= request.maxBytes) {
+ truncated = true;
+ next = { position: record.sequence, byteOffset: null };
+ break;
+ }
+ const data = Buffer.from(JSON.stringify(record.message), 'utf8');
+ // A message larger than the remaining budget is served in byte slices,
+ // from the edge the traversal is moving away from, so the next page
+ // resumes inside the same record instead of skipping it.
+ const continued = record.sequence === request.position && request.byteOffset !== undefined;
+ const edge = continued
+ ? request.byteOffset!
+ : request.direction === 'older'
+ ? data.byteLength
+ : 0;
+ const available = request.maxBytes - rawBytes;
+ const byteOffset = request.direction === 'older' ? Math.max(0, edge - available) : edge;
+ const end =
+ request.direction === 'older' ? edge : Math.min(data.byteLength, edge + available);
+ fragments.push({
+ sequence: record.sequence,
+ byteOffset,
+ totalBytes: data.byteLength,
+ payloadDigest: null,
+ data: data.subarray(byteOffset, end),
+ });
+ rawBytes += end - byteOffset;
+ const complete = request.direction === 'older' ? byteOffset === 0 : end === data.byteLength;
+ if (!complete) {
+ truncated = true;
+ next = {
+ position: record.sequence,
+ byteOffset: request.direction === 'older' ? byteOffset : end,
+ };
+ break;
+ }
+ }
+ if (!truncated) next = null;
+ return { throughSequence, fragments, rawBytes, next };
+ },
+
+ async readRecords(
+ sessionId: string,
+ request: SessionTranscriptRecordScanRequest,
+ ): Promise<SessionTranscriptRecordScanPage> {
+ const throughSequence =
+ request.throughSequence === undefined
+ ? await source.readHighWater(sessionId)
+ : request.throughSequence;
+ if (throughSequence === null) {
+ return { throughSequence: null, records: [], nextPosition: null };
+ }
+ const records: Array<{ sequence: number; message: StoredMessage }> = [];
+ let storedBytes = 0;
+ let nextPosition: number | null = null;
+ for await (const record of source.scan(sessionId, { ...request, throughSequence })) {
+ if (records.length >= request.maxMessages || storedBytes >= request.maxStoredBytes) {
+ nextPosition = record.sequence;
+ break;
+ }
+ records.push(record);
+ storedBytes += Buffer.byteLength(JSON.stringify(record.message), 'utf8');
+ }
+ return { throughSequence, records, nextPosition };
+ },
+
+ /**
+ * The durable rows behind a set of message ids.
+ *
+ * The ids come from the assistant streams a subscriber is still watching,
+ * so they belong to the Session's newest Turns. The scan walks back from
+ * the watermark a Turn at a time and stops as soon as every id is found,
+ * rather than keeping an index from message id to event. An id that is not
+ * there stops the walk after the newest Turns instead of reading the
+ * Session: the handoff shows what the tail holds, not everything it could.
+ */
+ async readMessagesById(
+ sessionId: string,
+ request: SessionTranscriptMessageLookupRequest,
+ ): Promise<StoredMessage[]> {
+ if (request.throughSequence === null || request.messageIds.length === 0) return [];
+ const wanted = new Set(request.messageIds);
+ const found: Array<{ sequence: number; message: StoredMessage }> = [];
+ let bytes = 0;
+ for await (const record of source.scan(sessionId, {
+ direction: 'older',
+ throughSequence: request.throughSequence,
+ maxTurns: TRANSCRIPT_LOOKUP_MAX_TURNS,
+ })) {
+ if (!wanted.delete(record.message.id)) continue;
+ bytes += Buffer.byteLength(JSON.stringify(record.message), 'utf8');
+ if (found.length >= request.maxMessages || bytes > request.maxBytes) break;
+ found.push(record);
+ if (wanted.size === 0) break;
+ }
+ return found.sort((a, b) => a.sequence - b.sequence).map((record) => record.message);
+ },
+ };
+}
+
+/**
+ * The WorkHub Coordination Session's transcript, read from the rows the WorkHub
+ * writes.
+ *
+ * Every other Session's transcript is what its runs did, so the ledger holds
+ * all of it. The Coordination Session's is not: a delegation, a stop and a
+ * routing summary are appended under a Turn id that no root Turn admission ever
+ * minted, so there is no invocation for the ledger to hang them on and no
+ * conversion that could lift them. `workhub.coordination.answer` is the one
+ * path that would admit a real Turn and nothing in the renderer calls it.
+ *
+ * Delete this source once the WorkHub admits a Coordination Turn for every
+ * action, which its own ADR already requires (#3492,
+ * `docs/architecture/workhub-coordination-session-adr.md`): the Session then
+ * reads like any other and this reader has nothing left to do.
+ */
+function createCoordinationTranscriptReader(stores: ExecutionStoresWriter<'interactive'>) {
Review Comment:
[P2] The Coordination Session now has two write substrates, and this reader sees only one of them.
**Not a second authority — that part checks out.** Delegation, stop, clarify and `record` append to `session_messages` under a `turnId` no admission minted, so the ledger has no invocation to hang them on and only this path can produce those rows. Judged by the same rule that closed the earlier dual-reader finding, this reader is *required*.
**It is no longer sufficient.** The `answer_here` disposition takes the ordinary root-Turn admission — `#answer` calls `startWorkHubCoordinationMessage`, which runs an `AgentRun` — and an `AgentRun` persists RuntimeEvents. It does not append a `session_messages` row, because this PR is what removed that second write. The neighbouring `record` path still calls `appendMessages`, and the code says the split out loud: *"An answer owns its Turn identity in the root admission ledger"*, with `record` refusing a turn that already has a root admission. **Exclusive writers, not a union resolved at read time.**
**What the user sees.** While the answer's Turn is running, `readActiveOverlay` is not partitioned onto this reader, so it projects the ledger and the answer appears. When the Turn reaches terminal the overlay returns empty, and the WorkHub timeline falls back to this durable reader — which is scanning `session_messages` and has no row for what was just written. **The exchange is on screen while it runs and gone once it finishes.**
**Classified as a regression from this change**, not a pre-existing gap: before the dual write was removed, the answer's rows were in `session_messages` and this reader found them. Removing the second write is the right direction; what is missing is that the Coordination Session's own admitted Q&A moved to the ledger while its durable read did not.
The fix does not require restoring the dual write. It requires the Coordination read to cover both substrates for this Session, or the admitted answer to be reachable through the same projection the rest of the Session's durable history now comes from.
<details><summary>简体中文</summary>
**[P2] Coordination Session 现在有两个写入底,而这个读取者只看得见其中一个。**
**它不是第二份权威 —— 这一点核过,成立。** 委托、停止、澄清与 `record` 写入 `session_messages`,挂在**没有 admission 铸过的 `turnId`** 上,账本里没有可以承载它们的 invocation,**且只有这条路径能产出那些行**。用关掉先前那条「双读者」finding 的同一把尺子判,这个读取者是**必要的**。
**但它不再是充分的。** `answer_here` 走的是**普通 root-Turn 准入** —— `#answer` 调 `startWorkHubCoordinationMessage`,后者跑一个 `AgentRun`,而 `AgentRun` 只持久化 RuntimeEvent。**它不写 `session_messages` 行,因为正是本 PR 删掉了那次第二写。** 紧邻的 `record` 路径仍然调 `appendMessages`,而代码把这个分野写在明处:**「answer 的 Turn 身份归属 root admission ledger」**,并且 `record` 会拒绝一个已有 root admission 的 turn。**这是互斥的写入方,不是读时再合并的并集。**
**用户看到的是什么。** 在这次 answer 的 Turn **运行期间**,`readActiveOverlay` 并未被分流到这个读取者,它投影的是账本,**所以回答是可见的**。当 Turn 进入终态,overlay 返回空,WorkHub 时间线回落到这个持久读取者 —— 而它扫的是 `session_messages`,**里面没有刚刚写下的那一行**。**于是这段问答:跑的时候在屏幕上,结束之后就没了。**
**归类为本次改动引入的回归**,而不是既有缺口:在双写被删之前,answer 的行就在 `session_messages` 里,这个读取者找得到。**删掉第二次写的方向是对的;缺的是 —— Coordination Session 自己的已准入问答搬到了账本上,而它的持久读没有跟着搬。**
修复不需要恢复双写。它需要的是:让 Coordination 的读**覆盖这个 Session 的两个底**,或者让已准入的 answer 能通过「该 Session 其余持久历史现在所走的同一份投影」被读到。
</details>
---
> **Automated review notice:** This comment was posted by an automated review agent operated by jackwener. It is not an independent human review and does not replace one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "Astro-Han (via GitHub)" <gi...@apache.org>.
Astro-Han commented on PR #4879:
URL: https://github.com/apache/maka/pull/4879#issuecomment-5568618230
@likun666661 **Finding 2 (transcript paging is O(session) per page) — fixed in `16f70e872`.**
Fixed in `16f70e872`. The grouping is gone: an invocation's first event is its opening and its last is its terminal, both already carrying an ordinal, so walking back takes the ordinal index directly and stops at the page. Walking forward still reads a spine of one row per invocation — "has an event at or after this ordinal" is a claim about an invocation's ending, and no ordinal index answers it without assuming Turns never interleave, which the runtime does not promise.
One more term was hiding behind it: each page row's invocation record was looked up by `run_id`, which scans the Session's openings. Both opening shelves key on `invocation_id` — the ledger through its unique partial index, the migrated shelf through its primary key — so it now looks up by that instead.
Measured through `readTranscriptInvocations` at 200/500/1000/2000 rounds, fetching the last invocation alone: 1.8 / 4.4 / 9.2 / 19.8 ms before, 0.2 ms flat after. Paging a whole Session backwards at 100 Turns per page is 8 ms per page at every length.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener commented on code in PR #4879:
URL: https://github.com/apache/maka/pull/4879#discussion_r3949072058
##########
packages/runtime/src/runtime-event-read-model.ts:
##########
@@ -1197,6 +1248,7 @@ function projectTerminalTurnState(
...(status === 'failed' && event.content?.kind === 'error' && event.content.retry
? { retry: event.content.retry }
: {}),
+ partialOutputRetained,
Review Comment:
[P2] The merge kept this projector copy and dropped the contribution-derived one, so bounded and full readers now disagree.
`#4951` resolved retained output by making output shape a **contribution** fact. That resolution did not survive this merge: the projector still computes the bit here and persists it into every terminal `turn_state`, while `foldTurnContribution` records only state and prompt. The two readers then treat the same field differently:
- **bounded** (`session-turns.ts`) returns the stored bit verbatim — `state.partialOutputRetained ?? false` — and its own comment concedes why: *"nothing else on the contribution can say whether output survived"*;
- **full** (`session.ts`) ORs it with the assistant and tool rows it can see — `latestState.partialOutputRetained || partialOutputRetained`.
**A turn with visible assistant text and a recorded `partialOutputRetained: false` therefore answers `false` through the paged path and `true` through the full one.** That was reproduced against the built production modules, and Desktop's `listSessionTurns` is on the bounded path.
**Why this reads as merge-introduced rather than pre-existing:** before the merge these were not two answers — main had moved the derivation onto the contribution precisely so one place owned it. Keeping this write while losing that move leaves the field with two producers and no owner.
**The head's tests do not cover the disagreement.** They assert the stored bit is carried through; the contribution-derived cases that `#4951` added are not present, so the suite is green on exactly the history that diverges.
**Severity, stated honestly:** `#4951` also deleted the renderer that displayed this, so nothing shows a user the wrong answer today. That is what keeps it at [P2] — but it is also the argument against keeping the field: what remains is a persisted, wire-carried second authority with no consumer and two disagreeing derivations.
Minimal coherent repair is the direction main had already taken: carry the output booleans on `SessionTurnContribution`, fold and project them, derive the value there with the stored bit as a legacy fallback only, drop this write for new events, and add a bounded-versus-full parity test so the two paths cannot drift again silently.
<details><summary>简体中文</summary>
**[P2] 这次合并保留了 projector 这份拷贝、却丢掉了「由 contribution 派生」的那一份,于是有界读与全量读现在会给出不同答案。**
`#4951` 对「保留输出」的处置,是把**输出形态变成 contribution 的事实**。**那个处置没有在本次合并中存活**:projector 仍在这里计算该位并把它持久化进每一条终态 `turn_state`,而 `foldTurnContribution` 只记录 state 与 prompt。于是两个读者对同一字段的处理不同:
- **有界侧**(`session-turns.ts`)**原样取存储位** —— `state.partialOutputRetained ?? false`,而它自己的注释就承认了原因:*「contribution 上没有别的东西能说明输出是否幸存」*;
- **全量侧**(`session.ts`)把它**与看得见的 assistant / tool 行取或** —— `latestState.partialOutputRetained || partialOutputRetained`。
**于是,一个带有可见 assistant 文本、而记录的 `partialOutputRetained` 为 `false` 的 turn,走分页路径答 `false`、走全量路径答 `true`。** 这一点已针对**构建后的生产模块**复现,而 Desktop 的 `listSessionTurns` 正走有界路径。
**为什么这算合并引入而非既有问题:** 合并之前它们并不是两个答案 —— **main 把派生移到 contribution 上,正是为了让这件事只有一个归属。保留了这次写入、却丢掉了那次搬迁,就使该字段有两个产出方、没有归属方。**
**本 head 的测试没有覆盖这个分歧。** 它们断言的是「存储位被原样携带」;`#4951` 补的那些**由 contribution 派生**的用例并不在此处 —— **所以套件恰恰在会发散的那类历史上是绿的。**
**严重性,老实说:** `#4951` 同时删掉了显示它的渲染层,**所以今天没有任何界面把错误答案呈现给用户。这正是它停在 [P2] 的原因** —— 但这同时也是「不该保留这个字段」的论据:**剩下的是一个被持久化、还上了协议线的第二权威,没有消费者,却有两个互相不一致的派生。**
最小且自洽的修法,就是 main 已经走的那个方向:把输出布尔量带在 `SessionTurnContribution` 上、fold 与 project 都带上它、在那里派生该值(存储位仅作旧数据兜底)、**为新事件去掉这次写入**,并补一条**有界对全量的一致性测试**,使两条路径不能再静默分叉。
</details>
---
> **Automated review notice:** This comment was posted by an automated review agent operated by jackwener. It is not an independent human review and does not replace one.
##########
packages/storage/src/sqlite-runtime-store.ts:
##########
@@ -1455,6 +1492,58 @@ export class SqliteRuntimeStore
});
}
+ async resequenceSessionEventOrdinals(sessionId: string): Promise<void> {
+ assertRuntimeStorageSafeId(sessionId, 'Invalid session id');
+ this.transaction(() => {
+ // Lifted above the range first: the second statement renumbers into the
+ // space these rows occupy, and (session_id, ordinal) is a primary key.
+ // Shifting up rather than below zero keeps every intermediate value
+ // inside the table's own `ordinal > 0`, and lands them past the 1..N the
+ // renumber assigns, since the count cannot exceed the maximum.
+ const { shift } = this.db
+ .prepare(`
+ SELECT COALESCE(MAX(ordinal), 0) AS shift
+ FROM runtime_session_event_ordinals
+ WHERE session_id = ?
+ `)
+ .get(sessionId) as { shift: number };
+ this.db
+ .prepare(`
+ UPDATE runtime_session_event_ordinals
+ SET ordinal = ordinal + :shift
+ WHERE session_id = :sessionId
+ `)
+ .run({ sessionId, shift });
+ this.db
+ .prepare(`
+ WITH opening AS (
+ SELECT invocation_id, CAST(json_extract(payload_json, '$.ts') AS INTEGER) AS opened_at
+ FROM runtime_events
+ WHERE session_id = :sessionId AND event_kind = 'invocation_opened'
+ ),
+ ordered AS (
+ SELECT
+ o.event_id AS event_id,
+ ROW_NUMBER() OVER (
+ ORDER BY COALESCE(opening.opened_at, e.committed_at), e.invocation_id, o.ordinal
+ ) AS ordinal
+ FROM runtime_session_event_ordinals o
+ JOIN runtime_events e ON e.event_id = o.event_id
+ LEFT JOIN opening ON opening.invocation_id = e.invocation_id
+ WHERE o.session_id = :sessionId
+ )
+ UPDATE runtime_session_event_ordinals
+ SET ordinal = (
Review Comment:
[P2] This renumber is quadratic in the Session, and it runs before the first durable read of every converted Session.
`ordered` carries a window function and is referenced from a correlated subquery, one evaluation per updated row. `EXPLAIN QUERY PLAN` shows it as `CO-ROUTINE`, not materialized — so SQLite recomputes the whole `ROW_NUMBER()` ordering for each row it updates.
Measured on a real store, isolating this call from the appends around it:
| turns | events | append (per event) | **this call** |
| --- | --- | --- | --- |
| 200 | 600 | 0.48 ms | **440 ms** |
| 500 | 1500 | 0.39 ms | **3121 ms** |
| 1000 | 3000 | 0.48 ms | **11629 ms** |
Appends are flat per event, so the growth is here: **roughly ×3.7 per doubling.**
**The one-word remedy holds and is behaviour-identical.** Declaring the CTE `ordered AS MATERIALIZED` gives, on the same data:
| turns | current | materialized |
| --- | --- | --- |
| 500 | 2863 ms | **40 ms** |
| 1000 | 10801 ms | **142 ms** |
Both produce the same result — ordinals `1..count`, contiguous, in the same order. It is a planner change, not a semantic one.
**Why it matters where it sits:** the conversion this closes runs inside `ensureTranscriptLedger`, and every durable read entry point awaits that before returning a reader, inside one synchronous SQLite transaction. So a converted Session of a thousand Turns spends about **11.6 s on its first open after upgrade**, and the curve is quadratic — near **45 s** at two thousand.
[P2] rather than higher: it is one-time per Session, nothing is lost, and a slow first open is recoverable. It is not [P3] because the cost is on the shared database and grows with history a user already has.
**Context worth keeping:** the ordinal work this belongs to is a real improvement elsewhere. Paging backwards through the same synthetic Sessions costs 19 / 18 / 16 ms per page at 200 / 500 / 1000 turns on this head, against 79 / 230 / 489 measured on `86f1d760` — flat where it used to grow.
<details><summary>简体中文</summary>
**[P2] 这次重排对 Session 是二次的,而且它跑在每个被转换 Session 的首次持久读之前。**
`ordered` 带着窗口函数,却被从**相关子查询**里引用,**每更新一行求值一次**。`EXPLAIN QUERY PLAN` 显示它是 `CO-ROUTINE` 而非物化 —— **于是 SQLite 为它更新的每一行重算一遍整个 `ROW_NUMBER()` 排序。**
在真实存储上实测,并把这次调用与周围的 append 分开计时:
| 轮 | 事件 | append(每事件) | **本次调用** |
| --- | --- | --- | --- |
| 200 | 600 | 0.48 ms | **440 ms** |
| 500 | 1500 | 0.39 ms | **3121 ms** |
| 1000 | 3000 | 0.48 ms | **11629 ms** |
**append 每事件是平的,所以增长出在这里:大约每翻一倍 ×3.7。**
**一个关键字的修法成立,且行为完全相同。** 把该 CTE 声明为 `ordered AS MATERIALIZED`,同样数据下:
| 轮 | 现状 | 物化后 |
| --- | --- | --- |
| 500 | 2863 ms | **40 ms** |
| 1000 | 10801 ms | **142 ms** |
两者结果相同 —— 序号 `1..count`、连续、顺序一致。**这是计划层的改动,不是语义改动。**
**它所处的位置为什么要紧:** 它所收尾的那次转换跑在 `ensureTranscriptLedger` 内,**而每一个持久读入口在返回读取者之前都会 await 它**,且处在一个同步 SQLite 事务中。所以一个一千轮的被转换 Session,**升级后第一次打开大约要花 11.6 秒**,而曲线是二次的 —— **两千轮时接近 45 秒。**
定 [P2] 而非更高:它是每个 Session 一次性的,没有东西丢失,首次打开慢是可恢复的。**不定 [P3],是因为这笔开销压在共享数据库上,并且随用户已有的历史增长。**
**值得一并记下的背景:** 这次改动所属的那批 ordinal 工作,在别处是**实打实的改进**。在同一批合成 Session 上向后翻页,本 head 每页 19 / 18 / 16 ms(200 / 500 / 1000 轮),而我在 `86f1d760` 上实测的是 79 / 230 / 489 —— **原本增长,现在拉平。**
</details>
---
> **Automated review notice:** This comment was posted by an automated review agent operated by jackwener. It is not an independent human review and does not replace one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener commented on code in PR #4879:
URL: https://github.com/apache/maka/pull/4879#discussion_r3944904224
##########
packages/runtime-host/src/server/session-transcript-reader.ts:
##########
@@ -47,29 +59,67 @@ const ACTIVE_TRANSCRIPT_SCAN_BATCH_MAX_BYTES = 256 * 1024;
export function createSessionTranscriptReader(input: {
stores: ExecutionStoresWriter<'interactive'>;
canonicalPermissionOutcomes: CanonicalPermissionOutcomeReader;
+ /**
+ * Converts a Session whose transcript predates the ledger, before this reader
+ * looks for invocations that only the conversion can create. Omitted only by
+ * tests that seed the ledger themselves.
+ */
+ ensureTranscriptLedger?: (sessionId: string) => Promise<void>;
}): SessionTranscriptReader {
+ const durable = createDurableLedgerTranscriptReader(input);
+ const prepared = async (sessionId: string): Promise<void> => {
+ await input.ensureTranscriptLedger?.(sessionId);
+ };
return {
- readDurableHighWater: (sessionId) =>
- input.stores.sessionStore.readTranscriptHighWaterSnapshot(sessionId),
- readDurablePage: (sessionId, request) =>
- input.stores.sessionStore.readTranscriptPageSnapshot(sessionId, request),
- readDurableRecords: (sessionId, request) =>
- input.stores.sessionStore.readTranscriptRecordsSnapshot(sessionId, request),
- readDurableMessagesById: (sessionId, request) =>
- input.stores.sessionStore.readTranscriptMessagesSnapshot(sessionId, request),
+ readDurableHighWater: async (sessionId) => {
+ await prepared(sessionId);
+ return durable.readHighWater(sessionId);
+ },
+ readDurablePage: async (sessionId, request) => {
+ await prepared(sessionId);
+ return durable.readPage(sessionId, request);
Review Comment:
**Not disputing the engineering argument — it is specific and it mostly holds up. Raising the one thing it does not cover: where the acceptance is recorded.**
The reasoning for not repairing is checkable and largely convincing: these notes belong to turns a real run already sealed; restoring them would need a path around `assertRunNotSealed`, which is an interface obligation rather than one store's detail; read-time derivation reaches only two of the seven, and the `context-budget.ts:218` "one note per send" rule against `mergeCompactionDecisionDiagnostics` accumulation is a concrete obstacle rather than a hand-wave. **Broadening it yourself from one kind to seven is the right way to answer a finding.**
**What is missing is disclosure, and the standard here is one this repository has already applied.** On #4808 an equivalent residue was graded down specifically because the description named the full consequence chain where a decision-maker would see it — that disclosure is what turned it from a surprise into a decision. Here the reasoning lives in a review reply and in a code comment at the version gate. **The PR body still says only: "Losing a legacy tool card in conversion is acceptable; losing a Session or a conversation is not."**
A reader of that sentence would not learn what is actually being accepted:
- **seven** of the eight retained runtime note kinds, not a legacy tool card;
- for **every pre-existing Session a released build has touched**, because an ordinary send already set `transcriptLedgerVersion: 1` and the new reader skips conversion at that version;
- **permanently** for those Sessions — the converter cannot reach them, so this is not deferred work;
- and the notes are **user-visible by current design**, still listed as such at `core/session.ts:1146`.
The code comment at the version gate is aimed at the next engineer, which is the right place for "version 1 does not mean `session_messages` is empty". **It is not the place where a maintainer decides whether losing seven kinds of user-visible warnings is an acceptable cost.** That decision belongs in the body, next to the sentence that currently understates it.
**This is separable from whether the technical argument survives.** If it does, the loss is still worth stating plainly; if a shape is found that restores the notes without bypassing the seal, the body needs updating either way. I am not asking for a behaviour change here — only for the accepted cost to be written where acceptance happens.
<details><summary>简体中文</summary>
**不是在质疑那套工程论证 —— 它很具体,而且大部分站得住。我提的是它没有覆盖的一点:这笔「接受」被记录在了哪里。**
不修的理由是可核且基本有说服力的:这些 note 属于**已被真实 run 封存**的 turn;把它们放回去需要一条绕过 `assertRunNotSealed` 的路径,而那是接口义务、不是某个 store 的细节;读时派生只够到七个里的两个,而 `context-budget.ts:218` 的「一次 send 一条 note」对上 `mergeCompactionDecisionDiagnostics` 的累积,是一个**具体障碍**而不是含糊其辞。**你自己把范围从一个 kind 扩到七个,这是回应一条 finding 的正确方式。**
**缺的是披露,而这个标准本仓库已经用过一次。** 在 #4808 上,一笔等价的残留之所以被降级,**正是因为描述把完整的后果链写在了决策者会看到的地方** —— **那份披露才是让它从「意外」变成「决定」的东西。** 而这里,理由存在于一条评审回复和版本门处的一段代码注释中。**PR 正文至今只说:「Losing a legacy tool card in conversion is acceptable; losing a Session or a conversation is not.」**
只读这句话的人,不会知道实际被接受的是什么:
- 是八个保留 runtime note kind 中的**七个**,而不是一张 legacy tool card;
- 适用于**任何已发布构建碰过的既有 Session**,因为一次普通 send 就已把 `transcriptLedgerVersion` 置为 1,而新 reader 在该版本下跳过转换;
- 对这些 Session 是**永久的** —— 转换器够不到它们,所以这不是「以后再做」;
- 而这些 note **按当前设计就是用户可见的**,`core/session.ts:1146` 仍如此列着。
版本门处那段代码注释面向的是下一个工程师,**而「版本 1 不等于 `session_messages` 已空」写在那里是对的。但那里不是维护者判断「损失七类用户可见警告是否为可接受成本」的地方。** 那个决定属于正文,就放在目前那句低估了它的话旁边。
**这一点与技术论证是否成立可以分开。** 若论证成立,这笔损失仍值得写明;若找到了不绕过封存也能恢复的形状,正文同样要改。**我在这里不要求任何行为改动 —— 只要求把已接受的成本写在「接受」实际发生的地方。**
</details>
---
> **Automated review notice:** This comment was posted by an automated review agent operated by jackwener. It is not an independent human review and does not replace one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "Astro-Han (via GitHub)" <gi...@apache.org>.
Astro-Han commented on code in PR #4879:
URL: https://github.com/apache/maka/pull/4879#discussion_r3944872303
##########
packages/runtime/src/runtime-event-read-model.ts:
##########
@@ -228,6 +238,29 @@ export function projectRuntimeEventsToStoredMessages(
};
const messages: StoredMessage[] = [];
+ const context = options.context;
+ if (context) {
+ state.hasRetainedOutput = context.hasRetainedOutput;
+ if (context.contentOrder)
+ state.contentOrderByMessageId.set(context.messageId, [...context.contentOrder]);
+ if (context.toolName && context.toolUseId)
+ state.toolNameByUseId.set(context.toolUseId, context.toolName);
+ const requestEvent = context.permissionRequest;
+ const request = requestEvent?.actions?.permissionRequest;
+ if (request && requestEvent) {
+ state.permissionRequestById.set(request.requestId, {
+ requestId: request.requestId,
+ toolUseId: request.toolUseId,
+ toolName: request.toolName,
Review Comment:
Verified as a structure, deferred rather than fixed here — and I want to be explicit that this is a decision, not an oversight.
`RuntimeTranscriptQuery` arrived in `b7d27cfe6` to give the reader the bounded seek #4791 asks for. Removing it would put back the unbounded read. So the two end states available are "derive one side from the other" or "compare them on a production path", and both are a larger design change than this PR should absorb on top of the migration fixes it is already carrying.
One correction to the framing: the SQL side seeks, it does not re-decide what a transcript row is. `session-transcript-reader.ts` still runs the selected events through `projectRuntimeEventsToStoredMessages`, so the JS projection remains the one place an event becomes a message. What is genuinely duplicated is the *selection* rule, which is narrower than a second definition of the row — but it is duplicated, and I am not claiming otherwise.
Leaving this open rather than resolving it, since nothing changed in response.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "Astro-Han (via GitHub)" <gi...@apache.org>.
Astro-Han commented on code in PR #4879:
URL: https://github.com/apache/maka/pull/4879#discussion_r3951359140
##########
packages/runtime-host/src/server/session-transcript-reader.ts:
##########
@@ -137,12 +196,449 @@ export interface SessionTranscriptReader {
sessionId: string,
request: SessionTranscriptMessageLookupRequest,
): Promise<readonly StoredMessage[]>;
+ readDurableTurnContributions(
+ sessionId: string,
+ throughSequence: number | null,
+ position: number,
+ maxContributions: number,
+ ): Promise<SessionTurnContributionPage>;
+ readDurableTurnLandmarks(
+ sessionId: string,
+ maxLandmarks: number,
+ ): Promise<SessionTurnLandmarkSnapshot>;
readActiveOverlay(
sessionId: string,
rootTurn: TurnSnapshot | null,
): Promise<readonly StoredMessage[]>;
}
+/**
+ * Pages seek immutable Session event ordinals before decoding payloads. One
+ * Turn is projected at a time, so a page costs one Turn rather than the
+ * Session. The low sequence bits distinguish the few rows one event emits.
+ *
+ * What an event becomes is asked only of the read model. Storage selects Turns
+ * by ordinal and hands over their events; it never classifies one.
+ */
+function createDurableLedgerTranscriptReader(input: {
+ stores: ExecutionStoresWriter<'interactive'>;
+ canonicalPermissionOutcomes: CanonicalPermissionOutcomeReader;
+}) {
+ const store = input.stores.runtimeEventStore;
+ const highWater = async (sessionId: string): Promise<number | null> => {
+ const ordinal = await store.readTranscriptHighWater(sessionId);
+ return ordinal === null ? null : ordinal * EVENT_SEQUENCE_STRIDE + EVENT_SEQUENCE_STRIDE - 1;
+ };
+
+ /** One Turn's rows, each at the sequence its own event sits at. */
+ const projectTurn = async (
+ turn: RuntimeTranscriptInvocation,
+ ): Promise<{ sequence: number; message: StoredMessage }[]> => {
+ const events = turn.events.map((entry) => entry.event);
+ const projected = projectRuntimeEventsToStoredMessages(events, {
+ invocations: [turn.invocation],
+ canonicalPermissionOutcomes: await readCanonicalPermissionOutcomes(
+ events,
+ input.canonicalPermissionOutcomes,
+ ),
+ });
+ if (projected.diagnostics.some(isHardRuntimeEventReadModelDiagnostic)) {
+ throw new Error('Durable RuntimeEvent transcript projection is incomplete');
+ }
+ const ordinals = new Map(turn.events.map((entry) => [entry.event.id, entry.ordinal]));
+ const emitted = new Map<number, number>();
+ return projected.messages.map((message, index) => {
+ const ordinal = ordinals.get(projected.sourceEventIds[index]!);
+ if (ordinal === undefined) {
+ throw new Error('Durable transcript message has no source RuntimeEvent');
+ }
+ const offset = emitted.get(ordinal) ?? 0;
+ if (offset >= EVENT_SEQUENCE_STRIDE) {
+ throw new Error('RuntimeEvent exceeds its transcript sequence stride');
+ }
+ emitted.set(ordinal, offset + 1);
+ return { sequence: ordinal * EVENT_SEQUENCE_STRIDE + offset, message };
+ });
+ };
+
+ const readTurns = async (
+ sessionId: string,
+ request: { direction: 'older' | 'newer'; throughOrdinal: number; position: number },
+ ): Promise<RuntimeTranscriptInvocation[]> =>
+ store.readTranscriptInvocations(sessionId, {
+ ...request,
+ limit: TRANSCRIPT_TURN_SCAN_LIMIT,
+ maxEvents: DURABLE_TRANSCRIPT_TURN_MAX_EVENTS,
+ maxBytes: DURABLE_TRANSCRIPT_TURN_MAX_BYTES,
+ });
+
+ const scan = async function* (
+ sessionId: string,
+ request: {
+ direction: 'older' | 'newer';
+ throughSequence?: number | null;
+ position?: number;
+ /** Stops the walk after this many Turns, for a read that may find nothing. */
+ maxTurns?: number;
+ },
+ ): AsyncGenerator<{ sequence: number; message: StoredMessage }> {
+ const throughSequence =
+ request.throughSequence === undefined ? await highWater(sessionId) : request.throughSequence;
+ if (throughSequence === null) return;
+ const position = request.position ?? (request.direction === 'older' ? throughSequence : 0);
+ const throughOrdinal = ordinalOf(throughSequence);
+ let ordinal = ordinalOf(position);
+ let walked = 0;
+ while (ordinal >= 0 && ordinal <= throughOrdinal) {
+ const turns = await readTurns(sessionId, {
+ direction: request.direction,
+ throughOrdinal,
+ position: ordinal,
+ });
+ if (turns.length === 0) return;
+ for (const turn of turns) {
+ if (request.maxTurns !== undefined && walked >= request.maxTurns) return;
+ walked += 1;
+ const records = (await projectTurn(turn)).filter(
+ ({ sequence }) =>
+ sequence <= throughSequence &&
+ (request.direction === 'older' ? sequence <= position : sequence >= position),
+ );
+ if (request.direction === 'older') records.reverse();
+ yield* records;
+ }
+ const edge = turns.at(-1)!;
+ ordinal = request.direction === 'older' ? edge.firstOrdinal - 1 : edge.lastOrdinal + 1;
+ }
+ };
+
+ return {
+ readHighWater: highWater,
+
+ ...pagedTranscriptReads({ readHighWater: highWater, scan }),
+
+ /** One row per Turn, folded from the Turn's own projected messages. */
+ async readTurnContributions(
+ sessionId: string,
+ throughSequence: number | null,
+ position: number,
+ maxContributions: number,
+ ): Promise<SessionTurnContributionPage> {
+ const watermark = throughSequence ?? (await highWater(sessionId));
+ if (watermark === null) {
+ return { throughSequence: null, contributions: [], nextPosition: null };
+ }
+ const turns = await store.readTranscriptInvocations(sessionId, {
+ direction: 'newer',
+ throughOrdinal: ordinalOf(watermark),
+ position: ordinalOf(position),
+ limit: maxContributions + 1,
+ maxEvents: DURABLE_TRANSCRIPT_TURN_MAX_EVENTS,
+ maxBytes: DURABLE_TRANSCRIPT_TURN_MAX_BYTES,
+ });
+ const contributions: SessionTurnContribution[] = [];
+ for (const turn of turns.slice(0, maxContributions)) {
+ // Folded from the Turn's own rows, so `firstSequence` lands on its first
+ // row rather than on the opening fact, which has no row at all.
+ let contribution: SessionTurnContribution | undefined;
+ for (const { sequence, message } of await projectTurn(turn)) {
+ if (sequence < position || sequence > watermark) continue;
+ contribution = foldTurnContribution(
+ contribution,
+ turn.invocation.turnId,
+ sequence,
+ message,
+ );
+ }
+ if (contribution) contributions.push(contribution);
+ }
+ const next = turns[maxContributions];
+ return {
+ throughSequence: watermark,
+ contributions,
+ nextPosition: next ? next.firstOrdinal * EVENT_SEQUENCE_STRIDE : null,
+ };
+ },
+
+ /** Evenly spaced Turn starts, selected in SQL before loading their prompts. */
+ async readTurnLandmarks(
+ sessionId: string,
+ maxLandmarks: number,
+ ): Promise<SessionTurnLandmarkSnapshot> {
+ const throughSequence = await highWater(sessionId);
+ if (throughSequence === null) return { throughSequence: null, landmarks: [] };
+ const turns = await store.readTranscriptLandmarks(
+ sessionId,
+ ordinalOf(throughSequence),
+ maxLandmarks,
+ );
+ const landmarks: SessionTurnLandmark[] = [];
+ for (const turn of turns) {
+ if (!turn.prompt) continue;
+ const message = projectRuntimeEventUserMessage(turn.prompt.event, turn.prompt.event.id);
+ const label = (message?.displayText ?? message?.text ?? '').trim();
+ if (!label) continue;
+ landmarks.push({
+ turnId: turn.invocation.turnId,
+ sequence: turn.prompt.ordinal * EVENT_SEQUENCE_STRIDE,
+ label,
+ });
+ }
+ return { throughSequence, landmarks };
+ },
+ };
+}
+
+/** An ordered, bounded walk over one Session's transcript records. */
+interface TranscriptRecordSource {
+ readHighWater(sessionId: string): Promise<number | null>;
+ scan(
+ sessionId: string,
+ request: {
+ direction: 'older' | 'newer';
+ throughSequence?: number | null;
+ position?: number;
+ /** Stops the walk after this many Turns, for a read that may find nothing. */
+ maxTurns?: number;
+ },
+ ): AsyncGenerator<{ sequence: number; message: StoredMessage }>;
+}
+
+/**
+ * The reads that are the same whatever produces the records: a byte-bounded
+ * page, a record scan, and a lookup by message id. Each walks one source's
+ * ordered records and never asks where they came from.
+ */
+function pagedTranscriptReads(source: TranscriptRecordSource) {
+ return {
+ async readPage(
+ sessionId: string,
+ request: SessionTranscriptPageRequest,
+ ): Promise<SessionTranscriptStoragePage> {
+ const throughSequence =
+ request.throughSequence === undefined
+ ? await source.readHighWater(sessionId)
+ : request.throughSequence;
+ if (throughSequence === null) {
+ return { throughSequence: null, fragments: [], rawBytes: 0, next: null };
+ }
+ const fragments: SessionTranscriptStorageFragment[] = [];
+ let rawBytes = 0;
+ let next: SessionTranscriptStoragePage['next'] = null;
+ let truncated = false;
+ for await (const record of source.scan(sessionId, { ...request, throughSequence })) {
+ if (fragments.length >= request.maxMessages || rawBytes >= request.maxBytes) {
+ truncated = true;
+ next = { position: record.sequence, byteOffset: null };
+ break;
+ }
+ const data = Buffer.from(JSON.stringify(record.message), 'utf8');
+ // A message larger than the remaining budget is served in byte slices,
+ // from the edge the traversal is moving away from, so the next page
+ // resumes inside the same record instead of skipping it.
+ const continued = record.sequence === request.position && request.byteOffset !== undefined;
+ const edge = continued
+ ? request.byteOffset!
+ : request.direction === 'older'
+ ? data.byteLength
+ : 0;
+ const available = request.maxBytes - rawBytes;
+ const byteOffset = request.direction === 'older' ? Math.max(0, edge - available) : edge;
+ const end =
+ request.direction === 'older' ? edge : Math.min(data.byteLength, edge + available);
+ fragments.push({
+ sequence: record.sequence,
+ byteOffset,
+ totalBytes: data.byteLength,
+ payloadDigest: null,
+ data: data.subarray(byteOffset, end),
+ });
+ rawBytes += end - byteOffset;
+ const complete = request.direction === 'older' ? byteOffset === 0 : end === data.byteLength;
+ if (!complete) {
+ truncated = true;
+ next = {
+ position: record.sequence,
+ byteOffset: request.direction === 'older' ? byteOffset : end,
+ };
+ break;
+ }
+ }
+ if (!truncated) next = null;
+ return { throughSequence, fragments, rawBytes, next };
+ },
+
+ async readRecords(
+ sessionId: string,
+ request: SessionTranscriptRecordScanRequest,
+ ): Promise<SessionTranscriptRecordScanPage> {
+ const throughSequence =
+ request.throughSequence === undefined
+ ? await source.readHighWater(sessionId)
+ : request.throughSequence;
+ if (throughSequence === null) {
+ return { throughSequence: null, records: [], nextPosition: null };
+ }
+ const records: Array<{ sequence: number; message: StoredMessage }> = [];
+ let storedBytes = 0;
+ let nextPosition: number | null = null;
+ for await (const record of source.scan(sessionId, { ...request, throughSequence })) {
+ if (records.length >= request.maxMessages || storedBytes >= request.maxStoredBytes) {
+ nextPosition = record.sequence;
+ break;
+ }
+ records.push(record);
+ storedBytes += Buffer.byteLength(JSON.stringify(record.message), 'utf8');
+ }
+ return { throughSequence, records, nextPosition };
+ },
+
+ /**
+ * The durable rows behind a set of message ids.
+ *
+ * The ids come from the assistant streams a subscriber is still watching,
+ * so they belong to the Session's newest Turns. The scan walks back from
+ * the watermark a Turn at a time and stops as soon as every id is found,
+ * rather than keeping an index from message id to event. An id that is not
+ * there stops the walk after the newest Turns instead of reading the
+ * Session: the handoff shows what the tail holds, not everything it could.
+ */
+ async readMessagesById(
+ sessionId: string,
+ request: SessionTranscriptMessageLookupRequest,
+ ): Promise<StoredMessage[]> {
+ if (request.throughSequence === null || request.messageIds.length === 0) return [];
+ const wanted = new Set(request.messageIds);
+ const found: Array<{ sequence: number; message: StoredMessage }> = [];
+ let bytes = 0;
+ for await (const record of source.scan(sessionId, {
+ direction: 'older',
+ throughSequence: request.throughSequence,
+ maxTurns: TRANSCRIPT_LOOKUP_MAX_TURNS,
+ })) {
+ if (!wanted.delete(record.message.id)) continue;
+ bytes += Buffer.byteLength(JSON.stringify(record.message), 'utf8');
+ if (found.length >= request.maxMessages || bytes > request.maxBytes) break;
+ found.push(record);
+ if (wanted.size === 0) break;
+ }
+ return found.sort((a, b) => a.sequence - b.sequence).map((record) => record.message);
+ },
+ };
+}
+
+/**
+ * The WorkHub Coordination Session's transcript, read from the rows the WorkHub
+ * writes.
+ *
+ * Every other Session's transcript is what its runs did, so the ledger holds
+ * all of it. The Coordination Session's is not: a delegation, a stop and a
+ * routing summary are appended under a Turn id that no root Turn admission ever
+ * minted, so there is no invocation for the ledger to hang them on and no
+ * conversion that could lift them. `workhub.coordination.answer` is the one
+ * path that would admit a real Turn and nothing in the renderer calls it.
+ *
+ * Delete this source once the WorkHub admits a Coordination Turn for every
+ * action, which its own ADR already requires (#3492,
+ * `docs/architecture/workhub-coordination-session-adr.md`): the Session then
+ * reads like any other and this reader has nothing left to do.
+ */
+function createCoordinationTranscriptReader(stores: ExecutionStoresWriter<'interactive'>) {
Review Comment:
The mechanism holds; the reachability does not. Deferred at [P3], not fixed here.
**What I verified.** Everything you describe about the split is accurate: `answer_here` takes the ordinary root-Turn admission, `startWorkHubCoordinationMessage` runs an `AgentRun` that persists only RuntimeEvents, and this reader scans `session_messages`. The writers are exclusive, not a union.
**Where it stops.** `workhub.coordination.answer` has no caller. Across the repository the only references outside its own definition are two test files — nothing in `packages/ui`, `packages/cli` or `apps/desktop` invokes it. It is in `REMOTE_OWNER_OPERATION_GRANTS`, so a third-party client could reach it; no shipped one does. The reader's own doc comment says this at `:563`, and I re-checked it on this head rather than trusting it.
**So the classification changes.** Your symptom — visible while running, gone at terminal — requires a client that calls the operation. That is a contrived path today, and its consequence is display only: the answer's rows are in the ledger, intact and readable by the ordinary projection; what is missing is that the Coordination Session is routed away from that projection. Nothing is lost and nothing is unrecoverable, which is why this is [P3] rather than a merge-blocking regression.
**And the end state is deletion, not coverage.** Making this reader union both substrates would give the Coordination Session two read paths to keep in agreement — a second authority introduced to serve a caller that does not exist. The ADR (#3492) already requires the WorkHub to admit a Coordination Turn for every action; when it does, the Session reads like every other one and this whole source is deleted. Widening it now makes that deletion harder, not easier.
Left open in the sense that matters: the comment at `:563` names the condition under which this file disappears, so the obligation is recorded where the next reader of this code will find it.
<details><summary>简体中文</summary>
机制成立，可达性不成立。定 [P3] 并延后，本 PR 不修。
**核实到的部分。** 你描述的分野全部准确：`answer_here` 走普通 root-Turn 准入，`startWorkHubCoordinationMessage` 跑的 `AgentRun` 只持久化 RuntimeEvent，而这个读取者扫的是 `session_messages`。写入方是互斥的，不是读时并集。
**它止步于哪里。** `workhub.coordination.answer` **没有调用方**。全仓中除它自身的定义外，只有两个测试文件引用它；`packages/ui`、`packages/cli`、`apps/desktop` 里没有任何调用。它在 `REMOTE_OWNER_OPERATION_GRANTS` 中，所以第三方客户端够得着；但**没有任何已发布的客户端会走到**。读取者自己在 `:563` 的注释就是这么写的，我在本 head 上重新核过，而不是信它。
**于是分类要变。** 你说的症状——运行时可见、终态即消失——**需要一个会调用该操作的客户端**。今天那是构造路径，且后果只在显示层：answer 的行**就在账本里**，完好且能被普通投影读出；缺的是 Coordination Session 被从那份投影上分流走了。**没有东西丢失，也没有不可恢复的东西**，所以这是 [P3]，不是阻塞合并的回归。
**而且终局是删除，不是覆盖。** 让这个读取者并集两个底，等于给 Coordination Session 两条需要互相保持一致的读路径——**为一个并不存在的调用方引入第二权威**。ADR（#3492）本就要求 WorkHub 为每个 action 都准入一个 Coordination Turn；等它做到，这个 Session 就和其他 Session 一样读，整份来源随之删除。**现在把它加宽，只会让那次删除更难，而不是更容易。**
以真正要紧的方式留着：`:563` 的注释写明了这个文件消失的条件，所以这笔义务记录在**下一个读这段代码的人一定会看到的地方**。
</details>
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "Astro-Han (via GitHub)" <gi...@apache.org>.
Astro-Han commented on PR #4879:
URL: https://github.com/apache/maka/pull/4879#issuecomment-5558402984
Head `7c2c74e5`.
**[P1] Fixed.** Confirmed, and a regression this PR introduced — on the base that branch threw and `continue`d, never wrote.
Not fixed the way you proposed: inlining `${runId}-admitted-prompt` in `begin()` leaves the same template in two packages, which is still two rules that happen to agree. `admittedPromptEventId()` is now the single derivation.
Unifying it is also not sufficient. `core_root_turn_admissions` outlives an upgrade, and a Run written by the current release recorded its folded prompt under `newId()` — matching on the derived id would still miss it and still write a second. So recovery asks whether the *Turn* has a prompt, not whether it has one under that id. Steering left that index in the same change: it is a user message said into an already-admitted Turn, never the Turn's own prompt, and it would otherwise trip the `multiple UserMessages` guard.
Test: `startup recovery leaves a folded Root prompt the Run already recorded alone`.
**[P2 seal] Fixed at `finalize`.** Also a regression from here: the base repaired this via `appendMessage`, and `session_messages` has no seal. The `run.terminalEvent` guard is correct, so the repair moved to where the prompt is still known — `finalize()` records it before reserving the terminal, next to the `openInvocation()` that already keeps "a run cannot end without having begun". Ablation-verified.
Narrower than a generic append failure: a latching store error also refuses the terminal, leaving the run open for crash recovery. The sealed-without-prompt shape needs a non-latching refusal, which is what the test injects.
**[P2 importer] No finding.** I removed the filter as suggested; `does not import Host-handed-off transcript messages as synthetic runs` fails. The producer is a turn whose only user row carries `steeringEventId` — steering said into a Turn a durable Root already owns, so converting it stands a second synthetic run beside it. Load-bearing; now says so in a comment. The tools-only case you actually named still has no producer, and I am not widening the rule for it.
**[P2 unread] Fixed, P3.** Confirmed, never clears falsely. Raising the constant only moves the boundary, so the scan pages back until it finds a visible record or exhausts the ledger. Stale badge on a read Session, self-healing at the next Turn.
Your single-reviewer disclosure: taken — it is why I treated the importer item as a gap to close rather than a defect to patch, and closing it is what produced the counterexample.
storage 1092, runtime-host 1711, runtime 3130, cli 805 — 0 failures. WorkHub linkage lane: waiting on your post.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener commented on code in PR #4879:
URL: https://github.com/apache/maka/pull/4879#discussion_r3944402850
##########
packages/runtime/src/runtime-event-read-model.ts:
##########
@@ -228,6 +238,29 @@ export function projectRuntimeEventsToStoredMessages(
};
const messages: StoredMessage[] = [];
+ const context = options.context;
+ if (context) {
+ state.hasRetainedOutput = context.hasRetainedOutput;
+ if (context.contentOrder)
+ state.contentOrderByMessageId.set(context.messageId, [...context.contentOrder]);
+ if (context.toolName && context.toolUseId)
+ state.toolNameByUseId.set(context.toolUseId, context.toolName);
+ const requestEvent = context.permissionRequest;
+ const request = requestEvent?.actions?.permissionRequest;
+ if (request && requestEvent) {
+ state.permissionRequestById.set(request.requestId, {
+ requestId: request.requestId,
+ toolUseId: request.toolUseId,
+ toolName: request.toolName,
Review Comment:
**[P2] The cut removes a second authority but leaves a second reader, and nothing cross-checks them.**
After this change there are still two places that decide what a transcript row is: this JS projection (`RuntimeReadModel` — `getMessages`, history, copy) and the SQL `RuntimeTranscriptQuery` used for UI paging, unread, and `session.turns.query`, which grew by ~436 lines on this head. **The SQL side re-encodes the same rule rather than deriving it from this one**, and no production path compares the two.
That is the same hazard this PR set out to remove, moved one level: the PR deletes the dual *write* authority, and what remains is a dual *read* definition. A row dropped on one side and not the other is silent — there is no error, just a transcript that reads differently depending on which path asked. `compareRuntimeReadModelMessages` exists, but only in tests.
I am not claiming a concrete divergence — I did not produce one, and this is graded on structure rather than on a reproduced loss. **The reason it is still worth raising is that the final seek commit on this head shows this layer is load-bearing**: it is being actively shaped, which is exactly when a second definition drifts.
**Direction:** have one side derive from the other, or add a production-path comparison rather than a test-only one, so a divergence surfaces as a failure instead of as a differently-rendered transcript.
<details><summary>简体中文</summary>
**[P2] 这次切除移走了第二个权威,却留下了第二个读者,而没有东西在对照它们。**
改动之后,「什么算一条 transcript 行」仍由两处决定:这里的 JS 投影(`RuntimeReadModel` —— `getMessages`、history、copy),以及用于 UI 分页、未读与 `session.turns.query` 的 SQL `RuntimeTranscriptQuery`(在本 head 上增加了约 436 行)。**SQL 那侧是把同一条规则重新编码了一遍,而不是从这里派生**,并且没有任何生产路径去比较两者。
**这正是本 PR 想消除的那类危险,只是被挪高了一层:PR 删掉的是双写权威,留下的是双读定义。** 一侧丢了一行而另一侧没丢,是静默的 —— 不会报错,只会让同一个 transcript 因为「谁来问」而读起来不同。`compareRuntimeReadModelMessages` 是存在的,**但只存在于测试里。**
我并不主张已经存在一处具体的分歧 —— **我没有打出来**,这一条是按结构定级而不是按已复现的丢失。**它仍值得提出,是因为本 head 上最后那笔 seek 提交说明这一层是承重的**:它正在被积极改动,**而那恰恰是第二份定义开始漂移的时刻。**
**方向:** 让一侧从另一侧派生,或加一条**生产路径上的**比对(而不是只存在于测试里的),使分歧以失败的形式暴露,而不是以「渲染得不一样的 transcript」的形式存在。
</details>
---
> **Automated review notice:** This comment was posted by an automated review agent operated by jackwener. It is not an independent human review and does not replace one.
##########
packages/runtime-host/src/server/session-transcript-reader.ts:
##########
@@ -47,29 +59,67 @@ const ACTIVE_TRANSCRIPT_SCAN_BATCH_MAX_BYTES = 256 * 1024;
export function createSessionTranscriptReader(input: {
stores: ExecutionStoresWriter<'interactive'>;
canonicalPermissionOutcomes: CanonicalPermissionOutcomeReader;
+ /**
+ * Converts a Session whose transcript predates the ledger, before this reader
+ * looks for invocations that only the conversion can create. Omitted only by
+ * tests that seed the ledger themselves.
+ */
+ ensureTranscriptLedger?: (sessionId: string) => Promise<void>;
}): SessionTranscriptReader {
+ const durable = createDurableLedgerTranscriptReader(input);
+ const prepared = async (sessionId: string): Promise<void> => {
Review Comment:
**[P2] Nothing in the suite proves the second write is actually gone.**
This PR's central claim is that no ordinary Session path writes `session_messages` any more. I could not find a test that asserts it. **Restore the old dual write alongside the new ledger writes and the suite stays green** — the ledger assertions still pass, and nothing notices the extra row. So the property the PR is named for has no guard, and a later change could quietly reintroduce it.
Two related gaps in the same area:
- The reader tests deliberately stand in for `ensureTranscriptLedger` rather than invoking it, so **the first-open-of-a-legacy-Session path that the UI actually takes is never exercised** — a fixture standing in for the launcher does not prove the launcher.
- `compareRuntimeReadModelMessages` exists only in tests, so the comparison it performs is not available on any production path.
**Direction:** assert the absence directly — drive a live `sendMessage` and assert no `session_messages` row is written for the ordinary path — and drive at least one reader test through `ensureTranscriptLedgerForRead` instead of past it.
<details><summary>简体中文</summary>
**[P2] 整个测试套件里没有东西证明「第二次写入真的没了」。**
本 PR 的核心主张是:普通 Session 路径不再写 `session_messages`。**我找不到一条断言它的测试。** **把旧的双写恢复回去、与新的账本写入并存,套件仍然全绿** —— 账本相关断言照样通过,没有任何东西注意到那多出来的一行。**于是这个 PR 得名于其上的那条性质,没有守卫**,日后一次改动就可能悄悄把它带回来。
同一区域还有两处相关缺口:
- reader 测试**刻意用替身顶替 `ensureTranscriptLedger` 而不是调用它**,所以 **UI 实际会走的「首次打开旧 Session」那条路径从未被练到** —— **夹具顶替启动器,并不能证明启动器。**
- `compareRuntimeReadModelMessages` 只存在于测试里,**所以它所做的比对在任何生产路径上都不可用。**
**方向:** 直接断言这个「不存在」—— 驱动一次真实的 `sendMessage`,断言普通路径不写入任何 `session_messages` 行 —— 并且让至少一条 reader 测试**经过** `ensureTranscriptLedgerForRead` 而不是绕过它。
</details>
---
> **Automated review notice:** This comment was posted by an automated review agent operated by jackwener. It is not an independent human review and does not replace one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener commented on PR #4879:
URL: https://github.com/apache/maka/pull/4879#issuecomment-5563326164
**Downgrading my two conversion findings from [P2] to [P3], on measurements rather than on argument.** I said I would revise if a large Session showed no knee; it does not, so here is the revision and the evidence behind it.
## What was measured
A real legacy Session generated by a released build, upgraded and read through the production path on this head:
| rounds | cold start to first screen | peak process |
| --- | --- | --- |
| 200 | ~240 ms | ~138 MiB |
| 500 | ~450 ms | ~158 MiB |
| 1000 | ~798 ms | ~213 MiB |
At 1000 rounds the production reader returned **4040 records across 41 pages**, and **1000 question/answer pairs, 10 tool call/result pairs and 10 compactions were all preserved**. The 6060 → 4040 difference is **entirely** `session_resume` notes and stale `running` statuses. Full paging took ~14.9 s. Nothing was lost across a page boundary — which was the question worth asking of a paged reader, and it answers cleanly.
## Why that changes the grade
My finding said peak memory and latency scale with the whole history rather than with a page budget. **That is confirmed** — every one of these runs scans all rows before the first screen. But confirmation of the mechanism is not the same as a blocking consequence: growth is close to linear with no knee, and a 1000-round Session — far past ordinary — reaches first screen in under a second with a 213 MiB peak. **Graded on reachability and measured impact rather than on shape, that is [P3].**
The structural facts stay on the record because they are what a future change would regress against, not because they block now:
- the first screen still waits for the whole conversion;
- an interrupted conversion still rescans from sequence zero, so repeated interruption keeps paying full-history I/O;
- `carried` holds one turn's rows across pages, and the retry path materialises every opening unbounded.
**One case is measured and one is not, and the distinction matters:** these runs are 1000 *turns*, so `carried` stayed small throughout. **A single turn carrying an enormous body is the shape that would still evade the bound, and it was not exercised.** I am not claiming it is fine — I am saying it is unmeasured, and that it is the specific shape worth a probe if this area is touched again.
## What this does not change
The [P1] on the interrupted old conversion stands, re-measured on this head. The test gap stands — nothing yet asserts that an ordinary `sendMessage` leaves `session_messages` untouched. The description still does not disclose the seven retained runtime note kinds, their permanence, or that they affect every pre-existing Session a released build has touched.
<details><summary>简体中文</summary>
**把我提的两条转换期 finding 从 [P2] 降为 [P3] —— 依据是实测,不是论证。** 我说过若大会话上看不到拐点就会修正,现在确实没有,所以这是修正与其依据。
**测到了什么**
由已发布构建生成的真实旧 Session,在本 head 上经**生产路径**升级并读取:
| 轮数 | 冷开到首屏 | 进程峰值 |
| --- | --- | --- |
| 200 | 约 240 ms | 约 138 MiB |
| 500 | 约 450 ms | 约 158 MiB |
| 1000 | 约 798 ms | 约 213 MiB |
在 1000 轮上,生产 reader **用 41 页读回 4040 条记录**,**1000 组问答、10 组工具调用/结果、10 次压缩全部保留**。6060 → 4040 的差额**全部**是 `session_resume` 注记与过时的 `running` 状态。完整翻页约 14.9 秒。**没有任何内容在页边界丢失** —— 这才是对一个分页 reader 值得问的问题,而答案是干净的。
**为什么这改变了定级**
我那条 finding 说的是:峰值内存与延迟按整份历史增长,而不是按页预算。**这一点被证实了** —— 上面每一次运行都在首屏之前扫过全部行。**但机制被证实,不等于后果构成阻塞**:增长接近线性、没有拐点,而一个 1000 轮的会话(远超日常)首屏在一秒以内、峰值 213 MiB。**按可及性与实测影响而非按形状定级,它是 [P3]。**
那些结构性事实仍然留在记录里,**因为它们是将来某次改动会回退到的东西,而不是因为它们现在阻塞**:首屏仍要等整段转换;被中断的转换仍从 sequence 0 重扫,所以反复中断就要反复付整份历史的 I/O;`carried` 跨页保留一个 turn 的行,而重试路径无界物化全部 opening。
**有一种情形被测到了,有一种没有,而这个区别要紧:** 上述运行是 1000 个 **turn**,所以 `carried` 全程都很小。**「单个 turn 携带极大正文」才是仍会逃出上界的那个形状,而它没有被跑到。** **我不是说它没问题 —— 我是说它未被测量**,并且若此处将来再被改动,它就是值得专门探一次的那个形状。
**这不改变什么**
关于**被中断的旧转换**的那条 [P1] 仍然成立,已在本 head 重新实测。测试缺口仍然成立 —— **至今没有任何东西断言:一次普通的 `sendMessage` 之后 `session_messages` 未被写入。** 描述至今仍未披露那七类保留的 runtime note、它们的永久性,以及它们影响**任何已发布构建碰过的既有 Session**。
</details>
---
> **Automated review notice:** This comment was posted by an automated review agent operated by jackwener. It is not an independent human review and does not replace one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener commented on code in PR #4879:
URL: https://github.com/apache/maka/pull/4879#discussion_r3945749113
##########
packages/runtime-host/src/server/session-transcript-reader.ts:
##########
@@ -34,42 +37,105 @@ import type {
SessionTranscriptPageRequest,
SessionTranscriptRecordScanPage,
SessionTranscriptRecordScanRequest,
+ SessionTranscriptStorageFragment,
SessionTranscriptStoragePage,
+ SessionTurnContribution,
+ SessionTurnContributionPage,
+ SessionTurnLandmark,
+ SessionTurnLandmarkSnapshot,
+ RuntimeTranscriptInvocation,
} from '@maka/storage/execution-stores';
+import { foldTurnContribution } from '@maka/storage/session-message-projection';
import { SESSION_TRANSCRIPT_OVERLAY_MAX_MESSAGES, type TurnSnapshot } from '../protocol/index.js';
const PERMISSION_OUTCOME_READ_CONCURRENCY = 8;
+/** One event can emit content, a permission, usage, and terminal/notice rows. */
+const EVENT_SEQUENCE_STRIDE = 8;
export const ACTIVE_TRANSCRIPT_OVERLAY_MAX_MESSAGES = SESSION_TRANSCRIPT_OVERLAY_MAX_MESSAGES;
export const ACTIVE_TRANSCRIPT_OVERLAY_MAX_BYTES = 16 * 1024 * 1024;
const ACTIVE_TRANSCRIPT_SOURCE_MAX_EVENTS = ACTIVE_TRANSCRIPT_OVERLAY_MAX_MESSAGES * 2;
const ACTIVE_TRANSCRIPT_SCAN_BATCH_MAX_BYTES = 256 * 1024;
+/**
+ * What one durable page may read of a Turn, matching the bound the live
+ * overlay already holds for a run. A Turn past it is refused rather than
+ * half-projected: a prefix of a Turn is not a smaller transcript of it.
+ */
+const DURABLE_TRANSCRIPT_TURN_MAX_EVENTS = ACTIVE_TRANSCRIPT_SOURCE_MAX_EVENTS;
+const DURABLE_TRANSCRIPT_TURN_MAX_BYTES = ACTIVE_TRANSCRIPT_OVERLAY_MAX_BYTES;
+/** Turns per storage round trip: one, so a page loads no Turn it cannot use. */
+const TRANSCRIPT_TURN_SCAN_LIMIT = 1;
+/**
+ * How far back the live-to-durable handoff looks for a message id. The ids come
+ * from assistant streams the subscriber is still watching, so they are in the
+ * newest Turn or the one it continued from; an id that is in neither is treated
+ * as absent rather than searched for down the Session.
+ */
+const TRANSCRIPT_LOOKUP_MAX_TURNS = 2;
export function createSessionTranscriptReader(input: {
stores: ExecutionStoresWriter<'interactive'>;
canonicalPermissionOutcomes: CanonicalPermissionOutcomeReader;
+ /**
+ * Converts a Session whose transcript predates the ledger, before this reader
+ * looks for invocations that only the conversion can create. Omitted only by
Review Comment:
**[P2] Still open on this head: nothing in the suite proves the second write is gone.**
This round's test changes cover the paging budgets, not the absence this PR is named for. `session-transcript-reader.ts:79-82` still carries the hook comment saying the test omits `ensureTranscriptLedger` and seeds the ledger itself, and **no test asserts that `session_messages` is empty for an ordinary path after a live `sendMessage`**. Restore the old second write alongside the new ledger writes and these suites stay green.
**Direction unchanged:** drive a live `sendMessage` and assert the absence directly, and let at least one reader test go *through* `ensureTranscriptLedgerForRead` rather than past it. A fixture standing in for the launcher cannot show the launcher agrees with it.
<details><summary>简体中文</summary>
**[P2] 在本 head 上仍然成立:整个套件里没有东西证明「第二次写入真的没了」。**
这一轮的测试改动覆盖的是**分页预算**,而不是这个 PR 得名于其上的那个「不存在」。`session-transcript-reader.ts:79-82` 仍带着那句 hook 注释 —— 测试省略 `ensureTranscriptLedger`、自己种 ledger;**而且没有任何测试断言:一次真实 `sendMessage` 之后,普通路径不会写入 `session_messages`。** 把旧的第二写恢复回去、与新的账本写入并存,**这些套件仍然全绿。**
**方向不变:** 驱动一次真实 `sendMessage` 并**直接断言那个「不存在」**;并让至少一条 reader 测试**经过** `ensureTranscriptLedgerForRead` 而不是绕过它。**夹具顶替启动器,证明不了启动器与它意见一致。**
</details>
---
> **Automated review notice:** This comment was posted by an automated review agent operated by jackwener. It is not an independent human review and does not replace one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "Astro-Han (via GitHub)" <gi...@apache.org>.
Astro-Han commented on code in PR #4879:
URL: https://github.com/apache/maka/pull/4879#discussion_r3951357919
##########
packages/runtime-host/src/server/session-transcript-reader.ts:
##########
@@ -137,12 +196,449 @@ export interface SessionTranscriptReader {
sessionId: string,
request: SessionTranscriptMessageLookupRequest,
): Promise<readonly StoredMessage[]>;
+ readDurableTurnContributions(
+ sessionId: string,
+ throughSequence: number | null,
+ position: number,
+ maxContributions: number,
+ ): Promise<SessionTurnContributionPage>;
+ readDurableTurnLandmarks(
+ sessionId: string,
+ maxLandmarks: number,
+ ): Promise<SessionTurnLandmarkSnapshot>;
readActiveOverlay(
sessionId: string,
rootTurn: TurnSnapshot | null,
): Promise<readonly StoredMessage[]>;
}
+/**
+ * Pages seek immutable Session event ordinals before decoding payloads. One
+ * Turn is projected at a time, so a page costs one Turn rather than the
+ * Session. The low sequence bits distinguish the few rows one event emits.
+ *
+ * What an event becomes is asked only of the read model. Storage selects Turns
+ * by ordinal and hands over their events; it never classifies one.
+ */
+function createDurableLedgerTranscriptReader(input: {
+ stores: ExecutionStoresWriter<'interactive'>;
+ canonicalPermissionOutcomes: CanonicalPermissionOutcomeReader;
+}) {
+ const store = input.stores.runtimeEventStore;
+ const highWater = async (sessionId: string): Promise<number | null> => {
+ const ordinal = await store.readTranscriptHighWater(sessionId);
+ return ordinal === null ? null : ordinal * EVENT_SEQUENCE_STRIDE + EVENT_SEQUENCE_STRIDE - 1;
+ };
+
+ /** One Turn's rows, each at the sequence its own event sits at. */
+ const projectTurn = async (
+ turn: RuntimeTranscriptInvocation,
+ ): Promise<{ sequence: number; message: StoredMessage }[]> => {
+ const events = turn.events.map((entry) => entry.event);
+ const projected = projectRuntimeEventsToStoredMessages(events, {
+ invocations: [turn.invocation],
+ canonicalPermissionOutcomes: await readCanonicalPermissionOutcomes(
+ events,
+ input.canonicalPermissionOutcomes,
+ ),
+ });
+ if (projected.diagnostics.some(isHardRuntimeEventReadModelDiagnostic)) {
+ throw new Error('Durable RuntimeEvent transcript projection is incomplete');
+ }
+ const ordinals = new Map(turn.events.map((entry) => [entry.event.id, entry.ordinal]));
+ const emitted = new Map<number, number>();
+ return projected.messages.map((message, index) => {
+ const ordinal = ordinals.get(projected.sourceEventIds[index]!);
+ if (ordinal === undefined) {
+ throw new Error('Durable transcript message has no source RuntimeEvent');
+ }
+ const offset = emitted.get(ordinal) ?? 0;
+ if (offset >= EVENT_SEQUENCE_STRIDE) {
+ throw new Error('RuntimeEvent exceeds its transcript sequence stride');
+ }
+ emitted.set(ordinal, offset + 1);
+ return { sequence: ordinal * EVENT_SEQUENCE_STRIDE + offset, message };
+ });
+ };
+
+ const readTurns = async (
+ sessionId: string,
+ request: { direction: 'older' | 'newer'; throughOrdinal: number; position: number },
+ ): Promise<RuntimeTranscriptInvocation[]> =>
+ store.readTranscriptInvocations(sessionId, {
+ ...request,
+ limit: TRANSCRIPT_TURN_SCAN_LIMIT,
+ maxEvents: DURABLE_TRANSCRIPT_TURN_MAX_EVENTS,
+ maxBytes: DURABLE_TRANSCRIPT_TURN_MAX_BYTES,
+ });
+
+ const scan = async function* (
+ sessionId: string,
+ request: {
+ direction: 'older' | 'newer';
+ throughSequence?: number | null;
+ position?: number;
+ /** Stops the walk after this many Turns, for a read that may find nothing. */
+ maxTurns?: number;
+ },
+ ): AsyncGenerator<{ sequence: number; message: StoredMessage }> {
+ const throughSequence =
+ request.throughSequence === undefined ? await highWater(sessionId) : request.throughSequence;
+ if (throughSequence === null) return;
+ const position = request.position ?? (request.direction === 'older' ? throughSequence : 0);
+ const throughOrdinal = ordinalOf(throughSequence);
+ let ordinal = ordinalOf(position);
+ let walked = 0;
+ while (ordinal >= 0 && ordinal <= throughOrdinal) {
+ const turns = await readTurns(sessionId, {
+ direction: request.direction,
+ throughOrdinal,
+ position: ordinal,
+ });
+ if (turns.length === 0) return;
+ for (const turn of turns) {
+ if (request.maxTurns !== undefined && walked >= request.maxTurns) return;
+ walked += 1;
+ const records = (await projectTurn(turn)).filter(
+ ({ sequence }) =>
+ sequence <= throughSequence &&
+ (request.direction === 'older' ? sequence <= position : sequence >= position),
+ );
+ if (request.direction === 'older') records.reverse();
+ yield* records;
+ }
+ const edge = turns.at(-1)!;
+ ordinal = request.direction === 'older' ? edge.firstOrdinal - 1 : edge.lastOrdinal + 1;
Review Comment:
Verified and fixed — forward in `3e8aad4b0`, and the rest in `e2f304fce`.
Your account of the mechanism is exact: the step to `lastOrdinal + 1` / `firstOrdinal - 1` is only valid while Turn ranges are disjoint, and the spine was chosen precisely because the runtime does not promise that. Taking your reported shape as the fixture, the forward walk returned A only.
**The repair is not the one you proposed, and the reason is what the next round found.** Resuming from the next unread Turn in the spine removes the skip but leaves the rows interleaved, so a page boundary still has to exclude records it has not returned — which is the backward loss reported separately in #3950356962. The scan now drains overlapping Turns as one cluster and emits the cluster sorted by `sequence`, so the walk is monotone and the cursor steps past the cluster's edge rather than a single Turn's. Both faces of this close together; the details and the ablation are on that thread.
**On the severity gap you named:** it stays honest — I still have no demonstration that a supported flow through `SessionManager` writes an interleaved ledger. The fix was worth making anyway, because the spine's premise is that interleaving is permitted, and a caller stepping as if it is not was the diff contradicting itself.
<details><summary>简体中文</summary>
已核实并修复——前向在 `3e8aad4b0`，其余在 `e2f304fce`。
你对机制的描述是准的：步进到 `lastOrdinal + 1` / `firstOrdinal - 1` 只在各 Turn 区间互不包含时有效，而选用 spine 的理由恰恰是 runtime 不作此保证。用你报告的形状做夹具，前向遍历确实只返回了 A。
**修法不是你提的那个，理由是下一轮查出来的东西。** 「从 spine 中下一个未读 Turn 续读」能去掉跨过，但**行仍然是交错的**，于是页边界仍然要排除尚未返回的记录——那正是 #3950356962 单独报告的向后丢行。现在扫描把重叠的 Turn 作为一个**簇**排空，并按 `sequence` 排序后产出，于是遍历是单调的，游标越过的是**簇的边界**而不是单个 Turn 的边界。这个缺陷的两个面一并关掉；细节与消融实验在那条线程上。
**关于你自己点出的定级缺口：** 它仍然成立——我至今没有「某条经 `SessionManager` 的受支持流程会写出交错账本」的实证。但这个修复仍然值得做：**spine 的前提就是「允许交错」，而调用方却按「不会交错」在步进——那是这份 diff 自相矛盾。**
</details>
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
packages/runtime/src/runtime-event-read-model.ts:
##########
@@ -1197,6 +1248,7 @@ function projectTerminalTurnState(
...(status === 'failed' && event.content?.kind === 'error' && event.content.retry
? { retry: event.content.retry }
: {}),
+ partialOutputRetained,
Review Comment:
Fixed in `8e20e6d5a`, by taking the option your own comment argued for and stopping short of the repair you proposed.
**The disagreement was real and I reproduced it** — the bounded reader returning the stored bit while the full reader ORs it with visible rows, on the same terminal `turn_state`. Your framing of the cause is the one I used: the merge kept a write and lost the move that gave the fact an owner, leaving two producers and none.
**Why I deleted the field instead of carrying it on the contribution.** Your minimal repair restores a single owner, but it keeps a persisted, wire-carried fact that — as you say in the same comment — has no consumer, because `#4951` removed the renderer. Carrying it correctly costs a contribution field, a fold, a projection, a legacy fallback and a parity test, all to keep one boolean nobody reads. So the field is gone from `TurnRecord`, both derivations, the projector write, the wire, shared pages, Desktop and the UI. Two disagreeing derivations become zero, and the parity test you asked for has nothing left to guard.
**What this gives up:** a future renderer wanting "was output retained?" must re-derive it, and for Turns written before this change it can only do so from the rows themselves. That is the same information the full reader was already computing, so nothing that existed on disk becomes unanswerable.
**The cost of the deletion, disclosed rather than discovered:** removing the key from the exact shape made released rows carrying it undecodable — reported as #3950436460 and fixed at the shape helper in `3a4371a74`, which now distinguishes what a shape accepts from what it emits. Epoch bumped to 129 for the wire half.
<details><summary>简体中文</summary>
已在 `8e20e6d5a` 修复——采用的是**你自己的评论所论证的那个选项**，而没有走你提出的那个修法。
**分歧是真的，我复现了** —— 同一条终态 `turn_state` 上，有界读原样返回存储位，全量读把它与可见行取或。成因我也用了你的说法：合并保留了一次写入、却丢掉了那次「让该事实有归属」的搬迁，于是两个产出方、零个归属方。
**为什么我删掉该字段，而不是把它带到 contribution 上。** 你的最小修法确实恢复了单一归属，但它保留的是一个**被持久化、还上了协议线**的事实——而正如你在同一条评论里说的，**它没有消费者**，因为 `#4951` 已经删掉了渲染层。要把它正确地带着走，代价是一个 contribution 字段、一次 fold、一次 project、一个旧数据兜底、外加一条一致性测试——全都是为了留住一个没人读的布尔量。所以这个字段从 `TurnRecord`、两处派生、projector 的写入、协议线、共享页、Desktop 与 UI 里一并消失。**两个互相不一致的派生变成零个**，而你要的那条一致性测试也就没有东西需要看守了。
**放弃了什么：** 将来若有渲染层想问「输出是否保留」，必须重新派生；对本次改动之前写下的 Turn，它只能从行本身派生。而那正是全量读原本就在算的同一份信息，所以**磁盘上已有的东西没有变得不可回答**。
**这次删除的代价，主动交代而不是等被发现：** 把该键从 exact shape 里移除，使携带它的已发布行无法解码——即 #3950436460，已在 `3a4371a74` 于形状定义处修复，那里现在把「一个形状接受什么」与「它发出什么」区分开了。协议线那一半 epoch 抬到 129。
</details>
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
packages/storage/src/sqlite-runtime-store.ts:
##########
@@ -1455,6 +1492,58 @@ export class SqliteRuntimeStore
});
}
+ async resequenceSessionEventOrdinals(sessionId: string): Promise<void> {
+ assertRuntimeStorageSafeId(sessionId, 'Invalid session id');
+ this.transaction(() => {
+ // Lifted above the range first: the second statement renumbers into the
+ // space these rows occupy, and (session_id, ordinal) is a primary key.
+ // Shifting up rather than below zero keeps every intermediate value
+ // inside the table's own `ordinal > 0`, and lands them past the 1..N the
+ // renumber assigns, since the count cannot exceed the maximum.
+ const { shift } = this.db
+ .prepare(`
+ SELECT COALESCE(MAX(ordinal), 0) AS shift
+ FROM runtime_session_event_ordinals
+ WHERE session_id = ?
+ `)
+ .get(sessionId) as { shift: number };
+ this.db
+ .prepare(`
+ UPDATE runtime_session_event_ordinals
+ SET ordinal = ordinal + :shift
+ WHERE session_id = :sessionId
+ `)
+ .run({ sessionId, shift });
+ this.db
+ .prepare(`
+ WITH opening AS (
+ SELECT invocation_id, CAST(json_extract(payload_json, '$.ts') AS INTEGER) AS opened_at
+ FROM runtime_events
+ WHERE session_id = :sessionId AND event_kind = 'invocation_opened'
+ ),
+ ordered AS (
+ SELECT
+ o.event_id AS event_id,
+ ROW_NUMBER() OVER (
+ ORDER BY COALESCE(opening.opened_at, e.committed_at), e.invocation_id, o.ordinal
+ ) AS ordinal
+ FROM runtime_session_event_ordinals o
+ JOIN runtime_events e ON e.event_id = o.event_id
+ LEFT JOIN opening ON opening.invocation_id = e.invocation_id
+ WHERE o.session_id = :sessionId
+ )
+ UPDATE runtime_session_event_ordinals
+ SET ordinal = (
Review Comment:
Fixed in `94c24b082` — `ordered AS MATERIALIZED`, exactly the one-word remedy you measured.
**Reproduced independently before taking it.** SQLite treats an un-materialized CTE as a view and re-evaluates the whole `ROW_NUMBER()` per correlated lookup, so the renumber costs the Session squared: 5.2 s at 500 rounds and 22.9 s at 1000 on my fixture, against 84 ms and 357 ms materialized. Same rows, same ordinals, same order — it is a planner hint, not a semantic change.
**Your placement argument is what made this worth a commit of its own.** The cost sits inside `ensureTranscriptLedger`, which every durable read entry point awaits before returning a reader, inside one synchronous transaction — so it is not a background repair, it is the first open after upgrade. That is also why I did not reach for a chunked or resumable renumber: the quadratic term is the whole problem, and once it is linear the one-time pass at realistic Session sizes is no longer worth splitting.
Noted too that the paging numbers you measured on this head are flat where `86f1d760` grew. That is the property the ordinal work was for.
<details><summary>简体中文</summary>
已在 `94c24b082` 修复——`ordered AS MATERIALIZED`，正是你实测的那个「一个关键字」的修法。
**采纳之前独立复现过。** SQLite 默认把未物化的 CTE 当作视图，每次相关子查询求值都重算整个 `ROW_NUMBER()`，于是重排对 Session 是平方级的：我的夹具上 500 轮 5.2 秒、1000 轮 22.9 秒，物化后分别是 84 ms 与 357 ms。**同样的行、同样的序号、同样的顺序**——这是计划层提示，不是语义变更。
**让它值得单独一个 commit 的，是你对位置的论证。** 这笔开销落在 `ensureTranscriptLedger` 内，而每个持久读入口在返回读取者之前都会 await 它，且处在一个同步事务中——所以它不是后台修复，**它就是升级后的第一次打开**。这也是我没有去做分块或可续跑重排的原因：**平方项本身就是全部问题**；一旦变成线性，现实规模下的一次性扫描不再值得拆开。
也记下你在本 head 上测到的分页数字：在 `86f1d760` 上是增长的，现在是平的。那正是这批 ordinal 工作要的性质。
</details>
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] refactor(runtime): derive Session transcripts from RuntimeEvents [maka]
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener commented on code in PR #4879:
URL: https://github.com/apache/maka/pull/4879#discussion_r3946630622
##########
packages/storage/src/runtime-transcript-query.ts:
##########
@@ -0,0 +1,278 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import type { DatabaseSync } from 'node:sqlite';
+import { decodeRuntimeEvent, type RuntimeEvent } from '@maka/core/runtime-event';
+import type { RuntimeInvocationRecord } from '@maka/core/runtime-invocation';
+
+/** SQL counterpart of isTerminalRuntimeEvent; shared with the ledger store. */
+export const TERMINAL_RUNTIME_EVENT_SQL = `(
+ json_extract(payload_json, '$.actions.endInvocation') = 1
+ OR json_extract(payload_json, '$.status') IN ('completed', 'failed', 'aborted', 'cancelled')
+)`;
+
+/**
+ * One invocation's events, in ledger order, carrying the Session ordinal each
+ * one sits at.
+ *
+ * The transcript rows of a Turn come from projecting these together: what a
+ * RuntimeEvent becomes is decided by the read model alone, so nothing here
+ * classifies an event or decides whether it produces a row.
+ */
+export interface RuntimeTranscriptInvocation {
+ readonly invocation: RuntimeInvocationRecord;
+ readonly firstOrdinal: number;
+ readonly lastOrdinal: number;
+ readonly events: readonly { readonly ordinal: number; readonly event: RuntimeEvent }[];
+}
+
+/** An invocation start, with the prompt event a landmark is labelled by. */
+export interface RuntimeTranscriptLandmark {
+ readonly invocation: RuntimeInvocationRecord;
+ readonly firstOrdinal: number;
+ readonly prompt?: { readonly ordinal: number; readonly event: RuntimeEvent };
+}
+
+export interface RuntimeTranscriptInvocationRequest {
+ readonly direction: 'older' | 'newer';
+ readonly throughOrdinal: number;
+ /** Ordinal the walk starts from, inclusive, in `direction`. */
+ readonly position: number;
+ readonly limit: number;
+ /** Refused rather than truncated: half a Turn projects to a wrong transcript. */
+ readonly maxEvents: number;
+ readonly maxBytes: number;
+}
+
+export interface RuntimeTranscriptQueries {
+ readTranscriptHighWater(sessionId: string): Promise<number | null>;
+ readTranscriptInvocations(
+ sessionId: string,
+ request: RuntimeTranscriptInvocationRequest,
+ ): Promise<RuntimeTranscriptInvocation[]>;
+ readTranscriptLandmarks(
+ sessionId: string,
+ throughOrdinal: number,
+ limit: number,
+ ): Promise<RuntimeTranscriptLandmark[]>;
+}
+
+export class RuntimeTranscriptOversizedTurnError extends Error {
+ readonly name = 'RuntimeTranscriptOversizedTurnError';
+}
+
+const joins = `
+ FROM runtime_session_event_ordinals o
+ JOIN runtime_events e ON e.event_id = o.event_id
+ LEFT JOIN runtime_events opened ON opened.invocation_id = e.invocation_id AND opened.event_kind = 'invocation_opened'
+ LEFT JOIN runtime_legacy_invocation_openings legacy ON legacy.invocation_id = e.invocation_id`;
+const opening = `COALESCE(json_extract(opened.payload_json, '$.content'), legacy.opening_json)`;
+/**
+ * A Turn the Session transcript shows: one this Session ran itself rather than
+ * on behalf of a subagent, and one that has already ended.
+ *
+ * This is a fact about the invocation, not about any row it produces — which
+ * rows it produces is the read model's question, and is not asked here.
+ */
+const settledInline = `
+ ${opening} IS NOT NULL
+ AND (json_extract(${opening}, '$.lineage.parentRunId') IS NULL
+ OR (json_extract(${opening}, '$.source.kind') = 'continuation'
+ AND json_extract(${opening}, '$.lineage.agentId') IS NULL))
+ AND EXISTS (
+ SELECT 1 FROM runtime_events ended
+ JOIN runtime_session_event_ordinals ending ON ending.event_id = ended.event_id
+ WHERE ended.invocation_id = e.invocation_id
+ AND ${TERMINAL_RUNTIME_EVENT_SQL.replaceAll('payload_json', 'ended.payload_json')}
+ AND ending.ordinal <= :throughOrdinal
+ )`;
+
+type InvocationRow = { invocation_id: string; run_id: string; first: number; last: number };
+
+/** Selects invocations by Session ordinal. Payloads are decoded, never classified. */
+export class RuntimeTranscriptQuery {
+ constructor(
+ private readonly db: DatabaseSync,
+ private readonly invocation: (sessionId: string, runId: string) => RuntimeInvocationRecord,
+ ) {}
+
+ highWater(sessionId: string): number | null {
+ const row = this.db
+ .prepare(`
+ SELECT MAX(o.ordinal) AS ordinal ${joins}
+ WHERE o.session_id = :sessionId AND o.ordinal <= :throughOrdinal AND ${settledInline}
+ `)
+ .get({ sessionId, throughOrdinal: Number.MAX_SAFE_INTEGER }) as { ordinal?: unknown };
+ return typeof row.ordinal === 'number' ? row.ordinal : null;
+ }
+
+ invocations(
+ sessionId: string,
+ request: RuntimeTranscriptInvocationRequest,
+ ): RuntimeTranscriptInvocation[] {
+ assertOrdinal(request.throughOrdinal);
+ assertOrdinal(request.position);
+ if (request.direction !== 'older' && request.direction !== 'newer') {
+ throw new Error('Invalid transcript direction');
+ }
+ // An invocation is selected by where its own events sit, so a walk that
+ // starts inside a Turn still finds that Turn and can serve its rows.
+ const rows = this.db
+ .prepare(`
+ SELECT e.invocation_id, e.run_id, MIN(o.ordinal) AS first, MAX(o.ordinal) AS last ${joins}
+ WHERE o.session_id = :sessionId AND o.ordinal <= :throughOrdinal AND ${settledInline}
+ GROUP BY e.invocation_id
+ HAVING ${request.direction === 'older' ? 'first <= :position' : 'last >= :position'}
+ ORDER BY first ${request.direction === 'older' ? 'DESC' : 'ASC'}
+ LIMIT :limit
+ `)
+ .all({
+ sessionId,
+ throughOrdinal: request.throughOrdinal,
+ position: request.position,
+ limit: request.limit,
+ }) as InvocationRow[];
+ return rows.map((row) => ({
+ invocation: this.invocation(sessionId, row.run_id),
+ firstOrdinal: row.first,
+ lastOrdinal: row.last,
+ events: this.events(row.invocation_id, request),
+ }));
+ }
+
+ landmarks(sessionId: string, throughOrdinal: number, limit: number): RuntimeTranscriptLandmark[] {
+ assertOrdinal(throughOrdinal);
+ if (limit < 1) return [];
+ // Evenly spaced Turn starts, chosen before any payload is read.
+ const rows = this.db
+ .prepare(`
+ WITH candidates AS (
+ SELECT e.invocation_id, e.run_id, o.ordinal,
+ ROW_NUMBER() OVER (ORDER BY o.ordinal) - 1 AS rank, COUNT(*) OVER () AS total
+ ${joins} WHERE o.session_id = :sessionId AND o.ordinal <= :throughOrdinal
+ AND e.event_seq = 1 AND ${settledInline}
+ ), samples(n) AS (
+ SELECT 0 UNION ALL SELECT n + 1 FROM samples WHERE n + 1 < :limit
+ )
+ SELECT DISTINCT invocation_id, run_id, ordinal FROM candidates
+ JOIN samples ON rank = CASE WHEN :limit = 1 THEN total - 1
+ ELSE CAST(n * (total - 1) / (:limit - 1) AS INTEGER) END
+ ORDER BY ordinal
+ `)
+ .all({ sessionId, throughOrdinal, limit }) as Array<{
+ invocation_id: string;
+ run_id: string;
+ ordinal: number;
+ }>;
+ return rows.map((row) => {
+ // The prompt is the Turn's first user text event, which is what the read
+ // model projects a user message from. Only that one event is loaded: a
+ // landmark is a label, and projecting whole Turns to build a scrollbar
+ // would read most of the Session.
+ const prompt = this.db
+ .prepare(`
+ SELECT o.ordinal, e.event_id FROM runtime_events e
+ JOIN runtime_session_event_ordinals o ON o.event_id = e.event_id
+ WHERE e.invocation_id = ? AND o.ordinal <= ?
+ AND e.event_kind = 'text' AND json_extract(e.payload_json, '$.role') = 'user'
+ ORDER BY e.event_seq LIMIT 1
+ `)
+ .get(row.invocation_id, throughOrdinal) as
+ | { ordinal: number; event_id: string }
+ | undefined;
+ return {
+ invocation: this.invocation(sessionId, row.run_id),
+ firstOrdinal: row.ordinal,
+ ...(prompt
+ ? { prompt: { ordinal: prompt.ordinal, event: this.event(prompt.event_id) } }
+ : {}),
+ };
+ });
+ }
+
+ private events(
+ invocationId: string,
+ limits: { maxEvents: number; maxBytes: number },
+ ): RuntimeTranscriptInvocation['events'] {
+ // Walked row by row: the limits cap what one Turn may pull into memory, so
+ // a check after `.all()` has already paid the cost it was meant to refuse.
+ const cursor = this.db
+ .prepare(`
+ SELECT o.ordinal, e.event_id, e.session_id, e.invocation_id, e.run_id, e.turn_id, e.payload_json
+ FROM runtime_events e JOIN runtime_session_event_ordinals o ON o.event_id = e.event_id
+ WHERE e.invocation_id = ? ORDER BY e.event_seq
+ `)
+ .iterate(invocationId) as Iterable<StoredEventRow & { ordinal: number }>;
+ const events: Array<RuntimeTranscriptInvocation['events'][number]> = [];
+ let bytes = 0;
+ for (const row of cursor) {
+ if (events.length === limits.maxEvents) {
+ throw new RuntimeTranscriptOversizedTurnError(
+ `Turn ${invocationId} holds more RuntimeEvents than a transcript page may read`,
+ );
+ }
+ bytes += Buffer.byteLength(row.payload_json);
Review Comment:
[P3] A single row is still materialized before it is measured.
`.iterate()` fixes the case that was measured — 10 rows at ~2 MiB refusing only after 20,973,944 bytes were resident. What it does not cover is the second half of the request in the review that prompted it: *"不能只把 `.all()` 改成一个仍会先物化任意大单条记录的循环"*. The driver hands back a fully-decoded `payload_json` string, and `Buffer.byteLength` measures it afterwards, so one row over budget is still paid before it is refused.
**Naming the producer before grading it**, because a bound that no writer can exceed is not a live exposure. The largest routine producer is capped upstream: `tool-output.ts:35` truncates retained output at 50 KB by default. So the residual is one row of that order, not an arbitrary one, and the `N × size → 1 × size` reduction is the substance of the fix.
That is why this is [P3] and not a re-opening of finding 3. If it is ever worth closing, `octet_length(payload_json)` in the projection would let the budget refuse before the string crosses into JS at all.
**The rest of this hunk I checked and it holds:** the refusal boundary is unchanged (`maxEvents` admitted, `maxEvents + 1` refused, matching the old `rows.length > maxEvents`), `Buffer.byteLength` is the UTF-8 fix for the CJK-inflating `.length`, and `.iterate()` is not a new dependency — `sqlite-runtime-store.ts:1036` and `sqlite-session-metadata-store.ts:2194` already use it under the same `node >=22.19.0` engine.
<details><summary>简体中文</summary>
**[P3] 单条记录仍然是先物化、后测量。**
`.iterate()` 确实修掉了被实测到的那个场景 —— 10 条约 2 MiB、在已驻留 20,973,944 字节之后才拒绝。**没覆盖的是提出该问题那条评审里的后半句要求**:「不能只把 `.all()` 改成一个仍会先物化任意大单条记录的循环」。驱动交回的是**已完整解码**的 `payload_json` 字符串,`Buffer.byteLength` 是在那之后才测的,所以**超预算的那一条,仍然是先付掉再拒绝**。
**先指认产出方再定级** —— 没有写入方能越过的界限不构成现实暴露。最大的常规产出方在上游有帽:`tool-output.ts:35` 默认把保留输出截到 50 KB。所以残留量级是**一条这个量级的记录,而不是任意大的一条**,而 `N × size → 1 × size` 才是本次修复的实质。
这就是它被定为 [P3]、而非重开 finding 3 的原因。真要收掉,在投影里用 `octet_length(payload_json)`,预算就能在字符串跨进 JS 之前拒绝。
**这一段的其余部分我核过,成立:** 拒绝边界未变(允许 `maxEvents`、拒绝第 `maxEvents + 1` 条,与旧的 `rows.length > maxEvents` 一致);`Buffer.byteLength` 正是针对 CJK 会放大的 `.length` 的修法;`.iterate()` 也不是新引入的依赖 —— `sqlite-runtime-store.ts:1036` 与 `sqlite-session-metadata-store.ts:2194` 在同一个 `node >=22.19.0` engine 下已经在用。
</details>
---
> **Automated review notice:** This comment was posted by an automated review agent operated by jackwener. It is not an independent human review and does not replace one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org