You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@maka.apache.org by "Sun-GLiang (via GitHub)" <gi...@apache.org> on 2026/08/30 11:11:08 UTC
[PR] feat(storage): add semantic transcript position snapshots [maka]
Sun-GLiang opened a new pull request, #4258:
URL: https://github.com/apache/maka/pull/4258
## Issue and approved plan
- Issue: https://github.com/apache/maka/issues/4123
- Plan: `issue-4123-four-pr-plan-v1.1-controller-approved-2026-08-30`
- Exact base: `8c491e64b35cd7af66437efc8020a88b171e2bb6`
- PR 1 of 4
## Scope
Adds the Storage-only semantic transcript position index and its single durable seam for later Runtime Host work:
- additive, lazy v35 normalized Turn membership/order/anchor metadata;
- persisted bounded scalar-identity recovery over existing inline/chunk physical storage;
- exact snapshot identity `{ throughSequence, authorityRevision, snapshotGeneration }`;
- atomic named recovery/build/ready phases, two retained generations, lease ownership, typed capacity, and monotonic ABA protection;
- ordinal-keyset position paging for tail/ordinal/sequence/turn anchors;
- exact-snapshot by-Turn complete-record lookup with bounded preflight and the existing complete body decoder;
- owner-epoch orphan snapshot/lease reclamation while preserving recovery/index state and generation monotonicity.
The recovery seam validates physical bytes, UTF-8/JSON structure, and only the top-level identity envelope. It never returns or retains transcript bodies. Complete body validation remains owned by `decodeStoredMessage`.
## Storage schema and API
v35 adds:
- `session_turn_authority_revisions`
- `session_turn_index_state`
- `session_turn_identity_recovery`
- `session_turn_metadata`
- `session_turn_memberships`
- `session_turn_position_snapshots`
- `session_turn_snapshot_positions`
- `session_turn_snapshot_leases`
Ready snapshot rows are protected by database guards.
Storage-only APIs:
- `readTurnPositionPageSnapshot({ sessionId, snapshotLeaseId, snapshotKey?, throughSequence?, anchor, maxPositions })`
- `readTranscriptRecordsByTurnIdsSnapshot({ sessionId, snapshotLeaseId, snapshotKey, turnIds, maxBytes, maxRecords })`
- `releaseTurnPositionSnapshot(sessionId, snapshotLeaseId, snapshotKey)`
## Resource evidence
- Recovery step maximum proven at 4,194,304 source payload bytes and 1,024 completed source records.
- Boundary fixtures cover 4 MiB - 1, exactly 4 MiB, 4 MiB + 1, and a record well above 4 MiB, including identity after a huge body and monotonic byte-offset resume.
- Persisted scanner state is capped below 64 KiB, captured identity below 32 KiB, and nesting below 4,096.
- Position pages are capped at 128 entries and below 64 KiB encoded.
- By-Turn reads cap unique Turn IDs at 128, complete records at 256, and stored bytes at 16 MiB, with no partial return.
- A 10,000-Turn fixture asserts indexed/keyset plans for position and membership paths with no OFFSET, ROW_NUMBER, COUNT DISTINCT, temporary B-tree, or transcript-wide hot scan.
- Reopen tests cover partial recovery, owner-epoch key rejection, lease sharing/release, two-slot capacity, and monotonic release/reallocation ABA generations.
## Verification
- `npm --workspace @maka/storage test`: 1051 total, 1044 passed, 7 skipped, 0 failed
- `npm run format:check`: 1777 files checked, no fixes
- `npm run lint`: 2961 files checked, no fixes
- `npm run typecheck`: passed
- `npm test`: all workspace tests passed
- `npm run check:asf-headers`: passed
- `npm run check:third-party-notices`: passed
- `npm run check:cli-third-party-notices`: passed
- `git diff --check 8c491e64b35cd7af66437efc8020a88b171e2bb6...HEAD`: passed
- Independent Standards review: no blocking/high findings remaining
- Independent Spec review against v1.1: no blocking/high findings remaining
## Out of scope / deferred
No Runtime Host public operation or wire schema, signed cursors, subscriptions, Desktop replica/Preload/IPC/Renderer/UI behavior, scroll/navigation/export/compatibility changes, or payload-limit changes are included.
PR2 remains responsible for Runtime Host subscription/public cursor and protocol work; PR3 and PR4 remain unchanged. This PR must not be merged as part of this task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] feat(storage): add semantic transcript position snapshots [maka]
Posted by "Sun-GLiang (via GitHub)" <gi...@apache.org>.
Sun-GLiang commented on PR #4258:
URL: https://github.com/apache/maka/pull/4258#issuecomment-5468825313
I have converted #4258 back to draft and will keep it unmerged while developing PR2.
PR2 will remain within the approved four-PR checklist and will be built on the exact PR1 head. Once the production Runtime Host implementation consumes the new Storage APIs and the Storage–Runtime Host integration coverage is ready, I will request a composition review of both the PR2 layer and the combined `main...PR2` diff.
PR1 will not be marked ready or merged before that review. The composition review will be used to verify whether the current recovery, two-generation retention, lease, and persisted-hash machinery is required by the real Runtime Host lifecycle and protocol guarantees. Anything that cannot be justified by the production call path and corresponding tests will be simplified or deferred.
After the composition is approved, I will reshape and land the minimum coherent Storage + Runtime Host slice together, with the exact merge procedure agreed with the maintainers before PR1 is marked ready.
This stage validates the Storage–Runtime Host contract and provides the merge gate for PR1 and PR2. The remaining Desktop behavior and final end-to-end acceptance will still be verified in PR3 and PR4.
<details>
<summary>简体中文</summary>
我已把 #4258 改回 draft，并将在开发 PR2 期间保持其未合入状态。
PR2 将继续遵循已经批准的四 PR checklist，并基于 PR1 的精确 head 开发。真实 Runtime Host 实现开始消费新的 Storage API，且 Storage–Runtime Host 集成测试完成后，我会同时提交 PR2 层级 diff 和完整的 `main...PR2` diff 进行组合审查。
在这次审查完成前，PR1 不会被标记为 ready，也不会单独合入。组合审查将验证当前的 recovery、双 generation、lease 和持久化 hash 机制是否确实由 Runtime Host 的真实生命周期与协议保证所需要。无法通过生产调用链和对应测试证明必要性的机制将被简化或后移。
组合设计获得认可后，我会将最小且连贯的 Storage + Runtime Host 切片一起落地，并在 PR1 被标记为 ready 前与维护者确认具体合入方式。
这一阶段验证 Storage–Runtime Host 契约，并作为 PR1 和 PR2 的合入门槛。其余 Desktop 行为和最终端到端验收仍将在 PR3 和 PR4 中完成。
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] feat(storage): add semantic transcript position snapshots [maka]
Posted by "Sun-GLiang (via GitHub)" <gi...@apache.org>.
Sun-GLiang closed pull request #4258: feat(storage): add semantic transcript position snapshots
URL: https://github.com/apache/maka/pull/4258
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org