[I] History compaction fails open with source_hash_mismatch whenever a stale tool-result transition covers the prefix [maka]

["me2seeks (via GitHub)" <gi...@apache.org>]

me2seeks opened a new issue, #4842:
URL: https://github.com/apache/maka/issues/4842

   ## What happened
   
   A history-compact checkpoint created by the standalone compaction path (Desktop "Compact" action / `RuntimeKernel.compactSession`) is **never used by the next turn** whenever the session has any durable model-projection transition (stale Tool Result archive — enabled by default) inside the covered prefix. The next turn's replay fails open with `source_hash_mismatch` and sends the **full** uncompacted history to the provider, so the user sees "Context compacted." while the context usage stays pinned at ~100% and every request keeps paying the full-history cost.
   
   The failure is silent in the UI until the turn settles (the `context_compaction_failed_open` note is only appended together with `token_usage` at settlement), so from the user's perspective compaction "succeeded" but did nothing.
   
   ## Root cause
   
   The checkpoint's `coverage.sourceDigest` is pinned over **raw** RuntimeEvents at creation, but the pre-turn replay matches it against **folded** events (after `reduceEffectiveModelProjections`):
   
   - Creation (`AiSdkCompaction.compactHistory`, and the mid-turn state's `priorContentEvents` from `buildMidTurnCapacityCompactState`) reads `begin.runtimeContext` / `input.runtimeContext` — the raw ledger, no transition fold. For an archived `function_response`, `effectiveDigestEvent` therefore hashes the raw `result` field.
   - Pre-turn replay (`buildPriorMessages` → `prepareContextBudgetPolicy`) folds the durable transitions first, so the same events carry `modelProjection` placeholders and `effectiveDigestEvent` hashes those instead → `matchHistoryCompactCheckpointPrefix` returns `source_hash_mismatch` → `applyRuntimeEventHistoryCompact` fails open → full replay.
   
   Verified against a live session's durable ledger (session `58da2f70-…`, checkpoint `hcheckpoint-eb286323…`, 863 covered events):
   
   - digest over the raw prefix → **matches** the checkpoint exactly
   - digest over the folded prefix (6 pre-existing stale-result transitions, 6 covered events rewritten) → `source_hash_mismatch`
   - the turn's own `model_stream_started` diagnostic recorded `compactionDecisions: [{ stage: 'priorReplay', decision: 'failedOpen', failOpenReason: 'source_hash_mismatch' }]` with `droppedEvents: 0`, and the provider then reported 268,212 tokens used against a 262,144 window.
   
   The same mismatch also kills **mid-turn** checkpoints at the next turn boundary: the mid-turn create/match pair is self-consistent over raw events, but the next turn's pre-turn replay matches over folded events.
   
   ## How to reproduce
   
   1. In any session, produce a tool result larger than 2,048 estimated tokens and let at least two more turns pass, so the default `staleToolResultPrune` commits a durable archive transition for it.
   2. Run manual context compaction (Desktop compact action). It succeeds and records a checkpoint ("Context compacted.").
   3. Send another message.
   4. Observe: the request still carries the full history (ctx indicator unchanged), and once the turn settles the transcript carries `context_compaction_failed_open`. The `token_usage.contextBudget.compactionDecisions` shows `decision: 'failedOpen', failOpenReason: 'source_hash_mismatch'`.
   
   ## Environment
   
   - Maka commit: c12205b8a0 (main)
   - Surface: Desktop / Runtime (ai-sdk backend)
   
   ## Proposed fix
   
   Make the two views agree. The least invasive, fleet-compatible option: **match the checkpoint against the raw durable prefix at pre-turn replay, then fold the projected `[block, tail]` through `reduceEffectiveModelProjections` before building messages** (mirroring what the mid-turn path already does at `ai-sdk-turn.ts` for the current-turn projection). Rationale:
   
   - every existing checkpoint in the field was hashed over raw events (both creation paths), so raw-side matching revives them instead of orphaning them;
   - raw ledger rows are immutable, so the pinned digest can no longer drift when a later transition commits;
   - the fold still runs before materialization, so the "model never sees content a transition removed" invariant (#4283) is preserved.
   
   Alternative (heavier): fold at creation in both `compactHistory` and `buildMidTurnCapacityCompactState` and keep folded matching — this matches the v2 policy comment's wording but leaves every existing raw-hashed checkpoint dead and requires the sync mid-turn state builder to become async.
   
   I can take this. Related but distinct: #4634 (fail-open from summarizer route), #3029/#3041 (summary content validation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org

Re: [I] History compaction fails open with source_hash_mismatch whenever a stale tool-result transition covers the prefix [maka]

["me2seeks (via GitHub)" <gi...@apache.org>]

me2seeks closed issue #4842: History compaction fails open with source_hash_mismatch whenever a stale tool-result transition covers the prefix
URL: https://github.com/apache/maka/issues/4842


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org