[PR] fix(runtime): measure Read pages and tool-result pruning in UTF-8 bytes [maka]

["Astro-Han (via GitHub)" <gi...@apache.org>]

Astro-Han opened a new pull request, #5676:
URL: https://github.com/apache/maka/pull/5676

   ## Summary
   
   Read pages and tool-result pruning share one 7,500 cap, but it was counted in UTF-16 characters. For English and code that is about 2K tokens. A Chinese character costs roughly three times as many tokens, so Chinese-heavy tool results reached about 3.3K tokens before being pruned, and Read pages grew by the same factor.
   
   This counts the same 7,500 limit in UTF-8 bytes (`READ_PAGE_MAX_CHARS` → `READ_PAGE_MAX_BYTES`). ASCII content is unaffected, since one character is one byte. Chinese returns to about 2K tokens. The page check, the prune threshold, and the placeholder and conversation-copy page budgets derived from the cap all use the same byte measure. The threshold value itself is intentionally unchanged.
   
   Basis, measured on 4,639 tool results from a local Maka session store:
   
   | Unit | Tokens at 7,500, Chinese ≥ 20% of text (DeepSeek tokenizer, median) | Code/English |
   | --- | --- | --- |
   | UTF-16 characters (before) | ~3,300 | ~2,100 |
   | UTF-8 bytes (after) | ~2,000 | ~2,100 |
   
   Offline token counts matched provider-reported `input_tokens` deltas between steps (DeepSeek 1.000×, GLM 1.000×). Pure Chinese prose triggers slightly earlier (~1,500 tokens), which errs toward pruning rather than letting large results through.
   
   ## Verification
   
   - Runtime, affected suites: `read-page`, `model-projection-transition-ledger`, `builtin-tools-file-worker`, `conversation-copy`, `ledger-tool-result-archive-reader`, `tool-result-archive-capability-backend`, `ai-sdk-backend`, `context-budget`, `context-budget-mid-turn-policy`, `mid-turn-capacity-backend`, `overflow-reactive-recovery`, `runtime-event-read-model` — 555/555 on a clean build.
   - `runtime-host` `execution-artifacts` 5/5; `cli` `pi-transcript` 113 pass, 0 fail.
   - With the checks temporarily reverted to character counting, the new Chinese prune-gate test and the existing Unicode/`界` page-bound tests (now asserting bytes) fail.
   - `npm run format`, `npm run lint` clean. Full-repo tests and E2E not run locally.
   
   ## AI use
   
   Select exactly one:
   
   - [ ] No generative tool made a substantive contribution
   - [x] Generative tooling made a substantive contribution
   
   Tool(s) and scope: Maka (claude-opus-5-5) — measurement, implementation, tests, and PR text.
   
   ## Checklist
   
   - [x] Tests cover the change and fail without it
   - [x] Lint, format, typecheck and the affected suites pass locally
   
   Does this PR entail a change in behavior?
   
   - [x] Yes — described under Summary above
   - [ ] No
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org

Re: [PR] fix(runtime): measure Read pages and tool-result pruning in UTF-8 bytes [maka]

["Astro-Han (via GitHub)" <gi...@apache.org>]

Astro-Han merged PR #5676:
URL: https://github.com/apache/maka/pull/5676


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org