[PR] feat(docs): make mode token costs traceable and reproducible [magpie]

["andreahlert (via GitHub)" <gi...@apache.org>]

andreahlert opened a new pull request, #1201:
URL: https://github.com/apache/magpie/pull/1201

   ## Summary
   
   The mode-economics page mixes planning estimates with numbers whose source and freshness are hard to verify. This change makes file counts reproducible and makes the limits of runtime evidence explicit.
   
   - Add a deterministic `skill-token-count` tool using pinned `tiktoken` / `cl100k_base`, with a generated table covering all 74 local skill entrypoints, a UTC measurement date, and content hashes. A path-filtered workflow and prek check reject stale counts. Tokenizer vocabulary preparation is explicit and checksum-verified; ordinary checks run offline.
   - Separate file measurements, bounded runtime samples, and unvalidated planning estimates. Publish 30 actual Claude Code calls across five skills and four modes, with per-skill p50/p90, worked examples, captured inputs, responses, and usage records. The optional collector validates mode labels before paid calls and never runs in CI.
   - Put measured replay p50 beside the existing estimates and explain discrepancies. Remove unsupported price multipliers, clarify Pairing reimbursement policy, and collapse the inactive mode section while preserving both legacy anchors.
   
   ## Type of change
   
   - [x] Python package
   - [x] Documentation
   - [x] CI / dev loop
   
   ## Test plan
   
   - [x] `prek run --all-files`, including workspace lint, typing, tests, validators, and documentation checks
   - [x] 32 tests in `tools/skill-token-count`, covering drift, offline tokenizer behavior, usage accounting, percentile calculation, mode validation, and published records/reports
   - [x] Verified all 30 recorded prompt hashes and source hashes against their captured corpus and current skill sources
   - [x] 30 opt-in model calls completed using Claude Code 2.1.268 and `claude-haiku-4-5-20251001`
   
   Model-call completion is not a correctness grade. The replays use synthetic inputs and captured observations, with live tools disabled. Patch-drafting samples do not measure applying patches or running tests. One generated draft claimed green tests without execution; that limitation is documented and is not treated as validation.
   
   ## RFC-AI-0004 compliance
   
   - [x] Vendor neutrality: backend-specific tokenization and CLI usage are isolated in the tool; no skill behavior changes
   - [x] Write-access discipline: benchmarks perform no external publication or repository mutations
   - [x] Privacy: benchmark inputs and outputs concern synthetic public-project scenarios
   
   ## Linked issues
   
   Closes [#327](https://github.com/apache/magpie/issues/327).
   
   ## Notes for reviewers
   
   Content hashes identify exact measurement inputs instead of embedding the current Git commit in a table that would change on every commit. The date is preserved until input drift requires regeneration. Git history supplies the publication revision.
   
   Unsupported cross-class price multipliers are removed rather than replaced with another dated vendor comparison. Apply current provider input/output/cache rates when budgeting.
   
   The initial sample mislabeled `good-first-issue-author` as Drafting. Its classification was corrected to Mentoring, preserving usage, timestamps, prompts, and responses; the original corpus hash is retained in correction metadata. Six additional calls measure `issue-fix-workflow` patch drafting. Samples are grouped by skill and do not establish typical costs for complete production workflows or multi-agent pipelines.
   
   The affected quality-tooling spec is updated. Local agent artifacts are excluded through `.gitignore`.
   
   ## AI assistance disclosure
   
   This change was developed with AI assistance from OpenAI Codex.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@magpie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(docs): make mode token costs traceable and reproducible [magpie]

["potiuk (via GitHub)" <gi...@apache.org>]

potiuk merged PR #1201:
URL: https://github.com/apache/magpie/pull/1201


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@magpie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org