[I] Optimize skill token cost across all families (setup-family recipe) [magpie]

["potiuk (via GitHub)" <gi...@apache.org>]

potiuk opened a new issue, #1342:
URL: https://github.com/apache/magpie/issues/1342

   Make every skill family leaner without changing what its skills do, using the recipe first applied to the `setup` family (#SETUP, PR #PR).
   
   ## Why
   
   Every skill costs tokens twice:
   
   - **Always-on** — `description` + `when_to_use` are loaded in *every* session for *every* installed skill, whether it runs or not. Budget: **200 tokens** per skill.
   - **Body** — `SKILL.md` is loaded on every invocation. Budget: **5,000 tokens**, pre-flight block excluded; `PRINCIPLES.md` P14's 500-line cap is the structural limit.
   
   Both budgets come from [`optimize-skill`](https://github.com/apache/magpie/blob/main/plugins/magpie-utilities/skills/optimize-skill/SKILL.md), which is also the tool for doing this work.
   
   ## Recipe (from the setup family)
   
   Run `/magpie-optimize-skill` per skill and apply the passes in this order:
   
   1. **Measure first.** `uv run --project tools/skill-token-count skill-token-count --write`, plus a chars ÷ 4 estimate of the frontmatter. Record before/after for both budgets.
   2. **Trim the always-on frontmatter** of any skill over 200 tokens. Drop lists that `argument-hint` or the body already carries; keep the routing phrases a model needs to pick the skill. Highest value per token, because it is paid in every session.
   3. **Split** (only for bodies over budget, or rules that serve one sub-action). Move such rules into the sibling file that uses them, **byte-for-byte**, leaving a one-line pointer. Keep a rule in `SKILL.md` if many places cite it. Repoint every `#anchor` reference to what moved.
   4. **Wording pass.** One sentence per line (SemBr); cut sentences that restate a nearby table or section; plain verbs; no hedges. Keep bold only on real prohibitions.
   5. **Fix stale pointers found on the way** (links to renamed files, "above" that is below, wrong counts). List anything eval-coupled that you did *not* fix in the PR description.
   
   ## Invariants — a PR that breaks one is not an optimization
   
   - Headings (`#`–`####`) and `**Golden rule N — …**` headlines stay **byte-identical**. They feed `surface_hash` (changing one tells every adopter their configuration is stale) and eval `step_heading` matching.
   - Frontmatter other than `description` / `when_to_use`, the generated pre-flight block, and every fenced code block stay byte-identical.
   - Every link target, command, flag, path, placeholder and condition survives. Check it mechanically: compare headings, fences, links and inline code spans between `HEAD` and the working tree, with whitespace normalised.
   - **Eval-extracted sections** (`step_heading` in `tools/skill-evals/evals/<skill>/*/step-config.json`): tighten only; every string that fixtures or `expected.json` rely on stays byte-identical.
   - The validator, `prek run --all-files` and `lychee` are green, and `docs/mode-economics.md` plus any family token figure in `docs/setup/marketplace.md` are regenerated.
   - **Run the eval suites before and after.** The harness launches the `claude` CLI, which fails inside the sandbox, so run it outside, or every case errors and nothing is graded.
   
   ## What to expect
   
   On setup, the family with no skill over either budget, the result was −2,709 body tokens across 7 skills (−8.5%) and the one frontmatter over budget cut from ~228 to ~167 tokens. The big win was `setup/SKILL.md` (−16%), where rules were moved as well as reworded; pure wording passes gave 4–9%. Families with skills far over budget (security, release-management, pr-management, issue) should gain much more from the split pass than setup did.
   
   ## Sub-issues
   
   One per family; each lists its skills with both budgets and marks the ones exceeded. Suggested priority: always-on first (it is paid by every session), then the largest bodies.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@magpie.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org