You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@magpie.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2026/09/17 10:11:02 UTC

[I] Measure Magpie skill economics against LLMAO's three models [magpie]

potiuk opened a new issue, #1261:
URL: https://github.com/apache/magpie/issues/1261

   ## What
   
   Run the mode-economics harness against LLMAO's three models and record both **tokens** and **wall-clock time** per skill, so the published economics stop being single-vendor.
   
   Depends on #1260 confirming the gateway is usable — in particular whether tool use works, since every Magpie skill is tool-driven.
   
   ## Why
   
   [`docs/mode-economics.md`](https://magpie.apache.org/docs/mode-economics/) exists because affordability is a mission commitment, not a nice-to-have. Right now every figure on it comes from one vendor's models. A second, ASF-hosted backend turns "what does this cost" from a vendor-specific number into a comparison, and it is the evidence the page's own *Long-term: the ASF inference endpoint* section already promises:
   
   > As pilot adopters accumulate real usage data, this page will be updated with observed ranges rather than theoretical estimates, so the endpoint sizing argument rests on evidence.
   
   It also runs in the other direction. ASF Tooling have capacity-planned from synthetic load and a few hundred recorded requests. Magpie's skills are a real maintainer workload, and Tooling asked for exactly this.
   
   ## Models
   
   | id | context | reasoning on by default |
   |---|---:|---|
   | `gemma4-26b` | 131,072 | no — the recommended default |
   | `qwen3.8-27b` | 131,072 | yes |
   | `qwen3-8b` | 40,960 | yes |
   
   A reasoning-on model emits nothing while it thinks, and agent clients abandon the stream and retry — so the box generates twice for one unread response. Expect that to distort naive wall-clock numbers on two of the three, and measure it rather than working around it.
   
   ## Workloads
   
   Two tiers, and the second is the one that matters:
   
   1. **The existing harness** — the bounded replay already used for the published figures, for a like-for-like comparison.
   2. **Real Apache Airflow work** — `issue-triage` and the security-family skills over actual Airflow issues. Airflow is a genuine, large maintainer workload rather than a constructed one, which is what makes the numbers worth anything to Tooling's capacity planning. Contrived prompts would tell them very little.
   
   ## Done when
   
   - Tokens and wall-clock recorded per skill, per model, for both tiers.
   - Failure modes recorded too — refusals, timeouts, abandoned streams, context-window overruns on `qwen3-8b`'s 40,960.
   - Results written up (see the ASF-scoped section issue).
   - Findings shared back with ASF Tooling, who want the real-load data.
   
   ## Constraint — this is a shared, unmetered resource
   
   There is **no** budgeting, quota, access control or fairness mechanism on the gateway. The `$100` in the UI is a placeholder and `spend_usd` reports `0.00` for self-hosted models. Capacity consumed is capacity another community cannot use, and Tooling expect only two people to be exercising it at all.
   
   So this run must be deliberately paced rather than opportunistic:
   
   - rate-limit it, and do not run the three models concurrently;
   - agree a window with Tooling before a large pass, rather than discovering the ceiling by hitting it;
   - `gemma4-26b` saturates at ~4 concurrent requests and `qwen3.8-27b` at 20+, so concurrency is not a free variable.
   
   Treat "we did not degrade the service for anyone else" as an acceptance criterion, not etiquette.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@magpie.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org