[DISCUSS] Opt-in for adopters to provide "mode economics/token data"

[Jarek Potiuk <ja...@potiuk.com>]

Hello here,

I was thinking about the best way to estimate token usage for Magpie
skills, and I believe gathering actual usage data is the best
approach.

I believe it's perfectly ok to ask our adopters for an opt-in to share
token usage with us so tha twe can optimise it and see how much the
skills actually use.

I think it should be double-opt-in:

a) opt-in to share usage data generally
b) once opted-in the operator will be asked whether to submit the data
after each session

I think an easy way to get that data is to have PR to the
"magpie-economics" repo (I created such repo) - this one is accessible
for practically all users, and they need to also make a deliberate
effort to create such PR - and we need to approve and merge it.

Here is a proposed PR: happy to discuss the idea:

https://github.com/apache/magpie/pull/750

J.

Re: [DISCUSS] Opt-in for adopters to provide "mode economics/token data"

[Alex <al...@gmail.com>]

Hi Jarek/Justin,

I was just typing the same points regarding UX annoyance as Justin.
- Once you enable per project sharing (first gate), users might
misinterpret it as "they share my data" by default.
- If the user is willing to share their data, they shouldn't need the
friction of saying yes with every session.  It's either safe to share or
isn't.

On the data itself, my own custom benchmark tournaments for similar things
have led me to see that I can compare wall time, success on evals/tests,
but something like tokens (which is only made more complicated by
read/write/cached granularities) is hardly comparable across agents (SDK or
CLI based).  Even if you can normalize tool calls/steps from traces on many
occasions, token reporting is obscure and I treat it with great
skepticism.  You can keep a lot of the same variables (harness, model,
config) and iterate on something like prompt to "optimize" token usage,
like a GEPA loop [1] but I'd very careful with aggregating data early and
concluding things based on such data.

I'd lean towards clean trace sharing for a more qualitative analysis.  In
the pi coding agent world, there were some interesting things there [2] and
people voluntarily shared their traces [3] for common improvement.  I think
collecting the stripped/privacy aware data in a way that preserves the
proper raw context of the sessions (So you know the coding agent, model,
config, etc) is probably the right way to go.  It presents many challenges
because things are non-standard, but it's easier to use for improvements.

[1] - https://github.com/egmaminta/GEPA-Lite
[2] - https://github.com/badlogic/pi-share-hf
[3] - https://traces.com/

On Sun, Jul 5, 2026 at 11:55 AM Justin Mclean <ju...@classsoftware.com>
wrote:

> Hi Jarek,
>
> Thanks for raising this. Token usage data would definitely be useful.
>
> But perhaps:
> - This should probably be run past the ASF privacy team first, given we'd
> be collecting and storing usage data from adopters.
> - Prompting after every session risks being quite annoying and may drive
> people away from participating altogether. A less frequent or purely
> passive opt-in might work better.
> - The collection mechanism isn't clear to me and ASF privacy would want to
> know this.
> - Token costs vary significantly by model and provider, and change over
> time, which limits how useful the data would be in practice.
>
> Kind Regards,
> Justin

Re: [DISCUSS] Opt-in for adopters to provide "mode economics/token data"

[Justin Mclean <ju...@classsoftware.com>]

Hi Jarek,

Thanks for raising this. Token usage data would definitely be useful.

But perhaps:
- This should probably be run past the ASF privacy team first, given we'd be collecting and storing usage data from adopters.
- Prompting after every session risks being quite annoying and may drive people away from participating altogether. A less frequent or purely passive opt-in might work better.
- The collection mechanism isn't clear to me and ASF privacy would want to know this.
- Token costs vary significantly by model and provider, and change over time, which limits how useful the data would be in practice.

Kind Regards,
Justin