You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "joeyutong (via GitHub)" <gi...@apache.org> on 2026/06/25 06:26:20 UTC
[I] [Feature] Track token usage metrics for embedding models [flink-agents]
joeyutong opened a new issue, #858:
URL: https://github.com/apache/flink-agents/issues/858
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/flink-agents/issues) and found nothing similar.
### Description
Embedding model calls currently do not consistently report token usage metrics.
Chat models already have a token accounting path: provider usage is attached to the chat response and later recorded as model-level token metrics. Embedding models return only vectors, so provider usage returned by OpenAI-compatible or DashScope-style embedding APIs can be dropped before it reaches the metrics layer.
Affected paths include:
- Direct Java or Python embedding model calls.
- Vector store and RAG paths that auto-generate embeddings during `add`, `update`, or `query`.
- Cross-language resource paths where the wrapper may receive an action metric group but the provider-side embedding resource performs the actual request.
This makes it harder to validate and compare embedding model cost/usage, especially when a job mixes chat, embedding, and vector store operations. Embedding metrics do not need `completionTokens`, but should expose input-side token usage, for example `promptTokens` and `totalTokens`, under the same model/provider metric dimensions used by chat metrics where possible.
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Track token usage metrics for embedding models [flink-agents]
Posted by "zxs1633079383 (via GitHub)" <gi...@apache.org>.
zxs1633079383 commented on issue #858:
URL: https://github.com/apache/flink-agents/issues/858#issuecomment-4881187842
Hi, I would like to work on this and have a first PR ready.
I took a first look at the current metric path and the related work in #860
and #861. My current understanding is that embedding token metrics should follow
the same production constraint as chat token metrics: record usage under the
request/action metric group that initiated the embedding call, not under a
mutable cached resource binding.
Proposed first PR scope:
- Add provider-neutral embedding token usage accounting for `promptTokens` and
`totalTokens` when provider usage is available.
- Cover both Python and Java embedding model paths with unit tests.
- Keep providers that do not expose usage as no-op, so embedding calls do not
fail or invent token counts.
- Align the metric dimensions with the existing chat model token metrics where
practical.
I noticed the related metric work in #860 and #861. I kept the first PR small
and based on current `main`, so it should be easy to review or rebase if the
chat request-scoped metric changes from #861 land first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Track token usage metrics for embedding models [flink-agents]
Posted by "richardchen874-sys (via GitHub)" <gi...@apache.org>.
richardchen874-sys commented on issue #858:
URL: https://github.com/apache/flink-agents/issues/858#issuecomment-5018300297
I like the way this issue frames tracking token usage metrics for embedding models, not only chat models. In real workloads, the hard part is usually not sending one request; it is keeping cost, latency, and response shape predictable once users switch providers.
I would record provider, model, dimensions, input batch size, token usage, and latency for embeddings just like chat responses. That makes embedding-heavy workflows visible before they become a hidden cost center.
I am testing an OpenAI-compatible multi-model API layer around official Chinese models, so this kind of routing, usage, and compatibility boundary is very relevant. Who do you expect to consume these metrics first: developers debugging agents, operators managing spend, or end users?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Track token usage metrics for embedding models [flink-agents]
Posted by "hulincup (via GitHub)" <gi...@apache.org>.
hulincup commented on issue #858:
URL: https://github.com/apache/flink-agents/issues/858#issuecomment-5394515519
Hi @joeyutong, I'd like to take this one. The embedding providers already populate `EmbeddingResult.tokenUsage` (Bedrock on the Java side, OpenAI/Tongyi on the Python cross-lang side), but nothing reads it to record metrics. I plan to mirror the chat path — add `recordTokenMetrics` to `BaseEmbeddingModelSetup` and record `promptTokens`/`totalTokens` at the `embedWithUsage` chokepoint, so direct calls and vector-store/RAG paths are both covered. Pure-Python embedding integrations stay unchanged, consistent with the chat side having no Python metric layer. Happy to adjust the scope if you had something broader in mind.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org