You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Zhuoxi2000 (via GitHub)" <gi...@apache.org> on 2026/07/27 00:45:11 UTC
[D] [Discuss][Observability] OpenTelemetry GenAI export for Agent Traces (follow-up to #900) [flink-agents]
GitHub user Zhuoxi2000 created a discussion: [Discuss][Observability] OpenTelemetry GenAI export for Agent Traces (follow-up to #900)
## Motivation
#900 defines how Agent Traces are recorded in the Event Log, and deliberately lists OTel export as a non-goal of the recording scope. In that thread, @joeyutong agreed that OpenTelemetry export should be a supported follow-up once the recording contract settles, with the exact GenAI attribute mapping to be worked out as part of the exporter design. This discussion is that follow-up: a config-gated exporter that maps recorded Execution Events to the [OpenTelemetry GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/), so any OTLP-compatible backend (Jaeger, Grafana Tempo, Langfuse, Phoenix, vendor APMs) can visualize and analyze agent runs without flink-agents shipping its own UI.
## Proposed mapping
Trace topology, aligned with the #900 recording model:
| flink-agents | OTel GenAI |
|---|---|
| Input run (`input_run_id`) | Trace (one run = one trace) |
| Execution (`execution_id` / `parent_execution_id`) | Span / parent Span |
| Agent execution | span `invoke_agent {agent_name}`; `gen_ai.operation.name = invoke_agent`, `gen_ai.agent.name` |
| Tool execution | span `execute_tool {tool_name}`; `gen_ai.operation.name = execute_tool`, `gen_ai.tool.name` |
| Model call | span `chat {model}`; `gen_ai.operation.name = chat`, `gen_ai.request.model`, `gen_ai.provider.name` |
| `business_key` | `gen_ai.conversation.id` |
| Lifecycle events (started/finished) | span start/end timestamps |
| `status` / `problem_category` | span status + `error.type` |
| Token usage | `gen_ai.usage.input_tokens` / `gen_ai.usage.output_tokens` |
**ID derivation.** Framework IDs stay opaque, as preferred in #900; the exporter derives fixed-width OTel IDs deterministically, e.g. `trace_id = truncate(SHA-256(input_run_id), 128 bits)` and `span_id = truncate(SHA-256(execution_id), 64 bits)`. Deterministic derivation keeps re-exports idempotent.
## Design points to agree on
1. **Where the exporter runs.** Two candidate shapes:
- **(a) In-process**: an Execution Event listener in the runtime feeds an async OTLP exporter (batching, bounded queue, drop-on-overflow so it can never backpressure the job).
- **(b) Out-of-band**: a separate consumer reads the Event Log and exports — zero impact on the running job, works retroactively, but requires event-log trace recording (#924) to be enabled.
I lean toward (b) first, since it derives everything from the recording contract and adds no runtime risk, keeping (a) as a later option. Opinions welcome.
2. **Config surface.** `event-log.trace.otel.enabled` (default `false`, mirroring the recording flag), OTLP endpoint/protocol/headers, `service.name`. Content capture (prompt/response bodies) stays off by default and is opted into separately — gen_ai content capture has privacy and volume implications.
3. **Semconv version.** The `gen_ai.*` conventions are still `development` stability; we should pin the exported `schema_url` and document the target version. I'm engaging in the OTel GenAI SIG, so we can track upstream changes as they stabilize.
4. **Scope for 0.4.** Phase 1 = Java, spans-only, out-of-band exporter + config; Phase 2 = usage-metrics enrichment; Phase 3 = Python parity. Is Phase 1 realistic for the 0.4 feature freeze (Sept 15)?
## What I'd deliver
I've already implemented the inverse mapping (OTel GenAI → storage) in a side project, so the attribute table above is grounded in working code. If maintainers agree with the general shape, I'll break this into concrete issues and start with Phase 1.
GitHub link: https://github.com/apache/flink-agents/discussions/929
----
This is an automatically sent email for issues@flink.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@flink.apache.org
Re: [D] [Discuss][Observability] OpenTelemetry GenAI export for Agent Traces (follow-up to #900) [flink-agents]
Posted by "weiqingy (via GitHub)" <gi...@apache.org>.
GitHub user weiqingy added a comment to the discussion: [Discuss][Observability] OpenTelemetry GenAI export for Agent Traces (follow-up to #900)
Thanks for working through these. I think this narrows the remaining questions considerably.
Assuming each converter invocation receives a fixed, complete set of already-collected files, the batch shape seems to resolve the ordering and timeout questions: the converter can group records by `inputRunId`, sort them, and use end-of-input rather than a timeout to determine that no more records are coming.
Incomplete executions may still be present, though. `EventLogWriter` calls append and flush for each record, but treats failures as best-effort and does not retry them. A crash can also leave a start record in the collected files without a terminal record. A restored Action execution keeps its `executionId` and `inputRunId`, since both are checkpointed with the `ActionTask`. By contrast, the state pairing nested LLM, Parser, and Tool starts with their terminal reports is transient, so recovery can produce new child executions under the same run.
How would we like those incomplete executions to appear in the exported trace? Would omitting them hide useful evidence of failed work, while synthesizing an end timestamp risk implying a duration we did not observe? Could emitting them as errors with an explicit "incomplete/synthetic close" marker preserve that distinction, or would a diagnostic-only record be clearer?
Could [`trace_tree.py`](https://github.com/apache/flink-agents/blob/1c55683deb5c7e8d44997851403e75b5c0fbdb57/python/flink_agents/cli/trace_tree.py) provide a useful precedent for input validation and diagnostics? It reports conditions such as `MISSING_PARENT` for business-event lineage and `MALFORMED_RECORD` for invalid input. Although it intentionally ignores execution lifecycle records and therefore does not answer the incomplete-span policy itself, might its warning model be reusable?
It also discovers only `events-*.log`, which is the `FileEventLogger` naming contract rather than the default SLF4J output. Would it help for Phase 1 to state whether the converter consumes specifically `FileEventLogger` JSONL files, including how a complete multi-subtask file set is identified, or normalized records collected from any Event Log sink?
I also wonder whether the Parser mapping could be explicit. The run/root/parent fields establish its topology, but the converter still needs to decide its span name, `gen_ai.operation.name`, and OTel span kind. Since #924 records PARSER as a first-class execution type, would defining one stable mapping here make traces more consistent across converter versions?
The OTel GenAI convention permits a custom `gen_ai.operation.name` when none of its well-known values applies, and OpenLLMetry uses that mechanism for names such as `execute_task` and `llm_request`. OpenInference's `CHAIN`, `GUARDRAIL`, and `EVALUATOR` values belong to the separate `openinference.span.kind` classification. Since Phoenix and Langfuse recognize that convention, would it be worth deciding whether Phase 1 emits only OTel GenAI attributes or also an OpenInference classification for AI-aware rendering?
Separately, for the OTel span kind, would `SpanKind.INTERNAL` fit Parser, and perhaps Action, as in-process work, while preserving `CLIENT` for spans that directly represent remote calls?
GitHub link: https://github.com/apache/flink-agents/discussions/929#discussioncomment-17956163
----
This is an automatically sent email for issues@flink.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@flink.apache.org
Re: [D] [Discuss][Observability] OpenTelemetry GenAI export for Agent Traces (follow-up to #900) [flink-agents]
Posted by "weiqingy (via GitHub)" <gi...@apache.org>.
GitHub user weiqingy added a comment to the discussion: [Discuss][Observability] OpenTelemetry GenAI export for Agent Traces (follow-up to #900)
Thanks for making the mapping explicit rather than leaving it implicit in #900. OTLP export is worth pursuing, and a standalone exporter looks like the right direction.
Could the proposal include a short rejected-alternatives note for Flink's `OpenTelemetryTraceReporterFactory`? Although it is available with the pinned Flink 2.3.0, it cannot preserve caller-derived IDs or span status, requires related spans to be reported together, and lacks consistent support across the 1.20, 2.0, and 2.1 distributions shipped here. That seems enough to explain why the native trace pipeline cannot represent an agent trace.
Before implementation, could we clarify the architecture and recording contract on which the exporter would depend?
**1. What durable source would the standalone exporter consume?**
Today, Event Log output is either SLF4J or TaskManager-local files under `java.io.tmpdir`. Both are local, and the logger type is a closed enum that does not support adding a durable sink alongside the existing one. A separate consumer would therefore amount to tailing files that may disappear during failover.
Could Phase 1 define durability, partition discovery, checkpoints, replay, retention, failover, and handling of incomplete or out-of-order lifecycle events? Would `event-log.trace.otel.*` also be clearer as consumer configuration, since an external process cannot access AgentPlan?
**2. Could #924 capture the telemetry-neutral metadata while its contract is still open?**
Several proposed OTel attributes currently have no reliable source:
- The LLM entity name is a resource alias such as `_default_chat_model`, not necessarily the exact requested model.
- Provider identity exists only indirectly in the resource descriptor.
- Token counts are cumulative Flink counters keyed by model, rather than per-execution values.
The existing `entityMetadata` channel is already serialized and used for tool executions, while both Java and Python LLM paths currently leave it empty. Would recording the resolved model, provider, and per-call token usage there keep the exporter focused on conversion rather than inference and metric joins?
Could the contract also define how agent, action, LLM, parser, and tool executions become spans? It currently records no agent entity, while action roots have no parent and parser executions have no proposed mapping. Without explicit run, root, and parent semantics, could chained actions appear as unrelated roots or recorded executions be omitted?
**3. Could the contract remain language-neutral from the start?**
Python actions already report through Java, and Java's `EventLogWriter` serializes their events. Could "Java first, Python parity later" therefore refer only to implementation work, with one serialized contract and one exporter serving both runtimes?
**4. Would an optional module provide a safer dependency boundary?**
The Elasticsearch integration already brings an unrelocated OTel API 1.29.0 into the uber JAR, and the default OTLP sender would add a second okhttp package alongside the one already there. Introducing the 1.54.1 SDK/exporter can produce incompatible dependency mediation, including the observed missing `ComponentLoader` failure. Could a standalone optional artifact align its OTel dependencies through the BOM and avoid exposing that conflict to every distribution?
**5. Could delivery semantics be defined separately from ID derivation?**
Stable derived IDs would help with reconstruction, but they would not make OTLP delivery idempotent. Could the proposal describe consumer checkpoints, retries, and replay behavior for complete and partial spans?
Would it make sense to frame Phase 1 as:
1. Finalize a language-neutral execution and metadata contract.
2. Define a durable, replayable Event Log source and consumer-checkpoint model.
3. Provide an optional standalone OTLP converter on top of that contract.
In-process export, content capture, and further enrichment could then build on that foundation in later phases.
GitHub link: https://github.com/apache/flink-agents/discussions/929#discussioncomment-17915174
----
This is an automatically sent email for issues@flink.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@flink.apache.org
Re: [D] [Discuss][Observability] OpenTelemetry GenAI export for Agent Traces (follow-up to #900) [flink-agents]
Posted by "Zhuoxi2000 (via GitHub)" <gi...@apache.org>.
GitHub user Zhuoxi2000 added a comment to the discussion: [Discuss][Observability] OpenTelemetry GenAI export for Agent Traces (follow-up to #900)
> Thanks for working through these. I think this narrows the remaining questions considerably.
>
> Assuming each converter invocation receives a fixed, complete set of already-collected files, the batch shape seems to resolve the ordering and timeout questions: the converter can group records by `inputRunId`, sort them, and use end-of-input rather than a timeout to determine that no more records are coming.
>
> Incomplete executions may still be present, though. `EventLogWriter` calls append and flush for each record, but treats failures as best-effort and does not retry them. A crash can also leave a start record in the collected files without a terminal record. A restored Action execution keeps its `executionId` and `inputRunId`, since both are checkpointed with the `ActionTask`. By contrast, the state pairing nested LLM, Parser, and Tool starts with their terminal reports is transient, so recovery can produce new child executions under the same run.
>
> How would we like those incomplete executions to appear in the exported trace? Would omitting them hide useful evidence of failed work, while synthesizing an end timestamp risk implying a duration we did not observe? Could emitting them as errors with an explicit "incomplete/synthetic close" marker preserve that distinction, or would a diagnostic-only record be clearer?
>
> Could [`trace_tree.py`](https://github.com/apache/flink-agents/blob/1c55683deb5c7e8d44997851403e75b5c0fbdb57/python/flink_agents/cli/trace_tree.py) provide a useful precedent for input validation and diagnostics? It reports conditions such as `MISSING_PARENT` for business-event lineage and `MALFORMED_RECORD` for invalid input. Although it intentionally ignores execution lifecycle records and therefore does not answer the incomplete-span policy itself, might its warning model be reusable?
>
> It also discovers only `events-*.log`, which is the `FileEventLogger` naming contract rather than the default SLF4J output. Would it help for Phase 1 to state whether the converter consumes specifically `FileEventLogger` JSONL files, including how a complete multi-subtask file set is identified, or normalized records collected from any Event Log sink?
>
> I also wonder whether the Parser mapping could be explicit. The run/root/parent fields establish its topology, but the converter still needs to decide its span name, `gen_ai.operation.name`, and OTel span kind. Since #924 records PARSER as a first-class execution type, would defining one stable mapping here make traces more consistent across converter versions?
>
> The OTel GenAI convention permits a custom `gen_ai.operation.name` when none of its well-known values applies, and OpenLLMetry uses that mechanism for names such as `execute_task` and `llm_request`. OpenInference's `CHAIN`, `GUARDRAIL`, and `EVALUATOR` values belong to the separate `openinference.span.kind` classification. Since Phoenix and Langfuse recognize that convention, would it be worth deciding whether Phase 1 emits only OTel GenAI attributes or also an OpenInference classification for AI-aware rendering?
>
> Separately, for the OTel span kind, would `SpanKind.INTERNAL` fit Parser, and perhaps Action, as in-process work, while preserving `CLIENT` for spans that directly represent remote calls?
Thanks! totally agreed that batch input gives us a clear ordering/completion model. The converter can group by `inputRunId`, sort by timestamp, and treat end-of-input as completion. I’ll make that explicit in #970.
**Incomplete executions.** I’d combine the two approaches: still emit the span, close it at the only timestamp we observed (so zero duration rather than inventing an end time), keep the status `UNSET`, add something like `flink_agents.execution.incomplete = true`, and surface a diagnostic as well.
I’d avoid `ERROR` because a missing terminal record doesn’t necessarily mean the execution failed — it could also be a dropped best-effort append or a recovery/replay artifact. Marking it incomplete keeps that evidence queryable without claiming an outcome we didn’t observe. A terminal record with no matching start would get the same treatment, with a `MISSING_START` diagnostic.
**Diagnostics.** I agree we should follow `trace_tree.py` here and reuse the same machine-readable shape (`code`, id, `message`, `filePath`, `lineNumber`). We can keep `MALFORMED_RECORD` and add `INCOMPLETE_EXECUTION` / `MISSING_START` rather than inventing a separate model.
**Input contract.** Phase 1 should operate on the JSON records themselves, not depend on a file naming convention. It can take explicit files, ignore unknown fields, and handle concatenated JSON objects. Directory discovery for `events-*.log` can just be a convenience matching `trace_tree.py`. For multi-subtask jobs, completeness remains the caller’s responsibility, but per-file counts and diagnostics should make missing input easier to spot.
**Parser mapping.** Agreed: `parse {entityName}`, `gen_ai.operation.name = "parse"`, `SpanKind.INTERNAL`.
**OpenInference.** I’d keep Phase 1 OTel GenAI-only. `openinference.span.kind` can be an additive opt-in follow-up for backends like Phoenix/Langfuse without changing topology or IDs.
**Span kind.** Agreed on `INTERNAL` for Parser and Action. I’d use `INTERNAL` for Tool as well for now, since the Event Log doesn’t tell us whether a tool is local or remote. LLM is the only clearly remote operation, so that can be `CLIENT`. If tool transport is recorded later in `entityMetadata`, remote/MCP tools can be classified as `CLIENT` then.
I’ll fold these into #970’s scope/design notes.
GitHub link: https://github.com/apache/flink-agents/discussions/929#discussioncomment-17956216
----
This is an automatically sent email for issues@flink.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@flink.apache.org
Re: [D] [Discuss][Observability] OpenTelemetry GenAI export for Agent Traces (follow-up to #900) [flink-agents]
Posted by "Zhuoxi2000 (via GitHub)" <gi...@apache.org>.
GitHub user Zhuoxi2000 edited a discussion: [Discuss][Observability] OpenTelemetry GenAI export for Agent Traces (follow-up to #900)
## Motivation
#900 defines how Agent Traces are recorded in the Event Log, and deliberately lists OTel export as a non-goal of the recording scope. In that thread, @joeyutong agreed that OpenTelemetry export should be a supported follow-up once the recording contract settles, with the exact GenAI attribute mapping to be worked out as part of the exporter design. This discussion is that follow-up: a config-gated exporter that maps recorded Execution Events to the [OpenTelemetry GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/), so any OTLP-compatible backend (Jaeger, Grafana Tempo, Langfuse, Phoenix, vendor APMs) can visualize and analyze agent runs without flink-agents shipping its own UI.
## Proposed mapping
Trace topology, aligned with the #900 recording model:
| flink-agents | OTel GenAI |
|---|---|
| Input run (`input_run_id`) | Trace (one run = one trace) |
| Execution (`execution_id` / `parent_execution_id`) | Span / parent Span |
| Agent execution | span `invoke_agent {agent_name}`; `gen_ai.operation.name = invoke_agent`, `gen_ai.agent.name` |
| Tool execution | span `execute_tool {tool_name}`; `gen_ai.operation.name = execute_tool`, `gen_ai.tool.name` |
| Model call | span `chat {model}`; `gen_ai.operation.name = chat`, `gen_ai.request.model`, `gen_ai.provider.name` |
| `business_key` | `gen_ai.conversation.id` |
| Lifecycle events (started/finished) | span start/end timestamps |
| `status` / `problem_category` | span status + `error.type` |
| Token usage | `gen_ai.usage.input_tokens` / `gen_ai.usage.output_tokens` |
**ID derivation.** Framework IDs stay opaque, as preferred in #900; the exporter derives fixed-width OTel IDs deterministically, e.g. `trace_id = truncate(SHA-256(input_run_id), 128 bits)` and `span_id = truncate(SHA-256(execution_id), 64 bits)`. Deterministic derivation keeps re-exports idempotent.
## Design points to agree on
1. **Where the exporter runs.** Two candidate shapes:
- **(a) In-process**: an Execution Event listener in the runtime feeds an async OTLP exporter (batching, bounded queue, drop-on-overflow so it can never backpressure the job).
- **(b) Out-of-band**: a separate consumer reads the Event Log and exports — zero impact on the running job, works retroactively, but requires event-log trace recording (#924) to be enabled.
I lean toward (b) first, since it derives everything from the recording contract and adds no runtime risk, keeping (a) as a later option. Opinions welcome.
2. **Config surface.** `event-log.trace.otel.enabled` (default `false`, mirroring the recording flag), OTLP endpoint/protocol/headers, `service.name`. Content capture (prompt/response bodies) stays off by default and is opted into separately — gen_ai content capture has privacy and volume implications.
3. **Semconv version.** The `gen_ai.*` conventions are still `development` stability; we should pin the exported `schema_url` and document the target version. I'm engaging in the OTel GenAI SIG, so we can track upstream changes as they stabilize.
4. **Scope for 0.4.** Phase 1 = Java, spans-only, out-of-band exporter + config; Phase 2 = usage-metrics enrichment; Phase 3 = Python parity. Is Phase 1 realistic for the 0.4 feature freeze (Sept 15)?
## What I'd deliver
I've already implemented the inverse mapping (OTel GenAI → storage) in a side project, so the attribute table above is grounded in working code. If maintainers agree with the general shape, I'll break this into concrete issues and start with Phase 1.
---
**Updates (2026-08-07)** — incorporating the review feedback above:
**Rejected alternative: Flink's `OpenTelemetryTraceReporterFactory`.** Flink 2.3 ships a native
OTel trace reporter, but it cannot serve as the export path for agent traces: it does not
preserve caller-derived trace/span ids or span status, it requires related spans to be reported
together, and it is not consistently available across the 1.20 / 2.0 / 2.1 distributions this
project ships for. A standalone converter avoids all three constraints.
**Scope clarifications:**
- Phase 1 is a **batch, offline converter** over already-collected Event Log files — not a
continuous tailer. Durability, partition discovery, checkpoints, and retention are properties
of a durable Event Log sink, which deserves its own design track; a continuous OTel consumer
would build on that. Delivery is at-least-once; deterministic ids make duplicates identifiable,
and idempotent coalescing is a backend property.
- Since the converter is an external process, its settings are plain converter arguments
(endpoint, protocol, service.name) — the earlier `event-log.trace.otel.*` naming is dropped.
- The contract is the serialized Event Log record, which both runtimes produce through the same
writer, so **one converter serves both Java and Python agents**. "Java" refers only to the
converter's implementation language; there is no separate Python-parity phase.
- The converter lives in an **optional standalone module that is not bundled into dist**, with
its OTel dependencies aligned through the `opentelemetry-bom`, keeping them off the
distribution classpath entirely.
- Attribute-quality gaps (resolved model vs resource alias, provider identity, per-call token
usage) are best addressed by recording them in `entityMetadata` at recording time, as raised
above — tracked as part of the recording-contract conversation on #924.
GitHub link: https://github.com/apache/flink-agents/discussions/929
----
This is an automatically sent email for issues@flink.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@flink.apache.org
Re: [D] [Discuss][Observability] OpenTelemetry GenAI export for Agent Traces (follow-up to #900) [flink-agents]
Posted by "Zhuoxi2000 (via GitHub)" <gi...@apache.org>.
GitHub user Zhuoxi2000 added a comment to the discussion: [Discuss][Observability] OpenTelemetry GenAI export for Agent Traces (follow-up to #900)
Thanks for the review! will open a tracking issue for the exporter and start with Phase 1: Java, spans only, reading from the Event Log, and gated by config. Usage-metrics enrichment and Python parity can follow as separate issues.
GitHub link: https://github.com/apache/flink-agents/discussions/929#discussioncomment-17908233
----
This is an automatically sent email for issues@flink.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@flink.apache.org
Re: [D] [Discuss][Observability] OpenTelemetry GenAI export for Agent Traces (follow-up to #900) [flink-agents]
Posted by "Zhuoxi2000 (via GitHub)" <gi...@apache.org>.
GitHub user Zhuoxi2000 edited a comment on the discussion: [Discuss][Observability] OpenTelemetry GenAI export for Agent Traces (follow-up to #900)
Thanks for the review! will open a tracking issue for the exporter and start with Phase 1: Java, spans only, reading from the Event Log, and gated by config. Usage-metrics enrichment and Python parity can follow as separate issues.
#970
GitHub link: https://github.com/apache/flink-agents/discussions/929#discussioncomment-17908233
----
This is an automatically sent email for issues@flink.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@flink.apache.org
Re: [D] [Discuss][Observability] OpenTelemetry GenAI export for Agent Traces (follow-up to #900) [flink-agents]
Posted by "wenjin272 (via GitHub)" <gi...@apache.org>.
GitHub user wenjin272 added a comment to the discussion: [Discuss][Observability] OpenTelemetry GenAI export for Agent Traces (follow-up to #900)
Thanks for driving this discussion. OpenTelemetry is becoming a widely adopted standard for agent observability, so I believe supporting the conversion of Flink Agents Event Logs into OpenTelemetry GenAI traces would be very valuable. It would also allow users to integrate with existing observability backends without Flink Agents having to provide its own visualization stack. The proposed mapping and overall design look reasonable to me, and I have no major concerns with this direction.
GitHub link: https://github.com/apache/flink-agents/discussions/929#discussioncomment-17907383
----
This is an automatically sent email for issues@flink.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@flink.apache.org
Re: [D] [Discuss][Observability] OpenTelemetry GenAI export for Agent Traces (follow-up to #900) [flink-agents]
Posted by "Zhuoxi2000 (via GitHub)" <gi...@apache.org>.
GitHub user Zhuoxi2000 added a comment to the discussion: [Discuss][Observability] OpenTelemetry GenAI export for Agent Traces (follow-up to #900)
Thanks for the thorough review — these are exactly the questions worth settling before implementation.
**Rejected alternative.** Agreed. I’ll document why `OpenTelemetryTraceReporterFactory` isn’t a good fit: no caller-defined IDs or status, related spans need to be exported together, and support differs across Flink distributions.
**1. Source and scope.** Phase 1 is a batch converter over already-collected Event Log files, not a continuous tailer. Durability, checkpoints, retention, and failover therefore belong to a separate durable Event Log sink design, which a future continuous OTel consumer could build on.
Since the converter is standalone, its settings will be regular arguments such as endpoint, protocol, and `service.name`, rather than `event-log.*` config.
**2. Recording metadata.** Strong +1 on recording the resolved model, provider, and per-call usage in `entityMetadata`. That keeps the exporter a pure converter without inference or metric joins. The run/root/parent relationships should also be explicit in the contract rather than synthesized by the exporter. I’m happy to help with the LLM metadata follow-up once #924 settles.
**3. Language neutrality.** Agreed. Both runtimes produce the same serialized Event Log records, so one converter serves both. “Java first” only refers to the converter implementation language.
**4. Dependency boundary.** The converter will remain an optional standalone module and won’t be bundled into the distribution, keeping its OTel SDK/exporter dependencies off the main classpath.
**5. Delivery semantics.** The batch converter is at-least-once. Deterministic IDs make duplicates identifiable, but idempotency is ultimately backend-dependent. Checkpointing and replay guarantees belong to the future continuous-consumer track.
So the revised plan is:
1. Clarify metadata and run/root/parent semantics in #924.
2. Track the durable, replayable Event Log sink separately.
3. Keep #970 focused on the optional batch converter, with a continuous consumer as follow-up work.
I’ll update #970 accordingly.
GitHub link: https://github.com/apache/flink-agents/discussions/929#discussioncomment-17924359
----
This is an automatically sent email for issues@flink.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@flink.apache.org