[jira] [Commented] (CAMEL-23708) OpenTelemetry2 - Missing RECEIVED spans for routes consuming from stub/seda in nested async paths

["Claus Ibsen (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CAMEL-23708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18086794#comment-18086794 ] 

Claus Ibsen commented on CAMEL-23708:
-------------------------------------

Investigation update: the missing RECEIVED spans are NOT caused by a framework bug in the OTel span lifecycle.

A reproducer unit test (MulticastSedaTest) was created that faithfully reproduces the production route-topology topology:
{code}
direct → direct → direct → stub:orders → multicast → stub:fulfillment/notifications → stub:warehouse/email
{code}

All 4 test scenarios pass — every RECEIVED span is properly created and ended, including those in nested async paths (seda consumer → multicast → seda/stub → leaf consumer routes).

*Root cause: DevSpanExporter capacity eviction*

The DevSpanExporter has a {{LinkedBlockingQueue}} with capacity 500 and FIFO eviction. With the route-topology example:
- Timer fires every 5 seconds generating ~33 spans per trace (with core processors enabled)
- 500 / 33 ≈ 15 complete traces fit in the queue
- After ~75 seconds, the queue is full and starts evicting from the head (oldest spans)
- The RECEIVED spans for leaf routes (fulfillment, notification) complete LAST in each trace due to async seda queueing, so they are the last to enter the exporter — but the earlier spans from older traces get evicted to make room, creating incomplete traces

*Action needed:* Reclassify this from a framework instrumentation bug to a DevSpanExporter improvement. Options:
# Increase capacity (e.g. 2000) or make it configurable
# Switch to trace-aware eviction: evict entire traces instead of individual spans, keeping the N most recent complete traces
# Add a "spans evicted" counter to the dev console response so the TUI can warn the user

The reproducer test ({{MulticastSedaTest}}) has been added to {{camel-opentelemetry2}} as a baseline regression test for multicast+seda/stub span completeness.

> OpenTelemetry2 - Missing RECEIVED spans for routes consuming from stub/seda in nested async paths
> -------------------------------------------------------------------------------------------------
>
>                 Key: CAMEL-23708
>                 URL: https://issues.apache.org/jira/browse/CAMEL-23708
>             Project: Camel
>          Issue Type: Bug
>          Components: camel-opentelemetry
>            Reporter: Claus Ibsen
>            Priority: Major
>
> When using the stub component to simulate Kafka (e.g. with {{--observe}} / dev profile), route consumer RECEIVED spans are not exported for routes that consume from stub endpoints when the producer side is inside a multicast on a seda consumer thread.
> h3. Steps to reproduce
> Run the route-topology example with OpenTelemetry dev mode:
> {code}
> camel run route-topology.camel.yaml --observe
> {code}
> The example has this flow:
> {code}
> timer -> order-generator -> direct:process-order -> process-order -> kafka:orders
>   -> order-dispatcher (multicast)
>     -> kafka:fulfillment -> fulfillment route -> kafka:warehouse-shipments
>     -> kafka:notifications -> notification route -> kafka:email-outbox
> {code}
> h3. Expected behavior
> Each trace should have 27 spans including EVENT_RECEIVED spans for all 6 routes:
> - timer (order-generator) - OK
> - direct:process-order (process-order) - OK
> - direct:validate-order (validate-order) - OK
> - kafka:orders via stub (order-dispatcher) - OK
> - kafka:fulfillment via stub (fulfillment) - MISSING
> - kafka:notifications via stub (notification) - MISSING
> h3. Actual behavior
> Each trace has exactly 25 spans. The EVENT_RECEIVED spans for the fulfillment and notification routes are never exported to the DevSpanExporter.
> The RECEIVED spans ARE created (processor spans log4, to7, log5, to8 correctly reference them as parentSpanId), but {{span.end()}} is apparently never called, so {{SimpleSpanProcessor.onEnd()}} never fires and they never reach the exporter.
> h3. Analysis
> The issue is specific to nested async routing: the fulfillment and notification routes consume from stub endpoints whose producers are inside a multicast within the order-dispatcher route, which is itself on a seda consumer thread. The {{onExchangeDone}} route policy callback ({{TracingRoutePolicy}} in {{camel-telemetry}}) does not fire for these exchanges, so the RECEIVED span is started but never ended/exported.
> The kafka:orders RECEIVED span (order-dispatcher route) works correctly - this is a direct stub producer to seda consumer path without the nested multicast.
> Key code path:
> - {{camel-telemetry}}: {{Tracer.TracingRoutePolicy.onExchangeDone()}} -> {{endEventSpan()}} -> {{spanLifecycleManager.deactivate(span)}} -> {{otelSpan.end()}}
> - {{camel-seda}}: {{SedaProducer.process()}} (InOnly path) -> {{addToQueue(exchange, true)}} -> async consumer pickup
> - {{camel-stub}}: extends seda, same behavior
> h3. Impact
> - Orphan spans in trace visualization (Jaeger, TUI waterfall) - processor spans reference non-existent parent
> - Missing routeId propagation for orphaned spans (dev console cannot walk parent chain)
> - Affects any scenario with stub/seda consumers inside multicast or other async EIPs
> h3. How this was discovered
> Using the TUI MCP {{tui_get_spans}} tool to fetch raw span JSON data from a running Camel TUI session, analyzing parent chain integrity across 20 consecutive traces (500 spans). Every trace showed the identical orphan pattern.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)