You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@dubbo.apache.org by "Alanxtl (via GitHub)" <gi...@apache.org> on 2026/05/20 05:15:10 UTC

[I] [FEATURE] Enhance OpenTelemetry tracing from usable spans to diagnostic spans [dubbo-go]

Alanxtl opened a new issue, #3338:
URL: https://github.com/apache/dubbo-go/issues/3338

   ## Background
   
   Dubbo-go already supports OpenTelemetry tracing through client/server filters, propagators, samplers, and multiple exporters such as stdout, Jaeger, Zipkin, OTLP HTTP, and OTLP gRPC. The current tracing path is usable, but production troubleshooting needs richer span names, attributes, events, and error information.
   
   ## Goals
   
   Enhance OpenTelemetry tracing so Dubbo-go traces are not only connected end-to-end, but also useful for diagnosing latency, routing, retry, timeout, serialization, registry, and downstream failure problems.
   
   ## Proposed Scope
   
   ### 1. Standardize span names and semantic attributes
   
   Review and standardize client/server span naming. Candidate format:
   
   - Consumer span: `dubbo.consumer <service>/<method>`
   - Provider span: `dubbo.provider <service>/<method>`
   
   Review and enrich span attributes, including:
   
   - `rpc.system=apache_dubbo`
   - `rpc.service`
   - `rpc.method`
   - `rpc.grpc.status_code` or Dubbo-specific equivalent where applicable
   - `network.peer.address` / `server.address` where available
   - Dubbo-specific attributes such as side, protocol, group, version, and error type
   
   The final attribute names should align with OpenTelemetry semantic conventions where available and use a stable `dubbo.*` namespace for Dubbo-specific data.
   
   ### 2. Add diagnostic span events
   
   Add span events for key runtime decisions and failure points where the information is available:
   
   - Retry attempts and final retry result.
   - Load-balance selection result.
   - Router or tag-route decision.
   - Timeout and cancellation.
   - Rate limit or circuit-breaking rejection.
   - Serialization/codec failure.
   - Registry lookup or provider-empty cases.
   
   These events should be low-noise and should not expose sensitive request payloads.
   
   ### 3. Improve error recording
   
   Current spans set error status from `result.Error()`. This can be made more useful by:
   
   - Recording a stable error type/category, reusing the RPC metrics error taxonomy where possible.
   - Adding error code attributes for Triple/gRPC and Dubbo protocol errors.
   - Avoiding lossy conversion of structured errors into plain strings where a typed error is available.
   - Ensuring span status, span events, and logs can be correlated.
   
   ### 4. Clarify propagation behavior
   
   Document and test propagation behavior for:
   
   - W3C trace context.
   - B3 propagation.
   - Dubbo attachments used as carriers.
   - Interaction with business attachments and baggage.
   - Consumer -> provider propagation across Triple and Dubbo protocol paths where supported.
   
   ### 5. Improve trace/log/metric correlation
   
   Coordinate tracing with existing observability features:
   
   - Correlate with `CtxLogger` trace fields (`trace_id`, `span_id`, `trace_flags`).
   - Reuse metrics error classification where applicable.
   - Document how a user moves from a Grafana panel to a trace and then to related logs.
   
   ## Acceptance Criteria
   
   - Client and server spans have documented, stable names and attributes.
   - Span events exist for at least retry/timeout/rejection/codec or equivalent diagnostic points where supported.
   - Error spans include stable category/code attributes in addition to human-readable messages.
   - Propagation behavior is covered by tests for supported propagators.
   - Documentation and samples explain how tracing connects with metrics and context-aware logging.
   
   ## Related Context
   
   - Existing OpenTelemetry implementation: `otel/trace/*`, `filter/otel/trace/*`, `config/otel_config.go`
   - Logger trace correlation: https://github.com/apache/dubbo-go/pull/3195
   - Logger sample: https://github.com/apache/dubbo-go-samples/pull/1030


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org


Re: [I] [FEATURE] Enhance OpenTelemetry tracing diagnostics / 增强 OpenTelemetry Trace 诊断能力 [dubbo-go]

Posted by "Tsukikage7 (via GitHub)" <gi...@apache.org>.
Tsukikage7 commented on issue #3338:
URL: https://github.com/apache/dubbo-go/issues/3338#issuecomment-5126900864

   Thanks @jiaming2li. To avoid duplicate work, I will treat the cluster-level logical `CLIENT` span migration as your implementation scope.
   
   I still recommend keeping #3551 focused on its current span-name and base-attribute changes, and delivering the cluster-level migration as a separate PR (or at least as clearly separated commits), because it changes the consumer span lifecycle and retry topology. Please let me know if you plan a different split.
   
   @Alanxtl @Oxidaner, unless there is other ongoing work that overlaps, I would like to coordinate and implement the remaining non-overlapping parts of #3338 after this boundary, including:
   
   - stable error category/code/source attributes;
   - remaining timeout, rejection, codec and registry diagnostic events not covered by the logical-span work;
   - W3C/B3/baggage/attachment propagation tests;
   - Triple and Dubbo protocol coverage;
   - trace/log/metric correlation;
   - related samples and documentation.
   
   Could you please confirm this ownership boundary? I will not duplicate the logical-span work or the span-name/base-attribute work already covered by @jiaming2li and #3551.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org


Re: [I] [FEATURE] Enhance OpenTelemetry tracing diagnostics / 增强 OpenTelemetry Trace 诊断能力 [dubbo-go]

Posted by "jiaming2li (via GitHub)" <gi...@apache.org>.
jiaming2li commented on issue #3338:
URL: https://github.com/apache/dubbo-go/issues/3338#issuecomment-5018051599

   Is this issue still active, can I take it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org


Re: [I] [FEATURE] Enhance OpenTelemetry tracing diagnostics / 增强 OpenTelemetry Trace 诊断能力 [dubbo-go]

Posted by "Alanxtl (via GitHub)" <gi...@apache.org>.
Alanxtl commented on issue #3338:
URL: https://github.com/apache/dubbo-go/issues/3338#issuecomment-5031502675

   > Is this issue still active, can I take it?
   
   sure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org


Re: [I] [FEATURE] Enhance OpenTelemetry tracing diagnostics / 增强 OpenTelemetry Trace 诊断能力 [dubbo-go]

Posted by "jiaming2li (via GitHub)" <gi...@apache.org>.
jiaming2li commented on issue #3338:
URL: https://github.com/apache/dubbo-go/issues/3338#issuecomment-5126883741

   @Tsukikage7 ok, I'll try in this way.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org


Re: [I] [FEATURE] Enhance OpenTelemetry tracing diagnostics / 增强 OpenTelemetry Trace 诊断能力 [dubbo-go]

Posted by "Tsukikage7 (via GitHub)" <gi...@apache.org>.
Tsukikage7 commented on issue #3338:
URL: https://github.com/apache/dubbo-go/issues/3338#issuecomment-5114413545

   Thanks for asking. Yes, I think a single upper logical `CLIENT` span is the right default model.
   
   The OpenTelemetry RPC semantic conventions describe the client span as covering the whole logical RPC, including internal retries: https://opentelemetry.io/docs/specs/semconv/rpc/rpc-spans/. In dubbo-go, this means the span should start before directory/router/load-balance/retry processing and end after the final result.
   
   I suggest the following boundaries:
   
   1. Keep the provider/server filter span unchanged.
   2. Create one consumer logical `CLIENT` span above the cluster invocation.
   3. Do not emit another Dubbo `CLIENT` span for every retry attempt by default.
   4. Propagate the logical span context to every actual provider attempt, so each reached provider creates its normal `SERVER` span as a child of the logical client span.
   5. Record retry, router, load-balance and provider-selection diagnostics as bounded events/attributes on the logical span.
   6. Set the logical span's final status from the final RPC outcome. Intermediate failed attempts should be events, especially when a later retry succeeds.
   
   I agree that this is larger than the current scope of #3551. To keep review and rollback manageable, I suggest preserving the span-name/semantic-attribute work in #3551 and handling the cluster-level logical span migration as a separate implementation slice or, if it must stay in one PR, as clearly separated commits.
   
   Before removing the per-attempt client filter, we should preserve the current tracing enable/disable configuration and context injection behavior, and add tests for:
   
   - one attempt -> one logical client span;
   - multiple retries -> still one logical client span covering the total duration;
   - retry failure followed by success;
   - all attempts failed;
   - timeout and cancellation;
   - propagation to every reached provider;
   - direct invocation / special cluster paths;
   - no duplicate client spans;
   - tracing-disabled behavior.
   
   For peer information, the logical span can describe the logical service target and the final contacted peer where appropriate; per-attempt provider and error details can stay in span events rather than becoming high-cardinality metric labels.
   
   So the architecture direction is OK, but I recommend implementing it incrementally instead of turning the current naming/attribute change into one large rewrite.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org


Re: [I] [FEATURE] Enhance OpenTelemetry tracing diagnostics / 增强 OpenTelemetry Trace 诊断能力 [dubbo-go]

Posted by "Alanxtl (via GitHub)" <gi...@apache.org>.
Alanxtl commented on issue #3338:
URL: https://github.com/apache/dubbo-go/issues/3338#issuecomment-5127278380

   > Thanks [@jiaming2li](https://github.com/jiaming2li). To avoid duplicate work, I will treat the cluster-level logical `CLIENT` span migration as your implementation scope.
   > 
   > I still recommend keeping [#3551](https://github.com/apache/dubbo-go/pull/3551) focused on its current span-name and base-attribute changes, and delivering the cluster-level migration as a separate PR (or at least as clearly separated commits), because it changes the consumer span lifecycle and retry topology. Please let me know if you plan a different split.
   > 
   > [@Alanxtl](https://github.com/Alanxtl) [@Oxidaner](https://github.com/Oxidaner), unless there is other ongoing work that overlaps, I would like to coordinate and implement the remaining non-overlapping parts of [#3338](https://github.com/apache/dubbo-go/issues/3338) after this boundary, including:
   > 
   > * stable error category/code/source attributes;
   > * remaining timeout, rejection, codec and registry diagnostic events not covered by the logical-span work;
   > * W3C/B3/baggage/attachment propagation tests;
   > * Triple and Dubbo protocol coverage;
   > * trace/log/metric correlation;
   > * related samples and documentation.
   > 
   > Could you please confirm this ownership boundary? I will not duplicate the logical-span work or the span-name/base-attribute work already covered by [@jiaming2li](https://github.com/jiaming2li) and [#3551](https://github.com/apache/dubbo-go/pull/3551).
   
   cool :) @Tsukikage7 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org


Re: [I] [FEATURE] Enhance OpenTelemetry tracing diagnostics / 增强 OpenTelemetry Trace 诊断能力 [dubbo-go]

Posted by "jiaming2li (via GitHub)" <gi...@apache.org>.
jiaming2li commented on issue #3338:
URL: https://github.com/apache/dubbo-go/issues/3338#issuecomment-5100831687

   @Alanxtl, I have a question: trace span is created inside the `OTel filter`, and on the consumer side that filter runs inside the cluster invoker. So retry, load-balance and router selection all happen before the span exists, so diagnostic events can't attach to span. Do I need to add a logical client span above the cluster to let retry/load-balance/router events attach?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org


Re: [I] [FEATURE] Enhance OpenTelemetry tracing diagnostics / 增强 OpenTelemetry Trace 诊断能力 [dubbo-go]

Posted by "Alanxtl (via GitHub)" <gi...@apache.org>.
Alanxtl commented on issue #3338:
URL: https://github.com/apache/dubbo-go/issues/3338#issuecomment-5111472586

   > > > [@Alanxtl](https://github.com/Alanxtl), I have a question: trace span is created inside the `OTel filter`, and on the consumer side that filter runs inside the cluster invoker. So retry, load-balance and router selection all happen before the span exists, so diagnostic events can't attach to span. Do I need to add a logical client span above the cluster to let retry/load-balance/router events attach?
   > > 
   > > 
   > > I understand your concern
   > > Wouldn't it be cleaner to keep only the upper logical client span? It represents the whole consumer RPC from the user's perspective, and retry/load-balance/router diagnostics can be recorded as events or attributes on that span.
   > > btw, are you in our wechat group?
   > 
   > Got it. So what I need to complete is to add a new logical span above the cluster as upper span doesn't exist at present and remove the per-attempt filter spans. It is a big change, do you think it's ok?
   
   @Tsukikage7 help reply this 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org


Re: [I] [FEATURE] Enhance OpenTelemetry tracing diagnostics / 增强 OpenTelemetry Trace 诊断能力 [dubbo-go]

Posted by "jiaming2li (via GitHub)" <gi...@apache.org>.
jiaming2li commented on issue #3338:
URL: https://github.com/apache/dubbo-go/issues/3338#issuecomment-5127324435

   @Tsukikage7 the tasks you want to implement are parts of #3338, but I am assigned to this issue and I have started some of them. There are lots of unfinished works in #3562, why not try to complete them?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org


Re: [I] [FEATURE] Enhance OpenTelemetry tracing diagnostics / 增强 OpenTelemetry Trace 诊断能力 [dubbo-go]

Posted by "Oxidaner (via GitHub)" <gi...@apache.org>.
Oxidaner commented on issue #3338:
URL: https://github.com/apache/dubbo-go/issues/3338#issuecomment-4594557873

   please assign to me


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org


Re: [I] [FEATURE] Enhance OpenTelemetry tracing diagnostics / 增强 OpenTelemetry Trace 诊断能力 [dubbo-go]

Posted by "jiaming2li (via GitHub)" <gi...@apache.org>.
jiaming2li commented on issue #3338:
URL: https://github.com/apache/dubbo-go/issues/3338#issuecomment-5109000744

   > > [@Alanxtl](https://github.com/Alanxtl), I have a question: trace span is created inside the `OTel filter`, and on the consumer side that filter runs inside the cluster invoker. So retry, load-balance and router selection all happen before the span exists, so diagnostic events can't attach to span. Do I need to add a logical client span above the cluster to let retry/load-balance/router events attach?
   > 
   > I understand your concern
   > 
   > Wouldn't it be cleaner to keep only the upper logical client span? It represents the whole consumer RPC from the user's perspective, and retry/load-balance/router diagnostics can be recorded as events or attributes on that span.
   > 
   > btw, are you in our wechat group?
   
   Got it. So what I need to complete is to add a new logical span above the cluster as upper span doesn't exist at present and remove the per-attempt filter spans. It is a big change, do you think it's ok?
   
   And how to join wechat group? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org


Re: [I] [FEATURE] Enhance OpenTelemetry tracing diagnostics / 增强 OpenTelemetry Trace 诊断能力 [dubbo-go]

Posted by "Alanxtl (via GitHub)" <gi...@apache.org>.
Alanxtl commented on issue #3338:
URL: https://github.com/apache/dubbo-go/issues/3338#issuecomment-5111474579

   wechat add me by iamalannnnn


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org


Re: [I] [FEATURE] Enhance OpenTelemetry tracing diagnostics / 增强 OpenTelemetry Trace 诊断能力 [dubbo-go]

Posted by "Alanxtl (via GitHub)" <gi...@apache.org>.
Alanxtl commented on issue #3338:
URL: https://github.com/apache/dubbo-go/issues/3338#issuecomment-5102653201

   > [@Alanxtl](https://github.com/Alanxtl), I have a question: trace span is created inside the `OTel filter`, and on the consumer side that filter runs inside the cluster invoker. So retry, load-balance and router selection all happen before the span exists, so diagnostic events can't attach to span. Do I need to add a logical client span above the cluster to let retry/load-balance/router events attach?
   
   I understand your concern
   
   Wouldn't it be cleaner to keep only the upper logical client span?
   It represents the whole consumer RPC from the user's perspective, and retry/load-balance/router diagnostics can be recorded as events or attributes on that span.
   
   btw, are you in our wechat group?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org