[jira] [Commented] (HBASE-26366) ZK interaction during Master startup produces loads of root-less spans

["Tak-Lon (Stephen) Wu (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-26366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17438454#comment-17438454 ] 

Tak-Lon (Stephen) Wu commented on HBASE-26366:
----------------------------------------------

I found the potential issue of this single span(s), basically it's due to the static class ZKUtil calls {{zkw.getRecoverableZooKeeper().create}} whenever  a znode or znode with its parent (normally /hbase) need to be created, e.g. {{ZKUtil.createAndWatch}}, {{ZKUtil.createWithParents}}, {{ZKUtil.getData}}

If we're going to group all these spans, we need to wrap a parent span at their parent level, e.g. I found at few places, 1/ ClientZKSyncer, 2/ Procedure / ProcedureMember, 3/ HMaster#startActiveMasterManager (this is a private method) and etc.

if this is the direction, the scope of this JIRAs are big and may need to create more sub-tasks.

Moreover, based few experiment I did to group spans for HMaster#startActiveMasterManager, IMO we should replace some current code of  {{Span span = TraceUtil.getGlobalTracer()}}or {{TraceUtil.trace()}}  with [{{@WithSpan}} annotation|https://github.com/open-telemetry/opentelemetry-java/blob/main/extensions/annotations/src/main/java/io/opentelemetry/extension/annotations/WithSpan.java] (see the [code example|https://github.com/open-telemetry/opentelemetry-java/blob/main/extensions/annotations/src/test/java/io/opentelemetry/extension/annotations/WithSpanUsageExamples.java] and don't hack too much code in TraceUtils)

> ZK interaction during Master startup produces loads of root-less spans
> ----------------------------------------------------------------------
>
>                 Key: HBASE-26366
>                 URL: https://issues.apache.org/jira/browse/HBASE-26366
>             Project: HBase
>          Issue Type: Bug
>          Components: tracing
>    Affects Versions: 2.5.0, 3.0.0-alpha-2
>            Reporter: Nick Dimiduk
>            Priority: Major
>         Attachments: image.png
>
>
> Enable tracing with {{-Dotel.traces.sampler=always_on}} and launch the standalone master, check the trace store. We see lots of single-span traces, each a call to different {{RecoverableZooKeeper}} methods, like {{create}} and {{exists}}. Whatever master startup thread this is should define its own root span.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)