You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2021/12/02 02:59:49 UTC
[GitHub] [druid] IvanVan opened a new pull request #12015: Add OpenTelemetry emitter extension
IvanVan opened a new pull request #12015:
URL: https://github.com/apache/druid/pull/12015
### Description
OpenTelemetry extension for distributed traces.
Use Druid emitter functionality to ingest metrics and create spans from query/time metrics. See [emitter](https://druid.apache.org/docs/latest/configuration/index.html#emitting-metrics) for more information
Use `context` field for propagation. See [druid context](https://druid.apache.org/docs/latest/querying/sql.html#http-post) for more information.
This PR has:
- [ ] been self-reviewed.
- [ ] added documentation for new or modified features or behaviors.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
- [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
- [ ] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson edited a comment on pull request #12015: OpenTelemetry emitter extension
Posted by GitBox <gi...@apache.org>.
jihoonson edited a comment on pull request #12015:
URL: https://github.com/apache/druid/pull/12015#issuecomment-1013618236
> [ERROR] Failed to execute goal de.thetaphi:forbiddenapis:3.1:check (default-cli) on project opentelemetry-emitter: Parsing signatures failed: Method not found while parsing signature: com.google.common.util.concurrent.MoreExecutors#sameThreadExecutor() -> [Help 1]
I'm not sure how this PR passed Travis CI. But [it broke the forbidden api checks](https://app.travis-ci.com/github/apache/druid/jobs/555724480). This seems like because of the different guava version that this extension uses. Any suggestions on how to fix this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on pull request #12015: OpenTelemetry emitter extension
Posted by GitBox <gi...@apache.org>.
jihoonson commented on pull request #12015:
URL: https://github.com/apache/druid/pull/12015#issuecomment-1013618236
> [ERROR] Failed to execute goal de.thetaphi:forbiddenapis:3.1:check (default-cli) on project opentelemetry-emitter: Parsing signatures failed: Method not found while parsing signature: com.google.common.util.concurrent.MoreExecutors#sameThreadExecutor() -> [Help 1]
I'm not sure how this PR passed Travis CI. But it broke the forbidden api checks. This seems like because of the different guava version that this extension uses. Any suggestions on how to fix this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] asdf2014 merged pull request #12015: OpenTelemetry emitter extension
Posted by GitBox <gi...@apache.org>.
asdf2014 merged pull request #12015:
URL: https://github.com/apache/druid/pull/12015
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] asdf2014 commented on pull request #12015: OpenTelemetry emitter extension
Posted by GitBox <gi...@apache.org>.
asdf2014 commented on pull request #12015:
URL: https://github.com/apache/druid/pull/12015#issuecomment-1013638175
@jihoonson At present, because of this https://github.com/apache/druid/pull/12034 PR, the `failOnUnresolvableSignatures` option has been deleted. After adding the old parameters back, there is no problem anymore. And the current PR's Travis runs before https://github.com/apache/druid/pull/12034, which explains why the current PR can succeed, but fails after the merge 😂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] xvrl commented on a change in pull request #12015: OpenTelemetry emitter extension
Posted by GitBox <gi...@apache.org>.
xvrl commented on a change in pull request #12015:
URL: https://github.com/apache/druid/pull/12015#discussion_r765207167
##########
File path: extensions-contrib/opentelemetry-emitter/src/main/java/org/apache/druid/emitter/opentelemetry/OpenTelemetryEmitter.java
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.emitter.opentelemetry;
+
+import io.opentelemetry.api.OpenTelemetry;
+import io.opentelemetry.api.trace.Span;
+import io.opentelemetry.api.trace.StatusCode;
+import io.opentelemetry.api.trace.Tracer;
+import io.opentelemetry.context.Context;
+import io.opentelemetry.context.Scope;
+import io.opentelemetry.context.propagation.TextMapPropagator;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.core.Emitter;
+import org.apache.druid.java.util.emitter.core.Event;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.joda.time.DateTime;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+public class OpenTelemetryEmitter implements Emitter
+{
+ static final DruidContextTextMapGetter DRUID_CONTEXT_TEXT_MAP_GETTER = new DruidContextTextMapGetter();
+ static final HashSet<String> TRACEPARENT_PROPAGATION_FIELDS = new HashSet<>(Arrays.asList(
+ "traceparent",
+ "tracestate"
Review comment:
nit, the docs mention traceparent, but they don't mention anything about tracestate or how it's being used. Can we add some details to clarify that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on pull request #12015: Add OpenTelemetry emitter extension
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on pull request #12015:
URL: https://github.com/apache/druid/pull/12015#issuecomment-986165486
@IvanVan Thanks for your contribution.
For distributed tracing, I have been working on a [side project](https://github.com/FrankChen021/bithon) by using java agent. This project not only collects metrics from any java processes, but also automatically injects opentelemetry standard tracing context in HTTP request to trace the call chain without any changes on Druid side.
Although it's still under development, I think you may be interested in it.
Here're some screenshots from my project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] xvrl commented on pull request #12015: OpenTelemetry emitter extension
Posted by GitBox <gi...@apache.org>.
xvrl commented on pull request #12015:
URL: https://github.com/apache/druid/pull/12015#issuecomment-989140362
@FrankChen021 I agree we can gain a lot by deploying the agent as well. While it does get us visibility into more things, it doesn't get us access to the query context metadata which is useful to get into the trace context.
Ultimately I think we want those two approaches to converge and and work better together, so this is a starting point.
Another reason to have this extension is to give us a place where we can expose all the other druid metrics via opentelemetry.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 edited a comment on pull request #12015: OpenTelemetry emitter extension
Posted by GitBox <gi...@apache.org>.
FrankChen021 edited a comment on pull request #12015:
URL: https://github.com/apache/druid/pull/12015#issuecomment-1012133966
@xvrl Sorry for the late response. It's not a hard thing for the agent to get the data in the query context, Here's an example:
BWT, I'm not giving objections on the merging of this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on pull request #12015: OpenTelemetry emitter extension
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on pull request #12015:
URL: https://github.com/apache/druid/pull/12015#issuecomment-1012133966
@xvrl Sorry for the late response. It's not a hard thing for the agent to get the data in the query context, Here's an example:
BWT, I'm not giving objections on the merging this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on pull request #12015: OpenTelemetry emitter extension
Posted by GitBox <gi...@apache.org>.
jihoonson commented on pull request #12015:
URL: https://github.com/apache/druid/pull/12015#issuecomment-1013621293
This is a blocker and we should fix it as soon as possible since it is blocking every PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] asdf2014 commented on pull request #12015: OpenTelemetry emitter extension
Posted by GitBox <gi...@apache.org>.
asdf2014 commented on pull request #12015:
URL: https://github.com/apache/druid/pull/12015#issuecomment-1013636027
@jihoonson How about this solution https://github.com/apache/druid/pull/12158
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org