You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/05/17 04:51:32 UTC
[GitHub] [incubator-doris] luozenglin opened a new pull request, #9608: [Feature] Introducing OpenTelemetry to improve the observability of Doris.
luozenglin opened a new pull request, #9608:
URL: https://github.com/apache/incubator-doris/pull/9608
The collection of query traces is implemented in fe and be, and the spans are exported to zipkin.
# Proposed changes
Issue Number: #9577
I plan to introduce OpenTelemetry to collect telemetry data, such as traces, metrics, logs, to improve the observability of Doris.
### The first phase uses OpenTelemetry to implement the collection and export of traces.
A trace represents the execution process of a single request in the system, span represents a logical operation unit with start time and execution duration in the system, and multiple spans form a trace.
1. support the collection and export of query traces
2. support the collection and export of load traces
### The second phase integrates metrics and logs to OpenTelemtry standard.
## Problem Summary:
This pr implements the collection and sending of query traces. fe turns on traces, collects spans and propagates the traces information to be via prc. eventually each node exports the collected spans to zipkin.
```
┌──────────────────────┐
│ fe │
│ │
│ span root │
│ │ │
│ ┌───────┴────┐ ├────────────────┐
│ │ │ │ │
│ span A span B │ │
│ │ │ │
│ │ │ │
└────────────────┼─────┘ │
│ │
┌─────┴───────────────┐ │
│ │ │
┌──────────┼───────────┐ │
│ │ be │ be │ ┌────────────┐
│ │ │ │ │ │
│ span C │ │ span │ │
│ │ ├────────────────┴──────────► zipkin │
│ ┌─────┴───────┐ │ │ │
│ │ │ │ │ │
│ span D span E │ │ │
│ │ └────────────┘
└──────────────────────┘
```
### Example：
1. Start zipkin:
```
curl -sSL https://zipkin.io/quickstart.sh | bash -s
java -jar zipkin.jar
```
2. Add Configuration：
fe.conf:
```
enable_tracing=true
trace_export_url=http://127.0.0.1:9411/api/v2/spans
```
be.conf:
```
enable_tracing=true
trace_export_url=http://127.0.0.1:9411/api/v2/spans
```
3. Execute a query
4. From zipkin UI：
## Checklist(Required)
1. Does it affect the original behavior: (No)
2. Has unit tests been added: (No)
3. Has document been added or modified: (No)
4. Does it need to update dependencies: (Yes)
5. Are there any changes that cannot be rolled back: (No)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] jackwener commented on a diff in pull request #9608: [Feature] Introducing OpenTelemetry to improve the observability of Doris.
Posted by GitBox <gi...@apache.org>.
jackwener commented on code in PR #9608:
URL: https://github.com/apache/incubator-doris/pull/9608#discussion_r877790244
##########
thirdparty/patches/opentelemetry-cpp-1.2.0.patch:
##########
@@ -0,0 +1,374 @@
+diff --git a/api/include/opentelemetry/common/threadlocal.h b/api/include/opentelemetry/common/threadlocal.h
Review Comment:
😂 you git push the `patch` file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] SaintBacchus commented on pull request #9608: [Feature] Introducing OpenTelemetry to improve the observability of Doris.
Posted by GitBox <gi...@apache.org>.
SaintBacchus commented on PR #9608:
URL: https://github.com/apache/incubator-doris/pull/9608#issuecomment-1135572749
Will this feature collect the plan frament execution metrics as the sql profile?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #9608: [Feature] Introducing OpenTelemetry to improve the observability of Doris.
Posted by GitBox <gi...@apache.org>.
morningman commented on PR #9608:
URL: https://github.com/apache/incubator-doris/pull/9608#issuecomment-1146570732
> Will this feature collect the plan frament execution metrics as the sql profile?
Work in progress
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] luozenglin closed pull request #9608: [Feature] Introducing OpenTelemetry to improve the observability of Doris.
Posted by GitBox <gi...@apache.org>.
luozenglin closed pull request #9608: [Feature] Introducing OpenTelemetry to improve the observability of Doris.
URL: https://github.com/apache/doris/pull/9608
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] spaces-X commented on pull request #9608: [Feature] Introducing OpenTelemetry to improve the observability of Doris.
Posted by GitBox <gi...@apache.org>.
spaces-X commented on PR #9608:
URL: https://github.com/apache/incubator-doris/pull/9608#issuecomment-1129766848
Nice work!
i just wonder what percentage of performance drawback will be caused by increasing the trace in general, looking forward to do a performance test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] luozenglin commented on pull request #9608: [Feature] Introducing OpenTelemetry to improve the observability of Doris.
Posted by GitBox <gi...@apache.org>.
luozenglin commented on PR #9608:
URL: https://github.com/apache/incubator-doris/pull/9608#issuecomment-1132560841
> Nice work! i just wonder what percentage of performance drawback will be caused by increasing the trace in general, looking forward to do a performance test
Yeah, I will do performance testing when this feature is about to be completed. In order to reduce performance loss, performance optimization may be required at that time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] luozenglin commented on a diff in pull request #9608: [Feature] Introducing OpenTelemetry to improve the observability of Doris.
Posted by GitBox <gi...@apache.org>.
luozenglin commented on code in PR #9608:
URL: https://github.com/apache/incubator-doris/pull/9608#discussion_r877826139
##########
thirdparty/patches/opentelemetry-cpp-1.2.0.patch:
##########
@@ -0,0 +1,374 @@
+diff --git a/api/include/opentelemetry/common/threadlocal.h b/api/include/opentelemetry/common/threadlocal.h
Review Comment:
This patch is to solve the problem that the opentelemtry-cpp library depends on `GLIBC_2.18`, and replaces the `thread_local` keyword in the opentelemtry-cpp library.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org