You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "mrproliu (via GitHub)" <gi...@apache.org> on 2023/03/22 06:54:47 UTC
[GitHub] [skywalking] mrproliu opened a new pull request, #10575: Support cross thread trace profiling
mrproliu opened a new pull request, #10575:
URL: https://github.com/apache/skywalking/pull/10575
Following https://github.com/apache/skywalking/issues/10373, support the query of the cross thread trace profiling.
Also, the `readNullableMetricsValue` query been added with default implementation, this should be update in other PR.
- [ ] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #<issue number>.
- [x] Update the [`CHANGES` log](https://github.com/apache/skywalking/blob/master/docs/en/changes/changes.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #10575: Support cross thread trace profiling
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #10575:
URL: https://github.com/apache/skywalking/pull/10575#issuecomment-1479084526
> FYI @apache/skywalking-committers After this gets merged, we are going to hold the demo update until we have UI side adoption. Otherwise, the profiling relative features are broken.
BTW, as @Fine0830 almost finished https://github.com/apache/skywalking-booster-ui/pull/243(new topology), I would like to merge that first and make an update for demo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng merged pull request #10575: Support cross thread trace profiling
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng merged PR #10575:
URL: https://github.com/apache/skywalking/pull/10575
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #10575: Support cross thread trace profiling
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #10575:
URL: https://github.com/apache/skywalking/pull/10575#discussion_r1144365902
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/profiling/trace/ProfileTaskQueryService.java:
##########
@@ -161,38 +182,126 @@ public List<ProfileTaskLog> getProfileTaskLogs(final String taskID) throws IOExc
return findMatchedLogs(taskID, taskLogList);
}
- /**
- * search profiled traces
- */
- public List<BasicTrace> getTaskTraces(String taskId) throws IOException {
- return getProfileThreadSnapshotQueryDAO().queryProfiledSegments(taskId);
+ public ProfileAnalyzation getProfileAnalyze(final List<SegmentProfileAnalyzeQuery> queries) throws IOException {
+ return profileAnalyzer.analyze(queries);
}
- public ProfileAnalyzation getProfileAnalyze(final String segmentId,
- final List<ProfileAnalyzeTimeRange> timeRanges) throws IOException {
- return profileAnalyzer.analyze(segmentId, timeRanges);
+ public List<SegmentRecord> getTaskSegments(String taskId) throws IOException {
+ final List<String> profiledSegmentIdList = getProfileThreadSnapshotQueryDAO().queryProfiledSegmentIdList(taskId);
+ return getTraceQueryDAO().queryBySegmentIdList(profiledSegmentIdList);
}
- public ProfiledSegment getProfiledSegment(String segmentId) throws IOException {
- SegmentRecord segmentRecord = getProfileThreadSnapshotQueryDAO().getProfiledSegment(segmentId);
- if (segmentRecord == null) {
- return null;
+ public List<ProfiledTraceSegments> getProfileTaskSegments(String taskId) throws IOException {
+ // query all profiled segments
+ final List<String> profiledSegmentIdList = getProfileThreadSnapshotQueryDAO().queryProfiledSegmentIdList(taskId);
+ final List<SegmentRecord> segmentRecords = getTraceQueryDAO().queryBySegmentIdList(profiledSegmentIdList);
+ if (CollectionUtils.isEmpty(segmentRecords)) {
+ return Collections.emptyList();
}
+ final Map<String, List<String>> traceWithInstances = segmentRecords.stream().collect(Collectors.toMap(
+ SegmentRecord::getTraceId,
+ s -> new ArrayList(List.of(s.getServiceInstanceId())),
+ (s1, s2) -> {
+ s1.addAll(s2);
+ return s1;
+ }));
+
+ // query all profiled segments related segments(same traceId and instanceId)
+ final Set<String> traceIdList = new HashSet<>(segmentRecords.size());
+ final Set<String> instanceIdList = new HashSet<>(segmentRecords.size());
+ for (SegmentRecord segment : segmentRecords) {
+ traceIdList.add(segment.getTraceId());
+ instanceIdList.add(segment.getServiceInstanceId());
+ }
+ final List<SegmentRecord> traceRelatedSegments = getTraceQueryDAO().queryByTraceIdWithInstanceId(
+ new ArrayList<>(traceIdList),
+ new ArrayList<>(instanceIdList));
+
+ // group by the traceId + service instanceId
+ final Map<String, List<SegmentRecord>> instanceTraceWithSegments = traceRelatedSegments.stream().filter(s -> {
+ final List<String> includingInstances = traceWithInstances.get(s.getTraceId());
+ return includingInstances.contains(s.getServiceInstanceId());
+ }).collect(Collectors.toMap(
+ s -> s.getTraceId() + s.getServiceInstanceId(),
+ s -> new ArrayList<>(List.of(s)),
+ (s1, s2) -> {
+ s1.addAll(s2);
+ return s1;
+ }));
+
+ // build result
+ return instanceTraceWithSegments.values().stream()
+ .flatMap(s -> buildProfiledSegmentsList(s, profiledSegmentIdList).stream())
+ .collect(Collectors.toList());
+ }
- ProfiledSegment profiledSegment = new ProfiledSegment();
- SegmentObject segmentObject = SegmentObject.parseFrom(segmentRecord.getDataBinary());
- profiledSegment.getSpans().addAll(buildProfiledSpanList(segmentObject));
+ protected List<ProfiledTraceSegments> buildProfiledSegmentsList(List<SegmentRecord> segmentRecords, List<String> profiledSegmentIdList) {
Review Comment:
Let's remove `cross_process` ref in the process to avoid UI confusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mrproliu commented on pull request #10575: Support cross thread trace profiling
Posted by "mrproliu (via GitHub)" <gi...@apache.org>.
mrproliu commented on PR #10575:
URL: https://github.com/apache/skywalking/pull/10575#issuecomment-1479598943
> @mrproliu There is a snapshot exporter tool. Do we need some update about that?
The tools have been updated to support multiple segments. If users want to use them, they need to update them too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #10575: Support cross thread trace profiling
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #10575:
URL: https://github.com/apache/skywalking/pull/10575#discussion_r1144328755
##########
oap-server/server-query-plugin/query-graphql-plugin/src/main/java/org/apache/skywalking/oap/query/graphql/resolver/MetricQuery.java:
##########
@@ -92,6 +93,14 @@ public IntValues getLinearIntValues(final MetricCondition metrics,
return metricsValues.getValues();
}
+ public NullableValue readNullableMetricsValue(MetricsCondition condition, Duration duration) {
+ // TODO default implantation
Review Comment:
FYI @wankai123
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #10575: Support cross thread trace profiling
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #10575:
URL: https://github.com/apache/skywalking/pull/10575#discussion_r1144921883
##########
docs/en/changes/changes.md:
##########
@@ -17,6 +17,7 @@
* Support continuous profiling feature.
* Support collect process level related metrics.
* Fix K8sRetag reads the wrong k8s service from the cache due to a possible namespace mismatch.
+* Support cross thread trace profiling.
Review Comment:
```suggestion
* [Breaking Change] Support cross-thread trace profiling. The data structure and query APIs are changed.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #10575: Support cross thread trace profiling
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #10575:
URL: https://github.com/apache/skywalking/pull/10575#issuecomment-1479473207
@mrproliu There is a snapshot exporter tool. Do we need some update about that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mrproliu commented on a diff in pull request #10575: Support cross thread trace profiling
Posted by "mrproliu (via GitHub)" <gi...@apache.org>.
mrproliu commented on code in PR #10575:
URL: https://github.com/apache/skywalking/pull/10575#discussion_r1144371945
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/profiling/trace/ProfileTaskQueryService.java:
##########
@@ -161,38 +182,126 @@ public List<ProfileTaskLog> getProfileTaskLogs(final String taskID) throws IOExc
return findMatchedLogs(taskID, taskLogList);
}
- /**
- * search profiled traces
- */
- public List<BasicTrace> getTaskTraces(String taskId) throws IOException {
- return getProfileThreadSnapshotQueryDAO().queryProfiledSegments(taskId);
+ public ProfileAnalyzation getProfileAnalyze(final List<SegmentProfileAnalyzeQuery> queries) throws IOException {
+ return profileAnalyzer.analyze(queries);
}
- public ProfileAnalyzation getProfileAnalyze(final String segmentId,
- final List<ProfileAnalyzeTimeRange> timeRanges) throws IOException {
- return profileAnalyzer.analyze(segmentId, timeRanges);
+ public List<SegmentRecord> getTaskSegments(String taskId) throws IOException {
+ final List<String> profiledSegmentIdList = getProfileThreadSnapshotQueryDAO().queryProfiledSegmentIdList(taskId);
+ return getTraceQueryDAO().queryBySegmentIdList(profiledSegmentIdList);
}
- public ProfiledSegment getProfiledSegment(String segmentId) throws IOException {
- SegmentRecord segmentRecord = getProfileThreadSnapshotQueryDAO().getProfiledSegment(segmentId);
- if (segmentRecord == null) {
- return null;
+ public List<ProfiledTraceSegments> getProfileTaskSegments(String taskId) throws IOException {
+ // query all profiled segments
+ final List<String> profiledSegmentIdList = getProfileThreadSnapshotQueryDAO().queryProfiledSegmentIdList(taskId);
+ final List<SegmentRecord> segmentRecords = getTraceQueryDAO().queryBySegmentIdList(profiledSegmentIdList);
+ if (CollectionUtils.isEmpty(segmentRecords)) {
+ return Collections.emptyList();
}
+ final Map<String, List<String>> traceWithInstances = segmentRecords.stream().collect(Collectors.toMap(
+ SegmentRecord::getTraceId,
+ s -> new ArrayList(List.of(s.getServiceInstanceId())),
+ (s1, s2) -> {
+ s1.addAll(s2);
+ return s1;
+ }));
+
+ // query all profiled segments related segments(same traceId and instanceId)
+ final Set<String> traceIdList = new HashSet<>(segmentRecords.size());
+ final Set<String> instanceIdList = new HashSet<>(segmentRecords.size());
+ for (SegmentRecord segment : segmentRecords) {
+ traceIdList.add(segment.getTraceId());
+ instanceIdList.add(segment.getServiceInstanceId());
+ }
+ final List<SegmentRecord> traceRelatedSegments = getTraceQueryDAO().queryByTraceIdWithInstanceId(
+ new ArrayList<>(traceIdList),
+ new ArrayList<>(instanceIdList));
+
+ // group by the traceId + service instanceId
+ final Map<String, List<SegmentRecord>> instanceTraceWithSegments = traceRelatedSegments.stream().filter(s -> {
+ final List<String> includingInstances = traceWithInstances.get(s.getTraceId());
+ return includingInstances.contains(s.getServiceInstanceId());
+ }).collect(Collectors.toMap(
+ s -> s.getTraceId() + s.getServiceInstanceId(),
+ s -> new ArrayList<>(List.of(s)),
+ (s1, s2) -> {
+ s1.addAll(s2);
+ return s1;
+ }));
+
+ // build result
+ return instanceTraceWithSegments.values().stream()
+ .flatMap(s -> buildProfiledSegmentsList(s, profiledSegmentIdList).stream())
+ .collect(Collectors.toList());
+ }
- ProfiledSegment profiledSegment = new ProfiledSegment();
- SegmentObject segmentObject = SegmentObject.parseFrom(segmentRecord.getDataBinary());
- profiledSegment.getSpans().addAll(buildProfiledSpanList(segmentObject));
+ protected List<ProfiledTraceSegments> buildProfiledSegmentsList(List<SegmentRecord> segmentRecords, List<String> profiledSegmentIdList) {
Review Comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mrproliu commented on pull request #10575: Support cross thread trace profiling
Posted by "mrproliu (via GitHub)" <gi...@apache.org>.
mrproliu commented on PR #10575:
URL: https://github.com/apache/skywalking/pull/10575#issuecomment-1479653202
> Is there anything expected to change for https://skywalking.apache.org/docs/main/next/en/guides/backend-profile-export/?
No, this file only descript how to use this export tool, and the usage has no change.
> Your next move should be about docs, right?
Yes, I will write the documentation for cross-thread profiling and continuous profiling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #10575: Support cross thread trace profiling
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #10575:
URL: https://github.com/apache/skywalking/pull/10575#issuecomment-1479015166
FYI @apache/skywalking-committers After this gets merged, we are going to hold the demo update until we have UI side adoption. Otherwise, the profiling relative features are broken.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mrproliu commented on pull request #10575: Support cross thread trace profiling
Posted by "mrproliu (via GitHub)" <gi...@apache.org>.
mrproliu commented on PR #10575:
URL: https://github.com/apache/skywalking/pull/10575#issuecomment-1479014285
> Is this going to break UI temporarily, right?
Yes, It need the UI to adapt with the new query protocol.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #10575: Support cross thread trace profiling
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #10575:
URL: https://github.com/apache/skywalking/pull/10575#issuecomment-1479012667
Is this going to break UI temporarily, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #10575: Support cross thread trace profiling
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #10575:
URL: https://github.com/apache/skywalking/pull/10575#issuecomment-1479636635
Is there anything expected to change for https://skywalking.apache.org/docs/main/next/en/guides/backend-profile-export/?
Notice, the profiling docs should be moved from contribute doc to new profiling doc. https://skywalking.apache.org/docs/main/next/en/guides/readme/#profile
Your next move should be about docs, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org