You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "mrproliu (via GitHub)" <gi...@apache.org> on 2023/06/02 11:35:40 UTC
[GitHub] [skywalking] mrproliu opened a new pull request, #10878: Adapt the continuous profiling task query GraphQL
mrproliu opened a new pull request, #10878:
URL: https://github.com/apache/skywalking/pull/10878
Following the https://github.com/apache/skywalking-query-protocol/pull/116, update the related implementation. I will do other PR to adapt the E2E.
- [ ] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #<issue number>.
- [x] Update the [`CHANGES` log](https://github.com/apache/skywalking/blob/master/docs/en/changes/changes.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #10878: Adapt the continuous profiling task query GraphQL
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #10878:
URL: https://github.com/apache/skywalking/pull/10878#discussion_r1214348346
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/profiling/continuous/ContinuousProfilingQueryService.java:
##########
@@ -76,10 +119,118 @@ public List<ContinuousProfilingPolicyTarget> queryContinuousProfilingServiceTarg
return result;
}).collect(Collectors.toList());
- return ContinuousProfilingPolicyTarget.builder()
+ final ContinuousProfilingPolicyTarget target = ContinuousProfilingPolicyTarget.builder()
.type(type)
.checkItems(items)
.build();
+
+ Optional.ofNullable(summaryMap.get(EBPFProfilingTargetType.valueOf(type).value()))
+ .ifPresent(summary -> {
+ target.setTriggeredCount(summary.getCount());
+ target.setLastTriggerTimestamp(summary.getLastTriggerTime());
+ });
+ return target;
}).collect(Collectors.toList());
}
+
+ public List<ContinuousProfilingMonitoringInstance> queryContinuousProfilingMonitoringInstances(String serviceId, ContinuousProfilingTargetType target) throws IOException {
+ // query all processes under same service
+ final List<Process> processes = getMetadataQueryDAO().listProcesses(serviceId, null, 0, 0);
+ if (CollectionUtils.isEmpty(processes)) {
+ return Collections.emptyList();
+ }
+ // query all triggered tasks
+ final List<EBPFProfilingTaskRecord> records = queryRecentTriggeredTasks(serviceId, List.of(target));
+
+ // query instances metadata
+ final Map<String, List<Process>> instancesProcesses = processes.stream().collect(Collectors.groupingBy(Process::getInstanceId));
+ final List<ServiceInstance> instanceIdWithMetadata = getMetadataQueryDAO().getInstances(Arrays.asList(instancesProcesses.keySet().toArray(new String[0])));
+
+ // build instance, process summary
+ final Map<String, EBPFProfilingTaskSummary> instanceSummary = buildSummaryByKey(records, EBPFProfilingTaskRecord::getInstanceId);
+ final Map<String, EBPFProfilingTaskSummary> processSummary = buildSummaryByKey(records, r -> {
+ final EBPFProfilingTaskContinuousProfiling continuousProfiling = GSON.fromJson(r.getContinuousProfilingJson(), EBPFProfilingTaskContinuousProfiling.class);
+ return continuousProfiling.getProcessId();
+ });
+
+ // build result
+ return instanceIdWithMetadata.stream().map(instance -> {
+ final ContinuousProfilingMonitoringInstance result = new ContinuousProfilingMonitoringInstance();
+ result.setId(instance.getId());
+ result.setName(instance.getName());
+ result.setAttributes(instance.getAttributes());
+ final EBPFProfilingTaskSummary summary = instanceSummary.get(instance.getId());
+ if (summary != null) {
+ result.setTriggeredCount(summary.getCount());
+ result.setLastTriggerTimestamp(summary.getLastTriggerTime());
+ }
+
+ result.setProcesses(instancesProcesses.getOrDefault(instance.getId(), List.of())
+ .stream().map(p -> {
+ final ContinuousProfilingMonitoringProcess process = new ContinuousProfilingMonitoringProcess();
+ process.setId(p.getId());
+ process.setName(p.getName());
+ process.setDetectType(p.getDetectType());
+ process.setLabels(p.getLabels());
+
+ final EBPFProfilingTaskSummary processSummaryItem = processSummary.get(p.getId());
+ if (processSummaryItem != null) {
+ process.setTriggeredCount(processSummaryItem.getCount());
+ process.setLastTriggerTimestamp(processSummaryItem.getLastTriggerTime());
+ }
+
+ return process;
+ }).collect(Collectors.toList()));
+ return result;
+ }).collect(Collectors.toList());
+ }
+
+ private <T> Map<T, EBPFProfilingTaskSummary> buildSummaryByKey(List<EBPFProfilingTaskRecord> records, Function<EBPFProfilingTaskRecord, T> groupBy) {
+ return records.stream().collect(Collectors.groupingByConcurrent(groupBy, buildSummaryCollector()));
+ }
+
+ private List<EBPFProfilingTaskRecord> queryRecentTriggeredTasks(String serviceId, Collection<ContinuousProfilingTargetType> targets) throws IOException {
+ final Calendar timeInstance = Calendar.getInstance();
+ timeInstance.add(Calendar.HOUR, -RECENT_TRIGGERED_HOURS);
+ return getEbpfProfilingTaskDAO().queryTasksByTargets(serviceId, null,
+ targets.stream().map(EBPFProfilingTargetType::valueOf).collect(Collectors.toList()),
+ EBPFProfilingTriggerType.CONTINUOUS_PROFILING, timeInstance.getTimeInMillis(), 0);
+ }
+
+ /**
+ * Summary all records to one summary
+ */
+ private Collector<EBPFProfilingTaskRecord, EBPFProfilingTaskSummary, EBPFProfilingTaskSummary> buildSummaryCollector() {
+ return Collector.of(EBPFProfilingTaskSummary::new,
+ (result, task) -> {
+ result.setCount(result.getCount() + 1);
+ if (task.getStartTime() > result.getLastTriggerTime()) {
+ result.setLastTriggerTime(task.getStartTime());
+ }
+ result.getRecords().add(task);
+ },
+ (result1, result2) -> {
+ result1.setCount(result1.getCount() + result2.getCount());
+ if (result2.getLastTriggerTime() > result1.getLastTriggerTime()) {
+ result1.setLastTriggerTime(result2.getLastTriggerTime());
+ }
+ result1.getRecords().addAll(result2.getRecords());
+ return result1;
+ });
+ }
+
+ @Data
+ private static class EBPFProfilingTaskSummary {
+ // count of triggered tasks
+ private int count;
+ // last trigger time
+ private long lastTriggerTime;
+ // all triggered tasks
+ private List<EBPFProfilingTaskRecord> records;
Review Comment:
I think in the summary page, we don't need this. Don't we expose the query for details separately?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #10878: Adapt the continuous profiling task query GraphQL
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #10878:
URL: https://github.com/apache/skywalking/pull/10878#discussion_r1214358813
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/profiling/continuous/ContinuousProfilingQueryService.java:
##########
@@ -76,10 +119,118 @@ public List<ContinuousProfilingPolicyTarget> queryContinuousProfilingServiceTarg
return result;
}).collect(Collectors.toList());
- return ContinuousProfilingPolicyTarget.builder()
+ final ContinuousProfilingPolicyTarget target = ContinuousProfilingPolicyTarget.builder()
.type(type)
.checkItems(items)
.build();
+
+ Optional.ofNullable(summaryMap.get(EBPFProfilingTargetType.valueOf(type).value()))
+ .ifPresent(summary -> {
+ target.setTriggeredCount(summary.getCount());
+ target.setLastTriggerTimestamp(summary.getLastTriggerTime());
+ });
+ return target;
}).collect(Collectors.toList());
}
+
+ public List<ContinuousProfilingMonitoringInstance> queryContinuousProfilingMonitoringInstances(String serviceId, ContinuousProfilingTargetType target) throws IOException {
+ // query all processes under same service
+ final List<Process> processes = getMetadataQueryDAO().listProcesses(serviceId, null, 0, 0);
+ if (CollectionUtils.isEmpty(processes)) {
+ return Collections.emptyList();
+ }
+ // query all triggered tasks
+ final List<EBPFProfilingTaskRecord> records = queryRecentTriggeredTasks(serviceId, List.of(target));
+
+ // query instances metadata
+ final Map<String, List<Process>> instancesProcesses = processes.stream().collect(Collectors.groupingBy(Process::getInstanceId));
+ final List<ServiceInstance> instanceIdWithMetadata = getMetadataQueryDAO().getInstances(Arrays.asList(instancesProcesses.keySet().toArray(new String[0])));
+
+ // build instance, process summary
+ final Map<String, EBPFProfilingTaskSummary> instanceSummary = buildSummaryByKey(records, EBPFProfilingTaskRecord::getInstanceId);
+ final Map<String, EBPFProfilingTaskSummary> processSummary = buildSummaryByKey(records, r -> {
+ final EBPFProfilingTaskContinuousProfiling continuousProfiling = GSON.fromJson(r.getContinuousProfilingJson(), EBPFProfilingTaskContinuousProfiling.class);
+ return continuousProfiling.getProcessId();
+ });
+
+ // build result
+ return instanceIdWithMetadata.stream().map(instance -> {
+ final ContinuousProfilingMonitoringInstance result = new ContinuousProfilingMonitoringInstance();
+ result.setId(instance.getId());
+ result.setName(instance.getName());
+ result.setAttributes(instance.getAttributes());
+ final EBPFProfilingTaskSummary summary = instanceSummary.get(instance.getId());
+ if (summary != null) {
+ result.setTriggeredCount(summary.getCount());
+ result.setLastTriggerTimestamp(summary.getLastTriggerTime());
+ }
+
+ result.setProcesses(instancesProcesses.getOrDefault(instance.getId(), List.of())
+ .stream().map(p -> {
+ final ContinuousProfilingMonitoringProcess process = new ContinuousProfilingMonitoringProcess();
+ process.setId(p.getId());
+ process.setName(p.getName());
+ process.setDetectType(p.getDetectType());
+ process.setLabels(p.getLabels());
+
+ final EBPFProfilingTaskSummary processSummaryItem = processSummary.get(p.getId());
+ if (processSummaryItem != null) {
+ process.setTriggeredCount(processSummaryItem.getCount());
+ process.setLastTriggerTimestamp(processSummaryItem.getLastTriggerTime());
+ }
+
+ return process;
+ }).collect(Collectors.toList()));
+ return result;
+ }).collect(Collectors.toList());
+ }
+
+ private <T> Map<T, EBPFProfilingTaskSummary> buildSummaryByKey(List<EBPFProfilingTaskRecord> records, Function<EBPFProfilingTaskRecord, T> groupBy) {
+ return records.stream().collect(Collectors.groupingByConcurrent(groupBy, buildSummaryCollector()));
+ }
+
+ private List<EBPFProfilingTaskRecord> queryRecentTriggeredTasks(String serviceId, Collection<ContinuousProfilingTargetType> targets) throws IOException {
+ final Calendar timeInstance = Calendar.getInstance();
+ timeInstance.add(Calendar.HOUR, -RECENT_TRIGGERED_HOURS);
+ return getEbpfProfilingTaskDAO().queryTasksByTargets(serviceId, null,
+ targets.stream().map(EBPFProfilingTargetType::valueOf).collect(Collectors.toList()),
+ EBPFProfilingTriggerType.CONTINUOUS_PROFILING, timeInstance.getTimeInMillis(), 0);
+ }
+
+ /**
+ * Summary all records to one summary
+ */
+ private Collector<EBPFProfilingTaskRecord, EBPFProfilingTaskSummary, EBPFProfilingTaskSummary> buildSummaryCollector() {
+ return Collector.of(EBPFProfilingTaskSummary::new,
+ (result, task) -> {
+ result.setCount(result.getCount() + 1);
+ if (task.getStartTime() > result.getLastTriggerTime()) {
+ result.setLastTriggerTime(task.getStartTime());
+ }
+ result.getRecords().add(task);
+ },
+ (result1, result2) -> {
+ result1.setCount(result1.getCount() + result2.getCount());
+ if (result2.getLastTriggerTime() > result1.getLastTriggerTime()) {
+ result1.setLastTriggerTime(result2.getLastTriggerTime());
+ }
+ result1.getRecords().addAll(result2.getRecords());
+ return result1;
+ });
+ }
+
+ @Data
+ private static class EBPFProfilingTaskSummary {
+ // count of triggered tasks
+ private int count;
+ // last trigger time
+ private long lastTriggerTime;
+ // all triggered tasks
+ private List<EBPFProfilingTaskRecord> records;
Review Comment:
Oops, this is a temp list only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #10878: Adapt the continuous profiling task query GraphQL
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #10878:
URL: https://github.com/apache/skywalking/pull/10878#discussion_r1214339128
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/profiling/continuous/ContinuousProfilingQueryService.java:
##########
@@ -76,10 +119,118 @@ public List<ContinuousProfilingPolicyTarget> queryContinuousProfilingServiceTarg
return result;
}).collect(Collectors.toList());
- return ContinuousProfilingPolicyTarget.builder()
+ final ContinuousProfilingPolicyTarget target = ContinuousProfilingPolicyTarget.builder()
.type(type)
.checkItems(items)
.build();
+
+ Optional.ofNullable(summaryMap.get(EBPFProfilingTargetType.valueOf(type).value()))
+ .ifPresent(summary -> {
+ target.setTriggeredCount(summary.getCount());
+ target.setLastTriggerTimestamp(summary.getLastTriggerTime());
+ });
+ return target;
}).collect(Collectors.toList());
}
+
+ public List<ContinuousProfilingMonitoringInstance> queryContinuousProfilingMonitoringInstances(String serviceId, ContinuousProfilingTargetType target) throws IOException {
+ // query all processes under same service
+ final List<Process> processes = getMetadataQueryDAO().listProcesses(serviceId, null, 0, 0);
+ if (CollectionUtils.isEmpty(processes)) {
+ return Collections.emptyList();
+ }
+ // query all triggered tasks
+ final List<EBPFProfilingTaskRecord> records = queryRecentTriggeredTasks(serviceId, List.of(target));
+
+ // query instances metadata
Review Comment:
```suggestion
// query the metadata of instances
```
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/profiling/continuous/ContinuousProfilingQueryService.java:
##########
@@ -76,10 +119,118 @@ public List<ContinuousProfilingPolicyTarget> queryContinuousProfilingServiceTarg
return result;
}).collect(Collectors.toList());
- return ContinuousProfilingPolicyTarget.builder()
+ final ContinuousProfilingPolicyTarget target = ContinuousProfilingPolicyTarget.builder()
.type(type)
.checkItems(items)
.build();
+
+ Optional.ofNullable(summaryMap.get(EBPFProfilingTargetType.valueOf(type).value()))
+ .ifPresent(summary -> {
+ target.setTriggeredCount(summary.getCount());
+ target.setLastTriggerTimestamp(summary.getLastTriggerTime());
+ });
+ return target;
}).collect(Collectors.toList());
}
+
+ public List<ContinuousProfilingMonitoringInstance> queryContinuousProfilingMonitoringInstances(String serviceId, ContinuousProfilingTargetType target) throws IOException {
+ // query all processes under same service
+ final List<Process> processes = getMetadataQueryDAO().listProcesses(serviceId, null, 0, 0);
+ if (CollectionUtils.isEmpty(processes)) {
+ return Collections.emptyList();
+ }
+ // query all triggered tasks
+ final List<EBPFProfilingTaskRecord> records = queryRecentTriggeredTasks(serviceId, List.of(target));
+
+ // query instances metadata
Review Comment:
```suggestion
// Query the metadata of instances
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #10878: Adapt the continuous profiling task query GraphQL
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #10878:
URL: https://github.com/apache/skywalking/pull/10878#discussion_r1214338786
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/profiling/continuous/ContinuousProfilingQueryService.java:
##########
@@ -76,10 +119,118 @@ public List<ContinuousProfilingPolicyTarget> queryContinuousProfilingServiceTarg
return result;
}).collect(Collectors.toList());
- return ContinuousProfilingPolicyTarget.builder()
+ final ContinuousProfilingPolicyTarget target = ContinuousProfilingPolicyTarget.builder()
.type(type)
.checkItems(items)
.build();
+
+ Optional.ofNullable(summaryMap.get(EBPFProfilingTargetType.valueOf(type).value()))
+ .ifPresent(summary -> {
+ target.setTriggeredCount(summary.getCount());
+ target.setLastTriggerTimestamp(summary.getLastTriggerTime());
+ });
+ return target;
}).collect(Collectors.toList());
}
+
+ public List<ContinuousProfilingMonitoringInstance> queryContinuousProfilingMonitoringInstances(String serviceId, ContinuousProfilingTargetType target) throws IOException {
+ // query all processes under same service
Review Comment:
```suggestion
// Query all processes of the given service
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #10878: Adapt the continuous profiling task query GraphQL
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #10878:
URL: https://github.com/apache/skywalking/pull/10878#discussion_r1214339556
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/profiling/continuous/ContinuousProfilingQueryService.java:
##########
@@ -76,10 +119,118 @@ public List<ContinuousProfilingPolicyTarget> queryContinuousProfilingServiceTarg
return result;
}).collect(Collectors.toList());
- return ContinuousProfilingPolicyTarget.builder()
+ final ContinuousProfilingPolicyTarget target = ContinuousProfilingPolicyTarget.builder()
.type(type)
.checkItems(items)
.build();
+
+ Optional.ofNullable(summaryMap.get(EBPFProfilingTargetType.valueOf(type).value()))
+ .ifPresent(summary -> {
+ target.setTriggeredCount(summary.getCount());
+ target.setLastTriggerTimestamp(summary.getLastTriggerTime());
+ });
+ return target;
}).collect(Collectors.toList());
}
+
+ public List<ContinuousProfilingMonitoringInstance> queryContinuousProfilingMonitoringInstances(String serviceId, ContinuousProfilingTargetType target) throws IOException {
+ // query all processes under same service
+ final List<Process> processes = getMetadataQueryDAO().listProcesses(serviceId, null, 0, 0);
+ if (CollectionUtils.isEmpty(processes)) {
+ return Collections.emptyList();
+ }
+ // query all triggered tasks
+ final List<EBPFProfilingTaskRecord> records = queryRecentTriggeredTasks(serviceId, List.of(target));
+
+ // query instances metadata
+ final Map<String, List<Process>> instancesProcesses = processes.stream().collect(Collectors.groupingBy(Process::getInstanceId));
+ final List<ServiceInstance> instanceIdWithMetadata = getMetadataQueryDAO().getInstances(Arrays.asList(instancesProcesses.keySet().toArray(new String[0])));
+
+ // build instance, process summary
Review Comment:
```suggestion
// Build instance & process summary
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng merged pull request #10878: Adapt the continuous profiling task query GraphQL
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng merged PR #10878:
URL: https://github.com/apache/skywalking/pull/10878
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org