You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "mrproliu (via GitHub)" <gi...@apache.org> on 2024/01/08 01:45:38 UTC
[PR] Support handle eBPF Access Logs and update related UI [skywalking]
mrproliu opened a new pull request, #11732:
URL: https://github.com/apache/skywalking/pull/11732
Following https://github.com/apache/skywalking/discussions/11677, adding eBPF access log protocol handler and related UI Templates.
- [ ] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #<issue number>.
- [x] Update the [`CHANGES` log](https://github.com/apache/skywalking/blob/master/docs/en/changes/changes.md).
# Kubernetes Root Template
## Basic Kubernetes Services Topology
## Service Mesh Topology
# Kubernetes Service Relation
## TCP Related Metrics
## HTTP Related Metrics
## Service Instance Topology
# Service Instance Relation
## TCP Related Metrics
## HTTP Related Metrics
# Endpoint
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] Support handle eBPF Access Logs and update related UI [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11732:
URL: https://github.com/apache/skywalking/pull/11732#discussion_r1450461079
##########
docs/menu.yml:
##########
@@ -203,7 +203,11 @@ catalog:
- name: "Kubernetes"
catalog:
- name: "Observe Kubernetes"
- path: "/en/setup/backend/backend-k8s-monitoring"
+ catalog:
+ - name: "kube-state-metrics + cAdvisor"
+ path: "/en/setup/backend/backend-k8s-monitoring-metrics-cadvisor"
+ - name: "Rover"
+ path: "/en/setup/backend/backend-k8s-monitoring-rover"
Review Comment:
```suggestion
- name: "Observe Kubernetes"
path: "/en/setup/backend/backend-k8s-monitoring"
- name: "kube-state-metrics + cAdvisor"
path: "/en/setup/backend/backend-k8s-monitoring-metrics-cadvisor"
- name: "Rover"
path: "/en/setup/backend/backend-k8s-monitoring-rover"
```
Please setup the docs in this way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] Support handle eBPF Access Logs and update related UI [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11732:
URL: https://github.com/apache/skywalking/pull/11732#discussion_r1450456218
##########
docs/en/setup/backend/backend-k8s-monitoring-rover.md:
##########
@@ -0,0 +1,103 @@
+# Kubernetes (K8s) monitoring from Rover
+
+SkyWalking uses the SkyWalking Rover system to collect access logs from Kubernetes clusters and hands them over to the [OAL system](./../../concepts-and-designs/oal.md) for metrics and entity analysis.
+
+## Data flow
+1. SkyWalking Rover monitoring access log data from K8s and send to the OAP.
+2. The SkyWalking OAP Server receive access log from Rover through gRPC, analysis the generate entity, and using [OAL](../../concepts-and-designs/oal.md) to generating metrics.
+
+## Setup
+1. Setup [Rover in the Kubernetes](https://skywalking.apache.org/docs/skywalking-rover/next/en/setup/examples/deploy/kubernetes/readme/) and enable [access log service](https://skywalking.apache.org/docs/skywalking-rover/next/en/setup/configuration/accesslog/).
Review Comment:
```suggestion
1. Setup [Rover in the Kubernetes](https://skywalking.apache.org/docs/skywalking-rover/next/en/setup/examples/deploy/kubernetes/readme/) and enable [access log service](https://skywalking.apache.org/docs/skywalking-rover/next/en/setup/configuration/accesslog/)(enabled by default).
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] Support handle eBPF Access Logs and update related UI [skywalking]
Posted by "mrproliu (via GitHub)" <gi...@apache.org>.
mrproliu commented on code in PR #11732:
URL: https://github.com/apache/skywalking/pull/11732#discussion_r1451852565
##########
docs/en/concepts-and-designs/scope-definitions.md:
##########
@@ -414,3 +414,80 @@ This calculates the endpoint dimensional metrics data from each request of MQ sy
| transmissionLatency | The latency from produce side to consume side . | | int(in ms) |
| status | Indicates the success or failure of the request. | | boolean |
| operation | Indicates this access is on `Produce` or `Consume` side | | enum |
+
+### SCOPES with `K8S` Prefix
+
+All metrics starting with `K8S` are derived from monitoring the Access Log in Kubernetes.
+
+#### Service, Service Instance and relations
+
+For all `K8SService`, `K8SInstance`, `K8SServiceRelation` and `K8SServiceInstanceRelation`, they all have the
Review Comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support handle eBPF Access Logs and update related UI [skywalking]
Posted by "mrproliu (via GitHub)" <gi...@apache.org>.
mrproliu merged PR #11732:
URL: https://github.com/apache/skywalking/pull/11732
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] Support handle eBPF Access Logs and update related UI [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11732:
URL: https://github.com/apache/skywalking/pull/11732#discussion_r1450454099
##########
docs/en/concepts-and-designs/scope-definitions.md:
##########
@@ -414,3 +414,80 @@ This calculates the endpoint dimensional metrics data from each request of MQ sy
| transmissionLatency | The latency from produce side to consume side . | | int(in ms) |
| status | Indicates the success or failure of the request. | | boolean |
| operation | Indicates this access is on `Produce` or `Consume` side | | enum |
+
+### SCOPES with `K8S` Prefix
+
+All metrics starting with `K8S` are derived from monitoring the Access Log in Kubernetes.
+
+#### Service, Service Instance and relations
+
+For all `K8SService`, `K8SInstance`, `K8SServiceRelation` and `K8SServiceInstanceRelation`, they all have the
Review Comment:
I think you missed those scopes' own field. Same for endpoint and endpoint relation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] Support handle eBPF Access Logs and update related UI [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11732:
URL: https://github.com/apache/skywalking/pull/11732#discussion_r1450471285
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/source/DefaultScopeDefine.java:
##########
@@ -129,6 +129,14 @@ public class DefaultScopeDefine {
public static final int SERVICE_HIERARCHY_RELATION = 70;
public static final int INSTANCE_HIERARCHY_RELATION = 71;
+
+ public static final int K8S_SERVICE = 72;
+ public static final int K8S_SERVICE_INSTANCE = 73;
+ public static final int K8S_SERVICE_RELATION = 74;
+ public static final int K8S_SERVICE_INSTANCE_RELATION = 75;
+ public static final int K8S_ENDPOINT = 76;
+ public static final int K8S_ENDPOINT_REALATION = 77;
Review Comment:
```suggestion
// Rover(eBPF agent) for k8s monitoring
public static final int K8S_SERVICE = 72;
public static final int K8S_SERVICE_INSTANCE = 73;
public static final int K8S_SERVICE_RELATION = 74;
public static final int K8S_SERVICE_INSTANCE_RELATION = 75;
public static final int K8S_ENDPOINT = 76;
public static final int K8S_ENDPOINT_REALATION = 77;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] Support handle eBPF Access Logs and update related UI [skywalking]
Posted by "mrproliu (via GitHub)" <gi...@apache.org>.
mrproliu commented on code in PR #11732:
URL: https://github.com/apache/skywalking/pull/11732#discussion_r1451852883
##########
docs/menu.yml:
##########
@@ -203,7 +203,11 @@ catalog:
- name: "Kubernetes"
catalog:
- name: "Observe Kubernetes"
- path: "/en/setup/backend/backend-k8s-monitoring"
Review Comment:
I have created a new summary document and linked these two documents together.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] Support handle eBPF Access Logs and update related UI [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11732:
URL: https://github.com/apache/skywalking/pull/11732#discussion_r1450456683
##########
docs/en/setup/backend/backend-k8s-monitoring-rover.md:
##########
@@ -0,0 +1,103 @@
+# Kubernetes (K8s) monitoring from Rover
+
+SkyWalking uses the SkyWalking Rover system to collect access logs from Kubernetes clusters and hands them over to the [OAL system](./../../concepts-and-designs/oal.md) for metrics and entity analysis.
+
+## Data flow
+1. SkyWalking Rover monitoring access log data from K8s and send to the OAP.
+2. The SkyWalking OAP Server receive access log from Rover through gRPC, analysis the generate entity, and using [OAL](../../concepts-and-designs/oal.md) to generating metrics.
+
+## Setup
+1. Setup [Rover in the Kubernetes](https://skywalking.apache.org/docs/skywalking-rover/next/en/setup/examples/deploy/kubernetes/readme/) and enable [access log service](https://skywalking.apache.org/docs/skywalking-rover/next/en/setup/configuration/accesslog/).
+2. Setup eBPF receiver module by the following configuration.
+```yaml
+```yaml
Review Comment:
```suggestion
```yaml
```
Misformated?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] Support handle eBPF Access Logs and update related UI [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11732:
URL: https://github.com/apache/skywalking/pull/11732#discussion_r1450463481
##########
oap-server/server-receiver-plugin/skywalking-ebpf-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/ebpf/provider/EBPFReceiverProvider.java:
##########
@@ -65,12 +67,19 @@ public void prepare() throws ServiceNotProvidedException, ModuleStartException {
@Override
public void start() throws ServiceNotProvidedException, ModuleStartException {
+ // load official analysis
+ getManager().find(CoreModule.NAME)
+ .provider()
+ .getService(OALEngineLoaderService.class)
+ .load(EBPFOALDefine.INSTANCE);
+
final GRPCHandlerRegister grpcHandlerRegister = getManager().find(SharingServerModule.NAME)
.provider()
.getService(GRPCHandlerRegister.class);
grpcHandlerRegister.addHandler(new EBPFProcessServiceHandler(getManager()));
grpcHandlerRegister.addHandler(new EBPFProfilingServiceHandler(getManager()));
grpcHandlerRegister.addHandler(new ContinuousProfilingServiceHandler(getManager(), this.config));
+ grpcHandlerRegister.addHandler(new AccessLogServiceHandler(getManager()));
Review Comment:
Please refer to https://github.com/apache/skywalking/pull/11749, what we learned from that performance benchmark is that, for heavy traffic service, especially like ALS, we should have an option to provide a separate port to help traffic limitation and control.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] Support handle eBPF Access Logs and update related UI [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11732:
URL: https://github.com/apache/skywalking/pull/11732#discussion_r1450460298
##########
docs/menu.yml:
##########
@@ -203,7 +203,11 @@ catalog:
- name: "Kubernetes"
catalog:
- name: "Observe Kubernetes"
- path: "/en/setup/backend/backend-k8s-monitoring"
Review Comment:
`/en/setup/backend/backend-k8s-monitoring` is referenced by blogs and SkyWalking UI, it should not be changed.
We should build a summary introduction doc for this path. And cAdvisor and eBPF should be two documents more about details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] Support handle eBPF Access Logs and update related UI [skywalking]
Posted by "mrproliu (via GitHub)" <gi...@apache.org>.
mrproliu commented on code in PR #11732:
URL: https://github.com/apache/skywalking/pull/11732#discussion_r1451853534
##########
oap-server/server-receiver-plugin/skywalking-ebpf-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/ebpf/provider/handler/AccessLogServiceHandler.java:
##########
@@ -0,0 +1,700 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.receiver.ebpf.provider.handler;
+
+import io.grpc.stub.StreamObserver;
+import io.vavr.Tuple2;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.apm.network.common.v3.DetectPoint;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.AccessLogConnection;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.AccessLogConnectionTLSMode;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.AccessLogHTTPProtocol;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.AccessLogKernelAcceptOperation;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.AccessLogKernelCloseOperation;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.AccessLogKernelConnectOperation;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.AccessLogKernelLog;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.AccessLogKernelReadOperation;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.AccessLogKernelWriteOperation;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.AccessLogProtocolLogs;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.AccessLogProtocolType;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.ConnectionAddress;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.EBPFAccessLogDownstream;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.EBPFAccessLogMessage;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.EBPFAccessLogServiceGrpc;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.EBPFAccessLogNodeInfo;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.EBPFAccessLogNodeNetInterface;
+import org.apache.skywalking.apm.network.common.v3.Instant;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.EBPFTimestamp;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.IPAddress;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.KubernetesProcessAddress;
+import org.apache.skywalking.library.kubernetes.ObjectID;
+import org.apache.skywalking.oap.meter.analyzer.k8s.K8sInfoRegistry;
+import org.apache.skywalking.oap.server.core.CoreModule;
+import org.apache.skywalking.oap.server.core.analysis.Layer;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.config.NamingControl;
+import org.apache.skywalking.oap.server.core.source.K8SEndpoint;
+import org.apache.skywalking.oap.server.core.source.K8SEndpointRelation;
+import org.apache.skywalking.oap.server.core.source.K8SMetrics;
+import org.apache.skywalking.oap.server.core.source.K8SService;
+import org.apache.skywalking.oap.server.core.source.K8SServiceInstance;
+import org.apache.skywalking.oap.server.core.source.K8SServiceInstanceRelation;
+import org.apache.skywalking.oap.server.core.source.K8SServiceRelation;
+import org.apache.skywalking.oap.server.core.source.SourceReceiver;
+import org.apache.skywalking.oap.server.library.module.ModuleManager;
+import org.apache.skywalking.oap.server.library.util.StringUtil;
+import org.apache.skywalking.oap.server.telemetry.TelemetryModule;
+import org.apache.skywalking.oap.server.telemetry.api.CounterMetrics;
+import org.apache.skywalking.oap.server.telemetry.api.HistogramMetrics;
+import org.apache.skywalking.oap.server.telemetry.api.MetricsCreator;
+import org.apache.skywalking.oap.server.telemetry.api.MetricsTag;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+@Slf4j
+public class AccessLogServiceHandler extends EBPFAccessLogServiceGrpc.EBPFAccessLogServiceImplBase {
+ private final SourceReceiver sourceReceiver;
+ private final NamingControl namingControl;
+
+ private final CounterMetrics inCounter;
+ private final HistogramMetrics processHistogram;
+ private final CounterMetrics dropCounter;
+ private final ConcurrentHashMap<String, DropDataReason> dropReasons = new ConcurrentHashMap<>();
+
+ public AccessLogServiceHandler(ModuleManager moduleManager) {
+ this.sourceReceiver = moduleManager.find(CoreModule.NAME).provider().getService(SourceReceiver.class);
+ this.namingControl = moduleManager.find(CoreModule.NAME).provider().getService(NamingControl.class);
+
+ MetricsCreator metricsCreator = moduleManager.find(TelemetryModule.NAME)
+ .provider()
+ .getService(MetricsCreator.class);
+ this.inCounter = metricsCreator.createCounter(
+ "k8s_als_in_count", "The count of envoy ALS message received", MetricsTag.EMPTY_KEY,
+ MetricsTag.EMPTY_VALUE);
+ this.processHistogram = metricsCreator.createHistogramMetric(
+ "k8s_als_in_latency", "The process latency of envoy ALS message received", MetricsTag.EMPTY_KEY,
+ MetricsTag.EMPTY_VALUE);
+ this.dropCounter = metricsCreator.createCounter(
+ "k8s_als_drop_count", "The count of envoy ALS message dropped", MetricsTag.EMPTY_KEY,
+ MetricsTag.EMPTY_VALUE);
Review Comment:
Update, please take a look at the screenshots.
##########
oap-server/server-receiver-plugin/skywalking-ebpf-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/ebpf/provider/EBPFReceiverProvider.java:
##########
@@ -65,12 +67,19 @@ public void prepare() throws ServiceNotProvidedException, ModuleStartException {
@Override
public void start() throws ServiceNotProvidedException, ModuleStartException {
+ // load official analysis
+ getManager().find(CoreModule.NAME)
+ .provider()
+ .getService(OALEngineLoaderService.class)
+ .load(EBPFOALDefine.INSTANCE);
+
final GRPCHandlerRegister grpcHandlerRegister = getManager().find(SharingServerModule.NAME)
.provider()
.getService(GRPCHandlerRegister.class);
grpcHandlerRegister.addHandler(new EBPFProcessServiceHandler(getManager()));
grpcHandlerRegister.addHandler(new EBPFProfilingServiceHandler(getManager()));
grpcHandlerRegister.addHandler(new ContinuousProfilingServiceHandler(getManager(), this.config));
+ grpcHandlerRegister.addHandler(new AccessLogServiceHandler(getManager()));
Review Comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] Support handle eBPF Access Logs and update related UI [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #11732:
URL: https://github.com/apache/skywalking/pull/11732#issuecomment-1880475704
Some polish is required.
1. HTTP metrics and legends of normal and red nodes are required
2. OAL docs, sources especially, should be updated.
3. Observe Kubernetes doc should be updated accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] Support handle eBPF Access Logs and update related UI [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11732:
URL: https://github.com/apache/skywalking/pull/11732#discussion_r1450452193
##########
docs/en/concepts-and-designs/scope-definitions.md:
##########
@@ -414,3 +414,80 @@ This calculates the endpoint dimensional metrics data from each request of MQ sy
| transmissionLatency | The latency from produce side to consume side . | | int(in ms) |
| status | Indicates the success or failure of the request. | | boolean |
| operation | Indicates this access is on `Produce` or `Consume` side | | enum |
+
+### SCOPES with `K8S` Prefix
+
+All metrics starting with `K8S` are derived from monitoring the Access Log in Kubernetes.
Review Comment:
```suggestion
All metrics starting with `K8S` are derived from Kubernetes monitoring by Rover(eBPF agent).
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] Support handle eBPF Access Logs and update related UI [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11732:
URL: https://github.com/apache/skywalking/pull/11732#discussion_r1450466459
##########
oap-server/server-receiver-plugin/skywalking-ebpf-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/ebpf/provider/handler/AccessLogServiceHandler.java:
##########
@@ -0,0 +1,700 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.receiver.ebpf.provider.handler;
+
+import io.grpc.stub.StreamObserver;
+import io.vavr.Tuple2;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.apm.network.common.v3.DetectPoint;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.AccessLogConnection;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.AccessLogConnectionTLSMode;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.AccessLogHTTPProtocol;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.AccessLogKernelAcceptOperation;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.AccessLogKernelCloseOperation;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.AccessLogKernelConnectOperation;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.AccessLogKernelLog;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.AccessLogKernelReadOperation;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.AccessLogKernelWriteOperation;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.AccessLogProtocolLogs;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.AccessLogProtocolType;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.ConnectionAddress;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.EBPFAccessLogDownstream;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.EBPFAccessLogMessage;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.EBPFAccessLogServiceGrpc;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.EBPFAccessLogNodeInfo;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.EBPFAccessLogNodeNetInterface;
+import org.apache.skywalking.apm.network.common.v3.Instant;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.EBPFTimestamp;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.IPAddress;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.KubernetesProcessAddress;
+import org.apache.skywalking.library.kubernetes.ObjectID;
+import org.apache.skywalking.oap.meter.analyzer.k8s.K8sInfoRegistry;
+import org.apache.skywalking.oap.server.core.CoreModule;
+import org.apache.skywalking.oap.server.core.analysis.Layer;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.config.NamingControl;
+import org.apache.skywalking.oap.server.core.source.K8SEndpoint;
+import org.apache.skywalking.oap.server.core.source.K8SEndpointRelation;
+import org.apache.skywalking.oap.server.core.source.K8SMetrics;
+import org.apache.skywalking.oap.server.core.source.K8SService;
+import org.apache.skywalking.oap.server.core.source.K8SServiceInstance;
+import org.apache.skywalking.oap.server.core.source.K8SServiceInstanceRelation;
+import org.apache.skywalking.oap.server.core.source.K8SServiceRelation;
+import org.apache.skywalking.oap.server.core.source.SourceReceiver;
+import org.apache.skywalking.oap.server.library.module.ModuleManager;
+import org.apache.skywalking.oap.server.library.util.StringUtil;
+import org.apache.skywalking.oap.server.telemetry.TelemetryModule;
+import org.apache.skywalking.oap.server.telemetry.api.CounterMetrics;
+import org.apache.skywalking.oap.server.telemetry.api.HistogramMetrics;
+import org.apache.skywalking.oap.server.telemetry.api.MetricsCreator;
+import org.apache.skywalking.oap.server.telemetry.api.MetricsTag;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+@Slf4j
+public class AccessLogServiceHandler extends EBPFAccessLogServiceGrpc.EBPFAccessLogServiceImplBase {
+ private final SourceReceiver sourceReceiver;
+ private final NamingControl namingControl;
+
+ private final CounterMetrics inCounter;
+ private final HistogramMetrics processHistogram;
+ private final CounterMetrics dropCounter;
+ private final ConcurrentHashMap<String, DropDataReason> dropReasons = new ConcurrentHashMap<>();
+
+ public AccessLogServiceHandler(ModuleManager moduleManager) {
+ this.sourceReceiver = moduleManager.find(CoreModule.NAME).provider().getService(SourceReceiver.class);
+ this.namingControl = moduleManager.find(CoreModule.NAME).provider().getService(NamingControl.class);
+
+ MetricsCreator metricsCreator = moduleManager.find(TelemetryModule.NAME)
+ .provider()
+ .getService(MetricsCreator.class);
+ this.inCounter = metricsCreator.createCounter(
+ "k8s_als_in_count", "The count of envoy ALS message received", MetricsTag.EMPTY_KEY,
+ MetricsTag.EMPTY_VALUE);
+ this.processHistogram = metricsCreator.createHistogramMetric(
+ "k8s_als_in_latency", "The process latency of envoy ALS message received", MetricsTag.EMPTY_KEY,
+ MetricsTag.EMPTY_VALUE);
+ this.dropCounter = metricsCreator.createCounter(
+ "k8s_als_drop_count", "The count of envoy ALS message dropped", MetricsTag.EMPTY_KEY,
+ MetricsTag.EMPTY_VALUE);
Review Comment:
As these important self-observability metrics, please update the native self-observability dashboard and Grafana dashboard, we should make them side by side with Envoy ALS and Agent traces metrics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org