You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2022/10/20 05:42:47 UTC
[GitHub] [skywalking] wankai123 opened a new pull request, #9817: Support export `Trace` and `Log` through Kafka.
wankai123 opened a new pull request, #9817:
URL: https://github.com/apache/skywalking/pull/9817
- [ ] If this is non-trivial feature, paste the links/URLs to the design doc.
- [X] Update the documentation to include this new feature.
- [X] Tests(including UT, IT, E2E) are added to verify the new feature.
- [ ] If it's UI related, attach the screenshots below.
- [X] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes (https://github.com/apache/skywalking/issues/9745).
- [X] Update the [`CHANGES` log](https://github.com/apache/skywalking/blob/master/docs/en/changes/changes.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #9817: Support export `Trace` and `Log` through Kafka.
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #9817:
URL: https://github.com/apache/skywalking/pull/9817#discussion_r1000200607
##########
docs/en/setup/backend/exporter.md:
##########
@@ -0,0 +1,161 @@
+# Exporter
+SkyWalking provides the essential functions of metrics aggregation, trace, log, alarm, and analysis.
+In many real-world scenarios, users may want to forward their data to a 3rd party system for further in-depth analysis.
+**Exporter** has made that possible.
+
+The exporter is an independent module that has to be manually activated.
+
+Right now, we provide the following exporters:
+1. gRPC Exporter
+- Metrics
+2. Kafka Exporter
+- Trace
+- Log
+
+## gRPC Exporter
+### Metrics gRPC Exporter
+Metrics gRPC exporter uses SkyWalking's native exporter service definition. Here is the proto definition.
+```proto
+service MetricExportService {
+ rpc export (stream ExportMetricValue) returns (ExportResponse) {
+ }
+
+ rpc subscription (SubscriptionReq) returns (SubscriptionsResp) {
+ }
+}
+
+message ExportMetricValue {
+ string metricName = 1;
+ string entityName = 2;
+ string entityId = 3;
+ ValueType type = 4;
+ int64 timeBucket = 5;
+ int64 longValue = 6;
+ double doubleValue = 7;
+ repeated int64 longValues = 8;
+}
+
+message SubscriptionsResp {
+ repeated SubscriptionMetric metrics = 1;
+}
+
+message SubscriptionMetric {
+ string metricName = 1;
+ EventType eventType = 2;
+}
+
+enum ValueType {
+ LONG = 0;
+ DOUBLE = 1;
+ MULTI_LONG = 2;
+}
+
+enum EventType {
+ // The metrics aggregated in this bulk, not include the existing persistent data.
+ INCREMENT = 0;
+ // Final result of the metrics at this moment.
+ TOTAL = 1;
+}
+
+message SubscriptionReq {
+
+}
+
+message ExportResponse {
+}
+```
+
+To activate the exporter, you should add this into your `application.yml`
Review Comment:
I believe this is copied from the previous doc. We should adjust words. These configurations exist in the YAML file already, we should say `you should set xxx to true to activate yyy`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #9817: Support export `Trace` and `Log` through Kafka.
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #9817:
URL: https://github.com/apache/skywalking/pull/9817#discussion_r1000201700
##########
docs/en/setup/backend/exporter.md:
##########
@@ -0,0 +1,161 @@
+# Exporter
+SkyWalking provides the essential functions of metrics aggregation, trace, log, alarm, and analysis.
+In many real-world scenarios, users may want to forward their data to a 3rd party system for further in-depth analysis.
+**Exporter** has made that possible.
+
+The exporter is an independent module that has to be manually activated.
+
+Right now, we provide the following exporters:
+1. gRPC Exporter
+- Metrics
+2. Kafka Exporter
+- Trace
+- Log
+
+## gRPC Exporter
+### Metrics gRPC Exporter
+Metrics gRPC exporter uses SkyWalking's native exporter service definition. Here is the proto definition.
+```proto
+service MetricExportService {
+ rpc export (stream ExportMetricValue) returns (ExportResponse) {
+ }
+
+ rpc subscription (SubscriptionReq) returns (SubscriptionsResp) {
+ }
+}
+
+message ExportMetricValue {
+ string metricName = 1;
+ string entityName = 2;
+ string entityId = 3;
+ ValueType type = 4;
+ int64 timeBucket = 5;
+ int64 longValue = 6;
+ double doubleValue = 7;
+ repeated int64 longValues = 8;
+}
+
+message SubscriptionsResp {
+ repeated SubscriptionMetric metrics = 1;
+}
+
+message SubscriptionMetric {
+ string metricName = 1;
+ EventType eventType = 2;
+}
+
+enum ValueType {
+ LONG = 0;
+ DOUBLE = 1;
+ MULTI_LONG = 2;
+}
+
+enum EventType {
+ // The metrics aggregated in this bulk, not include the existing persistent data.
+ INCREMENT = 0;
+ // Final result of the metrics at this moment.
+ TOTAL = 1;
+}
+
+message SubscriptionReq {
+
+}
+
+message ExportResponse {
+}
+```
+
+To activate the exporter, you should add this into your `application.yml`
+```yaml
+exporter:
+ default:
+ # gRPC exporter
+ enableGRPCMetrics: ${SW_EXPORTER_ENABLE_GRPC_METRICS:true}
+ gRPCTargetHost: ${SW_EXPORTER_GRPC_HOST:127.0.0.1}
+ gRPCTargetPort: ${SW_EXPORTER_GRPC_PORT:9870}
+ ...
+```
+
+- `gRPCTargetHost`:`gRPCTargetPort` is the expected target service address. You could set any gRPC server to receive the data.
+- Target gRPC service needs to go on standby; otherwise, the OAP startup may fail.
+
+#### Target exporter service
+1. Subscription implementation.
+Return the expected metrics name list with event type (incremental or total). All names must match the OAL/MAL script definition.
+Return empty list, if you want to export all metrics in the incremental event type.
+
+2. Export implementation.
+Stream service. All subscribed metrics will be sent here based on the OAP core schedule. Also, if the OAP is deployed as a cluster,
+this method will be called concurrently. For metrics value, you need to follow `#type` to choose `#longValue` or `#doubleValue`.
+
+## Kafka Exporter
+### Trace Kafka Exporter
+Trace kafka exporter pushes messages to the Kafka Broker and Topic `skywalking-trace` to export the trace. Here is the message:
+```
+ProducerRecord<String, Bytes>
+Key: TraceSegmentId
+Value: Bytes of SegmentObject
+```
+
+The `SegmentObject` definition follows the protocol:
+[SkyWalking data collect protocol#Tracing.proto](https://github.com/apache/skywalking-data-collect-protocol/blob/master/language-agent/Tracing.proto).
+```proto
+// The segment is a collection of spans. It includes all collected spans in a simple one request context, such as a HTTP request process.
+message SegmentObject {
+ string traceId = 1;
+ string traceSegmentId = 2;
+ repeated SpanObject spans = 3;
+ string service = 4;
+ string serviceInstance = 5;
+ bool isSizeLimited = 6;
+}
+```
+
+To activate the exporter, you should add this into your `application.yml`
+```yaml
+exporter:
+ default:
+ # Kafka exporter
+ enableKafkaTrace: ${SW_EXPORTER_ENABLE_KAFKA_TRACE:true}
+ kafkaBootstrapServers: ${SW_EXPORTER_KAFKA_SERVERS:localhost:9092}
+ # Kafka producer config, JSON format as Properties.
+ kafkaProducerConfig: ${SW_EXPORTER_KAFKA_PRODUCER_CONFIG:""}
+ kafkaTopicTrace: ${SW_EXPORTER_KAFKA_TOPIC_TRACE:skywalking-trace}
+ ...
+```
+
+### Log Kafka Exporter
+Log kafka exporter pushes messages to the Kafka Broker and Topic `skywalking-log` to export the log. Here is the message:
+```
+ProducerRecord<String, Bytes>
+Key: LogRecordId
+Value: Bytes of LogData
Review Comment:
@kezhenxu94 @Superskyyy I remember there was a discussion on Python agent, mentioning the encode in Python proto is UTF-16, rather than Java's UTF-8? Could you confirm that? If so, we should mention the encoding format.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] sonatype-lift[bot] commented on a diff in pull request #9817: Support export `Trace` and `Log` through Kafka.
Posted by GitBox <gi...@apache.org>.
sonatype-lift[bot] commented on code in PR #9817:
URL: https://github.com/apache/skywalking/pull/9817#discussion_r1000234391
##########
oap-server/exporter/src/main/java/org/apache/skywalking/oap/server/exporter/provider/kafka/log/KafkaLogExporter.java:
##########
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.exporter.provider.kafka.log;
+
+import com.google.protobuf.InvalidProtocolBufferException;
+import java.util.List;
+import java.util.Properties;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.skywalking.apm.network.logging.v3.JSONLog;
+import org.apache.skywalking.apm.network.logging.v3.LogData;
+import org.apache.skywalking.apm.network.logging.v3.LogDataBody;
+import org.apache.skywalking.apm.network.logging.v3.LogTags;
+import org.apache.skywalking.apm.network.logging.v3.TextLog;
+import org.apache.skywalking.apm.network.logging.v3.TraceContext;
+import org.apache.skywalking.apm.network.logging.v3.YAMLLog;
+import org.apache.skywalking.oap.server.core.UnexpectedException;
+import org.apache.skywalking.oap.server.core.analysis.IDManager;
+import org.apache.skywalking.oap.server.core.analysis.manual.log.LogRecord;
+import org.apache.skywalking.oap.server.core.exporter.LogExportService;
+import org.apache.skywalking.oap.server.core.query.type.ContentType;
+import org.apache.skywalking.oap.server.exporter.provider.ExporterSetting;
+import org.apache.skywalking.oap.server.exporter.provider.kafka.KafkaExportProducer;
+import org.apache.skywalking.oap.server.library.datacarrier.DataCarrier;
+import org.apache.skywalking.oap.server.library.datacarrier.buffer.BufferStrategy;
+import org.apache.skywalking.oap.server.library.datacarrier.consumer.IConsumer;
+import org.apache.skywalking.oap.server.library.module.ModuleManager;
+import org.apache.skywalking.oap.server.library.util.StringUtil;
+import org.apache.skywalking.oap.server.telemetry.TelemetryModule;
+import org.apache.skywalking.oap.server.telemetry.api.CounterMetrics;
+import org.apache.skywalking.oap.server.telemetry.api.MetricsCreator;
+import org.apache.skywalking.oap.server.telemetry.api.MetricsTag;
+
+@Slf4j
+public class KafkaLogExporter extends KafkaExportProducer implements LogExportService, IConsumer<LogRecord> {
+ private DataCarrier<LogRecord> exportBuffer;
+ private CounterMetrics successCounter;
+ private CounterMetrics errorCounter;
+ private final ModuleManager moduleManager;
+
+ public KafkaLogExporter(ModuleManager manager, ExporterSetting setting) {
+ super(setting);
+ this.moduleManager = manager;
+ }
+
+ @Override
+ public void start() {
+ super.getProducer();
+ exportBuffer = new DataCarrier<>(
+ "KafkaLogExporter", "KafkaLogExporter", setting.getBufferChannelNum(), setting.getBufferChannelSize(),
+ BufferStrategy.IF_POSSIBLE
+ );
+ exportBuffer.consume(this, 1, 200);
+ MetricsCreator metricsCreator = moduleManager.find(TelemetryModule.NAME)
+ .provider()
+ .getService(MetricsCreator.class);
+ successCounter = metricsCreator.createCounter(
+ "kafka_exporter_log_success_count", "The success number of log exported by kafka exporter.",
+ new MetricsTag.Keys("protocol"),
+ new MetricsTag.Values("kafka")
+ );
+ errorCounter = metricsCreator.createCounter(
+ "kafka_exporter_log_error_count", "The error number of log exported by kafka exporter",
+ new MetricsTag.Keys("protocol"), new MetricsTag.Values("kafka")
+ );
+ }
+
+ @Override
+ public void export(final LogRecord logRecord) {
+ if (logRecord != null) {
+ exportBuffer.produce(logRecord);
+ }
+ }
+
+ @Override
+ public boolean isEnabled() {
+ return setting.isEnableKafkaLog();
+ }
+
+ @Override
+ public void init(final Properties properties) {
+
+ }
+
+ @Override
+ public void consume(final List<LogRecord> data) {
+ for (LogRecord logRecord : data) {
+ if (logRecord != null) {
+ try {
+ LogData logData = transLogData(logRecord);
+ ProducerRecord<String, Bytes> record = new ProducerRecord<>(
+ setting.getKafkaTopicLog(),
+ logRecord.id(),
+ Bytes.wrap(logData.toByteArray())
+ );
+ super.getProducer().send(record, (metadata, ex) -> {
Review Comment:
*[FutureReturnValueIgnored](https://errorprone.info/bugpattern/FutureReturnValueIgnored):* Return value of methods returning Future must be checked. Ignoring returned Futures suppresses exceptions thrown from the code that completes the Future.
---
<details><summary><b>âšī¸ Learn about @sonatype-lift commands</b></summary>
You can reply with the following commands. For example, reply with ***@sonatype-lift ignoreall*** to leave out all findings.
| **Command** | **Usage** |
| ------------- | ------------- |
| `@sonatype-lift ignore` | Leave out the above finding from this PR |
| `@sonatype-lift ignoreall` | Leave out all the existing findings from this PR |
| `@sonatype-lift exclude <file\|issue\|path\|tool>` | Exclude specified `file\|issue\|path\|tool` from Lift findings by updating your config.toml file |
**Note:** When talking to LiftBot, you need to **refresh** the page to see its response.
<sub>[Click here](https://github.com/apps/sonatype-lift/installations/new) to add LiftBot to another repo.</sub></details>
---
Was this a good recommendation?
[ [đ Not relevant](https://www.sonatype.com/lift-comment-rating?comment=346683672&lift_comment_rating=1) ] - [ [đ Won't fix](https://www.sonatype.com/lift-comment-rating?comment=346683672&lift_comment_rating=2) ] - [ [đ Not critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=346683672&lift_comment_rating=3) ] - [ [đ Critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=346683672&lift_comment_rating=4) ] - [ [đ Critical, fixing now](https://www.sonatype.com/lift-comment-rating?comment=346683672&lift_comment_rating=5) ]
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/exporter/TraceExportService.java:
##########
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.core.exporter;
+
+import org.apache.skywalking.oap.server.core.analysis.manual.segment.SegmentRecord;
+import org.apache.skywalking.oap.server.library.module.Service;
+
+/**
+ * Export the traces from metrics through this service.
+ */
+public interface TraceExportService extends Service, ExporterService<SegmentRecord> {
+
+ void export(SegmentRecord segmentRecord);
Review Comment:
*[MissingOverride](https://errorprone.info/bugpattern/MissingOverride):* export implements method in ExporterService; expected @Override
---
```suggestion
@Override void export(SegmentRecord segmentRecord);
```
---
<details><summary><b>âšī¸ Learn about @sonatype-lift commands</b></summary>
You can reply with the following commands. For example, reply with ***@sonatype-lift ignoreall*** to leave out all findings.
| **Command** | **Usage** |
| ------------- | ------------- |
| `@sonatype-lift ignore` | Leave out the above finding from this PR |
| `@sonatype-lift ignoreall` | Leave out all the existing findings from this PR |
| `@sonatype-lift exclude <file\|issue\|path\|tool>` | Exclude specified `file\|issue\|path\|tool` from Lift findings by updating your config.toml file |
**Note:** When talking to LiftBot, you need to **refresh** the page to see its response.
<sub>[Click here](https://github.com/apps/sonatype-lift/installations/new) to add LiftBot to another repo.</sub></details>
---
Was this a good recommendation?
[ [đ Not relevant](https://www.sonatype.com/lift-comment-rating?comment=346683555&lift_comment_rating=1) ] - [ [đ Won't fix](https://www.sonatype.com/lift-comment-rating?comment=346683555&lift_comment_rating=2) ] - [ [đ Not critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=346683555&lift_comment_rating=3) ] - [ [đ Critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=346683555&lift_comment_rating=4) ] - [ [đ Critical, fixing now](https://www.sonatype.com/lift-comment-rating?comment=346683555&lift_comment_rating=5) ]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #9817: Support export `Trace` and `Log` through Kafka.
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #9817:
URL: https://github.com/apache/skywalking/pull/9817#discussion_r1000178869
##########
oap-server/server-starter/src/main/resources/application.yml:
##########
@@ -433,7 +433,7 @@ alarm:
default:
telemetry:
- selector: ${SW_TELEMETRY:none}
+ selector: ${SW_TELEMETRY:prometheus}
Review Comment:
No, we aren't. This should be reverted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] sonatype-lift[bot] commented on a diff in pull request #9817: Support export `Trace` and `Log` through Kafka.
Posted by GitBox <gi...@apache.org>.
sonatype-lift[bot] commented on code in PR #9817:
URL: https://github.com/apache/skywalking/pull/9817#discussion_r1000185647
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/exporter/LogExportService.java:
##########
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.core.exporter;
+
+import org.apache.skywalking.oap.server.core.analysis.manual.log.LogRecord;
+import org.apache.skywalking.oap.server.library.module.Service;
+
+/**
+ * Export the log from metrics through this service.
+ */
+public interface LogExportService extends Service, ExporterService<LogRecord> {
+
+ void export(LogRecord logRecord);
Review Comment:
*[MissingOverride](https://errorprone.info/bugpattern/MissingOverride):* export implements method in ExporterService; expected @Override
---
```suggestion
@Override void export(LogRecord logRecord);
```
---
<details><summary><b>âšī¸ Learn about @sonatype-lift commands</b></summary>
You can reply with the following commands. For example, reply with ***@sonatype-lift ignoreall*** to leave out all findings.
| **Command** | **Usage** |
| ------------- | ------------- |
| `@sonatype-lift ignore` | Leave out the above finding from this PR |
| `@sonatype-lift ignoreall` | Leave out all the existing findings from this PR |
| `@sonatype-lift exclude <file\|issue\|path\|tool>` | Exclude specified `file\|issue\|path\|tool` from Lift findings by updating your config.toml file |
**Note:** When talking to LiftBot, you need to **refresh** the page to see its response.
<sub>[Click here](https://github.com/apps/sonatype-lift/installations/new) to add LiftBot to another repo.</sub></details>
---
Was this a good recommendation?
[ [đ Not relevant](https://www.sonatype.com/lift-comment-rating?comment=346679505&lift_comment_rating=1) ] - [ [đ Won't fix](https://www.sonatype.com/lift-comment-rating?comment=346679505&lift_comment_rating=2) ] - [ [đ Not critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=346679505&lift_comment_rating=3) ] - [ [đ Critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=346679505&lift_comment_rating=4) ] - [ [đ Critical, fixing now](https://www.sonatype.com/lift-comment-rating?comment=346679505&lift_comment_rating=5) ]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #9817: Support export `Trace` and `Log` through Kafka.
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #9817:
URL: https://github.com/apache/skywalking/pull/9817#discussion_r1000202009
##########
docs/en/setup/backend/exporter.md:
##########
@@ -0,0 +1,161 @@
+# Exporter
+SkyWalking provides the essential functions of metrics aggregation, trace, log, alarm, and analysis.
+In many real-world scenarios, users may want to forward their data to a 3rd party system for further in-depth analysis.
+**Exporter** has made that possible.
+
+The exporter is an independent module that has to be manually activated.
+
+Right now, we provide the following exporters:
+1. gRPC Exporter
+- Metrics
+2. Kafka Exporter
+- Trace
+- Log
+
+## gRPC Exporter
+### Metrics gRPC Exporter
+Metrics gRPC exporter uses SkyWalking's native exporter service definition. Here is the proto definition.
+```proto
+service MetricExportService {
+ rpc export (stream ExportMetricValue) returns (ExportResponse) {
+ }
+
+ rpc subscription (SubscriptionReq) returns (SubscriptionsResp) {
+ }
Review Comment:
Agree, we used to copy them, but link should be better, just keep there service part, and leave the details into proto.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #9817: Support export `Trace` and `Log` through Kafka.
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #9817:
URL: https://github.com/apache/skywalking/pull/9817#discussion_r1000202009
##########
docs/en/setup/backend/exporter.md:
##########
@@ -0,0 +1,161 @@
+# Exporter
+SkyWalking provides the essential functions of metrics aggregation, trace, log, alarm, and analysis.
+In many real-world scenarios, users may want to forward their data to a 3rd party system for further in-depth analysis.
+**Exporter** has made that possible.
+
+The exporter is an independent module that has to be manually activated.
+
+Right now, we provide the following exporters:
+1. gRPC Exporter
+- Metrics
+2. Kafka Exporter
+- Trace
+- Log
+
+## gRPC Exporter
+### Metrics gRPC Exporter
+Metrics gRPC exporter uses SkyWalking's native exporter service definition. Here is the proto definition.
+```proto
+service MetricExportService {
+ rpc export (stream ExportMetricValue) returns (ExportResponse) {
+ }
+
+ rpc subscription (SubscriptionReq) returns (SubscriptionsResp) {
+ }
Review Comment:
Agree, we used to copy them, but copy should be better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #9817: Support export `Trace` and `Log` through Kafka.
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #9817:
URL: https://github.com/apache/skywalking/pull/9817#discussion_r1000200824
##########
docs/en/setup/backend/exporter.md:
##########
@@ -0,0 +1,161 @@
+# Exporter
+SkyWalking provides the essential functions of metrics aggregation, trace, log, alarm, and analysis.
+In many real-world scenarios, users may want to forward their data to a 3rd party system for further in-depth analysis.
+**Exporter** has made that possible.
+
+The exporter is an independent module that has to be manually activated.
+
+Right now, we provide the following exporters:
+1. gRPC Exporter
+- Metrics
+2. Kafka Exporter
+- Trace
+- Log
+
+## gRPC Exporter
+### Metrics gRPC Exporter
+Metrics gRPC exporter uses SkyWalking's native exporter service definition. Here is the proto definition.
+```proto
+service MetricExportService {
+ rpc export (stream ExportMetricValue) returns (ExportResponse) {
+ }
+
+ rpc subscription (SubscriptionReq) returns (SubscriptionsResp) {
+ }
+}
+
+message ExportMetricValue {
+ string metricName = 1;
+ string entityName = 2;
+ string entityId = 3;
+ ValueType type = 4;
+ int64 timeBucket = 5;
+ int64 longValue = 6;
+ double doubleValue = 7;
+ repeated int64 longValues = 8;
+}
+
+message SubscriptionsResp {
+ repeated SubscriptionMetric metrics = 1;
+}
+
+message SubscriptionMetric {
+ string metricName = 1;
+ EventType eventType = 2;
+}
+
+enum ValueType {
+ LONG = 0;
+ DOUBLE = 1;
+ MULTI_LONG = 2;
+}
+
+enum EventType {
+ // The metrics aggregated in this bulk, not include the existing persistent data.
+ INCREMENT = 0;
+ // Final result of the metrics at this moment.
+ TOTAL = 1;
+}
+
+message SubscriptionReq {
+
+}
+
+message ExportResponse {
+}
+```
+
+To activate the exporter, you should add this into your `application.yml`
Review Comment:
Same suggestions for trace and log exporter configurations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on a diff in pull request #9817: Support export `Trace` and `Log` through Kafka.
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on code in PR #9817:
URL: https://github.com/apache/skywalking/pull/9817#discussion_r1000168680
##########
oap-server/exporter/pom.xml:
##########
@@ -38,6 +38,11 @@
<artifactId>library-datacarrier-queue</artifactId>
<version>${project.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.kafka</groupId>
+ <artifactId>kafka-clients</artifactId>
+ <version>${kafka-clients.version}</version>
Review Comment:
Move the version management to oap-server-bom module
##########
oap-server/server-starter/src/main/resources/application.yml:
##########
@@ -433,7 +433,7 @@ alarm:
default:
telemetry:
- selector: ${SW_TELEMETRY:none}
+ selector: ${SW_TELEMETRY:prometheus}
Review Comment:
Do we want to change the default value?
##########
docs/en/setup/backend/exporter.md:
##########
@@ -0,0 +1,161 @@
+# Exporter
+SkyWalking provides the essential functions of metrics aggregation, trace, log, alarm, and analysis.
+In many real-world scenarios, users may want to forward their data to a 3rd party system for further in-depth analysis.
+**Exporter** has made that possible.
+
+The exporter is an independent module that has to be manually activated.
+
+Right now, we provide the following exporters:
+1. gRPC Exporter
+- Metrics
+2. Kafka Exporter
+- Trace
+- Log
+
+## gRPC Exporter
+### Metrics gRPC Exporter
+Metrics gRPC exporter uses SkyWalking's native exporter service definition. Here is the proto definition.
+```proto
+service MetricExportService {
+ rpc export (stream ExportMetricValue) returns (ExportResponse) {
+ }
+
+ rpc subscription (SubscriptionReq) returns (SubscriptionsResp) {
+ }
Review Comment:
I would just link to the proto file instead of copying them here as they might be outdated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #9817: Support export `Trace` and `Log` through Kafka.
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on PR #9817:
URL: https://github.com/apache/skywalking/pull/9817#issuecomment-1284981705
@Superskyyy This should help the aiops engine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] Superskyyy commented on a diff in pull request #9817: Support export `Trace` and `Log` through Kafka.
Posted by GitBox <gi...@apache.org>.
Superskyyy commented on code in PR #9817:
URL: https://github.com/apache/skywalking/pull/9817#discussion_r1001201145
##########
docs/en/setup/backend/exporter.md:
##########
@@ -0,0 +1,161 @@
+# Exporter
+SkyWalking provides the essential functions of metrics aggregation, trace, log, alarm, and analysis.
+In many real-world scenarios, users may want to forward their data to a 3rd party system for further in-depth analysis.
+**Exporter** has made that possible.
+
+The exporter is an independent module that has to be manually activated.
+
+Right now, we provide the following exporters:
+1. gRPC Exporter
+- Metrics
+2. Kafka Exporter
+- Trace
+- Log
+
+## gRPC Exporter
+### Metrics gRPC Exporter
+Metrics gRPC exporter uses SkyWalking's native exporter service definition. Here is the proto definition.
+```proto
+service MetricExportService {
+ rpc export (stream ExportMetricValue) returns (ExportResponse) {
+ }
+
+ rpc subscription (SubscriptionReq) returns (SubscriptionsResp) {
+ }
+}
+
+message ExportMetricValue {
+ string metricName = 1;
+ string entityName = 2;
+ string entityId = 3;
+ ValueType type = 4;
+ int64 timeBucket = 5;
+ int64 longValue = 6;
+ double doubleValue = 7;
+ repeated int64 longValues = 8;
+}
+
+message SubscriptionsResp {
+ repeated SubscriptionMetric metrics = 1;
+}
+
+message SubscriptionMetric {
+ string metricName = 1;
+ EventType eventType = 2;
+}
+
+enum ValueType {
+ LONG = 0;
+ DOUBLE = 1;
+ MULTI_LONG = 2;
+}
+
+enum EventType {
+ // The metrics aggregated in this bulk, not include the existing persistent data.
+ INCREMENT = 0;
+ // Final result of the metrics at this moment.
+ TOTAL = 1;
+}
+
+message SubscriptionReq {
+
+}
+
+message ExportResponse {
+}
+```
+
+To activate the exporter, you should add this into your `application.yml`
+```yaml
+exporter:
+ default:
+ # gRPC exporter
+ enableGRPCMetrics: ${SW_EXPORTER_ENABLE_GRPC_METRICS:true}
+ gRPCTargetHost: ${SW_EXPORTER_GRPC_HOST:127.0.0.1}
+ gRPCTargetPort: ${SW_EXPORTER_GRPC_PORT:9870}
+ ...
+```
+
+- `gRPCTargetHost`:`gRPCTargetPort` is the expected target service address. You could set any gRPC server to receive the data.
+- Target gRPC service needs to go on standby; otherwise, the OAP startup may fail.
+
+#### Target exporter service
+1. Subscription implementation.
+Return the expected metrics name list with event type (incremental or total). All names must match the OAL/MAL script definition.
+Return empty list, if you want to export all metrics in the incremental event type.
+
+2. Export implementation.
+Stream service. All subscribed metrics will be sent here based on the OAP core schedule. Also, if the OAP is deployed as a cluster,
+this method will be called concurrently. For metrics value, you need to follow `#type` to choose `#longValue` or `#doubleValue`.
+
+## Kafka Exporter
+### Trace Kafka Exporter
+Trace kafka exporter pushes messages to the Kafka Broker and Topic `skywalking-trace` to export the trace. Here is the message:
+```
+ProducerRecord<String, Bytes>
+Key: TraceSegmentId
+Value: Bytes of SegmentObject
+```
+
+The `SegmentObject` definition follows the protocol:
+[SkyWalking data collect protocol#Tracing.proto](https://github.com/apache/skywalking-data-collect-protocol/blob/master/language-agent/Tracing.proto).
+```proto
+// The segment is a collection of spans. It includes all collected spans in a simple one request context, such as a HTTP request process.
+message SegmentObject {
+ string traceId = 1;
+ string traceSegmentId = 2;
+ repeated SpanObject spans = 3;
+ string service = 4;
+ string serviceInstance = 5;
+ bool isSizeLimited = 6;
+}
+```
+
+To activate the exporter, you should add this into your `application.yml`
+```yaml
+exporter:
+ default:
+ # Kafka exporter
+ enableKafkaTrace: ${SW_EXPORTER_ENABLE_KAFKA_TRACE:true}
+ kafkaBootstrapServers: ${SW_EXPORTER_KAFKA_SERVERS:localhost:9092}
+ # Kafka producer config, JSON format as Properties.
+ kafkaProducerConfig: ${SW_EXPORTER_KAFKA_PRODUCER_CONFIG:""}
+ kafkaTopicTrace: ${SW_EXPORTER_KAFKA_TOPIC_TRACE:skywalking-trace}
+ ...
+```
+
+### Log Kafka Exporter
+Log kafka exporter pushes messages to the Kafka Broker and Topic `skywalking-log` to export the log. Here is the message:
+```
+ProducerRecord<String, Bytes>
+Key: LogRecordId
+Value: Bytes of LogData
Review Comment:
I don't recalI encoding problems on the Python side (protobuf), if you are referring to this [discussion](https://github.com/apache/skywalking/issues/8794) it seems unrelated. So we are good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] sonatype-lift[bot] commented on pull request #9817: Support export `Trace` and `Log` through Kafka.
Posted by GitBox <gi...@apache.org>.
sonatype-lift[bot] commented on PR #9817:
URL: https://github.com/apache/skywalking/pull/9817#issuecomment-1284999911
:warning: **6 God Classes** were detected by Lift in this project. [Visit the Lift web console](https://lift.sonatype.com/results/github.com/apache/skywalking/01GFSYMR7FKRCRQXT7HBHXY6WN?tab=technical-debt&utm_source=github.com&utm_campaign=lift-comment&utm_content=apache\%20skywalking) for more details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] Superskyyy commented on pull request #9817: Support export `Trace` and `Log` through Kafka.
Posted by GitBox <gi...@apache.org>.
Superskyyy commented on PR #9817:
URL: https://github.com/apache/skywalking/pull/9817#issuecomment-1286258428
Looking great! This is exactly what I need. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #9817: Support export `Trace` and `Log` through Kafka.
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #9817:
URL: https://github.com/apache/skywalking/pull/9817#discussion_r1000197189
##########
docs/en/setup/backend/exporter.md:
##########
@@ -0,0 +1,161 @@
+# Exporter
+SkyWalking provides the essential functions of metrics aggregation, trace, log, alarm, and analysis.
+In many real-world scenarios, users may want to forward their data to a 3rd party system for further in-depth analysis.
+**Exporter** has made that possible.
+
+The exporter is an independent module that has to be manually activated.
+
+Right now, we provide the following exporters:
+1. gRPC Exporter
+- Metrics
+2. Kafka Exporter
+- Trace
+- Log
+
+## gRPC Exporter
+### Metrics gRPC Exporter
+Metrics gRPC exporter uses SkyWalking's native exporter service definition. Here is the proto definition.
Review Comment:
```suggestion
Metrics gRPC exporter uses SkyWalking's native export service definition. Here is the proto definition.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wankai123 commented on a diff in pull request #9817: Support export `Trace` and `Log` through Kafka.
Posted by GitBox <gi...@apache.org>.
wankai123 commented on code in PR #9817:
URL: https://github.com/apache/skywalking/pull/9817#discussion_r1000179133
##########
oap-server/server-starter/src/main/resources/application.yml:
##########
@@ -433,7 +433,7 @@ alarm:
default:
telemetry:
- selector: ${SW_TELEMETRY:none}
+ selector: ${SW_TELEMETRY:prometheus}
Review Comment:
no, It's a mistake
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #9817: Support export `Trace` and `Log` through Kafka.
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #9817:
URL: https://github.com/apache/skywalking/pull/9817#discussion_r1000196318
##########
docs/en/setup/backend/exporter.md:
##########
@@ -0,0 +1,161 @@
+# Exporter
+SkyWalking provides the essential functions of metrics aggregation, trace, log, alarm, and analysis.
+In many real-world scenarios, users may want to forward their data to a 3rd party system for further in-depth analysis.
+**Exporter** has made that possible.
+
+The exporter is an independent module that has to be manually activated.
+
+Right now, we provide the following exporters:
Review Comment:
```suggestion
Right now, we provide the following exporting channels:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #9817: Support export `Trace` and `Log` through Kafka.
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #9817:
URL: https://github.com/apache/skywalking/pull/9817#discussion_r1000195727
##########
docs/en/setup/backend/exporter.md:
##########
@@ -0,0 +1,161 @@
+# Exporter
+SkyWalking provides the essential functions of metrics aggregation, trace, log, alarm, and analysis.
Review Comment:
```suggestion
SkyWalking provides the essential functions of observability, including metrics aggregation, trace, log, alerting, and profiling.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #9817: Support export `Trace` and `Log` through Kafka.
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #9817:
URL: https://github.com/apache/skywalking/pull/9817#discussion_r1000199233
##########
docs/en/setup/backend/exporter.md:
##########
@@ -0,0 +1,161 @@
+# Exporter
+SkyWalking provides the essential functions of metrics aggregation, trace, log, alarm, and analysis.
+In many real-world scenarios, users may want to forward their data to a 3rd party system for further in-depth analysis.
+**Exporter** has made that possible.
+
+The exporter is an independent module that has to be manually activated.
+
+Right now, we provide the following exporters:
+1. gRPC Exporter
+- Metrics
+2. Kafka Exporter
+- Trace
+- Log
Review Comment:
Please add `in-page` links to the parts of the doc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #9817: Support export `Trace` and `Log` through Kafka.
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #9817:
URL: https://github.com/apache/skywalking/pull/9817#discussion_r1000467468
##########
oap-server-bom/pom.xml:
##########
@@ -73,6 +73,8 @@
<httpcore.version>4.4.13</httpcore.version>
<commons-compress.version>1.21</commons-compress.version>
<banyandb-java-client.version>0.1.0</banyandb-java-client.version>
+ <kafka-clients.version>2.4.1</kafka-clients.version>
Review Comment:
Could you could the existing Kafka fetcher? Why doesn't this declear?
##########
docs/menu.yml:
##########
@@ -133,8 +133,8 @@ catalog:
path: "/en/setup/backend/on-demand-pod-log"
- name: "Extension"
catalog:
- - name: "Metrics Exporter"
- path: "/en/setup/backend/metrics-exporter"
Review Comment:
Let's keep this file with contents guiding user to the new exporter doc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng merged pull request #9817: Support export `Trace` and `Log` through Kafka.
Posted by GitBox <gi...@apache.org>.
wu-sheng merged PR #9817:
URL: https://github.com/apache/skywalking/pull/9817
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org