You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by wu...@apache.org on 2022/10/08 08:13:06 UTC
[skywalking] branch master updated: Limit the max length of trace and log tag's `key=value` (#9741)
This is an automated email from the ASF dual-hosted git repository.
wusheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/skywalking.git
The following commit(s) were added to refs/heads/master by this push:
new f1e0558ce3 Limit the max length of trace and log tag's `key=value` (#9741)
f1e0558ce3 is described below
commit f1e0558ce356cfabc28a6064a8098c771b990621
Author: Wan Kai <wa...@foxmail.com>
AuthorDate: Sat Oct 8 16:12:48 2022 +0800
Limit the max length of trace and log tag's `key=value` (#9741)
---
docs/en/changes/changes.md | 3 +++
docs/en/setup/backend/configuration-vocabulary.md | 8 ++++----
docs/en/setup/backend/zipkin-trace.md | 2 ++
.../trace/parser/listener/SegmentAnalysisListener.java | 6 ++++++
.../provider/log/listener/RecordSinkListener.java | 9 +++++++++
.../skywalking/oap/server/core/alarm/AlarmRecord.java | 3 ++-
.../oap/server/core/alarm/AlarmStandardPersistence.java | 7 +++++++
.../core/analysis/manual/log/AbstractLogRecord.java | 3 ++-
.../oap/server/core/analysis/manual/searchtag/Tag.java | 1 +
.../analysis/manual/searchtag/TagAutocompleteData.java | 2 +-
.../core/analysis/manual/segment/SegmentRecord.java | 3 ++-
.../oap/server/core/zipkin/ZipkinSpanRecord.java | 3 ++-
.../oap/server/receiver/zipkin/trace/SpanForward.java | 15 ++++++++++++---
.../server-starter/src/main/resources/application.yml | 5 +++++
14 files changed, 58 insertions(+), 12 deletions(-)
diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md
index 4ca4624d11..164c708a22 100644
--- a/docs/en/changes/changes.md
+++ b/docs/en/changes/changes.md
@@ -46,6 +46,9 @@
* Support skywalking-client-js report empty `service version` and `page path` , set default version as `latest` and
default page path as `/`(root). Fix the
error `fetching data (/browser_app_page_pv0) : Can't split endpoint id into 2 parts`.
+* [**Breaking Change**] Limit the max length of trace/log/alarm tag's `key=value`, set the max length of column `tags`
+ in tables`log_tag/segment_tag/alarm_record_tag` and column `query` in `zipkin_query` and column `tag_value` in `tag_autocomplete` to 256.
+ SQL-Database requires altering these columns' length or removing these tables before OAP starts, if bump up from previous releases.
#### UI
diff --git a/docs/en/setup/backend/configuration-vocabulary.md b/docs/en/setup/backend/configuration-vocabulary.md
index 72c7ea74ac..eb53ab9924 100644
--- a/docs/en/setup/backend/configuration-vocabulary.md
+++ b/docs/en/setup/backend/configuration-vocabulary.md
@@ -32,9 +32,9 @@ The Configuration Vocabulary lists all available configurations provided by `app
| - | - | serviceNameMaxLength | Maximum length limit of service names. [...]
| - | - | instanceNameMaxLength | Maximum length limit of service instance names. The maximum length of service + instance names should be less than 200. [...]
| - | - | endpointNameMaxLength | Maximum length limit of endpoint names. The maximum length of service + endpoint names should be less than 240. [...]
-| - | - | searchableTracesTags | Defines a set of span tag keys which are searchable through GraphQL. Multiple values are separated by commas. [...]
-| - | - | searchableLogsTags | Defines a set of log tag keys which are searchable through GraphQL. Multiple values are separated by commas. [...]
-| - | - | searchableAlarmTags | Defines a set of alarm tag keys which are searchable through GraphQL. Multiple values are separated by commas. [...]
+| - | - | searchableTracesTags | Defines a set of span tag keys which are searchable through GraphQL. Multiple values are separated by commas. The max length of key=value should be less than 256 or will be dropped. [...]
+| - | - | searchableLogsTags | Defines a set of log tag keys which are searchable through GraphQL. Multiple values are separated by commas. The max length of key=value should be less than 256 or will be dropped. [...]
+| - | - | searchableAlarmTags | Defines a set of alarm tag keys which are searchable through GraphQL. Multiple values are separated by commas. The max length of key=value should be less than 256 or will be dropped. [...]
| - | - | autocompleteTagKeysQueryMaxSize | The max size of tags keys for autocomplete select. [...]
| - | - | autocompleteTagValuesQueryMaxSize | The max size of tags values for autocomplete select. [...]
| - | - | gRPCThreadPoolSize | Pool size of gRPC server. [...]
@@ -199,7 +199,7 @@ The Configuration Vocabulary lists all available configurations provided by `app
| - | - | enabledOtelRules | Enabled metric rules for OC handler. [...]
| receiver-zipkin | default | A receiver for Zipkin traces. | - [...]
| - | - | sampleRate | The sample rate precision is 1/10000, should be between 0 and 10000 [...]
-| - | - | searchableTracesTags | Defines a set of span tag keys which are searchable. Multiple values are separated by commas. [...]
+| - | - | searchableTracesTags | Defines a set of span tag keys which are searchable. Multiple values are separated by commas. The max length of key=value should be less than 256 or will be dropped. [...]
| - | - | enableHttpCollector | Enable Http Collector. [...]
| - | - | restHost | Binding IP of RESTful services. [...]
| - | - | restPort | Binding port of RESTful services. [...]
diff --git a/docs/en/setup/backend/zipkin-trace.md b/docs/en/setup/backend/zipkin-trace.md
index 962eae97d8..132a688c3f 100644
--- a/docs/en/setup/backend/zipkin-trace.md
+++ b/docs/en/setup/backend/zipkin-trace.md
@@ -9,6 +9,8 @@ Set `enableHttpCollector` to enable HTTP collector and `enableKafkaCollector` to
receiver-zipkin:
selector: ${SW_RECEIVER_ZIPKIN:default}
default:
+ # Defines a set of span tag keys which are searchable.
+ # The max length of key=value should be less than 256 or will be dropped.
searchableTracesTags: ${SW_ZIPKIN_SEARCHABLE_TAG_KEYS:http.method}
# The sample rate precision is 1/10000, should be between 0 and 10000
sampleRate: ${SW_ZIPKIN_SAMPLE_RATE:10000}
diff --git a/oap-server/analyzer/agent-analyzer/src/main/java/org/apache/skywalking/oap/server/analyzer/provider/trace/parser/listener/SegmentAnalysisListener.java b/oap-server/analyzer/agent-analyzer/src/main/java/org/apache/skywalking/oap/server/analyzer/provider/trace/parser/listener/SegmentAnalysisListener.java
index b125ff319c..b19bbfd968 100644
--- a/oap-server/analyzer/agent-analyzer/src/main/java/org/apache/skywalking/oap/server/analyzer/provider/trace/parser/listener/SegmentAnalysisListener.java
+++ b/oap-server/analyzer/agent-analyzer/src/main/java/org/apache/skywalking/oap/server/analyzer/provider/trace/parser/listener/SegmentAnalysisListener.java
@@ -152,6 +152,12 @@ public class SegmentAnalysisListener implements FirstAnalysisListener, EntryAnal
span.getTagsList().forEach(tag -> {
if (searchableTagKeys.contains(tag.getKey())) {
final Tag spanTag = new Tag(tag.getKey(), tag.getValue());
+ if (tag.getValue().length() > Tag.TAG_LENGTH || spanTag.toString().length() > Tag.TAG_LENGTH) {
+ if (log.isDebugEnabled()) {
+ log.debug("Segment tag : {} length > : {}, dropped", spanTag, Tag.TAG_LENGTH);
+ }
+ return;
+ }
if (!segment.getTags().contains(spanTag)) {
segment.getTags().add(spanTag);
}
diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/RecordSinkListener.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/RecordSinkListener.java
index 8a866ef688..0de9759760 100644
--- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/RecordSinkListener.java
+++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/RecordSinkListener.java
@@ -45,6 +45,8 @@ import org.apache.skywalking.oap.server.core.query.type.ContentType;
import org.apache.skywalking.oap.server.core.source.Log;
import org.apache.skywalking.oap.server.core.source.SourceReceiver;
import org.apache.skywalking.oap.server.library.module.ModuleManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
import static org.apache.skywalking.oap.server.library.util.ProtoBufJsonUtils.toJSON;
@@ -53,6 +55,7 @@ import static org.apache.skywalking.oap.server.library.util.ProtoBufJsonUtils.to
*/
@RequiredArgsConstructor
public class RecordSinkListener implements LogSinkListener {
+ private static final Logger LOGGER = LoggerFactory.getLogger(RecordSinkListener.class);
private final SourceReceiver sourceReceiver;
private final NamingControl namingControl;
private final List<String> searchableTagKeys;
@@ -126,6 +129,12 @@ public class RecordSinkListener implements LogSinkListener {
logData.getTags().getDataList().forEach(tag -> {
if (searchableTagKeys.contains(tag.getKey())) {
final Tag logTag = new Tag(tag.getKey(), tag.getValue());
+ if (tag.getValue().length() > Tag.TAG_LENGTH || logTag.toString().length() > Tag.TAG_LENGTH) {
+ if (LOGGER.isDebugEnabled()) {
+ LOGGER.debug("Log tag : {} length > : {}, dropped", logTag, Tag.TAG_LENGTH);
+ }
+ return;
+ }
logTags.add(logTag);
}
});
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/alarm/AlarmRecord.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/alarm/AlarmRecord.java
index 178aaa5740..dd76b4a8a5 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/alarm/AlarmRecord.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/alarm/AlarmRecord.java
@@ -23,6 +23,7 @@ import lombok.Getter;
import lombok.Setter;
import org.apache.skywalking.oap.server.core.Const;
import org.apache.skywalking.oap.server.core.analysis.Stream;
+import org.apache.skywalking.oap.server.core.analysis.manual.searchtag.Tag;
import org.apache.skywalking.oap.server.core.analysis.record.Record;
import org.apache.skywalking.oap.server.core.analysis.worker.RecordStreamProcessor;
import org.apache.skywalking.oap.server.core.source.DefaultScopeDefine;
@@ -80,7 +81,7 @@ public class AlarmRecord extends Record {
@Column(columnName = TAGS, indexOnly = true)
@SQLDatabase.AdditionalEntity(additionalTables = {ADDITIONAL_TAG_TABLE})
private List<String> tagsInString;
- @Column(columnName = TAGS_RAW_DATA, storageOnly = true)
+ @Column(columnName = TAGS_RAW_DATA, storageOnly = true, length = Tag.TAG_LENGTH)
private byte[] tagsRawData;
public static class Builder implements StorageBuilder<AlarmRecord> {
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/alarm/AlarmStandardPersistence.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/alarm/AlarmStandardPersistence.java
index ba417c9787..cd417b2c31 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/alarm/AlarmStandardPersistence.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/alarm/AlarmStandardPersistence.java
@@ -78,6 +78,13 @@ public class AlarmStandardPersistence implements AlarmCallback {
tags.forEach(tag -> {
if (configService.getSearchableAlarmTags().contains(tag.getKey())) {
final Tag alarmTag = new Tag(tag.getKey(), tag.getValue());
+
+ if (tag.getValue().length() > Tag.TAG_LENGTH || alarmTag.toString().length() > Tag.TAG_LENGTH) {
+ if (LOGGER.isDebugEnabled()) {
+ LOGGER.debug("Alarm tag : {} length > : {}, dropped", alarmTag, Tag.TAG_LENGTH);
+ }
+ return;
+ }
alarmTags.add(alarmTag);
}
});
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/log/AbstractLogRecord.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/log/AbstractLogRecord.java
index cbfa36f854..5bbae2328a 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/log/AbstractLogRecord.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/log/AbstractLogRecord.java
@@ -22,6 +22,7 @@ import java.util.List;
import lombok.Getter;
import lombok.Setter;
import org.apache.skywalking.oap.server.core.UnexpectedException;
+import org.apache.skywalking.oap.server.core.analysis.manual.searchtag.Tag;
import org.apache.skywalking.oap.server.core.analysis.record.Record;
import org.apache.skywalking.oap.server.core.query.type.ContentType;
import org.apache.skywalking.oap.server.core.storage.annotation.BanyanDB;
@@ -99,7 +100,7 @@ public abstract class AbstractLogRecord extends Record {
private byte[] tagsRawData;
@Setter
@Getter
- @Column(columnName = TAGS, indexOnly = true)
+ @Column(columnName = TAGS, indexOnly = true, length = Tag.TAG_LENGTH)
@SQLDatabase.AdditionalEntity(additionalTables = {ADDITIONAL_TAG_TABLE})
private List<String> tagsInString;
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/searchtag/Tag.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/searchtag/Tag.java
index 893d4081fb..2f35cb2aa6 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/searchtag/Tag.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/searchtag/Tag.java
@@ -33,6 +33,7 @@ import org.apache.skywalking.oap.server.library.util.CollectionUtils;
@NoArgsConstructor
@AllArgsConstructor
public class Tag {
+ public static final int TAG_LENGTH = 256;
private String key;
private String value;
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/searchtag/TagAutocompleteData.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/searchtag/TagAutocompleteData.java
index 629163b727..78c717f4ef 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/searchtag/TagAutocompleteData.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/searchtag/TagAutocompleteData.java
@@ -55,7 +55,7 @@ public class TagAutocompleteData extends Metrics {
private String tagKey;
@Setter
@Getter
- @Column(columnName = TAG_VALUE)
+ @Column(columnName = TAG_VALUE, length = Tag.TAG_LENGTH)
private String tagValue;
@Setter
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/segment/SegmentRecord.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/segment/SegmentRecord.java
index a526b1fce8..e88f326e76 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/segment/SegmentRecord.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/segment/SegmentRecord.java
@@ -22,6 +22,7 @@ import java.util.List;
import lombok.Getter;
import lombok.Setter;
import org.apache.skywalking.oap.server.core.analysis.Stream;
+import org.apache.skywalking.oap.server.core.analysis.manual.searchtag.Tag;
import org.apache.skywalking.oap.server.core.analysis.record.Record;
import org.apache.skywalking.oap.server.core.analysis.worker.RecordStreamProcessor;
import org.apache.skywalking.oap.server.core.source.DefaultScopeDefine;
@@ -99,7 +100,7 @@ public class SegmentRecord extends Record {
private byte[] dataBinary;
@Setter
@Getter
- @Column(columnName = TAGS, indexOnly = true)
+ @Column(columnName = TAGS, indexOnly = true, length = Tag.TAG_LENGTH)
@SQLDatabase.AdditionalEntity(additionalTables = {ADDITIONAL_TAG_TABLE})
private List<String> tags;
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/zipkin/ZipkinSpanRecord.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/zipkin/ZipkinSpanRecord.java
index c5c611b060..d0f3a6e659 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/zipkin/ZipkinSpanRecord.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/zipkin/ZipkinSpanRecord.java
@@ -48,6 +48,7 @@ import static org.apache.skywalking.oap.server.core.analysis.record.Record.TIME_
@SQLDatabase.Sharding(shardingAlgorithm = ShardingAlgorithm.TIME_SEC_RANGE_SHARDING_ALGORITHM, dataSourceShardingColumn = TRACE_ID, tableShardingColumn = TIME_BUCKET)
public class ZipkinSpanRecord extends Record {
private static final Gson GSON = new Gson();
+ public static final int QUERY_LENGTH = 256;
public static final String INDEX_NAME = "zipkin_span";
public static final String ADDITIONAL_QUERY_TABLE = "zipkin_query";
public static final String TRACE_ID = "trace_id";
@@ -156,7 +157,7 @@ public class ZipkinSpanRecord extends Record {
private int shared;
@Setter
@Getter
- @Column(columnName = QUERY, indexOnly = true)
+ @Column(columnName = QUERY, indexOnly = true, length = QUERY_LENGTH)
@SQLDatabase.AdditionalEntity(additionalTables = {ADDITIONAL_QUERY_TABLE})
private List<String> query;
diff --git a/oap-server/server-receiver-plugin/zipkin-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/zipkin/trace/SpanForward.java b/oap-server/server-receiver-plugin/zipkin-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/zipkin/trace/SpanForward.java
index 9548b67a7b..d873e8e87f 100644
--- a/oap-server/server-receiver-plugin/zipkin-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/zipkin/trace/SpanForward.java
+++ b/oap-server/server-receiver-plugin/zipkin-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/zipkin/trace/SpanForward.java
@@ -24,10 +24,13 @@ import java.util.Arrays;
import java.util.List;
import java.util.Map;
+import lombok.extern.slf4j.Slf4j;
import org.apache.skywalking.oap.server.core.Const;
import org.apache.skywalking.oap.server.core.CoreModule;
+import org.apache.skywalking.oap.server.core.analysis.manual.searchtag.Tag;
import org.apache.skywalking.oap.server.core.analysis.manual.searchtag.TagType;
import org.apache.skywalking.oap.server.core.source.TagAutocomplete;
+import org.apache.skywalking.oap.server.core.zipkin.ZipkinSpanRecord;
import org.apache.skywalking.oap.server.core.zipkin.source.ZipkinService;
import org.apache.skywalking.oap.server.core.zipkin.source.ZipkinServiceRelation;
import org.apache.skywalking.oap.server.core.zipkin.source.ZipkinServiceSpan;
@@ -42,8 +45,8 @@ import org.apache.skywalking.oap.server.receiver.zipkin.ZipkinReceiverConfig;
import zipkin2.Annotation;
import zipkin2.Span;
import zipkin2.internal.HexCodec;
-import zipkin2.internal.RecyclableBuffers;
+@Slf4j
public class SpanForward {
private final ZipkinReceiverConfig config;
private final NamingControl namingControl;
@@ -110,7 +113,10 @@ public class SpanForward {
JsonObject tagsJson = new JsonObject();
for (Annotation annotation : span.annotations()) {
annotationsJson.addProperty(Long.toString(annotation.timestamp()), annotation.value());
- if (annotation.value().length() > RecyclableBuffers.SHORT_STRING_LENGTH) {
+ if (annotation.value().length() > ZipkinSpanRecord.QUERY_LENGTH) {
+ if (log.isDebugEnabled()) {
+ log.debug("Span annotation : {} length > : {}, dropped", annotation.value(), ZipkinSpanRecord.QUERY_LENGTH);
+ }
continue;
}
query.add(annotation.value());
@@ -119,7 +125,10 @@ public class SpanForward {
for (Map.Entry<String, String> tag : span.tags().entrySet()) {
String tagString = tag.getKey() + "=" + tag.getValue();
tagsJson.addProperty(tag.getKey(), tag.getValue());
- if (tagString.length() > RecyclableBuffers.SHORT_STRING_LENGTH) {
+ if (tag.getValue().length() > Tag.TAG_LENGTH || tagString.length() > Tag.TAG_LENGTH) {
+ if (log.isDebugEnabled()) {
+ log.debug("Span tag : {} length > : {}, dropped", tagString, Tag.TAG_LENGTH);
+ }
continue;
}
query.add(tag.getKey());
diff --git a/oap-server/server-starter/src/main/resources/application.yml b/oap-server/server-starter/src/main/resources/application.yml
index e2c3951903..d05739d1c4 100644
--- a/oap-server/server-starter/src/main/resources/application.yml
+++ b/oap-server/server-starter/src/main/resources/application.yml
@@ -116,10 +116,13 @@ core:
# The max length of service + endpoint names should be less than 240
endpointNameMaxLength: ${SW_ENDPOINT_NAME_MAX_LENGTH:150}
# Define the set of span tag keys, which should be searchable through the GraphQL.
+ # The max length of key=value should be less than 256 or will be dropped.
searchableTracesTags: ${SW_SEARCHABLE_TAG_KEYS:http.method,http.status_code,rpc.status_code,db.type,db.instance,mq.queue,mq.topic,mq.broker}
# Define the set of log tag keys, which should be searchable through the GraphQL.
+ # The max length of key=value should be less than 256 or will be dropped.
searchableLogsTags: ${SW_SEARCHABLE_LOGS_TAG_KEYS:level}
# Define the set of alarm tag keys, which should be searchable through the GraphQL.
+ # The max length of key=value should be less than 256 or will be dropped.
searchableAlarmTags: ${SW_SEARCHABLE_ALARM_TAG_KEYS:level}
# The max size of tags keys for autocomplete select.
autocompleteTagKeysQueryMaxSize: ${SW_AUTOCOMPLETE_TAG_KEYS_QUERY_MAX_SIZE:100}
@@ -351,6 +354,8 @@ receiver-otel:
receiver-zipkin:
selector: ${SW_RECEIVER_ZIPKIN:-}
default:
+ # Defines a set of span tag keys which are searchable.
+ # The max length of key=value should be less than 256 or will be dropped.
searchableTracesTags: ${SW_ZIPKIN_SEARCHABLE_TAG_KEYS:http.method}
# The sample rate precision is 1/10000, should be between 0 and 10000
sampleRate: ${SW_ZIPKIN_SAMPLE_RATE:10000}