You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by wa...@apache.org on 2026/09/23 04:57:24 UTC
(skywalking) 01/01: Keep the TraceQL scope in the OTLP tag autocomplete
This is an automated email from the ASF dual-hosted git repository.
wankai123 pushed a commit to branch fix/otlp-autocomplete-scope
in repository https://gitbox.apache.org/repos/asf/skywalking.git
commit 8a6659411e05cd174187365f907945d527cf24a8
Author: wankai123 <wa...@foxmail.com>
AuthorDate: Wed Sep 23 12:57:08 2026 +0800
Keep the TraceQL scope in the OTLP tag autocomplete
The OTLP tag autocomplete stored keys without their scope, so every catalog
key was listed under the `span` scope. A searchable key that is a resource
attribute, such as `service.namespace`, was offered as
`span.service.namespace`, a query that matches nothing, and `resource.<key>`
value lookups answered 400.
- OTLPSpanForward publishes autocomplete keys with the same `resource.` /
`span.` prefix as the `tags` search index.
- The OTLP TraceQL handler lists each key under its own scope, answers
`resource.<key>` values from the index, and looks up both scopes for an
unscoped `.key`.
- `otlpTraceSearchableTags` entries name their scope (`resource.<key>` or
`span.<key>`); an entry without one fails the boot in `otlp` mode. The
default gains low-cardinality resource attributes (deployment
environment, service namespace and version, Kubernetes namespace and
cluster).
---
docs/en/api/traceql-service.md | 9 +-
docs/en/setup/backend/configuration-vocabulary.md | 2 +-
docs/en/setup/backend/otlp-trace.md | 4 +-
.../traceql/handler/OTLPTraceQLApiHandler.java | 48 ++++++--
.../traceql/handler/OTLPTraceQLApiHandlerTest.java | 131 +++++++++++++++++++++
.../receiver/otel/OtelMetricReceiverConfig.java | 22 +++-
.../receiver/otel/OtelMetricReceiverProvider.java | 7 ++
.../server/receiver/otel/otlp/OTLPSpanForward.java | 17 +--
.../otel/OtelMetricReceiverProviderTest.java | 60 ++++++++++
.../receiver/otel/otlp/OTLPSpanForwardTest.java | 53 ++++++++-
.../src/main/resources/application.yml | 9 +-
test/e2e-v2/cases/storage/expected/config-dump.yml | 2 +-
12 files changed, 333 insertions(+), 31 deletions(-)
diff --git a/docs/en/api/traceql-service.md b/docs/en/api/traceql-service.md
index 99f81900b0..754d5f3c7a 100644
--- a/docs/en/api/traceql-service.md
+++ b/docs/en/api/traceql-service.md
@@ -858,12 +858,15 @@ table is one column set over every trace.
### Tag Names and Values
`/api/v2/search/tags` returns `service.name`, `service.instance.id` and `remote.service` under the `resource` scope,
-the keys configured by `receiver-otel.default.otlpTraceSearchableTags` plus `otel.scope.name` under the `span` scope,
-and `name`, `status`, `kind`, `duration` and their `span:` spellings under `intrinsic`. `scope` narrows the answer to
+`otel.scope.name` under the `span` scope, and `name`, `status`, `kind`, `duration` and their `span:` spellings under
+`intrinsic`. A key configured by `receiver-otel.default.otlpTraceSearchableTags` is written there with its
+scope, `resource.<key>` or `span.<key>`, and listed under that scope without the prefix, so a key picked from the
+list builds a query that matches. `scope` narrows the answer to
one scope (`event`, `link` and `instrumentation` are accepted and empty), `limit` caps the number of names per scope,
and the v1 `/api/search/tags` lists the same names without scopes. `service.name` values come from the service catalog, `name`
values from the span-name catalog of the service given in `q`, `remote.service` values from the peer catalog, and the
-other span keys from the tag autocomplete index; `limit` caps every value list. The three catalogs list every name the
+other `resource.` and `span.` keys from the tag autocomplete index of that scope, both scopes for an unscoped `.key`;
+`limit` caps every value list. The three catalogs list every name the
storage retains and ignore `start` and `end`: a catalog row is written once, when its name is first seen, so a time
window would hide every name older than it; the autocomplete index and the sample below honor the window. A `q`
narrower than a service name, for
diff --git a/docs/en/setup/backend/configuration-vocabulary.md b/docs/en/setup/backend/configuration-vocabulary.md
index 9dbf19e555..46b9b71558 100644
--- a/docs/en/setup/backend/configuration-vocabulary.md
+++ b/docs/en/setup/backend/configuration-vocabulary.md
@@ -210,7 +210,7 @@ It divided into several modules, each of which has its own settings. The followi
| - | - | enabledHandlers | Enabled handlers for otel. [...]
| - | - | enabledOtelMetricsRules | Enabled metric rules for OTLP handler. [...]
| - | - | otlpTraceStorage | Where the `otlp-traces` handler stores spans. `otlp` stores the native OTLP span in `otlp_span`, served by the TraceQL `/otlp` datasource; `zipkin` converts to Zipkin v2 and hands them to receiver-zipkin. | SW_OTEL_TRACE_STORAGE | otlp |
-| - | - | otlpTraceSearchableTags | `otlp` mode only. Attribute keys offered by tag autocomplete. Every resource and span attribute is indexed for equality search regardless of this list. | SW_OTEL_TRACE_SEARCHABLE_TAG_KEYS | http.request.method,http.response.status_code,url.path,rpc.method,rpc.grpc.status_code,db.system.name,db.namespace,messaging.system,messaging.destination.name,peer.service |
+| - | - | otlpTraceSearchableTags | `otlp` mode only. Attribute keys offered by tag autocomplete, each with its TraceQL scope, `resource.<key>` or `span.<key>`; an entry without a scope fails the boot. Every resource and span attribute is indexed for equality search regardless of this list. | SW_OTEL_TRACE_SEARCHABLE_TAG_KEYS | resource.deployment.environment.name,resource.deployment.environment,resource.service.namespace,resource.service.version,resourc [...]
| - | - | otlpTraceSampleRate | `otlp` mode only. Head sampling by trace id, precision 1/10000. 10000 keeps every trace. | SW_OTEL_TRACE_SAMPLE_RATE | 10000 |
| - | - | otlpTraceMaxSpansPerSecond | `otlp` mode only. Spans per second accepted before dropping. 0 means no limit. | SW_OTEL_TRACE_MAX_SPANS_PER_SECOND | 0 |
| receiver-zipkin | default | A receiver for Zipkin traces. | - [...]
diff --git a/docs/en/setup/backend/otlp-trace.md b/docs/en/setup/backend/otlp-trace.md
index 941fdef4d3..d935c4ac01 100644
--- a/docs/en/setup/backend/otlp-trace.md
+++ b/docs/en/setup/backend/otlp-trace.md
@@ -63,7 +63,7 @@ receiver-otel:
default:
enabledHandlers: otlp-traces
otlpTraceStorage: otlp
- otlpTraceSearchableTags: http.request.method,http.response.status_code,url.path,rpc.method,rpc.grpc.status_code,db.system.name,db.namespace,messaging.system,messaging.destination.name,peer.service
+ otlpTraceSearchableTags: resource.deployment.environment.name,resource.deployment.environment,resource.service.namespace,resource.service.version,resource.k8s.namespace.name,resource.k8s.cluster.name,span.http.request.method,span.http.response.status_code,span.url.path,span.rpc.method,span.rpc.grpc.status_code,span.db.system.name,span.db.namespace,span.messaging.system,span.messaging.destination.name,span.peer.service
otlpTraceSampleRate: 10000
otlpTraceMaxSpansPerSecond: 0
@@ -80,7 +80,7 @@ The Zipkin receiver and query modules are not needed in this mode.
| Setting | Environment variable | Meaning |
|------------------------------|--------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `otlpTraceStorage` | `SW_OTEL_TRACE_STORAGE` | `otlp`, the default, stores the native span. `zipkin` converts to Zipkin v2 as earlier releases did. Any other value fails the boot. |
-| `otlpTraceSearchableTags` | `SW_OTEL_TRACE_SEARCHABLE_TAG_KEYS` | Attribute keys offered by tag autocomplete (`/api/v2/search/tags`). Every resource and span attribute is indexed for equality search regardless of this list. |
+| `otlpTraceSearchableTags` | `SW_OTEL_TRACE_SEARCHABLE_TAG_KEYS` | Attribute keys offered by tag autocomplete (`/api/v2/search/tags`), each written with its TraceQL scope, `resource.<key>` or `span.<key>`, and listed under that scope; an entry without a scope fails the boot. List low-cardinality attributes only: every distinct value is stored once per day. Every resource and span attribute is indexed for equality search regardless of this list. |
| `otlpTraceSampleRate` | `SW_OTEL_TRACE_SAMPLE_RATE` | Head sampling by trace id, precision 1/10000. `10000` keeps every trace. |
| `otlpTraceMaxSpansPerSecond` | `SW_OTEL_TRACE_MAX_SPANS_PER_SECOND` | Spans per second accepted before the receiver drops the rest. `0` means no limit. Dropped spans are counted in the `otel_spans_dropped` self-observability metric. |
diff --git a/oap-server/server-query-plugin/traceql-plugin/src/main/java/org/apache/skywalking/oap/query/traceql/handler/OTLPTraceQLApiHandler.java b/oap-server/server-query-plugin/traceql-plugin/src/main/java/org/apache/skywalking/oap/query/traceql/handler/OTLPTraceQLApiHandler.java
index 206c90c209..070c2cd804 100644
--- a/oap-server/server-query-plugin/traceql-plugin/src/main/java/org/apache/skywalking/oap/query/traceql/handler/OTLPTraceQLApiHandler.java
+++ b/oap-server/server-query-plugin/traceql-plugin/src/main/java/org/apache/skywalking/oap/query/traceql/handler/OTLPTraceQLApiHandler.java
@@ -31,6 +31,7 @@ import java.io.IOException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashSet;
+import java.util.LinkedHashSet;
import java.util.List;
import java.util.Locale;
import java.util.Map;
@@ -51,6 +52,7 @@ import org.apache.skywalking.oap.server.core.CoreModule;
import org.apache.skywalking.oap.server.core.config.NamingControl;
import org.apache.skywalking.oap.server.core.analysis.manual.searchtag.Tag;
import org.apache.skywalking.oap.server.core.analysis.manual.searchtag.TagType;
+import org.apache.skywalking.oap.server.core.otlp.OTLPSpanRecord;
import org.apache.skywalking.oap.server.core.query.OTLPTraceQueryService;
import org.apache.skywalking.oap.server.core.query.TagAutoCompleteQueryService;
import org.apache.skywalking.oap.server.core.query.input.Duration;
@@ -77,6 +79,8 @@ public class OTLPTraceQLApiHandler extends TraceQLApiHandler {
NAME, STATUS, KIND, DURATION,
SPAN_INTRINSIC_PREFIX + NAME, SPAN_INTRINSIC_PREFIX + STATUS, SPAN_INTRINSIC_PREFIX + KIND, SPAN_INTRINSIC_PREFIX + DURATION);
private static final String OTEL_SCOPE_NAME = OTLPTraceAssembler.OTEL_SCOPE_NAME;
+ // SPAN_OTEL_SCOPE_NAME and RESOURCE_SERVICE_INSTANCE_ID are also the autocomplete keys OTLPSpanForward
+ // always publishes; every autocomplete key carries its scope prefix.
private static final String SPAN_OTEL_SCOPE_NAME = SPAN_PREFIX + OTEL_SCOPE_NAME;
private static final String RESOURCE_SERVICE_INSTANCE_ID = SCOPE_RESOURCE + Const.POINT + SERVICE_INSTANCE_ID;
private static final String PEER_SERVICE = "peer.service";
@@ -298,14 +302,27 @@ public class OTLPTraceQLApiHandler extends TraceQLApiHandler {
private List<String> namesOf(final String scope, final Duration duration) throws IOException {
switch (scope) {
case SCOPE_RESOURCE:
- return Arrays.asList(SERVICE_NAME, SERVICE_INSTANCE_ID, REMOTE_SERVICE);
+ final Set<String> names = new LinkedHashSet<>(Arrays.asList(SERVICE_NAME, SERVICE_INSTANCE_ID, REMOTE_SERVICE));
+ names.addAll(autocompleteKeys(OTLPSpanRecord.RESOURCE_TAG_PREFIX, duration));
+ return new ArrayList<>(names);
case SCOPE_SPAN:
- return new ArrayList<>(tagAutoCompleteQueryService.queryTagAutocompleteKeys(TagType.OTLP, duration));
+ return autocompleteKeys(OTLPSpanRecord.SPAN_TAG_PREFIX, duration);
default:
return INTRINSIC_TAG_NAMES;
}
}
+ /**
+ * The autocomplete keys of one scope, without the scope prefix OTLPSpanForward publishes them with.
+ */
+ private List<String> autocompleteKeys(final String scopePrefix, final Duration duration) throws IOException {
+ return tagAutoCompleteQueryService.queryTagAutocompleteKeys(TagType.OTLP, duration)
+ .stream()
+ .filter(key -> key.startsWith(scopePrefix))
+ .map(key -> key.substring(scopePrefix.length()))
+ .collect(Collectors.toList());
+ }
+
/**
* Tempo's {@code q} on a tag lookup: the newest {@link #TAG_FILTER_SAMPLE_TRACES} traces matching the filter,
* with the matcher that picks their matching spans.
@@ -340,6 +357,11 @@ public class OTLPTraceQLApiHandler extends TraceQLApiHandler {
final Optional<Long> end) throws IOException {
final Duration duration = buildDuration(start, end, config.getLookback());
final String tag = normalizeTagName(tagName);
+ // Tempo's unscoped `.key` is an attribute of either scope; normalizeTagName picks the span one for a key
+ // that is not a known resource key, so the resource one is looked up too.
+ final List<String> attributeTags = tagName.startsWith(".") && tag.startsWith(OTLPSpanRecord.SPAN_TAG_PREFIX)
+ ? Arrays.asList(tag, OTLPSpanRecord.RESOURCE_TAG_PREFIX + tag.substring(OTLPSpanRecord.SPAN_TAG_PREFIX.length()))
+ : List.of(tag);
final TraceQLParseResult filter = parseFilter(query);
if (filter != null && filter.hasError()) {
return badRequestResponse(filter.getErrorInfo());
@@ -350,7 +372,11 @@ public class OTLPTraceQLApiHandler extends TraceQLApiHandler {
&& !(onlyServiceName(filter.getParams()) && (NAME.equals(tag) || RESOURCE_REMOTE_SERVICE.equals(tag)))) {
try {
final Sample sample = sample(filter.getParams(), duration);
- return successResponse(stringValues(OTLPTraceAssembler.tagValues(sample.traces, sample.matcher, tag), limit));
+ final Set<String> values = new LinkedHashSet<>();
+ for (final String attributeTag : attributeTags) {
+ values.addAll(OTLPTraceAssembler.tagValues(sample.traces, sample.matcher, attributeTag));
+ }
+ return successResponse(stringValues(values, limit));
} catch (IllegalArgumentException e) {
return badRequestResponse(e.getMessage());
}
@@ -362,8 +388,8 @@ public class OTLPTraceQLApiHandler extends TraceQLApiHandler {
return successResponse(stringValues(queryService.getServiceNames(), limit));
case RESOURCE_SERVICE_INSTANCE_ID:
case RESOURCE_INSTANCE:
- return successResponse(stringValues(
- tagAutoCompleteQueryService.queryTagAutocompleteValues(TagType.OTLP, SERVICE_INSTANCE_ID, duration), limit));
+ return successResponse(stringValues(tagAutoCompleteQueryService.queryTagAutocompleteValues(
+ TagType.OTLP, RESOURCE_SERVICE_INSTANCE_ID, duration), limit));
case STATUS:
return successResponse(stringValues(Arrays.asList(ERROR, OK, UNSET), limit));
case KIND:
@@ -374,14 +400,18 @@ public class OTLPTraceQLApiHandler extends TraceQLApiHandler {
case OTEL_SCOPE_NAME:
case SPAN_OTEL_SCOPE_NAME:
return successResponse(stringValues(
- tagAutoCompleteQueryService.queryTagAutocompleteValues(TagType.OTLP, OTEL_SCOPE_NAME, duration), limit));
+ tagAutoCompleteQueryService.queryTagAutocompleteValues(TagType.OTLP, SPAN_OTEL_SCOPE_NAME, duration), limit));
case NAME:
case RESOURCE_REMOTE_SERVICE:
return scopedValues(tag, query, limit);
default:
- if (tag.startsWith(SPAN_PREFIX)) {
- return successResponse(stringValues(tagAutoCompleteQueryService.queryTagAutocompleteValues(
- TagType.OTLP, tag.substring(SPAN_PREFIX.length()), duration), limit));
+ if (tag.startsWith(OTLPSpanRecord.SPAN_TAG_PREFIX) || tag.startsWith(OTLPSpanRecord.RESOURCE_TAG_PREFIX)) {
+ final Set<String> values = new LinkedHashSet<>();
+ for (final String attributeTag : attributeTags) {
+ values.addAll(
+ tagAutoCompleteQueryService.queryTagAutocompleteValues(TagType.OTLP, attributeTag, duration));
+ }
+ return successResponse(stringValues(values, limit));
}
return badRequestResponse("Unsupported tag value query.");
}
diff --git a/oap-server/server-query-plugin/traceql-plugin/src/test/java/org/apache/skywalking/oap/query/traceql/handler/OTLPTraceQLApiHandlerTest.java b/oap-server/server-query-plugin/traceql-plugin/src/test/java/org/apache/skywalking/oap/query/traceql/handler/OTLPTraceQLApiHandlerTest.java
new file mode 100644
index 0000000000..85c1fea8c6
--- /dev/null
+++ b/oap-server/server-query-plugin/traceql-plugin/src/test/java/org/apache/skywalking/oap/query/traceql/handler/OTLPTraceQLApiHandlerTest.java
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.query.traceql.handler;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.linecorp.armeria.common.AggregatedHttpResponse;
+import com.linecorp.armeria.common.HttpStatus;
+import java.util.ArrayList;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.skywalking.oap.query.traceql.TraceQLConfig;
+import org.apache.skywalking.oap.server.core.CoreModule;
+import org.apache.skywalking.oap.server.core.analysis.manual.searchtag.TagType;
+import org.apache.skywalking.oap.server.core.config.NamingControl;
+import org.apache.skywalking.oap.server.core.query.TagAutoCompleteQueryService;
+import org.apache.skywalking.oap.server.library.module.ModuleManager;
+import org.apache.skywalking.oap.server.library.module.ModuleProviderHolder;
+import org.apache.skywalking.oap.server.library.module.ModuleServiceHolder;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * The tag endpoints over the OTLP autocomplete index, whose keys OTLPSpanForward publishes with their scope prefix.
+ */
+class OTLPTraceQLApiHandlerTest {
+ private static final ObjectMapper MAPPER = new ObjectMapper();
+
+ private TagAutoCompleteQueryService tagAutoCompleteQueryService;
+ private OTLPTraceQLApiHandler handler;
+
+ @BeforeEach
+ void setUp() throws Exception {
+ final ModuleManager moduleManager = mock(ModuleManager.class);
+ final ModuleProviderHolder providerHolder = mock(ModuleProviderHolder.class);
+ final ModuleServiceHolder serviceHolder = mock(ModuleServiceHolder.class);
+ tagAutoCompleteQueryService = mock(TagAutoCompleteQueryService.class);
+ when(moduleManager.find(CoreModule.NAME)).thenReturn(providerHolder);
+ when(providerHolder.provider()).thenReturn(serviceHolder);
+ when(serviceHolder.getService(NamingControl.class)).thenReturn(mock(NamingControl.class));
+ when(serviceHolder.getService(TagAutoCompleteQueryService.class)).thenReturn(tagAutoCompleteQueryService);
+ when(tagAutoCompleteQueryService.queryTagAutocompleteKeys(eq(TagType.OTLP), any())).thenReturn(new LinkedHashSet<>(List.of(
+ "resource.service.instance.id",
+ "resource.deployment.environment",
+ "span.otel.scope.name",
+ "span.http.request.method"
+ )));
+ handler = new OTLPTraceQLApiHandler(moduleManager, new TraceQLConfig());
+ }
+
+ @Test
+ void shouldListEachAutocompleteKeyUnderItsOwnScope() throws Exception {
+ final JsonNode body = MAPPER.readTree(handler.searchTagsV2(
+ Optional.empty(), Optional.empty(), Optional.empty(), Optional.empty(), Optional.empty()
+ ).aggregate().join().contentUtf8());
+
+ assertEquals(
+ List.of("service.name", "service.instance.id", "remote.service", "deployment.environment"),
+ tagsOf(body, "resource"));
+ assertEquals(List.of("otel.scope.name", "http.request.method"), tagsOf(body, "span"));
+ }
+
+ @Test
+ void shouldAnswerResourceAttributeValuesFromTheAutocompleteIndex() throws Exception {
+ when(tagAutoCompleteQueryService.queryTagAutocompleteValues(
+ eq(TagType.OTLP), eq("resource.deployment.environment"), any())).thenReturn(Set.of("prod"));
+
+ final AggregatedHttpResponse response = handler.searchTagValues(
+ "resource.deployment.environment", Optional.empty(), Optional.empty(), Optional.empty(), Optional.empty(),
+ Optional.empty()
+ ).aggregate().join();
+
+ assertEquals(HttpStatus.OK, response.status());
+ assertEquals(List.of("prod"), valuesOf(MAPPER.readTree(response.contentUtf8())));
+ }
+
+ @Test
+ void shouldLookUpAnUnscopedKeyInBothScopes() throws Exception {
+ when(tagAutoCompleteQueryService.queryTagAutocompleteValues(
+ eq(TagType.OTLP), eq("span.deployment.environment"), any())).thenReturn(Set.of("canary"));
+ when(tagAutoCompleteQueryService.queryTagAutocompleteValues(
+ eq(TagType.OTLP), eq("resource.deployment.environment"), any())).thenReturn(Set.of("prod"));
+
+ final JsonNode body = MAPPER.readTree(handler.searchTagValues(
+ ".deployment.environment", Optional.empty(), Optional.empty(), Optional.empty(), Optional.empty(),
+ Optional.empty()
+ ).aggregate().join().contentUtf8());
+
+ assertEquals(List.of("canary", "prod"), valuesOf(body));
+ }
+
+ private static List<String> tagsOf(final JsonNode body, final String scope) {
+ final List<String> tags = new ArrayList<>();
+ for (final JsonNode entry : body.get("scopes")) {
+ if (scope.equals(entry.get("name").asText())) {
+ entry.get("tags").forEach(tag -> tags.add(tag.asText()));
+ }
+ }
+ return tags;
+ }
+
+ private static List<String> valuesOf(final JsonNode body) {
+ final List<String> values = new ArrayList<>();
+ body.get("tagValues").forEach(value -> values.add(value.get("value").asText()));
+ return values;
+ }
+}
diff --git a/oap-server/server-receiver-plugin/otel-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/otel/OtelMetricReceiverConfig.java b/oap-server/server-receiver-plugin/otel-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/otel/OtelMetricReceiverConfig.java
index 3cb2b86df5..5b1ba19963 100644
--- a/oap-server/server-receiver-plugin/otel-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/otel/OtelMetricReceiverConfig.java
+++ b/oap-server/server-receiver-plugin/otel-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/otel/OtelMetricReceiverConfig.java
@@ -23,9 +23,11 @@ import com.google.common.base.Strings;
import lombok.Getter;
import lombok.Setter;
import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.oap.server.core.otlp.OTLPSpanRecord;
import org.apache.skywalking.oap.server.library.module.ModuleConfig;
import java.util.List;
+import java.util.stream.Collectors;
/**
* The configuration of every OTLP handler. The name is historical; the trace and log handlers read it too.
@@ -57,8 +59,9 @@ public class OtelMetricReceiverConfig extends ModuleConfig {
private String otlpTraceStorage = OTLP_TRACE_STORAGE_OTLP;
/**
- * Native mode only. Attribute keys offered by tag autocomplete. Every attribute is indexed for equality
- * search regardless of this list.
+ * Native mode only. Attribute keys offered by tag autocomplete, each with its TraceQL scope:
+ * {@code resource.<key>} or {@code span.<key>}. Every attribute is indexed for equality search regardless of
+ * this list.
*/
@Setter
private String otlpTraceSearchableTags;
@@ -85,6 +88,21 @@ public class OtelMetricReceiverConfig extends ModuleConfig {
return Splitter.on(",").trimResults().omitEmptyStrings().splitToList(Strings.nullToEmpty(otlpTraceSearchableTags));
}
+ /**
+ * The {@link #otlpTraceSearchableTags} entries that name no scope; they would match no attribute, so the boot
+ * refuses them instead of silently leaving their dropdown empty.
+ */
+ public List<String> getUnscopedOtlpTraceSearchableTags() {
+ return getOtlpTraceSearchableTags().stream()
+ .filter(key -> !isScoped(key, OTLPSpanRecord.RESOURCE_TAG_PREFIX)
+ && !isScoped(key, OTLPSpanRecord.SPAN_TAG_PREFIX))
+ .collect(Collectors.toList());
+ }
+
+ private static boolean isScoped(final String key, final String scopePrefix) {
+ return key.startsWith(scopePrefix) && key.length() > scopePrefix.length();
+ }
+
public boolean isOtlpTraceStorageNative() {
return OTLP_TRACE_STORAGE_OTLP.equalsIgnoreCase(Strings.nullToEmpty(otlpTraceStorage).trim());
}
diff --git a/oap-server/server-receiver-plugin/otel-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/otel/OtelMetricReceiverProvider.java b/oap-server/server-receiver-plugin/otel-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/otel/OtelMetricReceiverProvider.java
index 0f4fb99f67..8aed0f47a1 100644
--- a/oap-server/server-receiver-plugin/otel-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/otel/OtelMetricReceiverProvider.java
+++ b/oap-server/server-receiver-plugin/otel-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/otel/OtelMetricReceiverProvider.java
@@ -75,6 +75,13 @@ public class OtelMetricReceiverProvider extends ModuleProvider {
"receiver-otel.default.otlpTraceStorage must be `" + OtelMetricReceiverConfig.OTLP_TRACE_STORAGE_ZIPKIN
+ "` or `" + OtelMetricReceiverConfig.OTLP_TRACE_STORAGE_OTLP + "`, got `" + config.getOtlpTraceStorage() + "`");
}
+ // Only OTLPSpanForward, the native path, reads otlpTraceSearchableTags; in zipkin mode the list is dead config
+ // and receiver-zipkin.searchableTracesTags applies, so a stale value there must not fail the boot.
+ if (config.isOtlpTraceStorageNative() && !config.getUnscopedOtlpTraceSearchableTags().isEmpty()) {
+ throw new ModuleStartException(
+ "receiver-otel.default.otlpTraceSearchableTags entries must name their scope, `resource.<key>` or "
+ + "`span.<key>`, got " + config.getUnscopedOtlpTraceSearchableTags());
+ }
metricRequestProcessor = new OpenTelemetryMetricRequestProcessor(
getManager(), config);
registerServiceImplementation(OpenTelemetryMetricRequestProcessor.class, metricRequestProcessor);
diff --git a/oap-server/server-receiver-plugin/otel-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/otel/otlp/OTLPSpanForward.java b/oap-server/server-receiver-plugin/otel-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/otel/otlp/OTLPSpanForward.java
index a1af151fa0..16a3a2bdc8 100644
--- a/oap-server/server-receiver-plugin/otel-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/otel/otlp/OTLPSpanForward.java
+++ b/oap-server/server-receiver-plugin/otel-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/otel/otlp/OTLPSpanForward.java
@@ -63,14 +63,15 @@ public class OTLPSpanForward {
public static final String PEER_SERVICE = OTLPSpanRecord.PEER_SERVICE_ATTRIBUTE;
/**
* The tag autocomplete key the instrumentation scope name is always published under, so TraceQL can list
- * {@code otel.scope.name} values without configuration.
+ * {@code otel.scope.name} values without configuration. Like every autocomplete key it carries its TraceQL
+ * scope, so the TraceQL datasource lists each key under the scope a query must name it with.
*/
- public static final String SCOPE_NAME_TAG = "otel.scope.name";
+ public static final String SCOPE_NAME_TAG = OTLPSpanRecord.SPAN_TAG_PREFIX + "otel.scope.name";
/**
* The tag autocomplete key the resolved instance is always published under, so TraceQL's
* {@code resource.service.instance.id} dropdown fills regardless of {@code otlpTraceSearchableTags}.
*/
- public static final String INSTANCE_TAG = "service.instance.id";
+ public static final String INSTANCE_TAG = OTLPSpanRecord.RESOURCE_TAG_PREFIX + "service.instance.id";
private static final int TRACE_ID_LENGTH = 16;
private final ModuleManager moduleManager;
@@ -222,8 +223,9 @@ public class OTLPSpanForward {
/**
* Append every attribute as {@code <scope>.key=value} to the search index, skipping entries longer than
- * {@link Tag#TAG_LENGTH}, and publish the configured keys to tag autocomplete, which stays scope-less: it only
- * feeds the value dropdowns.
+ * {@link Tag#TAG_LENGTH}, and publish the attributes {@code otlpTraceSearchableTags} names with this scope to
+ * tag autocomplete under the same prefix, so the TraceQL datasource lists each key under the scope a query must
+ * name it with.
*
* @param scopePrefix {@link OTLPSpanRecord#RESOURCE_TAG_PREFIX} or {@link OTLPSpanRecord#SPAN_TAG_PREFIX}
* @return the {@code peer.service} value, or null when the attributes carry none
@@ -246,8 +248,9 @@ public class OTLPSpanForward {
continue;
}
tags.add(tag);
- if (searchableTagKeys.contains(attribute.getKey())) {
- addAutocompleteTag(minuteTimeBucket, attribute.getKey(), value);
+ final String scopedKey = scopePrefix + attribute.getKey();
+ if (searchableTagKeys.contains(scopedKey)) {
+ addAutocompleteTag(minuteTimeBucket, scopedKey, value);
}
}
return peerService;
diff --git a/oap-server/server-receiver-plugin/otel-receiver-plugin/src/test/java/org/apache/skywalking/oap/server/receiver/otel/OtelMetricReceiverProviderTest.java b/oap-server/server-receiver-plugin/otel-receiver-plugin/src/test/java/org/apache/skywalking/oap/server/receiver/otel/OtelMetricReceiverProviderTest.java
new file mode 100644
index 0000000000..4d10a697ee
--- /dev/null
+++ b/oap-server/server-receiver-plugin/otel-receiver-plugin/src/test/java/org/apache/skywalking/oap/server/receiver/otel/OtelMetricReceiverProviderTest.java
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.skywalking.oap.server.receiver.otel;
+
+import java.util.List;
+import org.apache.skywalking.oap.server.library.module.ModuleStartException;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+class OtelMetricReceiverProviderTest {
+ @Test
+ void shouldRefuseSearchableTagsWithoutAScope() {
+ final OtelMetricReceiverConfig config = new OtelMetricReceiverConfig();
+ config.setOtlpTraceSearchableTags("resource.service.namespace,http.request.method,span.");
+ final OtelMetricReceiverProvider provider = new OtelMetricReceiverProvider();
+ provider.newConfigCreator().onInitialized(config);
+
+ final ModuleStartException e = assertThrows(ModuleStartException.class, provider::prepare);
+ assertTrue(e.getMessage().contains("[http.request.method, span.]"), e.getMessage());
+ }
+
+ @Test
+ void shouldAcceptScopedSearchableTags() {
+ final OtelMetricReceiverConfig config = new OtelMetricReceiverConfig();
+ config.setOtlpTraceSearchableTags("resource.service.namespace, span.http.request.method");
+
+ assertEquals(List.of(), config.getUnscopedOtlpTraceSearchableTags());
+ }
+
+ @Test
+ void shouldNotCheckSearchableTagsInZipkinMode() {
+ final OtelMetricReceiverConfig config = new OtelMetricReceiverConfig();
+ config.setOtlpTraceStorage(OtelMetricReceiverConfig.OTLP_TRACE_STORAGE_ZIPKIN);
+ config.setOtlpTraceSearchableTags("http.request.method");
+ final OtelMetricReceiverProvider provider = new OtelMetricReceiverProvider();
+ provider.newConfigCreator().onInitialized(config);
+
+ // Zipkin mode never reads the list, so an unscoped entry there is not a reason to fail the boot.
+ assertDoesNotThrow(provider::prepare);
+ }
+}
diff --git a/oap-server/server-receiver-plugin/otel-receiver-plugin/src/test/java/org/apache/skywalking/oap/server/receiver/otel/otlp/OTLPSpanForwardTest.java b/oap-server/server-receiver-plugin/otel-receiver-plugin/src/test/java/org/apache/skywalking/oap/server/receiver/otel/otlp/OTLPSpanForwardTest.java
index c8b710913a..43a0f7c4ee 100644
--- a/oap-server/server-receiver-plugin/otel-receiver-plugin/src/test/java/org/apache/skywalking/oap/server/receiver/otel/otlp/OTLPSpanForwardTest.java
+++ b/oap-server/server-receiver-plugin/otel-receiver-plugin/src/test/java/org/apache/skywalking/oap/server/receiver/otel/otlp/OTLPSpanForwardTest.java
@@ -28,6 +28,7 @@ import io.opentelemetry.proto.trace.v1.Status;
import java.util.Arrays;
import java.util.List;
import java.util.Map;
+import java.util.Set;
import java.util.stream.Collectors;
import org.apache.skywalking.oap.server.core.CoreModule;
import org.apache.skywalking.oap.server.core.analysis.manual.searchtag.Tag;
@@ -85,7 +86,7 @@ class OTLPSpanForwardTest {
when(namingControl.formatEndpointName(anyString(), anyString())).thenAnswer(invocation -> invocation.getArgument(1));
config = new OtelMetricReceiverConfig();
- config.setOtlpTraceSearchableTags("http.request.method,peer.service");
+ config.setOtlpTraceSearchableTags("span.http.request.method,span.peer.service");
}
@Test
@@ -156,13 +157,59 @@ class OTLPSpanForwardTest {
.collect(Collectors.toMap(
TagAutocomplete::getTagKey, TagAutocomplete::getTagValue));
assertEquals(Map.of(
- "http.request.method", "GET",
- "peer.service", "payment",
+ "span.http.request.method", "GET",
+ "span.peer.service", "payment",
OTLPSpanForward.SCOPE_NAME_TAG, "io.opentelemetry.http",
OTLPSpanForward.INSTANCE_TAG, "checkout-1"
), autocomplete);
}
+ @Test
+ void shouldPublishAutocompleteKeysUnderTheScopeTheyWereSeenIn() {
+ config.setOtlpTraceSearchableTags(
+ "resource.deployment.environment,span.deployment.environment,span.http.request.method");
+ final OTLPSpanForward forward = new OTLPSpanForward(config, moduleManager);
+ final Resource resource = Resource.newBuilder()
+ .addAttributes(string("service.name", "checkout"))
+ .addAttributes(string("deployment.environment", "prod"))
+ .build();
+ final Span span = span().addAttributes(string("http.request.method", "GET"))
+ .addAttributes(string("deployment.environment", "canary"))
+ .build();
+
+ assertTrue(forward.send(resource, "", InstrumentationScope.getDefaultInstance(), "", span, "checkout",
+ Map.of("service.name", "checkout"), Map.of()));
+
+ // Each configured entry names one scope; a key listed for both is published once per scope.
+ final Set<String> autocomplete = received().stream()
+ .filter(TagAutocomplete.class::isInstance)
+ .map(TagAutocomplete.class::cast)
+ .map(tag -> tag.getTagKey() + "=" + tag.getTagValue())
+ .collect(Collectors.toSet());
+ assertEquals(Set.of(
+ "resource.deployment.environment=prod",
+ "span.deployment.environment=canary",
+ "span.http.request.method=GET"
+ ), autocomplete);
+ }
+
+ @Test
+ void shouldPublishOnlyTheScopeAnEntryNames() {
+ config.setOtlpTraceSearchableTags("resource.deployment.environment");
+ final OTLPSpanForward forward = new OTLPSpanForward(config, moduleManager);
+ final Resource resource = Resource.newBuilder().addAttributes(string("deployment.environment", "prod")).build();
+ final Span span = span().addAttributes(string("deployment.environment", "canary")).build();
+
+ assertTrue(forward.send(resource, "", InstrumentationScope.getDefaultInstance(), "", span, "checkout",
+ Map.of(), Map.of()));
+
+ assertEquals(Set.of("resource.deployment.environment=prod"), received().stream()
+ .filter(TagAutocomplete.class::isInstance)
+ .map(TagAutocomplete.class::cast)
+ .map(tag -> tag.getTagKey() + "=" + tag.getTagValue())
+ .collect(Collectors.toSet()));
+ }
+
@Test
void shouldDropSpansOverTheRateLimit() {
config.setOtlpTraceMaxSpansPerSecond(1);
diff --git a/oap-server/server-starter/src/main/resources/application.yml b/oap-server/server-starter/src/main/resources/application.yml
index 52c48c1361..47d57dc907 100644
--- a/oap-server/server-starter/src/main/resources/application.yml
+++ b/oap-server/server-starter/src/main/resources/application.yml
@@ -437,9 +437,12 @@ receiver-otel:
# otlp: store the native OTLP span in `otlp_span`, served by the TraceQL `/otlp` datasource
# zipkin: convert to Zipkin v2 and hand them to receiver-zipkin (needs receiver-zipkin and query-zipkin)
otlpTraceStorage: ${SW_OTEL_TRACE_STORAGE:otlp}
- # `otlp` mode only. Attribute keys offered by tag autocomplete (TraceQL /api/v2/search/tags). Every resource and span
- # attribute is indexed for equality search regardless of this list; same meaning as receiver-zipkin.searchableTracesTags.
- otlpTraceSearchableTags: ${SW_OTEL_TRACE_SEARCHABLE_TAG_KEYS:http.request.method,http.response.status_code,url.path,rpc.method,rpc.grpc.status_code,db.system.name,db.namespace,messaging.system,messaging.destination.name,peer.service}
+ # `otlp` mode only. Attribute keys offered by tag autocomplete (TraceQL /api/v2/search/tags), each written with its
+ # TraceQL scope, `resource.<key>` or `span.<key>`; an entry without a scope fails the boot. List low-cardinality
+ # attributes only: every distinct value is stored once per day. Every resource and span attribute is indexed for
+ # equality search regardless of this list. Not read in `zipkin` mode: the converted spans go through receiver-zipkin,
+ # whose `searchableTracesTags` decides autocomplete there.
+ otlpTraceSearchableTags: ${SW_OTEL_TRACE_SEARCHABLE_TAG_KEYS:resource.deployment.environment.name,resource.deployment.environment,resource.service.namespace,resource.service.version,resource.k8s.namespace.name,resource.k8s.cluster.name,span.http.request.method,span.http.response.status_code,span.url.path,span.rpc.method,span.rpc.grpc.status_code,span.db.system.name,span.db.namespace,span.messaging.system,span.messaging.destination.name,span.peer.service}
# `otlp` mode only. Head sampling by trace id, precision 1/10000.
otlpTraceSampleRate: ${SW_OTEL_TRACE_SAMPLE_RATE:10000}
# `otlp` mode only. Spans per second accepted before dropping. 0 means no limit.
diff --git a/test/e2e-v2/cases/storage/expected/config-dump.yml b/test/e2e-v2/cases/storage/expected/config-dump.yml
index 4d67024ea8..cd44392a91 100644
--- a/test/e2e-v2/cases/storage/expected/config-dump.yml
+++ b/test/e2e-v2/cases/storage/expected/config-dump.yml
@@ -197,7 +197,7 @@
"receiver-otel.default.enabledOtelMetricsRules": "apisix,nginx/*,k8s/*,istio-controlplane,vm,mysql/*,postgresql/*,oap,aws-eks/*,windows,aws-s3/*,aws-dynamodb/*,aws-gateway/*,redis/*,elasticsearch/*,rabbitmq/*,mongodb/*,kafka/*,pulsar/*,bookkeeper/*,rocketmq/*,clickhouse/*,activemq/*,kong/*,flink/*,airflow/*,banyandb/*,envoy-ai-gateway/*,ios/*,miniprogram/*,ai-agent/*",
"receiver-otel.default.otlpTraceMaxSpansPerSecond": "0",
"receiver-otel.default.otlpTraceSampleRate": "10000",
- "receiver-otel.default.otlpTraceSearchableTags": "http.request.method,http.response.status_code,url.path,rpc.method,rpc.grpc.status_code,db.system.name,db.namespace,messaging.system,messaging.destination.name,peer.service",
+ "receiver-otel.default.otlpTraceSearchableTags": "resource.deployment.environment.name,resource.deployment.environment,resource.service.namespace,resource.service.version,resource.k8s.namespace.name,resource.k8s.cluster.name,span.http.request.method,span.http.response.status_code,span.url.path,span.rpc.method,span.rpc.grpc.status_code,span.db.system.name,span.db.namespace,span.messaging.system,span.messaging.destination.name,span.peer.service",
"receiver-otel.default.otlpTraceStorage": "otlp",
"receiver-otel.provider": "default",
"receiver-pprof.default.memoryParserEnabled": "true",