You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by wu...@apache.org on 2026/10/19 00:02:28 UTC
(skywalking) branch genai-evaluation-trace-storage created (now 407d328b7b)
This is an automated email from the ASF dual-hosted git repository.
wu-sheng pushed a change to branch genai-evaluation-trace-storage
in repository https://gitbox.apache.org/repos/asf/skywalking.git
at 407d328b7b Record which store holds the span a GenAI evaluation was judged from
This branch includes the following new commits:
new 407d328b7b Record which store holds the span a GenAI evaluation was judged from
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
(skywalking) 01/01: Record which store holds the span a GenAI evaluation was judged from
Posted by wu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
wu-sheng pushed a commit to branch genai-evaluation-trace-storage
in repository https://gitbox.apache.org/repos/asf/skywalking.git
commit 407d328b7b7d4c5b9e3f54bded0e45692b69749a
Author: Wu Sheng <wu...@foxmail.com>
AuthorDate: Mon Oct 19 08:02:17 2026 +0800
Record which store holds the span a GenAI evaluation was judged from
`traceRef.type` names how the judged span is addressed, and an `OTLP`
reference can point into either the native OTLP store or the Zipkin store.
Each listener path already knows where the span went, so the record now
keeps it as `trace_storage` and the query exposes it as `traceRef.storage`.
---
docs/en/changes/changes.md | 1 +
docs/en/setup/backend/ai-evaluation.md | 10 ++
.../ai/evaluation/context/AIEvaluationContext.java | 2 +
.../listener/AIEvaluationSpanListener.java | 11 +-
.../strategy/span/SpanAIEvaluationStrategy.java | 2 +
.../listener/GenAIEvaluationAnalysisListener.java | 2 +
.../listener/AIEvaluationSpanListenerTest.java | 141 +++++++++++++++++++++
.../src/test/resources/gen-ai-config.yml | 39 ++----
.../manual/genai/GenAIEvaluationRecord.java | 25 +++-
.../GenAITraceStorage.java} | 20 ++-
.../oap/server/core/query/type/GenAITraceRef.java | 2 +
.../manual/genai/GenAIEvaluationRecordTest.java | 71 +++++++++++
.../src/main/resources/query-protocol | 2 +-
.../BanyanDBGenAIEvaluationRecordQueryDAO.java | 2 +
.../query/GenAIEvaluationRecordQueryEsDAO.java | 1 +
.../dao/JDBCGenAIEvaluationRecordQueryDAO.java | 2 +
test/e2e-v2/cases/otlp-virtual-genai/e2e.yaml | 1 +
.../otlp-virtual-genai/evaluation-storage.yaml | 26 ++++
.../{e2e.yaml => expected/evaluation-storage.yml} | 32 +----
.../cases/otlp-virtual-genai/native/e2e.yaml | 1 +
.../native/evaluation-storage.yaml | 26 ++++
.../expected/evaluation-storage.yml} | 32 +----
test/e2e-v2/cases/virtual-genai/e2e.yaml | 1 +
.../cases/virtual-genai/evaluation-storage.yaml | 26 ++++
.../expected/evaluation-storage.yml} | 32 +----
.../cases/zipkin-virtual-genai/docker-compose.yml | 7 +
test/e2e-v2/cases/zipkin-virtual-genai/e2e.yaml | 1 +
.../zipkin-virtual-genai/evaluation-storage.yaml | 26 ++++
.../expected/evaluation-storage.yml} | 32 +----
29 files changed, 410 insertions(+), 166 deletions(-)
diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md
index 8bc2676a28..4f6ca1f1d4 100644
--- a/docs/en/changes/changes.md
+++ b/docs/en/changes/changes.md
@@ -56,6 +56,7 @@
* Fix the Envoy ALS sidecar internal request and response latencies dropping whole seconds: only the `nanos` part of each protobuf `Duration` was read. The internal response latency is now left out of the internal latency metrics when the access log lacks the upstream or downstream timing it is computed from, such as when Envoy answers an upstream timeout with its own 504, instead of counting the whole wait or going negative.
* Sync the Envoy access log proto, `envoy/data/accesslog/v3/accesslog.proto`, with Envoy v1.40.0, adding the `envoy/annotations/deprecation.proto` it imports. With it, the Envoy ALS sidecar internal latencies leave out what the access log cannot measure instead of counting it: the internal request latency of an HTTP request Envoy retried or hedged, or of a TCP connection that took more than one connect attempt (`upstream_request_attempt_count` above 1), as Envoy measures the upstream tim [...]
* Fix OAL `>`, `>=`, `<` and `<=` filters on a `long` or `double` source field, and `==` on a `long` one, failing to compile when they compare it with a number, such as `filter(sideCar.internalRequestLatencyNanos >= 0)`. The generated code passed the number as an `int`, and the comparison matchers have no `match(long, int)` or `match(double, int)` overload, as Javassist does not widen an argument. The number is now written in the field's type, `0L` or `0.0`; filters on `int` fields, whic [...]
+* Record which store holds the span a GenAI evaluation was produced from, as the new `storage` field of the evaluation record's `traceRef`: `SKYWALKING_NATIVE`, `OTLP` for OTLP spans stored natively, or `ZIPKIN` for spans received by `receiver-zipkin` and OTLP spans converted with `otlpTraceStorage: zipkin`. `traceRef.type` names how the span is addressed, and an `OTLP` reference can point into either the OTLP or the Zipkin store, so a client had no way to tell which one to read the trac [...]
#### UI
* Add a Virtual GenAI evaluation-record page and evaluation-score chart in Horizon UI, so operators can inspect evaluation result, level, reason, judge model, timestamp, trace linkage, and the `gen_ai_model_evaluation_score_ppm` trend for evaluated records.
diff --git a/docs/en/setup/backend/ai-evaluation.md b/docs/en/setup/backend/ai-evaluation.md
index 13b8a956c9..52ce39fae9 100644
--- a/docs/en/setup/backend/ai-evaluation.md
+++ b/docs/en/setup/backend/ai-evaluation.md
@@ -98,6 +98,16 @@ Evaluation records are available through the `queryGenAIEvaluationRecord` GraphQ
The metric value is PPM. Divide it by `1000000` when displaying the original score.
+The trace reference names both how the judged span is addressed and where it is stored, so a client reads the trace from the right place. `type` is the addressing: `SKYWALKING_NATIVE` by segment ID and span index, `OTLP` by span ID. `storage` is the store holding the span:
+
+| `storage` | Spans | Read through |
+|---|---|---|
+| `SKYWALKING_NATIVE` | Received from SkyWalking agents | The SkyWalking trace query |
+| `OTLP` | Received over OTLP with `receiver-otel` `otlpTraceStorage: otlp`, the default | The TraceQL OTLP datasource |
+| `ZIPKIN` | Received by `receiver-zipkin`, or over OTLP with `otlpTraceStorage: zipkin` | The Zipkin query API |
+
+Both OTLP and Zipkin spans are addressed by span ID, and one OAP can hold both at once, since `receiver-zipkin` runs independently of `otlpTraceStorage`. The store is decided when a span is received, so changing `otlpTraceStorage` later does not move the spans of earlier records. `storage` is empty only on an OTLP-addressed record written before OAP recorded the store.
+
## Operational considerations
Evaluation runs asynchronously after a GenAI span is sampled. A slow or unavailable judge affects evaluation throughput, not trace ingestion, but it can fill the local queue and increase dropped-evaluation counters. Monitor the `ai_evaluation_dropped_count` and `ai_evaluation_error_count` telemetry metrics when tuning the queue or judge settings.
diff --git a/oap-server/analyzer/ai-evaluation/src/main/java/org/apache/skywalking/oap/server/ai/evaluation/context/AIEvaluationContext.java b/oap-server/analyzer/ai-evaluation/src/main/java/org/apache/skywalking/oap/server/ai/evaluation/context/AIEvaluationContext.java
index cc4b75e6a8..939bc4a390 100644
--- a/oap-server/analyzer/ai-evaluation/src/main/java/org/apache/skywalking/oap/server/ai/evaluation/context/AIEvaluationContext.java
+++ b/oap-server/analyzer/ai-evaluation/src/main/java/org/apache/skywalking/oap/server/ai/evaluation/context/AIEvaluationContext.java
@@ -21,6 +21,7 @@ package org.apache.skywalking.oap.server.ai.evaluation.context;
import lombok.Builder;
import lombok.Getter;
import org.apache.skywalking.oap.server.core.query.enumeration.GenAITraceRefType;
+import org.apache.skywalking.oap.server.core.query.enumeration.GenAITraceStorage;
import java.util.Collections;
import java.util.Map;
@@ -30,6 +31,7 @@ import java.util.Map;
public class AIEvaluationContext {
private final String traceId;
private final GenAITraceRefType traceRefType;
+ private final GenAITraceStorage traceStorage;
private final String spanId;
private final Integer spanIndex;
private final String segmentId;
diff --git a/oap-server/analyzer/ai-evaluation/src/main/java/org/apache/skywalking/oap/server/ai/evaluation/listener/AIEvaluationSpanListener.java b/oap-server/analyzer/ai-evaluation/src/main/java/org/apache/skywalking/oap/server/ai/evaluation/listener/AIEvaluationSpanListener.java
index 1bc9afc883..6732fda64a 100644
--- a/oap-server/analyzer/ai-evaluation/src/main/java/org/apache/skywalking/oap/server/ai/evaluation/listener/AIEvaluationSpanListener.java
+++ b/oap-server/analyzer/ai-evaluation/src/main/java/org/apache/skywalking/oap/server/ai/evaluation/listener/AIEvaluationSpanListener.java
@@ -28,6 +28,7 @@ import org.apache.skywalking.oap.server.core.trace.OTLPSpanReader;
import org.apache.skywalking.oap.server.core.trace.SpanListener;
import org.apache.skywalking.oap.server.core.trace.SpanListenerResult;
import org.apache.skywalking.oap.server.core.query.enumeration.GenAITraceRefType;
+import org.apache.skywalking.oap.server.core.query.enumeration.GenAITraceStorage;
import org.apache.skywalking.oap.server.core.zipkin.source.ZipkinSpan;
import org.apache.skywalking.oap.server.library.module.ModuleManager;
import org.apache.skywalking.oap.server.library.util.StringUtil;
@@ -77,7 +78,8 @@ public class AIEvaluationSpanListener implements SpanListener {
if (!shouldSample(span.getTraceId())) {
return SpanListenerResult.CONTINUE;
}
- sample(span.getTraceId(),
+ sample(GenAITraceStorage.ZIPKIN,
+ span.getTraceId(),
span.getSpanId(),
span.getLocalEndpointServiceName(),
span.getName(),
@@ -116,7 +118,8 @@ public class AIEvaluationSpanListener implements SpanListener {
if (!shouldSample(span.traceId())) {
return SpanListenerResult.CONTINUE;
}
- sample(span.traceId(),
+ sample(GenAITraceStorage.OTLP,
+ span.traceId(),
span.spanId(),
resolveServiceName(resourceAttributes),
span.spanName(),
@@ -141,7 +144,8 @@ public class AIEvaluationSpanListener implements SpanListener {
return "";
}
- private void sample(final String traceId,
+ private void sample(final GenAITraceStorage traceStorage,
+ final String traceId,
final String spanId,
final String serviceName,
final String operationName,
@@ -153,6 +157,7 @@ public class AIEvaluationSpanListener implements SpanListener {
evaluationService.sample(AIEvaluationContext.builder()
.traceId(traceId)
.traceRefType(GenAITraceRefType.OTLP)
+ .traceStorage(traceStorage)
.spanId(spanId)
.serviceName(serviceName)
.operationName(operationName)
diff --git a/oap-server/analyzer/ai-evaluation/src/main/java/org/apache/skywalking/oap/server/ai/evaluation/service/strategy/span/SpanAIEvaluationStrategy.java b/oap-server/analyzer/ai-evaluation/src/main/java/org/apache/skywalking/oap/server/ai/evaluation/service/strategy/span/SpanAIEvaluationStrategy.java
index 1dbf417417..3d8f2c2bf3 100644
--- a/oap-server/analyzer/ai-evaluation/src/main/java/org/apache/skywalking/oap/server/ai/evaluation/service/strategy/span/SpanAIEvaluationStrategy.java
+++ b/oap-server/analyzer/ai-evaluation/src/main/java/org/apache/skywalking/oap/server/ai/evaluation/service/strategy/span/SpanAIEvaluationStrategy.java
@@ -146,6 +146,7 @@ public class SpanAIEvaluationStrategy implements AIEvaluationStrategy {
record.setServiceName(namingControl.formatServiceName(context.getServiceName()));
record.setOperationName(operationName(context));
record.setRefType(context.getTraceRefType().name());
+ record.setTraceStorage(context.getTraceStorage().name());
record.setSegmentId(context.getSegmentId());
record.setSpanIndex(context.getSpanIndex());
record.setSpanId(context.getSpanId());
@@ -172,6 +173,7 @@ public class SpanAIEvaluationStrategy implements AIEvaluationStrategy {
private static boolean completeLLMCallSpan(final AIEvaluationContext context) {
if (StringUtil.isBlank(context.getTraceId())
|| context.getTraceRefType() == null
+ || context.getTraceStorage() == null
|| StringUtil.isBlank(context.getServiceName())
|| StringUtil.isBlank(context.getProviderName())
|| StringUtil.isBlank(context.getModelName())) {
diff --git a/oap-server/analyzer/ai-evaluation/src/main/java/org/apache/skywalking/oap/server/analyzer/provider/trace/parser/listener/GenAIEvaluationAnalysisListener.java b/oap-server/analyzer/ai-evaluation/src/main/java/org/apache/skywalking/oap/server/analyzer/provider/trace/parser/listener/GenAIEvaluationAnalysisListener.java
index face16850c..17fe4342ea 100644
--- a/oap-server/analyzer/ai-evaluation/src/main/java/org/apache/skywalking/oap/server/analyzer/provider/trace/parser/listener/GenAIEvaluationAnalysisListener.java
+++ b/oap-server/analyzer/ai-evaluation/src/main/java/org/apache/skywalking/oap/server/analyzer/provider/trace/parser/listener/GenAIEvaluationAnalysisListener.java
@@ -30,6 +30,7 @@ import org.apache.skywalking.oap.server.ai.evaluation.context.AIEvaluationContex
import org.apache.skywalking.oap.server.ai.evaluation.service.IAIEvaluationService;
import org.apache.skywalking.oap.server.analyzer.provider.AnalyzerModuleConfig;
import org.apache.skywalking.oap.server.core.query.enumeration.GenAITraceRefType;
+import org.apache.skywalking.oap.server.core.query.enumeration.GenAITraceStorage;
import org.apache.skywalking.oap.server.library.module.ModuleManager;
import org.apache.skywalking.oap.server.library.util.StringUtil;
import org.apache.skywalking.oap.server.library.util.genai.GenAIContextResolver;
@@ -85,6 +86,7 @@ public class GenAIEvaluationAnalysisListener implements EntryAnalysisListener, E
evaluationService.sample(AIEvaluationContext.builder()
.traceId(segmentObject.getTraceId())
.traceRefType(GenAITraceRefType.SKYWALKING_NATIVE)
+ .traceStorage(GenAITraceStorage.SKYWALKING_NATIVE)
.segmentId(segmentObject.getTraceSegmentId())
.spanIndex(span.getSpanId())
.serviceName(segmentObject.getService())
diff --git a/oap-server/analyzer/ai-evaluation/src/test/java/org/apache/skywalking/oap/server/ai/evaluation/listener/AIEvaluationSpanListenerTest.java b/oap-server/analyzer/ai-evaluation/src/test/java/org/apache/skywalking/oap/server/ai/evaluation/listener/AIEvaluationSpanListenerTest.java
new file mode 100644
index 0000000000..d247c6bbe5
--- /dev/null
+++ b/oap-server/analyzer/ai-evaluation/src/test/java/org/apache/skywalking/oap/server/ai/evaluation/listener/AIEvaluationSpanListenerTest.java
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.ai.evaluation.listener;
+
+import com.google.gson.JsonObject;
+import java.util.Collections;
+import java.util.Map;
+import org.apache.skywalking.oap.server.ai.evaluation.AIEvaluationModule;
+import org.apache.skywalking.oap.server.ai.evaluation.context.AIEvaluationContext;
+import org.apache.skywalking.oap.server.ai.evaluation.service.IAIEvaluationService;
+import org.apache.skywalking.oap.server.core.query.enumeration.GenAITraceRefType;
+import org.apache.skywalking.oap.server.core.query.enumeration.GenAITraceStorage;
+import org.apache.skywalking.oap.server.core.trace.OTLPSpanReader;
+import org.apache.skywalking.oap.server.core.zipkin.source.ZipkinSpan;
+import org.apache.skywalking.oap.server.library.module.ModuleManager;
+import org.apache.skywalking.oap.server.library.module.ModuleProviderHolder;
+import org.apache.skywalking.oap.server.library.module.ModuleServiceHolder;
+import org.apache.skywalking.oap.server.library.util.genai.GenAISemanticAttributes;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.mockito.ArgumentCaptor;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+class AIEvaluationSpanListenerTest {
+ private static final String TRACE_ID = "4bf92f3577b34da6a3ce929d0e0e4736";
+ private static final String SPAN_ID = "00f067aa0ba902b7";
+ private static final String MODEL = "gpt-4.1-mini";
+
+ private IAIEvaluationService evaluationService;
+ private AIEvaluationSpanListener listener;
+
+ @BeforeEach
+ void setUp() {
+ evaluationService = mock(IAIEvaluationService.class);
+ when(evaluationService.shouldSample(anyString())).thenReturn(true);
+ final ModuleServiceHolder provider = mock(ModuleServiceHolder.class);
+ when(provider.getService(IAIEvaluationService.class)).thenReturn(evaluationService);
+ final ModuleProviderHolder module = mock(ModuleProviderHolder.class);
+ when(module.provider()).thenReturn(provider);
+ final ModuleManager moduleManager = mock(ModuleManager.class);
+ when(moduleManager.find(AIEvaluationModule.NAME)).thenReturn(module);
+ listener = new AIEvaluationSpanListener();
+ listener.init(moduleManager);
+ }
+
+ @Test
+ void recordsAZipkinSpanAsStoredInZipkin() {
+ final JsonObject tags = new JsonObject();
+ tags.addProperty(GenAISemanticAttributes.RESPONSE_MODEL, MODEL);
+ final ZipkinSpan span = new ZipkinSpan();
+ span.setTraceId(TRACE_ID);
+ span.setSpanId(SPAN_ID);
+ span.setTags(tags);
+
+ listener.onZipkinSpan(span);
+
+ final AIEvaluationContext context = sampledContext();
+ assertEquals(GenAITraceRefType.OTLP, context.getTraceRefType());
+ assertEquals(GenAITraceStorage.ZIPKIN, context.getTraceStorage());
+ assertEquals(SPAN_ID, context.getSpanId());
+ }
+
+ @Test
+ void recordsANativeOtlpSpanAsStoredInOtlp() {
+ final OTLPSpanReader span = new OTLPSpanReader() {
+ @Override
+ public String traceId() {
+ return TRACE_ID;
+ }
+
+ @Override
+ public String spanId() {
+ return SPAN_ID;
+ }
+
+ @Override
+ public String spanName() {
+ return "chat " + MODEL;
+ }
+
+ @Override
+ public String spanKind() {
+ return "SPAN_KIND_CLIENT";
+ }
+
+ @Override
+ public long startTimeNanos() {
+ return 0;
+ }
+
+ @Override
+ public long endTimeNanos() {
+ return 0;
+ }
+
+ @Override
+ public String getAttribute(final String key) {
+ return attributes().get(key);
+ }
+
+ @Override
+ public Map<String, String> attributes() {
+ return Collections.singletonMap(GenAISemanticAttributes.RESPONSE_MODEL, MODEL);
+ }
+ };
+
+ listener.onNativeOTLPSpan(span, Collections.emptyMap(), "opentelemetry.instrumentation.openai_v2", "2.0");
+
+ final AIEvaluationContext context = sampledContext();
+ assertEquals(GenAITraceRefType.OTLP, context.getTraceRefType());
+ assertEquals(GenAITraceStorage.OTLP, context.getTraceStorage());
+ assertEquals(SPAN_ID, context.getSpanId());
+ }
+
+ private AIEvaluationContext sampledContext() {
+ final ArgumentCaptor<AIEvaluationContext> captor = ArgumentCaptor.forClass(AIEvaluationContext.class);
+ verify(evaluationService).sample(captor.capture());
+ return captor.getValue();
+ }
+}
diff --git a/test/e2e-v2/cases/otlp-virtual-genai/e2e.yaml b/oap-server/analyzer/ai-evaluation/src/test/resources/gen-ai-config.yml
similarity index 52%
copy from test/e2e-v2/cases/otlp-virtual-genai/e2e.yaml
copy to oap-server/analyzer/ai-evaluation/src/test/resources/gen-ai-config.yml
index 0a14830ffb..420a8c65f4 100644
--- a/test/e2e-v2/cases/otlp-virtual-genai/e2e.yaml
+++ b/oap-server/analyzer/ai-evaluation/src/test/resources/gen-ai-config.yml
@@ -13,33 +13,12 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-# This file is used to show how to write configuration files and can be used to test.
-
-setup:
- env: compose
- file: docker-compose.yml
- timeout: 20m
- init-system-environment: ../../script/env
- steps:
- - name: set PATH
- command: export PATH=/tmp/skywalking-infra-e2e/bin:$PATH
- - name: install yq
- command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh yq
- - name: install swctl
- command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh swctl
-
-trigger:
- action: http
- interval: 3s
- times: 5
- url: http://localhost:9091/
- method: GET
-
-verify:
- retry:
- count: 60
- interval: 3s
- cases:
- - includes:
- - ./otlp-virtual-genai.yaml
-
+# The span listeners resolve the provider of a GenAI span from these rules.
+providers:
+ - provider: openai
+ prefix-match:
+ - gpt
+ models:
+ - name: gpt-4.1-mini
+ input-estimated-cost-per-m: 0.4
+ output-estimated-cost-per-m: 1.6
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/genai/GenAIEvaluationRecord.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/genai/GenAIEvaluationRecord.java
index 50b5b514f2..23ff2b9202 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/genai/GenAIEvaluationRecord.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/genai/GenAIEvaluationRecord.java
@@ -33,6 +33,7 @@ import org.apache.skywalking.oap.server.core.analysis.record.Record;
import org.apache.skywalking.oap.server.core.analysis.worker.RecordStreamProcessor;
import org.apache.skywalking.oap.server.core.query.enumeration.GenAIEvaluationValueType;
import org.apache.skywalking.oap.server.core.query.enumeration.GenAITraceRefType;
+import org.apache.skywalking.oap.server.core.query.enumeration.GenAITraceStorage;
import org.apache.skywalking.oap.server.core.query.type.GenAITraceRef;
import org.apache.skywalking.oap.server.core.source.DefaultScopeDefine;
import org.apache.skywalking.oap.server.core.source.ScopeDeclaration;
@@ -65,6 +66,7 @@ public class GenAIEvaluationRecord extends Record {
public static final String EVAL_NUMBER_VALUE = "eval_number_value";
public static final long SCORE_SCALE = 1_000_000L;
public static final String REF_TYPE = "ref_type";
+ public static final String TRACE_STORAGE = "trace_storage";
public static final String SEGMENT_ID = "segment_id";
public static final String SPAN_INDEX = "span_index";
public static final String SPAN_ID = "span_id";
@@ -114,6 +116,10 @@ public class GenAIEvaluationRecord extends Record {
@Column(name = REF_TYPE, length = 32)
private String refType;
+ @Getter(AccessLevel.NONE)
+ @Column(name = TRACE_STORAGE, length = 32, storageOnly = true)
+ private String traceStorage;
+
@Column(name = SEGMENT_ID, length = 150)
private String segmentId;
@@ -184,7 +190,22 @@ public class GenAIEvaluationRecord extends Record {
} catch (IllegalArgumentException | NullPointerException e) {
type = segmentId == null ? GenAITraceRefType.OTLP : GenAITraceRefType.SKYWALKING_NATIVE;
}
- return new GenAITraceRef(type, traceId, segmentId, spanIndex, spanId);
+ return new GenAITraceRef(type, resolveTraceStorage(type), traceId, segmentId, spanIndex, spanId);
+ }
+
+ /**
+ * A native reference has only one store. An OTLP reference written before the store was recorded may be in
+ * either OTLP or Zipkin storage, so it is left unknown rather than guessed.
+ */
+ private GenAITraceStorage resolveTraceStorage(final GenAITraceRefType type) {
+ if (traceStorage != null) {
+ try {
+ return GenAITraceStorage.valueOf(traceStorage);
+ } catch (IllegalArgumentException e) {
+ return null;
+ }
+ }
+ return type == GenAITraceRefType.SKYWALKING_NATIVE ? GenAITraceStorage.SKYWALKING_NATIVE : null;
}
public GenAIEvaluationValueType getValueType() {
@@ -289,6 +310,7 @@ public class GenAIEvaluationRecord extends Record {
final Number numberValue = (Number) converter.get(EVAL_NUMBER_VALUE);
record.setEvalNumberValue(numberValue == null ? null : numberValue.longValue());
record.setRefType((String) converter.get(REF_TYPE));
+ record.setTraceStorage((String) converter.get(TRACE_STORAGE));
record.setSegmentId((String) converter.get(SEGMENT_ID));
final Number spanIndex = (Number) converter.get(SPAN_INDEX);
record.setSpanIndex(spanIndex == null ? null : spanIndex.intValue());
@@ -314,6 +336,7 @@ public class GenAIEvaluationRecord extends Record {
converter.accept(OPERATION_NAME, storageData.getOperationName());
converter.accept(EVAL_NUMBER_VALUE, storageData.getEvalNumberValue());
converter.accept(REF_TYPE, storageData.refType);
+ converter.accept(TRACE_STORAGE, storageData.traceStorage);
converter.accept(SEGMENT_ID, storageData.getSegmentId());
converter.accept(SPAN_INDEX, storageData.getSpanIndex());
converter.accept(SPAN_ID, storageData.getSpanId());
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/type/GenAITraceRef.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/enumeration/GenAITraceStorage.java
similarity index 68%
copy from oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/type/GenAITraceRef.java
copy to oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/enumeration/GenAITraceStorage.java
index a84773f5d1..5736e80c30 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/type/GenAITraceRef.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/enumeration/GenAITraceStorage.java
@@ -16,16 +16,14 @@
*
*/
-package org.apache.skywalking.oap.server.core.query.type;
+package org.apache.skywalking.oap.server.core.query.enumeration;
-import lombok.Value;
-import org.apache.skywalking.oap.server.core.query.enumeration.GenAITraceRefType;
-
-@Value
-public class GenAITraceRef {
- GenAITraceRefType type;
- String traceId;
- String segmentId;
- Integer spanIndex;
- String spanId;
+/**
+ * The store holding the span a GenAI evaluation was produced from. {@link GenAITraceRefType#OTLP} addresses spans
+ * in both {@link #OTLP} and {@link #ZIPKIN}, so the reference type alone does not say where to read the trace.
+ */
+public enum GenAITraceStorage {
+ SKYWALKING_NATIVE,
+ OTLP,
+ ZIPKIN
}
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/type/GenAITraceRef.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/type/GenAITraceRef.java
index a84773f5d1..82746d5158 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/type/GenAITraceRef.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/type/GenAITraceRef.java
@@ -20,10 +20,12 @@ package org.apache.skywalking.oap.server.core.query.type;
import lombok.Value;
import org.apache.skywalking.oap.server.core.query.enumeration.GenAITraceRefType;
+import org.apache.skywalking.oap.server.core.query.enumeration.GenAITraceStorage;
@Value
public class GenAITraceRef {
GenAITraceRefType type;
+ GenAITraceStorage storage;
String traceId;
String segmentId;
Integer spanIndex;
diff --git a/oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/analysis/manual/genai/GenAIEvaluationRecordTest.java b/oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/analysis/manual/genai/GenAIEvaluationRecordTest.java
new file mode 100644
index 0000000000..7f165c3d50
--- /dev/null
+++ b/oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/analysis/manual/genai/GenAIEvaluationRecordTest.java
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.core.analysis.manual.genai;
+
+import org.apache.skywalking.oap.server.core.query.enumeration.GenAITraceRefType;
+import org.apache.skywalking.oap.server.core.query.enumeration.GenAITraceStorage;
+import org.apache.skywalking.oap.server.core.query.type.GenAITraceRef;
+import org.apache.skywalking.oap.server.core.storage.type.HashMapConverter;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNull;
+
+class GenAIEvaluationRecordTest {
+
+ @Test
+ void keepsTheTraceStorageThroughStorage() {
+ final GenAIEvaluationRecord record = GenAIEvaluationRecord.create("openai", "gpt-4.1-mini");
+ record.setUniqueId("unique");
+ record.setTraceId("trace");
+ record.setRefType(GenAITraceRefType.OTLP.name());
+ record.setTraceStorage(GenAITraceStorage.ZIPKIN.name());
+ record.setSpanId("0123456789abcdef");
+ record.setEvaluationTime(1_790_000_000_000L);
+ record.setTimeBucket(20261019000000L);
+
+ final GenAIEvaluationRecord.Builder builder = new GenAIEvaluationRecord.Builder();
+ final HashMapConverter.ToStorage toStorage = new HashMapConverter.ToStorage();
+ builder.entity2Storage(record, toStorage);
+ final GenAITraceRef ref = builder.storage2Entity(new HashMapConverter.ToEntity(toStorage.obtain()))
+ .getTraceRef();
+
+ assertEquals(GenAITraceRefType.OTLP, ref.getType());
+ assertEquals(GenAITraceStorage.ZIPKIN, ref.getStorage());
+ assertEquals("0123456789abcdef", ref.getSpanId());
+ }
+
+ @Test
+ void resolvesTheStorageOfANativeRecordWrittenWithoutOne() {
+ final GenAIEvaluationRecord record = new GenAIEvaluationRecord();
+ record.setRefType(GenAITraceRefType.SKYWALKING_NATIVE.name());
+ record.setSegmentId("segment");
+
+ assertEquals(GenAITraceStorage.SKYWALKING_NATIVE, record.getTraceRef().getStorage());
+ }
+
+ @Test
+ void leavesTheStorageOfAnOtlpRecordWrittenWithoutOneUnknown() {
+ final GenAIEvaluationRecord record = new GenAIEvaluationRecord();
+ record.setRefType(GenAITraceRefType.OTLP.name());
+ record.setSpanId("0123456789abcdef");
+
+ assertNull(record.getTraceRef().getStorage());
+ }
+}
diff --git a/oap-server/server-query-plugin/query-graphql-plugin/src/main/resources/query-protocol b/oap-server/server-query-plugin/query-graphql-plugin/src/main/resources/query-protocol
index 689ddfea3e..62f655d8a0 160000
--- a/oap-server/server-query-plugin/query-graphql-plugin/src/main/resources/query-protocol
+++ b/oap-server/server-query-plugin/query-graphql-plugin/src/main/resources/query-protocol
@@ -1 +1 @@
-Subproject commit 689ddfea3e7ccf32400a5be34a0538a013d23eb9
+Subproject commit 62f655d8a04785fa87e6878b89e3464d4396d967
diff --git a/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/stream/BanyanDBGenAIEvaluationRecordQueryDAO.java b/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/stream/BanyanDBGenAIEvaluationRecordQueryDAO.java
index 15d2e01062..550a457582 100644
--- a/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/stream/BanyanDBGenAIEvaluationRecordQueryDAO.java
+++ b/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/stream/BanyanDBGenAIEvaluationRecordQueryDAO.java
@@ -51,6 +51,7 @@ public class BanyanDBGenAIEvaluationRecordQueryDAO extends AbstractBanyanDBDAO i
GenAIEvaluationRecord.OPERATION_NAME,
GenAIEvaluationRecord.EVAL_NUMBER_VALUE,
GenAIEvaluationRecord.REF_TYPE,
+ GenAIEvaluationRecord.TRACE_STORAGE,
GenAIEvaluationRecord.SEGMENT_ID,
GenAIEvaluationRecord.SPAN_INDEX,
GenAIEvaluationRecord.SPAN_ID,
@@ -152,6 +153,7 @@ public class BanyanDBGenAIEvaluationRecordQueryDAO extends AbstractBanyanDBDAO i
final Number numberValue = rowEntity.getTagValue(GenAIEvaluationRecord.EVAL_NUMBER_VALUE);
evaluationRecord.setEvalNumberValue(numberValue == null ? null : numberValue.longValue());
evaluationRecord.setRefType(rowEntity.getTagValue(GenAIEvaluationRecord.REF_TYPE));
+ evaluationRecord.setTraceStorage(rowEntity.getTagValue(GenAIEvaluationRecord.TRACE_STORAGE));
evaluationRecord.setSegmentId(rowEntity.getTagValue(GenAIEvaluationRecord.SEGMENT_ID));
final Number spanIndex = rowEntity.getTagValue(GenAIEvaluationRecord.SPAN_INDEX);
evaluationRecord.setSpanIndex(spanIndex == null ? null : spanIndex.intValue());
diff --git a/oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/GenAIEvaluationRecordQueryEsDAO.java b/oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/GenAIEvaluationRecordQueryEsDAO.java
index 34b54b1b4a..5aaa2bbf6a 100644
--- a/oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/GenAIEvaluationRecordQueryEsDAO.java
+++ b/oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/GenAIEvaluationRecordQueryEsDAO.java
@@ -157,6 +157,7 @@ public class GenAIEvaluationRecordQueryEsDAO extends EsDAO implements IGenAIEval
final Number numberValue = (Number) source.get(GenAIEvaluationRecord.EVAL_NUMBER_VALUE);
record.setEvalNumberValue(numberValue == null ? null : numberValue.longValue());
record.setRefType((String) source.get(GenAIEvaluationRecord.REF_TYPE));
+ record.setTraceStorage((String) source.get(GenAIEvaluationRecord.TRACE_STORAGE));
record.setSegmentId((String) source.get(GenAIEvaluationRecord.SEGMENT_ID));
final Number spanIndex = (Number) source.get(GenAIEvaluationRecord.SPAN_INDEX);
record.setSpanIndex(spanIndex == null ? null : spanIndex.intValue());
diff --git a/oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/common/dao/JDBCGenAIEvaluationRecordQueryDAO.java b/oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/common/dao/JDBCGenAIEvaluationRecordQueryDAO.java
index 641214b233..76c5b4d9d8 100644
--- a/oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/common/dao/JDBCGenAIEvaluationRecordQueryDAO.java
+++ b/oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/common/dao/JDBCGenAIEvaluationRecordQueryDAO.java
@@ -59,6 +59,7 @@ public class JDBCGenAIEvaluationRecordQueryDAO implements IGenAIEvaluationRecord
GenAIEvaluationRecord.OPERATION_NAME,
GenAIEvaluationRecord.EVAL_NUMBER_VALUE,
GenAIEvaluationRecord.REF_TYPE,
+ GenAIEvaluationRecord.TRACE_STORAGE,
GenAIEvaluationRecord.SEGMENT_ID,
GenAIEvaluationRecord.SPAN_INDEX,
GenAIEvaluationRecord.SPAN_ID,
@@ -143,6 +144,7 @@ public class JDBCGenAIEvaluationRecordQueryDAO implements IGenAIEvaluationRecord
final long numberValue = resultSet.getLong(GenAIEvaluationRecord.EVAL_NUMBER_VALUE);
record.setEvalNumberValue(resultSet.wasNull() ? null : numberValue);
record.setRefType(resultSet.getString(GenAIEvaluationRecord.REF_TYPE));
+ record.setTraceStorage(resultSet.getString(GenAIEvaluationRecord.TRACE_STORAGE));
record.setSegmentId(resultSet.getString(GenAIEvaluationRecord.SEGMENT_ID));
final int spanIndex = resultSet.getInt(GenAIEvaluationRecord.SPAN_INDEX);
record.setSpanIndex(resultSet.wasNull() ? null : spanIndex);
diff --git a/test/e2e-v2/cases/otlp-virtual-genai/e2e.yaml b/test/e2e-v2/cases/otlp-virtual-genai/e2e.yaml
index 0a14830ffb..d3964b0b59 100644
--- a/test/e2e-v2/cases/otlp-virtual-genai/e2e.yaml
+++ b/test/e2e-v2/cases/otlp-virtual-genai/e2e.yaml
@@ -42,4 +42,5 @@ verify:
cases:
- includes:
- ./otlp-virtual-genai.yaml
+ - ./evaluation-storage.yaml
diff --git a/test/e2e-v2/cases/otlp-virtual-genai/evaluation-storage.yaml b/test/e2e-v2/cases/otlp-virtual-genai/evaluation-storage.yaml
new file mode 100644
index 0000000000..0721049e12
--- /dev/null
+++ b/test/e2e-v2/cases/otlp-virtual-genai/evaluation-storage.yaml
@@ -0,0 +1,26 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# The store holding the judged span, which depends on how this case's OAP receives spans:
+# OTLP converted to Zipkin with otlpTraceStorage: zipkin.
+cases:
+ - query: |
+ START=$(date -u -v-1H '+%Y-%m-%d %H%M' 2>/dev/null || date -u -d '1 hour ago' '+%Y-%m-%d %H%M')
+ END=$(date -u '+%Y-%m-%d %H%M')
+ curl -s -X POST http://${oap_host}:${oap_12800}/graphql \
+ -H 'Content-Type: application/json' \
+ -d '{"query":"query ($condition: GenAIEvaluationRecordQueryCondition!) { queryGenAIEvaluationRecord(condition: $condition) { genAIEvaluationRecordList { traceRef { type storage } } } }","variables":{"condition":{"providerId":"b3BlbmFp.0","queryDuration":{"start":"'"$START"'","end":"'"$END"'","step":"MINUTE"},"paging":{"pageNum":1,"pageSize":1}}}}' \
+ | yq -P '.data.queryGenAIEvaluationRecord.genAIEvaluationRecordList[0].traceRef'
+ expected: expected/evaluation-storage.yml
diff --git a/test/e2e-v2/cases/otlp-virtual-genai/e2e.yaml b/test/e2e-v2/cases/otlp-virtual-genai/expected/evaluation-storage.yml
similarity index 52%
copy from test/e2e-v2/cases/otlp-virtual-genai/e2e.yaml
copy to test/e2e-v2/cases/otlp-virtual-genai/expected/evaluation-storage.yml
index 0a14830ffb..6422672272 100644
--- a/test/e2e-v2/cases/otlp-virtual-genai/e2e.yaml
+++ b/test/e2e-v2/cases/otlp-virtual-genai/expected/evaluation-storage.yml
@@ -13,33 +13,5 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-# This file is used to show how to write configuration files and can be used to test.
-
-setup:
- env: compose
- file: docker-compose.yml
- timeout: 20m
- init-system-environment: ../../script/env
- steps:
- - name: set PATH
- command: export PATH=/tmp/skywalking-infra-e2e/bin:$PATH
- - name: install yq
- command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh yq
- - name: install swctl
- command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh swctl
-
-trigger:
- action: http
- interval: 3s
- times: 5
- url: http://localhost:9091/
- method: GET
-
-verify:
- retry:
- count: 60
- interval: 3s
- cases:
- - includes:
- - ./otlp-virtual-genai.yaml
-
+type: OTLP
+storage: ZIPKIN
diff --git a/test/e2e-v2/cases/otlp-virtual-genai/native/e2e.yaml b/test/e2e-v2/cases/otlp-virtual-genai/native/e2e.yaml
index 155957ffaa..ec50dfef04 100644
--- a/test/e2e-v2/cases/otlp-virtual-genai/native/e2e.yaml
+++ b/test/e2e-v2/cases/otlp-virtual-genai/native/e2e.yaml
@@ -43,3 +43,4 @@ verify:
cases:
- includes:
- ../otlp-virtual-genai.yaml
+ - ./evaluation-storage.yaml
diff --git a/test/e2e-v2/cases/otlp-virtual-genai/native/evaluation-storage.yaml b/test/e2e-v2/cases/otlp-virtual-genai/native/evaluation-storage.yaml
new file mode 100644
index 0000000000..700001f078
--- /dev/null
+++ b/test/e2e-v2/cases/otlp-virtual-genai/native/evaluation-storage.yaml
@@ -0,0 +1,26 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# The store holding the judged span, which depends on how this case's OAP receives spans:
+# OTLP stored natively with otlpTraceStorage: otlp.
+cases:
+ - query: |
+ START=$(date -u -v-1H '+%Y-%m-%d %H%M' 2>/dev/null || date -u -d '1 hour ago' '+%Y-%m-%d %H%M')
+ END=$(date -u '+%Y-%m-%d %H%M')
+ curl -s -X POST http://${oap_host}:${oap_12800}/graphql \
+ -H 'Content-Type: application/json' \
+ -d '{"query":"query ($condition: GenAIEvaluationRecordQueryCondition!) { queryGenAIEvaluationRecord(condition: $condition) { genAIEvaluationRecordList { traceRef { type storage } } } }","variables":{"condition":{"providerId":"b3BlbmFp.0","queryDuration":{"start":"'"$START"'","end":"'"$END"'","step":"MINUTE"},"paging":{"pageNum":1,"pageSize":1}}}}' \
+ | yq -P '.data.queryGenAIEvaluationRecord.genAIEvaluationRecordList[0].traceRef'
+ expected: expected/evaluation-storage.yml
diff --git a/test/e2e-v2/cases/otlp-virtual-genai/e2e.yaml b/test/e2e-v2/cases/otlp-virtual-genai/native/expected/evaluation-storage.yml
similarity index 52%
copy from test/e2e-v2/cases/otlp-virtual-genai/e2e.yaml
copy to test/e2e-v2/cases/otlp-virtual-genai/native/expected/evaluation-storage.yml
index 0a14830ffb..063475191e 100644
--- a/test/e2e-v2/cases/otlp-virtual-genai/e2e.yaml
+++ b/test/e2e-v2/cases/otlp-virtual-genai/native/expected/evaluation-storage.yml
@@ -13,33 +13,5 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-# This file is used to show how to write configuration files and can be used to test.
-
-setup:
- env: compose
- file: docker-compose.yml
- timeout: 20m
- init-system-environment: ../../script/env
- steps:
- - name: set PATH
- command: export PATH=/tmp/skywalking-infra-e2e/bin:$PATH
- - name: install yq
- command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh yq
- - name: install swctl
- command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh swctl
-
-trigger:
- action: http
- interval: 3s
- times: 5
- url: http://localhost:9091/
- method: GET
-
-verify:
- retry:
- count: 60
- interval: 3s
- cases:
- - includes:
- - ./otlp-virtual-genai.yaml
-
+type: OTLP
+storage: OTLP
diff --git a/test/e2e-v2/cases/virtual-genai/e2e.yaml b/test/e2e-v2/cases/virtual-genai/e2e.yaml
index 3416deb91c..ce6959b2af 100644
--- a/test/e2e-v2/cases/virtual-genai/e2e.yaml
+++ b/test/e2e-v2/cases/virtual-genai/e2e.yaml
@@ -42,4 +42,5 @@ verify:
cases:
- includes:
- ./virtual-genai.yaml
+ - ./evaluation-storage.yaml
diff --git a/test/e2e-v2/cases/virtual-genai/evaluation-storage.yaml b/test/e2e-v2/cases/virtual-genai/evaluation-storage.yaml
new file mode 100644
index 0000000000..a33cc75ce5
--- /dev/null
+++ b/test/e2e-v2/cases/virtual-genai/evaluation-storage.yaml
@@ -0,0 +1,26 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# The store holding the judged span, which depends on how this case's OAP receives spans:
+# SkyWalking agent segments.
+cases:
+ - query: |
+ START=$(date -u -v-1H '+%Y-%m-%d %H%M' 2>/dev/null || date -u -d '1 hour ago' '+%Y-%m-%d %H%M')
+ END=$(date -u '+%Y-%m-%d %H%M')
+ curl -s -X POST http://${oap_host}:${oap_12800}/graphql \
+ -H 'Content-Type: application/json' \
+ -d '{"query":"query ($condition: GenAIEvaluationRecordQueryCondition!) { queryGenAIEvaluationRecord(condition: $condition) { genAIEvaluationRecordList { traceRef { type storage } } } }","variables":{"condition":{"providerId":"b3BlbmFp.0","queryDuration":{"start":"'"$START"'","end":"'"$END"'","step":"MINUTE"},"paging":{"pageNum":1,"pageSize":1}}}}' \
+ | yq -P '.data.queryGenAIEvaluationRecord.genAIEvaluationRecordList[0].traceRef'
+ expected: expected/evaluation-storage.yml
diff --git a/test/e2e-v2/cases/otlp-virtual-genai/e2e.yaml b/test/e2e-v2/cases/virtual-genai/expected/evaluation-storage.yml
similarity index 52%
copy from test/e2e-v2/cases/otlp-virtual-genai/e2e.yaml
copy to test/e2e-v2/cases/virtual-genai/expected/evaluation-storage.yml
index 0a14830ffb..6987b11b0b 100644
--- a/test/e2e-v2/cases/otlp-virtual-genai/e2e.yaml
+++ b/test/e2e-v2/cases/virtual-genai/expected/evaluation-storage.yml
@@ -13,33 +13,5 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-# This file is used to show how to write configuration files and can be used to test.
-
-setup:
- env: compose
- file: docker-compose.yml
- timeout: 20m
- init-system-environment: ../../script/env
- steps:
- - name: set PATH
- command: export PATH=/tmp/skywalking-infra-e2e/bin:$PATH
- - name: install yq
- command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh yq
- - name: install swctl
- command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh swctl
-
-trigger:
- action: http
- interval: 3s
- times: 5
- url: http://localhost:9091/
- method: GET
-
-verify:
- retry:
- count: 60
- interval: 3s
- cases:
- - includes:
- - ./otlp-virtual-genai.yaml
-
+type: SKYWALKING_NATIVE
+storage: SKYWALKING_NATIVE
diff --git a/test/e2e-v2/cases/zipkin-virtual-genai/docker-compose.yml b/test/e2e-v2/cases/zipkin-virtual-genai/docker-compose.yml
index 87d47057e2..b1a3f9acb1 100644
--- a/test/e2e-v2/cases/zipkin-virtual-genai/docker-compose.yml
+++ b/test/e2e-v2/cases/zipkin-virtual-genai/docker-compose.yml
@@ -21,6 +21,13 @@ services:
environment:
SW_QUERY_ZIPKIN: default
SW_RECEIVER_ZIPKIN: default
+ SW_AI_EVALUATION: default
+ SW_AI_EVALUATION_SAMPLE_RATE: 1000000
+ AI_EVALUATION_ENDPOINT: http://mock-llm-server:9090/llm/evaluation/v1/chat/completions
+ AI_EVALUATION_MODEL: e2e-judge
+ AI_EVALUATION_API_KEY: e2e-api-key
+ volumes:
+ - ../virtual-genai/ai-evaluation.yml:/skywalking/config/ai-evaluation.yml
ports:
- "12800:12800"
- "11800:11800"
diff --git a/test/e2e-v2/cases/zipkin-virtual-genai/e2e.yaml b/test/e2e-v2/cases/zipkin-virtual-genai/e2e.yaml
index 0147b6d000..7e9be223fb 100644
--- a/test/e2e-v2/cases/zipkin-virtual-genai/e2e.yaml
+++ b/test/e2e-v2/cases/zipkin-virtual-genai/e2e.yaml
@@ -42,3 +42,4 @@ verify:
cases:
- includes:
- ./zipkin-virtual-genai.yaml
+ - ./evaluation-storage.yaml
diff --git a/test/e2e-v2/cases/zipkin-virtual-genai/evaluation-storage.yaml b/test/e2e-v2/cases/zipkin-virtual-genai/evaluation-storage.yaml
new file mode 100644
index 0000000000..6a77431009
--- /dev/null
+++ b/test/e2e-v2/cases/zipkin-virtual-genai/evaluation-storage.yaml
@@ -0,0 +1,26 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# The store holding the judged span, which depends on how this case's OAP receives spans:
+# Zipkin spans received by receiver-zipkin.
+cases:
+ - query: |
+ START=$(date -u -v-1H '+%Y-%m-%d %H%M' 2>/dev/null || date -u -d '1 hour ago' '+%Y-%m-%d %H%M')
+ END=$(date -u '+%Y-%m-%d %H%M')
+ curl -s -X POST http://${oap_host}:${oap_12800}/graphql \
+ -H 'Content-Type: application/json' \
+ -d '{"query":"query ($condition: GenAIEvaluationRecordQueryCondition!) { queryGenAIEvaluationRecord(condition: $condition) { genAIEvaluationRecordList { traceRef { type storage } } } }","variables":{"condition":{"providerId":"b3BlbmFp.0","queryDuration":{"start":"'"$START"'","end":"'"$END"'","step":"MINUTE"},"paging":{"pageNum":1,"pageSize":1}}}}' \
+ | yq -P '.data.queryGenAIEvaluationRecord.genAIEvaluationRecordList[0].traceRef'
+ expected: expected/evaluation-storage.yml
diff --git a/test/e2e-v2/cases/otlp-virtual-genai/e2e.yaml b/test/e2e-v2/cases/zipkin-virtual-genai/expected/evaluation-storage.yml
similarity index 52%
copy from test/e2e-v2/cases/otlp-virtual-genai/e2e.yaml
copy to test/e2e-v2/cases/zipkin-virtual-genai/expected/evaluation-storage.yml
index 0a14830ffb..6422672272 100644
--- a/test/e2e-v2/cases/otlp-virtual-genai/e2e.yaml
+++ b/test/e2e-v2/cases/zipkin-virtual-genai/expected/evaluation-storage.yml
@@ -13,33 +13,5 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-# This file is used to show how to write configuration files and can be used to test.
-
-setup:
- env: compose
- file: docker-compose.yml
- timeout: 20m
- init-system-environment: ../../script/env
- steps:
- - name: set PATH
- command: export PATH=/tmp/skywalking-infra-e2e/bin:$PATH
- - name: install yq
- command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh yq
- - name: install swctl
- command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh swctl
-
-trigger:
- action: http
- interval: 3s
- times: 5
- url: http://localhost:9091/
- method: GET
-
-verify:
- retry:
- count: 60
- interval: 3s
- cases:
- - includes:
- - ./otlp-virtual-genai.yaml
-
+type: OTLP
+storage: ZIPKIN