You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by wu...@apache.org on 2026/10/16 10:01:58 UTC
(skywalking) branch master updated: Fix eBPF access-log timestamps truncating nanosecond offsets to whole seconds. (#14145)
This is an automated email from the ASF dual-hosted git repository.
wu-sheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/skywalking.git
The following commit(s) were added to refs/heads/master by this push:
new 36803d76f0 Fix eBPF access-log timestamps truncating nanosecond offsets to whole seconds. (#14145)
36803d76f0 is described below
commit 36803d76f09d42d311c412f62183ef4ef1bd3108
Author: songzhendong <xi...@qq.com>
AuthorDate: Fri Oct 16 18:01:53 2026 +0800
Fix eBPF access-log timestamps truncating nanosecond offsets to whole seconds. (#14145)
Mesh HTTP latency was quantized to 0 or 1000ms and minute buckets lost sub-second precision; convert boot Instant plus offset nanos like Envoy ALS.
---
docs/en/changes/changes.md | 1 +
.../provider/handler/AccessLogServiceHandler.java | 9 +-
.../AccessLogServiceHandlerMeshMetricsTest.java | 119 +++++++++++++++++++++
.../AccessLogServiceHandlerNodeInfoTest.java | 118 ++++++++++++++++++++
4 files changed, 243 insertions(+), 4 deletions(-)
diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md
index 4c056ee923..323e561fab 100644
--- a/docs/en/changes/changes.md
+++ b/docs/en/changes/changes.md
@@ -49,6 +49,7 @@
* Fix Elasticsearch reads of a metric a peer OAP reinstalls without a schema change, such as a bundled MAL rule restored after the runtime rule overriding it is deleted. The peer kept the index it had registered for the runtime rule. This only matters with `logicSharding` enabled and when the two rules use different functions, since the function decides the index there; with the default merged indices every metric stays in `metrics-all`.
* Build the dated Elasticsearch index names in one place, `IndexNames`, with the day step decided once per physical table from its models, replacing `TimeSeriesUtils` and `TimeRangeIndexNameGenerator`. Apart from the fixes above, index names are unchanged.
* Fix a service instance dropping out of the instance list queried with a time range when a record carrying an older minute updates it, such as the segment of a request that started minutes earlier arriving after the instance's heartbeat. `InstanceTraffic` took the `last_ping` of the last record written instead of keeping the latest one, as `EndpointTraffic` now does.
+* Fix eBPF access-log timestamps losing all sub-second precision. `NodeInfo.parseTimestamp` truncated the nanosecond-since-boot offset to whole seconds and also dropped `bootTime.nanos` before converting to millis, so short mesh HTTP latency (`end - start`) was quantized to 0 or 1000ms depending on whether the request crossed a whole-second boundary, and minute buckets could land in the wrong minute near minute boundaries. The conversion is now `Instant.ofEpochSecond(bootSeconds, bootNan [...]
#### UI
* Add a Virtual GenAI evaluation-record page and evaluation-score chart in Horizon UI, so operators can inspect evaluation result, level, reason, judge model, timestamp, trace linkage, and the `gen_ai_model_evaluation_score_ppm` trend for evaluated records.
diff --git a/oap-server/server-receiver-plugin/skywalking-ebpf-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/ebpf/provider/handler/AccessLogServiceHandler.java b/oap-server/server-receiver-plugin/skywalking-ebpf-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/ebpf/provider/handler/AccessLogServiceHandler.java
index 1b231622b1..53c9859d2f 100644
--- a/oap-server/server-receiver-plugin/skywalking-ebpf-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/ebpf/provider/handler/AccessLogServiceHandler.java
+++ b/oap-server/server-receiver-plugin/skywalking-ebpf-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/ebpf/provider/handler/AccessLogServiceHandler.java
@@ -521,13 +521,14 @@ public class AccessLogServiceHandler extends EBPFAccessLogServiceGrpc.EBPFAccess
}
public long parseMinuteTimeBucket(EBPFTimestamp timestamp) {
- final long seconds = bootTime.getSeconds() + TimeUnit.NANOSECONDS.toSeconds(timestamp.getOffset().getOffset());
- return TimeBucket.getMinuteTimeBucket(seconds * 1000);
+ return TimeBucket.getMinuteTimeBucket(parseTimestamp(timestamp));
}
+ // bootTime + offset(ns since boot) → epoch millis; do not truncate offset to whole seconds
public long parseTimestamp(EBPFTimestamp timestamp) {
- return TimeUnit.SECONDS.toMillis(bootTime.getSeconds() +
- TimeUnit.NANOSECONDS.toSeconds(timestamp.getOffset().getOffset()));
+ return java.time.Instant.ofEpochSecond(bootTime.getSeconds(), bootTime.getNanos())
+ .plusNanos(timestamp.getOffset().getOffset())
+ .toEpochMilli();
}
public boolean shouldExcludeNamespace(String namespace) {
diff --git a/oap-server/server-receiver-plugin/skywalking-ebpf-receiver-plugin/src/test/java/org/apache/skywalking/oap/server/receiver/ebpf/provider/handler/AccessLogServiceHandlerMeshMetricsTest.java b/oap-server/server-receiver-plugin/skywalking-ebpf-receiver-plugin/src/test/java/org/apache/skywalking/oap/server/receiver/ebpf/provider/handler/AccessLogServiceHandlerMeshMetricsTest.java
new file mode 100644
index 0000000000..a4fb18a09e
--- /dev/null
+++ b/oap-server/server-receiver-plugin/skywalking-ebpf-receiver-plugin/src/test/java/org/apache/skywalking/oap/server/receiver/ebpf/provider/handler/AccessLogServiceHandlerMeshMetricsTest.java
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.receiver.ebpf.provider.handler;
+
+import org.apache.skywalking.apm.network.common.v3.DetectPoint;
+import org.apache.skywalking.apm.network.common.v3.Instant;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.AccessLogConnection;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.AccessLogConnectionTLSMode;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.AccessLogHTTPProtocol;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.AccessLogHTTPProtocolRequest;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.AccessLogHTTPProtocolRequestMethod;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.AccessLogHTTPProtocolResponse;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.AccessLogHTTPProtocolVersion;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.AccessLogProtocolType;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.ConnectionAddress;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.EBPFAccessLogNodeInfo;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.EBPFOffsetTimestamp;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.EBPFTimestamp;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.KubernetesProcessAddress;
+import org.apache.skywalking.apm.network.servicemesh.v3.HTTPServiceMeshMetric;
+import org.apache.skywalking.oap.server.core.config.NamingControl;
+import org.apache.skywalking.oap.server.core.config.group.EndpointNameGrouping;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.lang.reflect.Field;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.mockito.Mockito.CALLS_REAL_METHODS;
+import static org.mockito.Mockito.mock;
+
+public class AccessLogServiceHandlerMeshMetricsTest {
+
+ private AccessLogServiceHandler handler;
+ private NamingControl namingControl;
+
+ @BeforeEach
+ public void setUp() throws Exception {
+ // Avoid the production constructor (scheduled drop-reason printer + ModuleManager).
+ handler = mock(AccessLogServiceHandler.class, CALLS_REAL_METHODS);
+ namingControl = new NamingControl(512, 512, 512, new EndpointNameGrouping());
+ Field namingControlField = AccessLogServiceHandler.class.getDeclaredField("namingControl");
+ namingControlField.setAccessible(true);
+ namingControlField.set(handler, namingControl);
+ }
+
+ @Test
+ public void generateHTTPServiceMeshMetricsKeepsSubSecondLatencyAcrossSecondBoundary() {
+ // boot 1000s + 500ms; start 1.9s / end 2.023s after boot → 123ms wall latency.
+ // Old second-truncation path would have reported 1000ms (offsets floor to 1s and 2s).
+ final AccessLogServiceHandler.NodeInfo node = new AccessLogServiceHandler.NodeInfo(
+ EBPFAccessLogNodeInfo.newBuilder()
+ .setName("node-1")
+ .setBootTime(Instant.newBuilder().setSeconds(1000L).setNanos(500_000_000).build())
+ .build());
+ final AccessLogServiceHandler.ConnectionInfo connection = handler.new ConnectionInfo(
+ namingControl, node, AccessLogConnection.newBuilder()
+ .setRole(DetectPoint.client)
+ .setTlsMode(AccessLogConnectionTLSMode.Plain)
+ .setProtocol(AccessLogProtocolType.HTTP_1)
+ .setLocal(k8sAddress("productpage.default", "productpage-0"))
+ .setRemote(k8sAddress("details.default", "details-0"))
+ .build());
+
+ final AccessLogHTTPProtocol http = AccessLogHTTPProtocol.newBuilder()
+ .setStartTime(offset(1_900_000_000L))
+ .setEndTime(offset(2_023_000_000L))
+ .setVersion(AccessLogHTTPProtocolVersion.HTTP1)
+ .setRequest(AccessLogHTTPProtocolRequest.newBuilder()
+ .setMethod(AccessLogHTTPProtocolRequestMethod.Get)
+ .setPath("/details")
+ .build())
+ .setResponse(AccessLogHTTPProtocolResponse.newBuilder().setStatusCode(200).build())
+ .build();
+
+ final HTTPServiceMeshMetric metric = handler.generateHTTPServiceMeshMetrics(
+ node, connection, http, AccessLogConnectionTLSMode.Plain.name());
+
+ assertEquals(1_002_400L, metric.getStartTime());
+ assertEquals(1_002_523L, metric.getEndTime());
+ assertEquals(123, metric.getLatency());
+ assertEquals("productpage.default", metric.getSourceServiceName());
+ assertEquals("details.default", metric.getDestServiceName());
+ assertEquals(200, metric.getResponseCode());
+ }
+
+ private static ConnectionAddress k8sAddress(String service, String pod) {
+ return ConnectionAddress.newBuilder()
+ .setKubernetes(KubernetesProcessAddress.newBuilder()
+ .setServiceName(service)
+ .setPodName(pod)
+ .setContainerName("app")
+ .setProcessName("app")
+ .build())
+ .build();
+ }
+
+ private static EBPFTimestamp offset(long nanosSinceBoot) {
+ return EBPFTimestamp.newBuilder()
+ .setOffset(EBPFOffsetTimestamp.newBuilder().setOffset(nanosSinceBoot).build())
+ .build();
+ }
+}
diff --git a/oap-server/server-receiver-plugin/skywalking-ebpf-receiver-plugin/src/test/java/org/apache/skywalking/oap/server/receiver/ebpf/provider/handler/AccessLogServiceHandlerNodeInfoTest.java b/oap-server/server-receiver-plugin/skywalking-ebpf-receiver-plugin/src/test/java/org/apache/skywalking/oap/server/receiver/ebpf/provider/handler/AccessLogServiceHandlerNodeInfoTest.java
new file mode 100644
index 0000000000..2c582cee2b
--- /dev/null
+++ b/oap-server/server-receiver-plugin/skywalking-ebpf-receiver-plugin/src/test/java/org/apache/skywalking/oap/server/receiver/ebpf/provider/handler/AccessLogServiceHandlerNodeInfoTest.java
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.receiver.ebpf.provider.handler;
+
+import org.apache.skywalking.apm.network.common.v3.Instant;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.EBPFAccessLogNodeInfo;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.EBPFOffsetTimestamp;
+import org.apache.skywalking.apm.network.ebpf.accesslog.v3.EBPFTimestamp;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.junit.jupiter.api.Test;
+
+import java.util.concurrent.TimeUnit;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotEquals;
+
+public class AccessLogServiceHandlerNodeInfoTest {
+
+ @Test
+ public void parseTimestampKeepsSubSecondOffset() {
+ // boot at epoch 1000s; offset 1.5s → 1001500 ms (old code truncated to 1001000)
+ final AccessLogServiceHandler.NodeInfo node = nodeInfo(1000L, 0);
+ final long millis = node.parseTimestamp(offset(1_500_000_000L));
+ assertEquals(1_001_500L, millis);
+ }
+
+ @Test
+ public void parseTimestampIncludesBootTimeNanos() {
+ // boot at 1000s + 500ms; offset 200ms → 1000700 ms
+ final AccessLogServiceHandler.NodeInfo node = nodeInfo(1000L, 500_000_000);
+ final long millis = node.parseTimestamp(offset(200_000_000L));
+ assertEquals(1_000_700L, millis);
+ }
+
+ @Test
+ public void parseTimestampCarriesBootNanosAcrossSecond() {
+ // boot 1000s + 999ms; offset 2ms → 1001001 ms
+ final AccessLogServiceHandler.NodeInfo node = nodeInfo(1000L, 999_000_000);
+ assertEquals(1_001_001L, node.parseTimestamp(offset(2_000_000L)));
+ }
+
+ @Test
+ public void parseTimestampWholeSecondOffsetMatchesLegacyWhenBootNanosZero() {
+ // regression: exact whole-second offsets with zero boot nanos must match the old formula
+ final long bootSeconds = 1_700_000_000L;
+ final long offsetNanos = 42L * 1_000_000_000L;
+ final AccessLogServiceHandler.NodeInfo node = nodeInfo(bootSeconds, 0);
+ final long legacy = TimeUnit.SECONDS.toMillis(bootSeconds + TimeUnit.NANOSECONDS.toSeconds(offsetNanos));
+ assertEquals(legacy, node.parseTimestamp(offset(offsetNanos)));
+ }
+
+ @Test
+ public void parseTimestampFloorsSubMillisecondRemainder() {
+ // toEpochMilli drops sub-ms nanos, same as Envoy ALS formatAsLong
+ final AccessLogServiceHandler.NodeInfo node = nodeInfo(1000L, 0);
+ assertEquals(1_000_000L, node.parseTimestamp(offset(999_999L)));
+ assertEquals(1_000_001L, node.parseTimestamp(offset(1_000_000L)));
+ }
+
+ @Test
+ public void parseTimestampMeshLatencyUsesMillisDifference() {
+ final AccessLogServiceHandler.NodeInfo node = nodeInfo(1_700_000_000L, 0);
+ final long start = node.parseTimestamp(offset(10_000_000_000L));
+ final long end = node.parseTimestamp(offset(10_123_000_000L));
+ assertEquals(123L, end - start);
+ }
+
+ @Test
+ public void parseMinuteTimeBucketUsesSameEpochMillis() {
+ final AccessLogServiceHandler.NodeInfo node = nodeInfo(1000L, 0);
+ final EBPFTimestamp ts = offset(1_500_000_000L);
+ assertEquals(TimeBucket.getMinuteTimeBucket(1_001_500L), node.parseMinuteTimeBucket(ts));
+ }
+
+ @Test
+ public void parseMinuteTimeBucketDiffersFromSecondTruncationNearBoundary() {
+ // Align boot to a minute boundary so fractional seconds can cross into the next minute.
+ final long bootSeconds = 1_699_999_980L; // 1700000000 - 20; % 60 == 0
+ final AccessLogServiceHandler.NodeInfo node = nodeInfo(bootSeconds, 300_000_000);
+ // correct: 0.3s + 59.8s = 60.1s past boot → next minute
+ // legacy: floor(59.8s) = 59s, ignores boot nanos → still previous minute
+ final long offsetNanos = 59_800_000_000L;
+ final long correctBucket = node.parseMinuteTimeBucket(offset(offsetNanos));
+ final long legacySeconds = bootSeconds + TimeUnit.NANOSECONDS.toSeconds(offsetNanos);
+ final long legacyBucket = TimeBucket.getMinuteTimeBucket(legacySeconds * 1000);
+ assertNotEquals(legacyBucket, correctBucket);
+ assertEquals(TimeBucket.getMinuteTimeBucket(node.parseTimestamp(offset(offsetNanos))), correctBucket);
+ }
+
+ private static AccessLogServiceHandler.NodeInfo nodeInfo(long seconds, int nanos) {
+ return new AccessLogServiceHandler.NodeInfo(EBPFAccessLogNodeInfo.newBuilder()
+ .setName("node-1")
+ .setBootTime(Instant.newBuilder().setSeconds(seconds).setNanos(nanos).build())
+ .build());
+ }
+
+ private static EBPFTimestamp offset(long nanosSinceBoot) {
+ return EBPFTimestamp.newBuilder()
+ .setOffset(EBPFOffsetTimestamp.newBuilder().setOffset(nanosSinceBoot).build())
+ .build();
+ }
+}