You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by wu...@apache.org on 2026/09/09 01:31:20 UTC
(skywalking) branch master updated: Fix JDBC reads missing the second day of a range shorter than a day that crosses midnight (#14061)
This is an automated email from the ASF dual-hosted git repository.
wu-sheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/skywalking.git
The following commit(s) were added to refs/heads/master by this push:
new 98b9ed6baa Fix JDBC reads missing the second day of a range shorter than a day that crosses midnight (#14061)
98b9ed6baa is described below
commit 98b9ed6baab5d2f2e390f8046e95761f3f8218f2
Author: 吴晟 Wu Sheng <wu...@foxmail.com>
AuthorDate: Wed Sep 9 09:31:14 2026 +0800
Fix JDBC reads missing the second day of a range shorter than a day that crosses midnight (#14061)
TableHelper walked a time range in 24-hour steps from its first instant
to pick the day tables to read, so a range that began late on one day
and ended early on the next never reached the second day. Every
time-ranged read on MySQL, PostgreSQL and H2, metrics, traces, logs,
alarms, records and the AI agent conversation list, was missing the
newer day's rows for the length of the range after each midnight; the
last thirty minutes read at 00:10 saw only yesterday's table, and on a
fresh install with no yesterday's table, nothing at all.
The range is now walked by calendar day from the start of its first day,
in both the model-based and the raw-table variants.
---
docs/en/changes/changes.md | 1 +
.../storage/plugin/jdbc/common/TableHelper.java | 51 ++++++++------
.../plugin/jdbc/common/TableHelperTest.java | 78 ++++++++++++++++++++++
3 files changed, 110 insertions(+), 20 deletions(-)
diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md
index 6cb72c8d5c..7bc3dc51c4 100644
--- a/docs/en/changes/changes.md
+++ b/docs/en/changes/changes.md
@@ -14,6 +14,7 @@
* Add AI agent conversations landed by the AI Sessionizer: the `AI_AGENT` layer, the bundled `lal/ai-agent.yaml` rule with the `ConversationFile` output builder that verifies and stores Session Data and Session Flow files, the `ai_agent_session_data` and `ai_agent_session_flow` models in a new BanyanDB group `recordsAIAgent`, the `ai-agent-conversation` module that folds a conversation into one `asz.view` document, and the `listConversations` (with optional `conversation` and `title` con [...]
* Support querying Zipkin traces from the BanyanDB cold stage through the Zipkin HTTP query API. `/api/v2/traces` takes an optional `coldStage` parameter, and `/api/v2/trace/{traceId}` and `/api/v2/traceMany` take optional `coldStage`, `endTs` and `lookback` parameters; the two by-id lookups had no time range at all before. All of them are SkyWalking additions to the Zipkin API and default to the previous behavior, so existing Zipkin clients such as the Lens UI keep working unchanged, an [...]
* BanyanDB: a query without a time range now covers everything the group's hot/warm stages retain instead of only the last 24 hours. The hard-coded day was a fallback added with the trace model, which requires a time range; it made `queryTrace(traceId)` without a `duration`, the Zipkin `/api/v2/trace/{traceId}` and `/api/v2/traceMany` lookups, and the TraceQL by-id lookup return "not found" for any trace older than a day even though it was still retained. The unbounded range is now bound [...]
+* Fix the JDBC storages, MySQL, PostgreSQL and H2, reading only the first day's table of a time range shorter than a day that crosses midnight. `TableHelper` walked the range in 24-hour steps from its first instant, so a range that began late on one day and ended early on the next never reached the second day; every time-ranged read, metrics, traces, logs, alarms, records and the AI agent conversation list, was missing the newer day's rows for the length of the range after each midnight, [...]
#### UI
* Add a Virtual GenAI evaluation-record page and evaluation-score chart in Horizon UI, so operators can inspect evaluation result, level, reason, judge model, timestamp, trace linkage, and the `gen_ai_model_evaluation_score_ppm` trend for evaluated records.
diff --git a/oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/common/TableHelper.java b/oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/common/TableHelper.java
index e770ef6d34..84409c408d 100644
--- a/oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/common/TableHelper.java
+++ b/oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/common/TableHelper.java
@@ -41,6 +41,7 @@ import org.apache.skywalking.oap.server.storage.plugin.jdbc.TableMetaInfo;
import java.time.Duration;
import java.util.ArrayList;
+import java.util.Calendar;
import java.util.Collections;
import java.util.List;
import java.util.concurrent.TimeUnit;
@@ -151,20 +152,36 @@ public class TableHelper {
return Collections.singletonList(rawTableName);
}
- final var timestampStart = TimeBucket.getTimestamp(timeBucketStart);
- final var timestampEnd = TimeBucket.getTimestamp(timeBucketEnd);
- final var timeBuckets = LongStream.builder();
- for (var timestamp = timestampStart; timestamp <= timestampEnd; timestamp += TimeUnit.DAYS.toMillis(1)) {
- timeBuckets.add(TimeBucket.getTimeBucket(timestamp, DownSampling.Day));
- }
-
- return timeBuckets
- .build()
- .distinct()
- .mapToObj(timeBucket -> getTable(rawTableName, timeBucket))
+ return dayTimeBucketsBetween(TimeBucket.getTimestamp(timeBucketStart), TimeBucket.getTimestamp(timeBucketEnd))
+ .stream()
+ .map(timeBucket -> getTable(rawTableName, timeBucket))
.collect(toList());
}
+ /**
+ * The day buckets a time range touches, first day to last. The range is walked by calendar days from the start
+ * of its first day, not by 24-hour steps from its first instant: a range shorter than a day that crosses
+ * midnight, the last thirty minutes read a few minutes after it, would otherwise never reach its second day.
+ *
+ * @param timestampStart the range's first instant, in milliseconds
+ * @param timestampEnd the range's last instant, in milliseconds
+ * @return the day buckets in order, empty when the range ends before it starts
+ */
+ static List<Long> dayTimeBucketsBetween(final long timestampStart, final long timestampEnd) {
+ final Calendar day = Calendar.getInstance();
+ day.setTimeInMillis(timestampStart);
+ day.set(Calendar.HOUR_OF_DAY, 0);
+ day.set(Calendar.MINUTE, 0);
+ day.set(Calendar.SECOND, 0);
+ day.set(Calendar.MILLISECOND, 0);
+ final List<Long> buckets = new ArrayList<>();
+ while (day.getTimeInMillis() <= timestampEnd) {
+ buckets.add(TimeBucket.getTimeBucket(day.getTimeInMillis(), DownSampling.Day));
+ day.add(Calendar.DAY_OF_MONTH, 1);
+ }
+ return buckets;
+ }
+
public List<String> getTablesWithinTTL(String modelName) {
final var model = TableMetaInfo.get(modelName);
final var rawTableName = getTableName(model);
@@ -215,15 +232,9 @@ public class TableHelper {
* the foreign-metric inspect probe across the node's known function tables.
*/
public List<String> getExistingDayTables(String rawTableName, long timeBucketStart, long timeBucketEnd) {
- final var timestampStart = TimeBucket.getTimestamp(timeBucketStart);
- final var timestampEnd = TimeBucket.getTimestamp(timeBucketEnd);
- final var timeBuckets = LongStream.builder();
- for (var timestamp = timestampStart; timestamp <= timestampEnd; timestamp += TimeUnit.DAYS.toMillis(1)) {
- timeBuckets.add(TimeBucket.getTimeBucket(timestamp, DownSampling.Day));
- }
- return timeBuckets.build()
- .distinct()
- .mapToObj(timeBucket -> getTable(rawTableName, timeBucket))
+ return dayTimeBucketsBetween(TimeBucket.getTimestamp(timeBucketStart), TimeBucket.getTimestamp(timeBucketEnd))
+ .stream()
+ .map(timeBucket -> getTable(rawTableName, timeBucket))
.filter(table -> {
try {
return tableExistence.get(table);
diff --git a/oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/common/TableHelperTest.java b/oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/common/TableHelperTest.java
new file mode 100644
index 0000000000..d7cf71cdd1
--- /dev/null
+++ b/oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/common/TableHelperTest.java
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.jdbc.common;
+
+import java.util.Arrays;
+import java.util.Calendar;
+import java.util.Collections;
+import org.apache.skywalking.oap.server.core.analysis.DownSampling;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class TableHelperTest {
+ /**
+ * A range shorter than a day that crosses midnight touches two day tables: the last thirty minutes, read a
+ * few minutes after midnight, must find today's table and not only yesterday's.
+ */
+ @Test
+ public void aRangeShorterThanADayAcrossMidnightTouchesBothDays() {
+ final long start = at(2026, 9, 8, 23, 40);
+ final long end = at(2026, 9, 9, 0, 10);
+ assertEquals(
+ Arrays.asList(day(2026, 9, 8), day(2026, 9, 9)),
+ TableHelper.dayTimeBucketsBetween(start, end));
+ }
+
+ @Test
+ public void aRangeWithinOneDayTouchesThatDayOnce() {
+ assertEquals(
+ Collections.singletonList(day(2026, 9, 8)),
+ TableHelper.dayTimeBucketsBetween(at(2026, 9, 8, 10, 0), at(2026, 9, 8, 11, 0)));
+ assertEquals(
+ Collections.singletonList(day(2026, 9, 8)),
+ TableHelper.dayTimeBucketsBetween(at(2026, 9, 8, 10, 0), at(2026, 9, 8, 10, 0)));
+ }
+
+ @Test
+ public void aRangeOverSeveralDaysTouchesEveryDayInOrder() {
+ assertEquals(
+ Arrays.asList(day(2026, 9, 8), day(2026, 9, 9), day(2026, 9, 10)),
+ TableHelper.dayTimeBucketsBetween(at(2026, 9, 8, 23, 0), at(2026, 9, 10, 1, 0)));
+ }
+
+ @Test
+ public void aRangeThatEndsBeforeItStartsTouchesNothing() {
+ assertEquals(
+ Collections.emptyList(),
+ TableHelper.dayTimeBucketsBetween(at(2026, 9, 9, 1, 0), at(2026, 9, 8, 23, 0)));
+ }
+
+ private static long at(final int year, final int month, final int day, final int hour, final int minute) {
+ final Calendar calendar = Calendar.getInstance();
+ calendar.clear();
+ calendar.set(year, month - 1, day, hour, minute, 0);
+ return calendar.getTimeInMillis();
+ }
+
+ private static long day(final int year, final int month, final int day) {
+ return TimeBucket.getTimeBucket(at(year, month, day, 0, 0), DownSampling.Day);
+ }
+}