You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "mymeiyi (via GitHub)" <gi...@apache.org> on 2026/03/31 02:59:15 UTC
[PR] [fix](cloud) add get_version and get_tablet_stats case [doris]
mymeiyi opened a new pull request, #61915:
URL: https://github.com/apache/doris/pull/61915
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) add get_version and get_tablet_stats case [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61915:
URL: https://github.com/apache/doris/pull/61915#issuecomment-4161263009
# FE UT Coverage Report
Increment line coverage ` 40.00% (4/10)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/61915_5103a9a4a98863363863271b7d81ea2ddca4d037/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/61915_5103a9a4a98863363863271b7d81ea2ddca4d037/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) add get_version and get_tablet_stats case [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #61915:
URL: https://github.com/apache/doris/pull/61915#issuecomment-4174106954
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) add get_version and get_tablet_stats case [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on PR #61915:
URL: https://github.com/apache/doris/pull/61915#issuecomment-4159597656
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) add get_version and get_tablet_stats case [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61915:
URL: https://github.com/apache/doris/pull/61915#issuecomment-4159514902
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) add get_version and get_tablet_stats case [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #61915:
URL: https://github.com/apache/doris/pull/61915#issuecomment-4159602821
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) add get_version and get_tablet_stats case [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61915:
URL: https://github.com/apache/doris/pull/61915#issuecomment-4161009561
<details>
<summary>TPC-H: <b>Total hot run time: 26665 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5103a9a4a98863363863271b7d81ea2ddca4d037, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17621 4463 4307 4307
q2 q3 10680 845 533 533
q4 4681 364 252 252
q5 7588 1218 1034 1034
q6 184 177 150 150
q7 813 869 694 694
q8 9328 1497 1413 1413
q9 4955 4571 4727 4571
q10 6243 1929 1662 1662
q11 495 265 250 250
q12 728 600 479 479
q13 18047 2764 1953 1953
q14 232 228 216 216
q15 q16 729 752 691 691
q17 740 867 449 449
q18 6053 5381 5264 5264
q19 1102 971 639 639
q20 539 516 382 382
q21 4370 1889 1469 1469
q22 351 311 257 257
Total cold run time: 95479 ms
Total hot run time: 26665 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4459 4336 4368 4336
q2 q3 3841 4336 3812 3812
q4 883 1169 785 785
q5 4014 4381 4329 4329
q6 189 176 145 145
q7 1757 1622 1513 1513
q8 2471 2658 2557 2557
q9 7807 7561 7400 7400
q10 3799 4112 3650 3650
q11 523 442 442 442
q12 504 600 483 483
q13 2606 2977 2126 2126
q14 302 309 292 292
q15 q16 738 923 770 770
q17 1214 1425 1389 1389
q18 7287 6824 6664 6664
q19 986 992 965 965
q20 2084 2143 1989 1989
q21 4038 3494 3421 3421
q22 503 464 397 397
Total cold run time: 50005 ms
Total hot run time: 47465 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) add get_version and get_tablet_stats case [doris]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #61915:
URL: https://github.com/apache/doris/pull/61915#discussion_r3013204787
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/CloudTabletStatMgr.java:
##########
@@ -560,6 +561,12 @@ public void addActiveTablets(List<Long> tabletIds) {
if (Config.cloud_get_tablet_stats_version == 1 || tabletIds == null || tabletIds.isEmpty()) {
return;
}
+ String ignoreTablets = DebugPointUtil.getDebugParamOrDefault(
+ "FE.CloudTabletStatMgr.addActiveTablets.ignore.tablets", "");
+ if (!ignoreTablets.isEmpty() && tabletIds.stream().anyMatch(id -> ignoreTablets.contains(String.valueOf(id)))) {
+ LOG.info("ignore adding active tablets: {}, debug param: {}", tabletIds, ignoreTablets);
Review Comment:
`ignoreTablets.contains(String.valueOf(id))` can match substrings (e.g., id=12 matches "112,113"), causing unrelated tablets to be ignored. Parse the debug param as a delimiter-separated set (e.g., split by comma and compare as longs/strings with exact equality) to avoid false positives.
##########
regression-test/suites/cloud_p0/version/test_version_syncer.groovy:
##########
@@ -0,0 +1,105 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_version_syncer", "nonConcurrent") {
+ if (!isCloudMode()) {
+ return
+ }
+
+ def tbl = 'test_version_syncer_tbl'
+ sql """ DROP TABLE IF EXISTS ${tbl} """
+
+ def enableSyncerConfig = sql_return_maparray """ ADMIN SHOW FRONTEND CONFIG LIKE '%cloud_enable_version_syncer%' """
+ assertTrue(enableSyncerConfig.size() > 0, "Expected to find cloud_enable_version_syncer config")
+ def enableSyncer = enableSyncerConfig[0].Value.toBoolean()
+ logger.info("cloud_enable_version_syncer = ${enableSyncer}")
+ if (!enableSyncer) {
+ logger.info("FE.CloudGlobalTransactionMgr.updateVersion.disabled")
+ return
+ }
+ def configResult = sql_return_maparray """ ADMIN SHOW FRONTEND CONFIG LIKE '%cloud_version_syncer_interval_second%' """
+ assertTrue(configResult.size() > 0, "Expected to find cloud_version_syncer_interval_second config")
+ def syncerInterval = configResult[0].Value.toInteger()
+ logger.info("cloud_version_syncer_interval_second = ${syncerInterval}")
+
+ onFinish {
+ GetDebugPoint().clearDebugPointsForAllFEs()
+ }
+
+ sql """
+ CREATE TABLE IF NOT EXISTS ${tbl} (
+ id INT NOT NULL,
+ name VARCHAR(50) NOT NULL,
+ value INT NOT NULL
+ )
+ DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES (
+ "replication_num" = "1"
+ );
+ """
+
+ def result = sql_return_maparray """ select * from ${tbl} """
+ assertEquals(0, result.size())
+
+ sql """ INSERT INTO ${tbl} VALUES (1, 'initial', 100) """
+ result = sql_return_maparray """ select * from ${tbl} where id = 1 """
+ assertEquals(1, result.size())
+
+ DebugPoint.enableDebugPointForAllFEs('FE.CloudGlobalTransactionMgr.updateVersion.disabled')
+
+ sql """ INSERT INTO ${tbl} VALUES (2, 'test', 200) """
+ result = sql_return_maparray """ select * from ${tbl} where id = 2 """
+ assertEquals(0, result.size(), "Data should not be visible before version sync")
+
+ def maxWaitMs = (syncerInterval * 2 + 5) * 1000
+ def intervalMs = 1000
+ def startTime = System.currentTimeMillis()
+ def found = false
+ while (true) {
+ result = sql_return_maparray """ select * from ${tbl} where id = 2 """
+ if (result.size() == 1) {
+ assertEquals('test', result[0].name)
+ assertEquals(200, result[0].value)
+ found = true
+ break
+ }
+ if (System.currentTimeMillis() - startTime > maxWaitMs) {
+ throw new IllegalStateException("Timeout waiting for data to be visible. Waited ${maxWaitMs}ms")
+ }
+ Thread.sleep(intervalMs)
+ }
+ assertTrue(found, "Data should eventually be visible after version syncer runs")
+
+ result = sql_return_maparray """ select * from ${tbl} order by id """
+ assertEquals(2, result.size())
+
+ // Test cloud_force_sync_version session variable
+ // Step 1: Insert data with version sync disabled, query should return no data
+ sql """ INSERT INTO ${tbl} VALUES (3, 'force_sync', 300) """
+ result = sql_return_maparray """ select * from ${tbl} where id = 3 """
+ assertEquals(0, result.size(), "Data should not be visible before force sync")
+
+ // Step 2: Enable session variable cloud_force_sync_version = true and execute show partitions
+ sql """ set cloud_force_sync_version = true """
+ def showPartitionsResult = sql_return_maparray """ show partitions from ${tbl} """
+ logger.info("Show partitions result: ${showPartitionsResult}")
+
+ // Step 3: Query again, data should now be visible
+ result = sql_return_maparray """ select * from ${tbl} where id = 3 """
+ assertEquals(1, result.size(), "Data should be visible after force sync")
Review Comment:
`set cloud_force_sync_version = true` is not reset to its default value. To avoid leaking session state into later statements/tests executed on the same connection, reset it (e.g., set to false) after the force-sync check (ideally in `onFinish`).
```suggestion
try {
def showPartitionsResult = sql_return_maparray """ show partitions from ${tbl} """
logger.info("Show partitions result: ${showPartitionsResult}")
// Step 3: Query again, data should now be visible
result = sql_return_maparray """ select * from ${tbl} where id = 3 """
assertEquals(1, result.size(), "Data should be visible after force sync")
} finally {
// Reset session variable to avoid leaking state into other statements/tests
sql """ set cloud_force_sync_version = false """
}
```
##########
regression-test/suites/cloud_p0/tablets/test_tablet_stat_syncer.groovy:
##########
@@ -0,0 +1,149 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import org.apache.doris.regression.suite.ClusterOptions
+
+suite("test_tablet_stat_syncer", "docker") {
+ if (!isCloudMode()) {
+ return
+ }
+
+ def options = new ClusterOptions()
+ options.setFeNum(1)
+ options.setBeNum(1)
+ options.cloudMode = true
+ options.enableDebugPoints()
+
+ docker(options) {
+ def tbl = 'test_tablet_stat_syncer_tbl'
+ sql """ DROP TABLE IF EXISTS ${tbl} """
+
+ def configResult = sql_return_maparray """ ADMIN SHOW FRONTEND CONFIG LIKE '%cloud_get_tablet_stats_version%' """
+ assertTrue(configResult.size() > 0, "Expected to find cloud_get_tablet_stats_version config")
+ def tabletStatsVersion = configResult[0].Value.toInteger()
+ logger.info("cloud_get_tablet_stats_version = ${tabletStatsVersion}")
+ if (tabletStatsVersion != 2) {
+ logger.info("cloud_get_tablet_stats_version is not 2, skip test")
+ return
+ }
+
+ def intervalConfigResult = sql_return_maparray """ ADMIN SHOW FRONTEND CONFIG LIKE '%tablet_stat_update_interval_second%' """
+ assertTrue(intervalConfigResult.size() > 0, "Expected to find tablet_stat_update_interval_second config")
+ def statUpdateInterval = intervalConfigResult[0].Value.toInteger()
+ logger.info("tablet_stat_update_interval_second = ${statUpdateInterval}")
+
+ sql """
+ CREATE TABLE IF NOT EXISTS ${tbl} (
+ id INT NOT NULL,
+ name VARCHAR(50) NOT NULL,
+ value INT NOT NULL
+ )
+ ENGINE = olap
+ DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES (
+ "replication_num" = "1"
+ );
+ """
+
+ def result = sql_return_maparray """ select * from ${tbl} """
+ assertEquals(0, result.size())
+
+ sql """ INSERT INTO ${tbl} VALUES (1, 'initial', 100) """
+ result = sql_return_maparray """ select * from ${tbl} where id = 1 """
+ assertEquals(1, result.size())
+
+ def tablets = sql_return_maparray """ show tablets from ${tbl} """
+ assertTrue(tablets.size() > 0, "Expected at least one tablet")
+ def tabletIds = tablets.collect { it.TabletId }.join(",")
+ logger.info("Tablet IDs: ${tabletIds}")
+
+ def initialRowCount = tablets[0].RowCount
+ logger.info("Initial RowCount: ${initialRowCount}")
+
+ GetDebugPoint().enableDebugPointForAllFEs("FE.CloudTabletStatMgr.addActiveTablets.ignore.tablets", [value: tabletIds])
+
+ sql """ INSERT INTO ${tbl} VALUES (2, 'test', 200) """
+ result = sql_return_maparray """ select * from ${tbl} where id = 2 """
+ assertEquals(1, result.size(), "Data should be visible in query")
+
+ def tabletsAfterInsert = sql_return_maparray """ show tablets from ${tbl} """
+ def rowCountAfterInsert = tabletsAfterInsert[0].RowCount
+ logger.info("RowCount after insert (before sync): ${rowCountAfterInsert}")
+
+ GetDebugPoint().disableDebugPointForAllFEs("FE.CloudTabletStatMgr.addActiveTablets.ignore.tablets")
Review Comment:
The debug point is intended to keep tablet stats stale, but the test doesn’t assert that `RowCount` stayed unchanged while it was enabled (and `initialRowCount` becomes unused). Add an assertion on `rowCountAfterInsert` (e.g., equals `initialRowCount`/1) before disabling the debug point so the test can’t pass if stats update immediately.
##########
regression-test/suites/cloud_p0/version/test_version_syncer.groovy:
##########
@@ -0,0 +1,105 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_version_syncer", "nonConcurrent") {
+ if (!isCloudMode()) {
+ return
+ }
Review Comment:
This test relies on FE debug points, but it doesn’t check that `enable_debug_points` is enabled on the target cluster. Consider skipping the test (or running in a docker cluster with `options.enableDebugPoints()`) when `getFeConfig('enable_debug_points') != 'true'`, otherwise the debug point may have no effect and assertions can fail.
```suggestion
}
def enableDebugPoints = getFeConfig("enable_debug_points")
if (enableDebugPoints != "true") {
logger.info("enable_debug_points is not enabled, skip test_version_syncer")
return
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) add get_version and get_tablet_stats case [doris]
Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on PR #61915:
URL: https://github.com/apache/doris/pull/61915#issuecomment-4161675737
run external
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) add get_version and get_tablet_stats case [doris]
Posted by "w41ter (via GitHub)" <gi...@apache.org>.
w41ter merged PR #61915:
URL: https://github.com/apache/doris/pull/61915
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) add get_version and get_tablet_stats case [doris]
Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on PR #61915:
URL: https://github.com/apache/doris/pull/61915#issuecomment-4160816780
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) add get_version and get_tablet_stats case [doris]
Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on PR #61915:
URL: https://github.com/apache/doris/pull/61915#issuecomment-4161435380
run nonConcurrent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) add get_version and get_tablet_stats case [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #61915:
URL: https://github.com/apache/doris/pull/61915#discussion_r3013274419
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/CloudTabletStatMgr.java:
##########
@@ -560,6 +561,12 @@ public void addActiveTablets(List<Long> tabletIds) {
if (Config.cloud_get_tablet_stats_version == 1 || tabletIds == null || tabletIds.isEmpty()) {
return;
}
+ String ignoreTablets = DebugPointUtil.getDebugParamOrDefault(
+ "FE.CloudTabletStatMgr.addActiveTablets.ignore.tablets", "");
+ if (!ignoreTablets.isEmpty() && tabletIds.stream().anyMatch(id -> ignoreTablets.contains(String.valueOf(id)))) {
+ LOG.info("ignore adding active tablets: {}, debug param: {}", tabletIds, ignoreTablets);
Review Comment:
**Bug: Substring false-positive in tablet ID matching**
`ignoreTablets.contains(String.valueOf(id))` performs substring matching on a comma-separated string. If `ignoreTablets = "12345"`, then tablet IDs `123`, `1234`, `2345`, `234`, `345`, `45`, etc. would all falsely match.
Other debug points in the codebase (e.g., `Backend.isQueryAvailable` in `Backend.java:545`, `HeartbeatMgr.java:329`) handle this correctly by splitting on comma first:
```java
Arrays.stream(ignoreTablets.split(",")).anyMatch(s -> s.equals(String.valueOf(id)))
```
Please use the split-then-exact-match pattern to avoid false positives.
##########
regression-test/suites/cloud_p0/tablets/test_tablet_stat_syncer.groovy:
##########
@@ -0,0 +1,149 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import org.apache.doris.regression.suite.ClusterOptions
+
+suite("test_tablet_stat_syncer", "docker") {
+ if (!isCloudMode()) {
+ return
+ }
+
+ def options = new ClusterOptions()
+ options.setFeNum(1)
+ options.setBeNum(1)
+ options.cloudMode = true
+ options.enableDebugPoints()
+
+ docker(options) {
+ def tbl = 'test_tablet_stat_syncer_tbl'
+ sql """ DROP TABLE IF EXISTS ${tbl} """
+
+ def configResult = sql_return_maparray """ ADMIN SHOW FRONTEND CONFIG LIKE '%cloud_get_tablet_stats_version%' """
+ assertTrue(configResult.size() > 0, "Expected to find cloud_get_tablet_stats_version config")
+ def tabletStatsVersion = configResult[0].Value.toInteger()
+ logger.info("cloud_get_tablet_stats_version = ${tabletStatsVersion}")
+ if (tabletStatsVersion != 2) {
+ logger.info("cloud_get_tablet_stats_version is not 2, skip test")
+ return
+ }
+
+ def intervalConfigResult = sql_return_maparray """ ADMIN SHOW FRONTEND CONFIG LIKE '%tablet_stat_update_interval_second%' """
+ assertTrue(intervalConfigResult.size() > 0, "Expected to find tablet_stat_update_interval_second config")
+ def statUpdateInterval = intervalConfigResult[0].Value.toInteger()
+ logger.info("tablet_stat_update_interval_second = ${statUpdateInterval}")
+
+ sql """
+ CREATE TABLE IF NOT EXISTS ${tbl} (
+ id INT NOT NULL,
+ name VARCHAR(50) NOT NULL,
+ value INT NOT NULL
+ )
+ ENGINE = olap
+ DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES (
+ "replication_num" = "1"
+ );
+ """
+
+ def result = sql_return_maparray """ select * from ${tbl} """
+ assertEquals(0, result.size())
+
+ sql """ INSERT INTO ${tbl} VALUES (1, 'initial', 100) """
+ result = sql_return_maparray """ select * from ${tbl} where id = 1 """
+ assertEquals(1, result.size())
+
+ def tablets = sql_return_maparray """ show tablets from ${tbl} """
+ assertTrue(tablets.size() > 0, "Expected at least one tablet")
+ def tabletIds = tablets.collect { it.TabletId }.join(",")
+ logger.info("Tablet IDs: ${tabletIds}")
+
+ def initialRowCount = tablets[0].RowCount
+ logger.info("Initial RowCount: ${initialRowCount}")
+
+ GetDebugPoint().enableDebugPointForAllFEs("FE.CloudTabletStatMgr.addActiveTablets.ignore.tablets", [value: tabletIds])
+
+ sql """ INSERT INTO ${tbl} VALUES (2, 'test', 200) """
+ result = sql_return_maparray """ select * from ${tbl} where id = 2 """
+ assertEquals(1, result.size(), "Data should be visible in query")
+
+ def tabletsAfterInsert = sql_return_maparray """ show tablets from ${tbl} """
+ def rowCountAfterInsert = tabletsAfterInsert[0].RowCount
+ logger.info("RowCount after insert (before sync): ${rowCountAfterInsert}")
+
+ GetDebugPoint().disableDebugPointForAllFEs("FE.CloudTabletStatMgr.addActiveTablets.ignore.tablets")
+
+ def maxWaitMs = (statUpdateInterval * 3 + 5) * 1000
+ def intervalMs = 1000
+ def startTime = System.currentTimeMillis()
+ def updated = false
+ while (true) {
+ def tabletsNow = sql_return_maparray """ show tablets from ${tbl} """
+ def rowCountNow = tabletsNow[0].RowCount.toInteger()
+ logger.info("RowCount get: ${rowCountNow}")
+ if (rowCountNow == 2) {
+ logger.info("RowCount updated: ${rowCountNow}")
+ updated = true
+ break
+ }
+ if (System.currentTimeMillis() - startTime > maxWaitMs) {
+ throw new IllegalStateException("Timeout waiting for tablet stat update. Waited ${maxWaitMs}ms, RowCount still ${rowCountNow}")
+ }
+ Thread.sleep(intervalMs)
+ }
+ assertTrue(updated, "Tablet stat should eventually update after debug point disabled")
+ tabletsAfterInsert = sql_return_maparray """ show tablets from ${tbl} """
+ rowCountAfterInsert = tabletsAfterInsert[0].RowCount.toInteger()
+ logger.info("RowCount after stat sync: ${rowCountAfterInsert}")
+ assertEquals(2, rowCountAfterInsert, "RowCount should reflect the new row after stat sync")
+
+ sql """ INSERT INTO ${tbl} VALUES (3, 'force_sync', 300) """
+ def tabletsBeforeSync = sql_return_maparray """ show tablets from ${tbl} """
+ def localDataSizeBeforeSync = tabletsBeforeSync[0].LocalDataSize
+ def rowCountBeforeSync = tabletsBeforeSync[0].RowCount
+ logger.info("LocalDataSize before force sync: ${localDataSizeBeforeSync}, RowCount: ${rowCountBeforeSync}")
+
+ sql """ set cloud_force_sync_tablet_stats = true """
+ def tabletsAfterForceSync = sql_return_maparray """ show tablets from ${tbl} """
+ def localDataSizeAfterForceSync = tabletsAfterForceSync[0].LocalDataSize
+ def rowCountAfterForceSync = tabletsAfterForceSync[0].RowCount
+ logger.info("LocalDataSize after force sync: ${localDataSizeAfterForceSync}, RowCount: ${rowCountAfterForceSync}")
Review Comment:
**Test weakness: force-sync section doesn't assert the immediate effect**
After setting `cloud_force_sync_tablet_stats = true` and reading tablet stats (lines 120-124), the test only logs the results but never asserts that `rowCountAfterForceSync == 3`. It then falls through to a polling loop that waits for eventual consistency — which is the same verification pattern as the non-force-sync path above.
To actually verify that force sync works (i.e., provides an *immediate* update), consider adding:
```groovy
assertEquals(3, rowCountAfterForceSync.toInteger(), "RowCount should be updated immediately after force sync")
```
Otherwise this section tests nothing beyond what the first section already tests.
##########
regression-test/suites/cloud_p0/version/test_version_syncer.groovy:
##########
@@ -0,0 +1,105 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_version_syncer", "nonConcurrent") {
+ if (!isCloudMode()) {
+ return
+ }
+
+ def tbl = 'test_version_syncer_tbl'
+ sql """ DROP TABLE IF EXISTS ${tbl} """
+
+ def enableSyncerConfig = sql_return_maparray """ ADMIN SHOW FRONTEND CONFIG LIKE '%cloud_enable_version_syncer%' """
+ assertTrue(enableSyncerConfig.size() > 0, "Expected to find cloud_enable_version_syncer config")
+ def enableSyncer = enableSyncerConfig[0].Value.toBoolean()
+ logger.info("cloud_enable_version_syncer = ${enableSyncer}")
+ if (!enableSyncer) {
+ logger.info("FE.CloudGlobalTransactionMgr.updateVersion.disabled")
+ return
+ }
+ def configResult = sql_return_maparray """ ADMIN SHOW FRONTEND CONFIG LIKE '%cloud_version_syncer_interval_second%' """
+ assertTrue(configResult.size() > 0, "Expected to find cloud_version_syncer_interval_second config")
+ def syncerInterval = configResult[0].Value.toInteger()
+ logger.info("cloud_version_syncer_interval_second = ${syncerInterval}")
+
+ onFinish {
+ GetDebugPoint().clearDebugPointsForAllFEs()
+ }
+
+ sql """
+ CREATE TABLE IF NOT EXISTS ${tbl} (
+ id INT NOT NULL,
+ name VARCHAR(50) NOT NULL,
+ value INT NOT NULL
+ )
+ DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES (
+ "replication_num" = "1"
+ );
+ """
+
+ def result = sql_return_maparray """ select * from ${tbl} """
+ assertEquals(0, result.size())
+
+ sql """ INSERT INTO ${tbl} VALUES (1, 'initial', 100) """
+ result = sql_return_maparray """ select * from ${tbl} where id = 1 """
+ assertEquals(1, result.size())
+
+ DebugPoint.enableDebugPointForAllFEs('FE.CloudGlobalTransactionMgr.updateVersion.disabled')
+
+ sql """ INSERT INTO ${tbl} VALUES (2, 'test', 200) """
+ result = sql_return_maparray """ select * from ${tbl} where id = 2 """
+ assertEquals(0, result.size(), "Data should not be visible before version sync")
+
+ def maxWaitMs = (syncerInterval * 2 + 5) * 1000
+ def intervalMs = 1000
+ def startTime = System.currentTimeMillis()
+ def found = false
+ while (true) {
+ result = sql_return_maparray """ select * from ${tbl} where id = 2 """
+ if (result.size() == 1) {
+ assertEquals('test', result[0].name)
+ assertEquals(200, result[0].value)
+ found = true
+ break
+ }
+ if (System.currentTimeMillis() - startTime > maxWaitMs) {
+ throw new IllegalStateException("Timeout waiting for data to be visible. Waited ${maxWaitMs}ms")
+ }
+ Thread.sleep(intervalMs)
+ }
+ assertTrue(found, "Data should eventually be visible after version syncer runs")
+
+ result = sql_return_maparray """ select * from ${tbl} order by id """
+ assertEquals(2, result.size())
+
+ // Test cloud_force_sync_version session variable
+ // Step 1: Insert data with version sync disabled, query should return no data
+ sql """ INSERT INTO ${tbl} VALUES (3, 'force_sync', 300) """
+ result = sql_return_maparray """ select * from ${tbl} where id = 3 """
+ assertEquals(0, result.size(), "Data should not be visible before force sync")
+
+ // Step 2: Enable session variable cloud_force_sync_version = true and execute show partitions
+ sql """ set cloud_force_sync_version = true """
+ def showPartitionsResult = sql_return_maparray """ show partitions from ${tbl} """
Review Comment:
**Minor: `cloud_force_sync_version` is never reset to `false`**
Line 98 sets `cloud_force_sync_version = true` but it is never set back to `false`. The tablet stat syncer test (line 125) properly resets its equivalent variable. While session-scoped variables are cleaned up when the session ends, it's good practice to reset it for consistency and to avoid affecting subsequent queries in the same session.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) add get_version and get_tablet_stats case [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #61915:
URL: https://github.com/apache/doris/pull/61915#issuecomment-4159602923
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) add get_version and get_tablet_stats case [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61915:
URL: https://github.com/apache/doris/pull/61915#issuecomment-4161076895
<details>
<summary>TPC-DS: <b>Total hot run time: 169593 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5103a9a4a98863363863271b7d81ea2ddca4d037, data reload: false
query5 4345 629 490 490
query6 332 220 206 206
query7 4221 465 258 258
query8 341 244 220 220
query9 8697 2682 2702 2682
query10 531 379 335 335
query11 7002 5133 4880 4880
query12 188 131 123 123
query13 1278 465 380 380
query14 5750 3741 3483 3483
query14_1 2854 2846 2843 2843
query15 211 194 177 177
query16 988 460 451 451
query17 1123 721 623 623
query18 2452 462 362 362
query19 225 218 190 190
query20 133 127 126 126
query21 210 136 111 111
query22 13137 14097 15112 14097
query23 16789 16171 16151 16151
query23_1 16264 16288 16129 16129
query24 7579 1688 1297 1297
query24_1 1317 1271 1277 1271
query25 568 477 418 418
query26 1419 303 167 167
query27 3379 585 306 306
query28 4652 1822 1790 1790
query29 818 558 473 473
query30 299 225 192 192
query31 1011 959 880 880
query32 82 69 69 69
query33 516 325 282 282
query34 891 877 525 525
query35 633 688 588 588
query36 1118 1099 936 936
query37 130 92 85 85
query38 2968 2889 2848 2848
query39 857 825 816 816
query39_1 799 784 792 784
query40 231 148 137 137
query41 64 66 57 57
query42 257 254 251 251
query43 250 248 217 217
query44
query45 193 199 185 185
query46 895 971 603 603
query47 2103 2128 2073 2073
query48 306 320 223 223
query49 624 439 389 389
query50 699 280 214 214
query51 4030 4072 4045 4045
query52 281 272 254 254
query53 295 325 277 277
query54 304 270 257 257
query55 92 85 80 80
query56 304 318 313 313
query57 1927 1725 1700 1700
query58 292 276 268 268
query59 2797 2966 2756 2756
query60 363 338 325 325
query61 162 153 148 148
query62 638 597 556 556
query63 309 291 275 275
query64 5048 1290 997 997
query65
query66 1456 450 358 358
query67 24384 24378 24245 24245
query68
query69 412 305 279 279
query70 938 976 879 879
query71 330 290 294 290
query72 2802 2736 2491 2491
query73 539 539 315 315
query74 9681 9643 9471 9471
query75 2889 2739 2461 2461
query76 2282 1029 669 669
query77 354 379 315 315
query78 11127 11280 10540 10540
query79 1068 844 584 584
query80 1465 625 526 526
query81 531 258 230 230
query82 1336 152 120 120
query83 347 271 235 235
query84 251 121 99 99
query85 1121 550 480 480
query86 395 347 293 293
query87 3179 3114 3013 3013
query88 3572 2641 2583 2583
query89 431 367 342 342
query90 1773 176 165 165
query91 183 167 133 133
query92 83 74 73 73
query93 929 853 498 498
query94 538 312 316 312
query95 587 340 381 340
query96 647 517 226 226
query97 2460 2466 2435 2435
query98 244 232 228 228
query99 1024 1009 928 928
Total cold run time: 252156 ms
Total hot run time: 169593 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org