You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "mymeiyi (via GitHub)" <gi...@apache.org> on 2026/03/31 02:59:15 UTC

[PR] [fix](cloud) add get_version and get_tablet_stats case [doris]

mymeiyi opened a new pull request, #61915:
URL: https://github.com/apache/doris/pull/61915

   ### What problem does this PR solve?
   
   Issue Number: close #xxx
   
   Related PR: #xxx
   
   Problem Summary:
   
   ### Release note
   
   None
   
   ### Check List (For Author)
   
   - Test <!-- At least one of them must be included. -->
       - [ ] Regression test
       - [ ] Unit Test
       - [ ] Manual test (add detailed scripts or steps below)
       - [ ] No need to test or manual test. Explain why:
           - [ ] This is a refactor/code format and no logic has been changed.
           - [ ] Previous test can cover this change.
           - [ ] No code files have been changed.
           - [ ] Other reason <!-- Add your reason?  -->
   
   - Behavior changed:
       - [ ] No.
       - [ ] Yes. <!-- Explain the behavior change -->
   
   - Does this need documentation?
       - [ ] No.
       - [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
   
   ### Check List (For Reviewer who merge this PR)
   
   - [ ] Confirm the release note
   - [ ] Confirm test cases
   - [ ] Confirm document
   - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](cloud) add get_version and get_tablet_stats case [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61915:
URL: https://github.com/apache/doris/pull/61915#issuecomment-4161263009

   # FE UT Coverage Report
   Increment line coverage `  40.00% (4/10)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/61915_5103a9a4a98863363863271b7d81ea2ddca4d037/fe_increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/61915_5103a9a4a98863363863271b7d81ea2ddca4d037/fe_report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](cloud) add get_version and get_tablet_stats case [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #61915:
URL: https://github.com/apache/doris/pull/61915#issuecomment-4174106954

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](cloud) add get_version and get_tablet_stats case [doris]

Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on PR #61915:
URL: https://github.com/apache/doris/pull/61915#issuecomment-4159597656

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](cloud) add get_version and get_tablet_stats case [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61915:
URL: https://github.com/apache/doris/pull/61915#issuecomment-4159514902

   
   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
   
   Please clearly describe your PR:
   1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
   2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
   3. What features were added. Why was this function added?
   4. Which code was refactored and why was this part of the code refactored?
   5. Which functions were optimized and what is the difference before and after the optimization?
       


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](cloud) add get_version and get_tablet_stats case [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #61915:
URL: https://github.com/apache/doris/pull/61915#issuecomment-4159602821

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](cloud) add get_version and get_tablet_stats case [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61915:
URL: https://github.com/apache/doris/pull/61915#issuecomment-4161009561

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 26665 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 5103a9a4a98863363863271b7d81ea2ddca4d037, data reload: false
   
   ------ Round 1 ----------------------------------
   orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
   ============================================
   q1	17621	4463	4307	4307
   q2	q3	10680	845	533	533
   q4	4681	364	252	252
   q5	7588	1218	1034	1034
   q6	184	177	150	150
   q7	813	869	694	694
   q8	9328	1497	1413	1413
   q9	4955	4571	4727	4571
   q10	6243	1929	1662	1662
   q11	495	265	250	250
   q12	728	600	479	479
   q13	18047	2764	1953	1953
   q14	232	228	216	216
   q15	q16	729	752	691	691
   q17	740	867	449	449
   q18	6053	5381	5264	5264
   q19	1102	971	639	639
   q20	539	516	382	382
   q21	4370	1889	1469	1469
   q22	351	311	257	257
   Total cold run time: 95479 ms
   Total hot run time: 26665 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
   ============================================
   q1	4459	4336	4368	4336
   q2	q3	3841	4336	3812	3812
   q4	883	1169	785	785
   q5	4014	4381	4329	4329
   q6	189	176	145	145
   q7	1757	1622	1513	1513
   q8	2471	2658	2557	2557
   q9	7807	7561	7400	7400
   q10	3799	4112	3650	3650
   q11	523	442	442	442
   q12	504	600	483	483
   q13	2606	2977	2126	2126
   q14	302	309	292	292
   q15	q16	738	923	770	770
   q17	1214	1425	1389	1389
   q18	7287	6824	6664	6664
   q19	986	992	965	965
   q20	2084	2143	1989	1989
   q21	4038	3494	3421	3421
   q22	503	464	397	397
   Total cold run time: 50005 ms
   Total hot run time: 47465 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](cloud) add get_version and get_tablet_stats case [doris]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #61915:
URL: https://github.com/apache/doris/pull/61915#discussion_r3013204787


##########
fe/fe-core/src/main/java/org/apache/doris/catalog/CloudTabletStatMgr.java:
##########
@@ -560,6 +561,12 @@ public void addActiveTablets(List<Long> tabletIds) {
         if (Config.cloud_get_tablet_stats_version == 1 || tabletIds == null || tabletIds.isEmpty()) {
             return;
         }
+        String ignoreTablets = DebugPointUtil.getDebugParamOrDefault(
+                "FE.CloudTabletStatMgr.addActiveTablets.ignore.tablets", "");
+        if (!ignoreTablets.isEmpty() && tabletIds.stream().anyMatch(id -> ignoreTablets.contains(String.valueOf(id)))) {
+            LOG.info("ignore adding active tablets: {}, debug param: {}", tabletIds, ignoreTablets);

Review Comment:
   `ignoreTablets.contains(String.valueOf(id))` can match substrings (e.g., id=12 matches "112,113"), causing unrelated tablets to be ignored. Parse the debug param as a delimiter-separated set (e.g., split by comma and compare as longs/strings with exact equality) to avoid false positives.



##########
regression-test/suites/cloud_p0/version/test_version_syncer.groovy:
##########
@@ -0,0 +1,105 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_version_syncer", "nonConcurrent") {
+    if (!isCloudMode()) {
+        return
+    }
+
+    def tbl = 'test_version_syncer_tbl'
+    sql """ DROP TABLE IF EXISTS ${tbl} """
+
+    def enableSyncerConfig = sql_return_maparray """ ADMIN SHOW FRONTEND CONFIG LIKE '%cloud_enable_version_syncer%' """
+    assertTrue(enableSyncerConfig.size() > 0, "Expected to find cloud_enable_version_syncer config")
+    def enableSyncer = enableSyncerConfig[0].Value.toBoolean()
+    logger.info("cloud_enable_version_syncer = ${enableSyncer}")
+    if (!enableSyncer) {
+        logger.info("FE.CloudGlobalTransactionMgr.updateVersion.disabled")
+        return
+    }
+    def configResult = sql_return_maparray """ ADMIN SHOW FRONTEND CONFIG LIKE '%cloud_version_syncer_interval_second%' """
+    assertTrue(configResult.size() > 0, "Expected to find cloud_version_syncer_interval_second config")
+    def syncerInterval = configResult[0].Value.toInteger()
+    logger.info("cloud_version_syncer_interval_second = ${syncerInterval}")
+
+    onFinish {
+        GetDebugPoint().clearDebugPointsForAllFEs()
+    }
+
+    sql """
+            CREATE TABLE IF NOT EXISTS ${tbl} (
+                id INT NOT NULL,
+                name VARCHAR(50) NOT NULL,
+                value INT NOT NULL
+            )
+            DUPLICATE KEY(id)
+            DISTRIBUTED BY HASH(id) BUCKETS 1
+            PROPERTIES (
+                "replication_num" = "1"
+            );
+    """
+
+    def result = sql_return_maparray """ select * from ${tbl} """
+    assertEquals(0, result.size())
+
+    sql """ INSERT INTO ${tbl} VALUES (1, 'initial', 100) """
+    result = sql_return_maparray """ select * from ${tbl} where id = 1 """
+    assertEquals(1, result.size())
+
+    DebugPoint.enableDebugPointForAllFEs('FE.CloudGlobalTransactionMgr.updateVersion.disabled')
+
+    sql """ INSERT INTO ${tbl} VALUES (2, 'test', 200) """
+    result = sql_return_maparray """ select * from ${tbl} where id = 2 """
+    assertEquals(0, result.size(), "Data should not be visible before version sync")
+
+    def maxWaitMs = (syncerInterval * 2 + 5) * 1000
+    def intervalMs = 1000
+    def startTime = System.currentTimeMillis()
+    def found = false
+    while (true) {
+        result = sql_return_maparray """ select * from ${tbl} where id = 2 """
+        if (result.size() == 1) {
+            assertEquals('test', result[0].name)
+            assertEquals(200, result[0].value)
+            found = true
+            break
+        }
+        if (System.currentTimeMillis() - startTime > maxWaitMs) {
+            throw new IllegalStateException("Timeout waiting for data to be visible. Waited ${maxWaitMs}ms")
+        }
+        Thread.sleep(intervalMs)
+    }
+    assertTrue(found, "Data should eventually be visible after version syncer runs")
+
+    result = sql_return_maparray """ select * from ${tbl} order by id """
+    assertEquals(2, result.size())
+
+    // Test cloud_force_sync_version session variable
+    // Step 1: Insert data with version sync disabled, query should return no data
+    sql """ INSERT INTO ${tbl} VALUES (3, 'force_sync', 300) """
+    result = sql_return_maparray """ select * from ${tbl} where id = 3 """
+    assertEquals(0, result.size(), "Data should not be visible before force sync")
+
+    // Step 2: Enable session variable cloud_force_sync_version = true and execute show partitions
+    sql """ set cloud_force_sync_version = true """
+    def showPartitionsResult = sql_return_maparray """ show partitions from ${tbl} """
+    logger.info("Show partitions result: ${showPartitionsResult}")
+
+    // Step 3: Query again, data should now be visible
+    result = sql_return_maparray """ select * from ${tbl} where id = 3 """
+    assertEquals(1, result.size(), "Data should be visible after force sync")

Review Comment:
   `set cloud_force_sync_version = true` is not reset to its default value. To avoid leaking session state into later statements/tests executed on the same connection, reset it (e.g., set to false) after the force-sync check (ideally in `onFinish`).
   ```suggestion
       try {
           def showPartitionsResult = sql_return_maparray """ show partitions from ${tbl} """
           logger.info("Show partitions result: ${showPartitionsResult}")
   
           // Step 3: Query again, data should now be visible
           result = sql_return_maparray """ select * from ${tbl} where id = 3 """
           assertEquals(1, result.size(), "Data should be visible after force sync")
       } finally {
           // Reset session variable to avoid leaking state into other statements/tests
           sql """ set cloud_force_sync_version = false """
       }
   ```



##########
regression-test/suites/cloud_p0/tablets/test_tablet_stat_syncer.groovy:
##########
@@ -0,0 +1,149 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import org.apache.doris.regression.suite.ClusterOptions
+
+suite("test_tablet_stat_syncer", "docker") {
+    if (!isCloudMode()) {
+        return
+    }
+
+    def options = new ClusterOptions()
+    options.setFeNum(1)
+    options.setBeNum(1)
+    options.cloudMode = true
+    options.enableDebugPoints()
+
+    docker(options) {
+        def tbl = 'test_tablet_stat_syncer_tbl'
+        sql """ DROP TABLE IF EXISTS ${tbl} """
+
+        def configResult = sql_return_maparray """ ADMIN SHOW FRONTEND CONFIG LIKE '%cloud_get_tablet_stats_version%' """
+        assertTrue(configResult.size() > 0, "Expected to find cloud_get_tablet_stats_version config")
+        def tabletStatsVersion = configResult[0].Value.toInteger()
+        logger.info("cloud_get_tablet_stats_version = ${tabletStatsVersion}")
+        if (tabletStatsVersion != 2) {
+            logger.info("cloud_get_tablet_stats_version is not 2, skip test")
+            return
+        }
+
+        def intervalConfigResult = sql_return_maparray """ ADMIN SHOW FRONTEND CONFIG LIKE '%tablet_stat_update_interval_second%' """
+        assertTrue(intervalConfigResult.size() > 0, "Expected to find tablet_stat_update_interval_second config")
+        def statUpdateInterval = intervalConfigResult[0].Value.toInteger()
+        logger.info("tablet_stat_update_interval_second = ${statUpdateInterval}")
+
+        sql """
+            CREATE TABLE IF NOT EXISTS ${tbl} (
+                id INT NOT NULL,
+                name VARCHAR(50) NOT NULL,
+                value INT NOT NULL
+            )
+            ENGINE = olap
+            DUPLICATE KEY(id)
+            DISTRIBUTED BY HASH(id) BUCKETS 1
+            PROPERTIES (
+                "replication_num" = "1"
+            );
+        """
+
+        def result = sql_return_maparray """ select * from ${tbl} """
+        assertEquals(0, result.size())
+
+        sql """ INSERT INTO ${tbl} VALUES (1, 'initial', 100) """
+        result = sql_return_maparray """ select * from ${tbl} where id = 1 """
+        assertEquals(1, result.size())
+
+        def tablets = sql_return_maparray """ show tablets from ${tbl} """
+        assertTrue(tablets.size() > 0, "Expected at least one tablet")
+        def tabletIds = tablets.collect { it.TabletId }.join(",")
+        logger.info("Tablet IDs: ${tabletIds}")
+
+        def initialRowCount = tablets[0].RowCount
+        logger.info("Initial RowCount: ${initialRowCount}")
+
+        GetDebugPoint().enableDebugPointForAllFEs("FE.CloudTabletStatMgr.addActiveTablets.ignore.tablets", [value: tabletIds])
+
+        sql """ INSERT INTO ${tbl} VALUES (2, 'test', 200) """
+        result = sql_return_maparray """ select * from ${tbl} where id = 2 """
+        assertEquals(1, result.size(), "Data should be visible in query")
+
+        def tabletsAfterInsert = sql_return_maparray """ show tablets from ${tbl} """
+        def rowCountAfterInsert = tabletsAfterInsert[0].RowCount
+        logger.info("RowCount after insert (before sync): ${rowCountAfterInsert}")
+
+        GetDebugPoint().disableDebugPointForAllFEs("FE.CloudTabletStatMgr.addActiveTablets.ignore.tablets")

Review Comment:
   The debug point is intended to keep tablet stats stale, but the test doesn’t assert that `RowCount` stayed unchanged while it was enabled (and `initialRowCount` becomes unused). Add an assertion on `rowCountAfterInsert` (e.g., equals `initialRowCount`/1) before disabling the debug point so the test can’t pass if stats update immediately.



##########
regression-test/suites/cloud_p0/version/test_version_syncer.groovy:
##########
@@ -0,0 +1,105 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_version_syncer", "nonConcurrent") {
+    if (!isCloudMode()) {
+        return
+    }

Review Comment:
   This test relies on FE debug points, but it doesn’t check that `enable_debug_points` is enabled on the target cluster. Consider skipping the test (or running in a docker cluster with `options.enableDebugPoints()`) when `getFeConfig('enable_debug_points') != 'true'`, otherwise the debug point may have no effect and assertions can fail.
   ```suggestion
       }
       def enableDebugPoints = getFeConfig("enable_debug_points")
       if (enableDebugPoints != "true") {
           logger.info("enable_debug_points is not enabled, skip test_version_syncer")
           return
       }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](cloud) add get_version and get_tablet_stats case [doris]

Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on PR #61915:
URL: https://github.com/apache/doris/pull/61915#issuecomment-4161675737

   run external


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](cloud) add get_version and get_tablet_stats case [doris]

Posted by "w41ter (via GitHub)" <gi...@apache.org>.
w41ter merged PR #61915:
URL: https://github.com/apache/doris/pull/61915


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](cloud) add get_version and get_tablet_stats case [doris]

Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on PR #61915:
URL: https://github.com/apache/doris/pull/61915#issuecomment-4160816780

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](cloud) add get_version and get_tablet_stats case [doris]

Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on PR #61915:
URL: https://github.com/apache/doris/pull/61915#issuecomment-4161435380

   run nonConcurrent


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](cloud) add get_version and get_tablet_stats case [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #61915:
URL: https://github.com/apache/doris/pull/61915#discussion_r3013274419


##########
fe/fe-core/src/main/java/org/apache/doris/catalog/CloudTabletStatMgr.java:
##########
@@ -560,6 +561,12 @@ public void addActiveTablets(List<Long> tabletIds) {
         if (Config.cloud_get_tablet_stats_version == 1 || tabletIds == null || tabletIds.isEmpty()) {
             return;
         }
+        String ignoreTablets = DebugPointUtil.getDebugParamOrDefault(
+                "FE.CloudTabletStatMgr.addActiveTablets.ignore.tablets", "");
+        if (!ignoreTablets.isEmpty() && tabletIds.stream().anyMatch(id -> ignoreTablets.contains(String.valueOf(id)))) {
+            LOG.info("ignore adding active tablets: {}, debug param: {}", tabletIds, ignoreTablets);

Review Comment:
   **Bug: Substring false-positive in tablet ID matching**
   
   `ignoreTablets.contains(String.valueOf(id))` performs substring matching on a comma-separated string. If `ignoreTablets = "12345"`, then tablet IDs `123`, `1234`, `2345`, `234`, `345`, `45`, etc. would all falsely match.
   
   Other debug points in the codebase (e.g., `Backend.isQueryAvailable` in `Backend.java:545`, `HeartbeatMgr.java:329`) handle this correctly by splitting on comma first:
   
   ```java
   Arrays.stream(ignoreTablets.split(",")).anyMatch(s -> s.equals(String.valueOf(id)))
   ```
   
   Please use the split-then-exact-match pattern to avoid false positives.



##########
regression-test/suites/cloud_p0/tablets/test_tablet_stat_syncer.groovy:
##########
@@ -0,0 +1,149 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import org.apache.doris.regression.suite.ClusterOptions
+
+suite("test_tablet_stat_syncer", "docker") {
+    if (!isCloudMode()) {
+        return
+    }
+
+    def options = new ClusterOptions()
+    options.setFeNum(1)
+    options.setBeNum(1)
+    options.cloudMode = true
+    options.enableDebugPoints()
+
+    docker(options) {
+        def tbl = 'test_tablet_stat_syncer_tbl'
+        sql """ DROP TABLE IF EXISTS ${tbl} """
+
+        def configResult = sql_return_maparray """ ADMIN SHOW FRONTEND CONFIG LIKE '%cloud_get_tablet_stats_version%' """
+        assertTrue(configResult.size() > 0, "Expected to find cloud_get_tablet_stats_version config")
+        def tabletStatsVersion = configResult[0].Value.toInteger()
+        logger.info("cloud_get_tablet_stats_version = ${tabletStatsVersion}")
+        if (tabletStatsVersion != 2) {
+            logger.info("cloud_get_tablet_stats_version is not 2, skip test")
+            return
+        }
+
+        def intervalConfigResult = sql_return_maparray """ ADMIN SHOW FRONTEND CONFIG LIKE '%tablet_stat_update_interval_second%' """
+        assertTrue(intervalConfigResult.size() > 0, "Expected to find tablet_stat_update_interval_second config")
+        def statUpdateInterval = intervalConfigResult[0].Value.toInteger()
+        logger.info("tablet_stat_update_interval_second = ${statUpdateInterval}")
+
+        sql """
+            CREATE TABLE IF NOT EXISTS ${tbl} (
+                id INT NOT NULL,
+                name VARCHAR(50) NOT NULL,
+                value INT NOT NULL
+            )
+            ENGINE = olap
+            DUPLICATE KEY(id)
+            DISTRIBUTED BY HASH(id) BUCKETS 1
+            PROPERTIES (
+                "replication_num" = "1"
+            );
+        """
+
+        def result = sql_return_maparray """ select * from ${tbl} """
+        assertEquals(0, result.size())
+
+        sql """ INSERT INTO ${tbl} VALUES (1, 'initial', 100) """
+        result = sql_return_maparray """ select * from ${tbl} where id = 1 """
+        assertEquals(1, result.size())
+
+        def tablets = sql_return_maparray """ show tablets from ${tbl} """
+        assertTrue(tablets.size() > 0, "Expected at least one tablet")
+        def tabletIds = tablets.collect { it.TabletId }.join(",")
+        logger.info("Tablet IDs: ${tabletIds}")
+
+        def initialRowCount = tablets[0].RowCount
+        logger.info("Initial RowCount: ${initialRowCount}")
+
+        GetDebugPoint().enableDebugPointForAllFEs("FE.CloudTabletStatMgr.addActiveTablets.ignore.tablets", [value: tabletIds])
+
+        sql """ INSERT INTO ${tbl} VALUES (2, 'test', 200) """
+        result = sql_return_maparray """ select * from ${tbl} where id = 2 """
+        assertEquals(1, result.size(), "Data should be visible in query")
+
+        def tabletsAfterInsert = sql_return_maparray """ show tablets from ${tbl} """
+        def rowCountAfterInsert = tabletsAfterInsert[0].RowCount
+        logger.info("RowCount after insert (before sync): ${rowCountAfterInsert}")
+
+        GetDebugPoint().disableDebugPointForAllFEs("FE.CloudTabletStatMgr.addActiveTablets.ignore.tablets")
+
+        def maxWaitMs = (statUpdateInterval * 3 + 5) * 1000
+        def intervalMs = 1000
+        def startTime = System.currentTimeMillis()
+        def updated = false
+        while (true) {
+            def tabletsNow = sql_return_maparray """ show tablets from ${tbl} """
+            def rowCountNow = tabletsNow[0].RowCount.toInteger()
+            logger.info("RowCount get: ${rowCountNow}")
+            if (rowCountNow == 2) {
+                logger.info("RowCount updated: ${rowCountNow}")
+                updated = true
+                break
+            }
+            if (System.currentTimeMillis() - startTime > maxWaitMs) {
+                throw new IllegalStateException("Timeout waiting for tablet stat update. Waited ${maxWaitMs}ms, RowCount still ${rowCountNow}")
+            }
+            Thread.sleep(intervalMs)
+        }
+        assertTrue(updated, "Tablet stat should eventually update after debug point disabled")
+        tabletsAfterInsert = sql_return_maparray """ show tablets from ${tbl} """
+        rowCountAfterInsert = tabletsAfterInsert[0].RowCount.toInteger()
+        logger.info("RowCount after stat sync: ${rowCountAfterInsert}")
+        assertEquals(2, rowCountAfterInsert, "RowCount should reflect the new row after stat sync")
+
+        sql """ INSERT INTO ${tbl} VALUES (3, 'force_sync', 300) """
+        def tabletsBeforeSync = sql_return_maparray """ show tablets from ${tbl} """
+        def localDataSizeBeforeSync = tabletsBeforeSync[0].LocalDataSize
+        def rowCountBeforeSync = tabletsBeforeSync[0].RowCount
+        logger.info("LocalDataSize before force sync: ${localDataSizeBeforeSync}, RowCount: ${rowCountBeforeSync}")
+
+        sql """ set cloud_force_sync_tablet_stats = true """
+        def tabletsAfterForceSync = sql_return_maparray """ show tablets from ${tbl} """
+        def localDataSizeAfterForceSync = tabletsAfterForceSync[0].LocalDataSize
+        def rowCountAfterForceSync = tabletsAfterForceSync[0].RowCount
+        logger.info("LocalDataSize after force sync: ${localDataSizeAfterForceSync}, RowCount: ${rowCountAfterForceSync}")

Review Comment:
   **Test weakness: force-sync section doesn't assert the immediate effect**
   
   After setting `cloud_force_sync_tablet_stats = true` and reading tablet stats (lines 120-124), the test only logs the results but never asserts that `rowCountAfterForceSync == 3`. It then falls through to a polling loop that waits for eventual consistency — which is the same verification pattern as the non-force-sync path above.
   
   To actually verify that force sync works (i.e., provides an *immediate* update), consider adding:
   ```groovy
   assertEquals(3, rowCountAfterForceSync.toInteger(), "RowCount should be updated immediately after force sync")
   ```
   Otherwise this section tests nothing beyond what the first section already tests.



##########
regression-test/suites/cloud_p0/version/test_version_syncer.groovy:
##########
@@ -0,0 +1,105 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_version_syncer", "nonConcurrent") {
+    if (!isCloudMode()) {
+        return
+    }
+
+    def tbl = 'test_version_syncer_tbl'
+    sql """ DROP TABLE IF EXISTS ${tbl} """
+
+    def enableSyncerConfig = sql_return_maparray """ ADMIN SHOW FRONTEND CONFIG LIKE '%cloud_enable_version_syncer%' """
+    assertTrue(enableSyncerConfig.size() > 0, "Expected to find cloud_enable_version_syncer config")
+    def enableSyncer = enableSyncerConfig[0].Value.toBoolean()
+    logger.info("cloud_enable_version_syncer = ${enableSyncer}")
+    if (!enableSyncer) {
+        logger.info("FE.CloudGlobalTransactionMgr.updateVersion.disabled")
+        return
+    }
+    def configResult = sql_return_maparray """ ADMIN SHOW FRONTEND CONFIG LIKE '%cloud_version_syncer_interval_second%' """
+    assertTrue(configResult.size() > 0, "Expected to find cloud_version_syncer_interval_second config")
+    def syncerInterval = configResult[0].Value.toInteger()
+    logger.info("cloud_version_syncer_interval_second = ${syncerInterval}")
+
+    onFinish {
+        GetDebugPoint().clearDebugPointsForAllFEs()
+    }
+
+    sql """
+            CREATE TABLE IF NOT EXISTS ${tbl} (
+                id INT NOT NULL,
+                name VARCHAR(50) NOT NULL,
+                value INT NOT NULL
+            )
+            DUPLICATE KEY(id)
+            DISTRIBUTED BY HASH(id) BUCKETS 1
+            PROPERTIES (
+                "replication_num" = "1"
+            );
+    """
+
+    def result = sql_return_maparray """ select * from ${tbl} """
+    assertEquals(0, result.size())
+
+    sql """ INSERT INTO ${tbl} VALUES (1, 'initial', 100) """
+    result = sql_return_maparray """ select * from ${tbl} where id = 1 """
+    assertEquals(1, result.size())
+
+    DebugPoint.enableDebugPointForAllFEs('FE.CloudGlobalTransactionMgr.updateVersion.disabled')
+
+    sql """ INSERT INTO ${tbl} VALUES (2, 'test', 200) """
+    result = sql_return_maparray """ select * from ${tbl} where id = 2 """
+    assertEquals(0, result.size(), "Data should not be visible before version sync")
+
+    def maxWaitMs = (syncerInterval * 2 + 5) * 1000
+    def intervalMs = 1000
+    def startTime = System.currentTimeMillis()
+    def found = false
+    while (true) {
+        result = sql_return_maparray """ select * from ${tbl} where id = 2 """
+        if (result.size() == 1) {
+            assertEquals('test', result[0].name)
+            assertEquals(200, result[0].value)
+            found = true
+            break
+        }
+        if (System.currentTimeMillis() - startTime > maxWaitMs) {
+            throw new IllegalStateException("Timeout waiting for data to be visible. Waited ${maxWaitMs}ms")
+        }
+        Thread.sleep(intervalMs)
+    }
+    assertTrue(found, "Data should eventually be visible after version syncer runs")
+
+    result = sql_return_maparray """ select * from ${tbl} order by id """
+    assertEquals(2, result.size())
+
+    // Test cloud_force_sync_version session variable
+    // Step 1: Insert data with version sync disabled, query should return no data
+    sql """ INSERT INTO ${tbl} VALUES (3, 'force_sync', 300) """
+    result = sql_return_maparray """ select * from ${tbl} where id = 3 """
+    assertEquals(0, result.size(), "Data should not be visible before force sync")
+
+    // Step 2: Enable session variable cloud_force_sync_version = true and execute show partitions
+    sql """ set cloud_force_sync_version = true """
+    def showPartitionsResult = sql_return_maparray """ show partitions from ${tbl} """

Review Comment:
   **Minor: `cloud_force_sync_version` is never reset to `false`**
   
   Line 98 sets `cloud_force_sync_version = true` but it is never set back to `false`. The tablet stat syncer test (line 125) properly resets its equivalent variable. While session-scoped variables are cleaned up when the session ends, it's good practice to reset it for consistency and to avoid affecting subsequent queries in the same session.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](cloud) add get_version and get_tablet_stats case [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #61915:
URL: https://github.com/apache/doris/pull/61915#issuecomment-4159602923

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](cloud) add get_version and get_tablet_stats case [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61915:
URL: https://github.com/apache/doris/pull/61915#issuecomment-4161076895

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 169593 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 5103a9a4a98863363863271b7d81ea2ddca4d037, data reload: false
   
   query5	4345	629	490	490
   query6	332	220	206	206
   query7	4221	465	258	258
   query8	341	244	220	220
   query9	8697	2682	2702	2682
   query10	531	379	335	335
   query11	7002	5133	4880	4880
   query12	188	131	123	123
   query13	1278	465	380	380
   query14	5750	3741	3483	3483
   query14_1	2854	2846	2843	2843
   query15	211	194	177	177
   query16	988	460	451	451
   query17	1123	721	623	623
   query18	2452	462	362	362
   query19	225	218	190	190
   query20	133	127	126	126
   query21	210	136	111	111
   query22	13137	14097	15112	14097
   query23	16789	16171	16151	16151
   query23_1	16264	16288	16129	16129
   query24	7579	1688	1297	1297
   query24_1	1317	1271	1277	1271
   query25	568	477	418	418
   query26	1419	303	167	167
   query27	3379	585	306	306
   query28	4652	1822	1790	1790
   query29	818	558	473	473
   query30	299	225	192	192
   query31	1011	959	880	880
   query32	82	69	69	69
   query33	516	325	282	282
   query34	891	877	525	525
   query35	633	688	588	588
   query36	1118	1099	936	936
   query37	130	92	85	85
   query38	2968	2889	2848	2848
   query39	857	825	816	816
   query39_1	799	784	792	784
   query40	231	148	137	137
   query41	64	66	57	57
   query42	257	254	251	251
   query43	250	248	217	217
   query44	
   query45	193	199	185	185
   query46	895	971	603	603
   query47	2103	2128	2073	2073
   query48	306	320	223	223
   query49	624	439	389	389
   query50	699	280	214	214
   query51	4030	4072	4045	4045
   query52	281	272	254	254
   query53	295	325	277	277
   query54	304	270	257	257
   query55	92	85	80	80
   query56	304	318	313	313
   query57	1927	1725	1700	1700
   query58	292	276	268	268
   query59	2797	2966	2756	2756
   query60	363	338	325	325
   query61	162	153	148	148
   query62	638	597	556	556
   query63	309	291	275	275
   query64	5048	1290	997	997
   query65	
   query66	1456	450	358	358
   query67	24384	24378	24245	24245
   query68	
   query69	412	305	279	279
   query70	938	976	879	879
   query71	330	290	294	290
   query72	2802	2736	2491	2491
   query73	539	539	315	315
   query74	9681	9643	9471	9471
   query75	2889	2739	2461	2461
   query76	2282	1029	669	669
   query77	354	379	315	315
   query78	11127	11280	10540	10540
   query79	1068	844	584	584
   query80	1465	625	526	526
   query81	531	258	230	230
   query82	1336	152	120	120
   query83	347	271	235	235
   query84	251	121	99	99
   query85	1121	550	480	480
   query86	395	347	293	293
   query87	3179	3114	3013	3013
   query88	3572	2641	2583	2583
   query89	431	367	342	342
   query90	1773	176	165	165
   query91	183	167	133	133
   query92	83	74	73	73
   query93	929	853	498	498
   query94	538	312	316	312
   query95	587	340	381	340
   query96	647	517	226	226
   query97	2460	2466	2435	2435
   query98	244	232	228	228
   query99	1024	1009	928	928
   Total cold run time: 252156 ms
   Total hot run time: 169593 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org