You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Gabriel39 (via GitHub)" <gi...@apache.org> on 2026/10/08 07:06:30 UTC

[PR] [feature](lance) Add synchronous index prewarm SQL (branch-4.1) [doris]

Gabriel39 opened a new pull request, #68775:
URL: https://github.com/apache/doris/pull/68775

   ### What problem does this PR solve?
   
   Related issue: #68692 (M2); parent tracker: #66340.
   Related PR: #68698 (merged dependency baseline).
   
   Lance index reads currently enter the BE cache during foreground queries. This adds synchronous SQL to load a named logical index into the same shared Lance session before queries run:
   
   ```sql
   WARM UP INDEX vector_idx ON lake_catalog.demo.items;
   WARM UP INDEX vector_idx ON lake_catalog.demo.items WITH COMPUTE GROUP search_group;
   ```
   
   The FE checks ADMIN and table SELECT privileges, plus compute-group USAGE in cloud mode. It resolves the normal catalog access path (including vended credentials), pins one dataset version and eligible BE set, and dispatches at most eight RPCs at a time. Success requires every target to acknowledge the pinned snapshot. The result contains table, index, version, backend count, and elapsed milliseconds.
   
   The BE uses the existing bounded heavy-work pool and allows one concurrent prewarm operation per shared session. It opens the pinned dataset through `LanceSessionManager` and delegates to the upstream synchronous C API. Missing indexes, storage failures, unavailable or older BEs, and missing acknowledgements fail explicitly; provider error details and storage credentials are excluded from prewarm errors.
   
   The statement observes the SQL timeout and KILL/connection cancellation while waiting. Cancelling a caller does not guarantee immediate interruption of running SDK IO or undo successful cache fills. Cached entries remain evictable. Explicit compute-group selection is supported in cloud mode; otherwise the current resource-tag execution context is used.
   
   ### Release note
   
   Add `WARM UP INDEX ... ON ... [WITH COMPUTE GROUP ...]` for synchronous Lance index prewarm on the eligible query backends.
   
   ### Validation
   
   - Passed 12 focused FE tests covering parsing, permissions, default/explicit compute groups, snapshot/index resolution, target selection, bounded fan-out, acknowledgements, errors, timeout, and cancellation.
   - FE Checkstyle: passed with zero violations.
   - clang-format 16: passed for all five affected C++ source/header files.
   - Affected C++ translation units passed syntax compilation against the pinned Lance-C headers.
   - Real pinned Lance-C SDK smoke test passed: scalar-index prewarm, cache hits through a separately opened shared-session handle, and historical-snapshot prewarm after dropping the current index.
   - Added BE manager tests for invalid inputs, sanitized errors, shared-session cache reuse, and fixed snapshots. Full BE unit-test execution was blocked locally because the available third-party bundle is missing `libsimdutf.a`; these tests are not reported as passing.
   - Added external regression coverage for indexed query equivalence, synchronous results, retry, explicit cloud groups, permission denial, and one-target failure injection. Groovy syntax validation passed; the external suite has not been run against a cluster locally.
   
   M3's supported-format/catalog matrix, broader lifecycle/performance verification, and bilingual website documentation remain tracked in #68692. This PR does not close that issue.
   
   ### Check List (For Author)
   
   - Test
       - [x] Regression test (added; cluster execution pending)
       - [x] Unit Test (FE passed; BE execution blocked as described above)
       - [x] Manual test (SDK smoke test described above)
   - Behavior changed:
       - [x] Yes: new synchronous administrative SQL command.
   - Does this need documentation?
       - [x] Yes: tracked separately in #68692 M3.
   
   ### Check List (For Reviewer who merge this PR)
   
   - [ ] Confirm the release note
   - [ ] Confirm test cases
   - [ ] Confirm document
   - [ ] Add branch pick label
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](lance) Add synchronous index prewarm SQL (branch-4.1) [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68775:
URL: https://github.com/apache/doris/pull/68775#issuecomment-6062932427

   # Cloud UT Coverage Report
   Increment line coverage ` ` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/a6be4b70adf2f42ca6f0516eae311896333ca103_a6be4b70adf2f42ca6f0516eae311896333ca103_cloud/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/a6be4b70adf2f42ca6f0516eae311896333ca103_a6be4b70adf2f42ca6f0516eae311896333ca103_cloud/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 78.49% (1960/2497) |
   | Line Coverage     | 65.46% (35325/53964) |
   | Region Coverage   | 53.09% (34002/64043) |
   | Branch Coverage   | 55.86% (10481/18762) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](lance) Add synchronous index prewarm SQL (branch-4.1) [doris]

Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68775:
URL: https://github.com/apache/doris/pull/68775#issuecomment-6061913502

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](lance) Add synchronous index prewarm SQL (branch-4.1) [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68775:
URL: https://github.com/apache/doris/pull/68775#discussion_r4220410219


##########
be/test/service/lance_index_prewarm_test.cpp:
##########
@@ -0,0 +1,123 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+
+#include <chrono>
+#include <future>
+#include <utility>
+
+#include "common/config.h"
+#include "load/channel/load_stream_mgr.h"
+#include "runtime/exec_env.h"
+#include "service/internal_service.h"
+
+namespace doris {
+namespace {
+
+class PrewarmClosure : public google::protobuf::Closure {
+public:
+    void Run() override { completed.set_value(); }
+    std::promise<void> completed;
+};
+
+class LanceIndexPrewarmServiceTest : public testing::Test {
+protected:
+    void SetUp() override {
+        auto heavy = std::exchange(config::brpc_heavy_work_pool_threads, 1);
+        auto light = std::exchange(config::brpc_light_work_pool_threads, 1);
+        auto peer = std::exchange(config::brpc_peer_fetch_pool_threads, 1);
+        auto flight = std::exchange(config::brpc_arrow_flight_work_pool_threads, 1);
+        auto* env = ExecEnv::GetInstance();
+        previous_load_mgr = std::move(env->_load_stream_mgr);
+        env->_load_stream_mgr = std::make_unique<LoadStreamMgr>(1);

Review Comment:
   [P1] Make the new BE service test compile. This fixture directly moves and assigns private `ExecEnv::_load_stream_mgr` here and later accesses protected `PInternalService` pools, even though it only derives from `testing::Test`. The file is globbed into `doris_be_test`, so that target fails compilation before these tests can run. Use public test hooks or suitable friend/test subclass access for both objects.



##########
regression-test/suites/external_table_p0/lance/test_lance_index_prewarm.groovy:
##########
@@ -0,0 +1,143 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import groovy.json.JsonSlurper
+
+import org.apache.doris.regression.util.DebugPoint
+import org.apache.doris.regression.util.NodeType
+
+suite("test_lance_index_prewarm", "p0,external") {
+    if (!"true".equalsIgnoreCase(context.config.otherConfigs.get("enableIcebergTest"))) {
+        logger.info("Lance prewarm requires the Iceberg MinIO fixtures")
+        return
+    }
+    String externalEnvIp = context.config.otherConfigs.get("externalEnvIp")
+    String minioPort = context.config.otherConfigs.get("iceberg_minio_port")
+    String catalogName = "test_lance_index_prewarm"
+    String tableName = "${catalogName}.doris.vs_ivf_flat_f32"
+    sql """DROP CATALOG IF EXISTS `${catalogName}`"""
+    sql """
+        CREATE CATALOG `${catalogName}` PROPERTIES (
+            "type" = "lance",
+            "lance.catalog.type" = "filesystem",
+            "warehouse" = "s3://warehouse/lance",
+            "s3.endpoint" = "http://${externalEnvIp}:${minioPort}",
+            "s3.access_key" = "admin",
+            "s3.secret_key" = "password",
+            "s3.region" = "us-east-1",
+            "use_path_style" = "true"
+        )
+    """
+    sql "SET enable_file_scanner_v2 = true"
+    def entries = sql """SELECT IndexName, DatasetVersion FROM lance_index_entries("table"="${tableName}")"""
+    assertFalse(entries.isEmpty())
+    String indexName = entries[0][0]
+    String statement = "WARM UP INDEX `${indexName}` ON ${tableName}"
+    String query = """
+        SELECT row_id, _distance FROM vector_search(
+            "table"="${tableName}", "column"="embedding",
+            "query_vector"="[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15]",
+            "top_k"="10", "metric"="l2", "nprobes"="4", "use_index"="true")
+        ORDER BY _distance, row_id
+    """
+    def before = sql(query)
+    def warm = sql(statement)
+    assertEquals(1, warm.size())
+    assertEquals(indexName, warm[0][1])
+    // Physical entries record the index build version; prewarm pins the current dataset snapshot.
+    assertTrue(warm[0][2].toLong() >= entries[0][1].toLong())
+    assertTrue(warm[0][3].toInteger() > 0)
+    assertEquals(before, sql(query), "Prewarm must preserve query results")
+    def retry = sql(statement)
+    assertEquals(warm[0][2], retry[0][2], "Prewarm must not change the dataset version")
+    assertEquals(warm[0][3], retry[0][3], "Retry must cover the same eligible backend set")
+
+    // Binary PREPARE and EXECUTE must agree on the result schema, including repeated executions.
+    // Force server preparation even when the JDBC URL defaults to client-side emulation.
+    def prepared = context.getConnection().unwrap(com.mysql.cj.jdbc.JdbcConnection)
+            .serverPrepareStatement(statement)
+    try {
+        assertTrue(prepared instanceof com.mysql.cj.jdbc.ServerPreparedStatement)
+        assertEquals(5, prepared.getMetaData().getColumnCount())
+        assertEquals("DatasetVersion", prepared.getMetaData().getColumnName(3))
+        for (int execution = 0; execution < 2; execution++) {
+            def result = exec(prepared)
+            assertEquals(1, result.size())
+            assertEquals(indexName, result[0][1])
+            assertEquals(warm[0][2].toString(), result[0][2].toString())
+            assertEquals(warm[0][3].toString(), result[0][3].toString())
+        }
+    } finally {
+        prepared.close()
+    }
+
+    test {
+        sql "WARM UP INDEX missing_index ON ${tableName}"
+        exception "does not exist in dataset version"
+    }
+    if (isCloudMode()) {
+        def groups = sql "SHOW CLUSTERS"
+        assertFalse(groups.isEmpty())
+        def explicit = sql "${statement} WITH COMPUTE GROUP `${groups[0][0]}`"

Review Comment:
   [P2] Choose an available compute group for this success assertion. `SHOW CLUSTERS` is sorted by name and includes groups with zero or unavailable backends, so `groups[0][0]` can differ from the healthy current group used by the preceding successful prewarm. In that valid multi-group state, `selectBackends()` throws `No available backends` here and the assertion fails. Select the current group from `is_current`, or choose a group with an eligible BE.



##########
regression-test/suites/external_table_p0/lance/test_lance_index_prewarm.groovy:
##########
@@ -0,0 +1,143 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import groovy.json.JsonSlurper
+
+import org.apache.doris.regression.util.DebugPoint
+import org.apache.doris.regression.util.NodeType
+
+suite("test_lance_index_prewarm", "p0,external") {
+    if (!"true".equalsIgnoreCase(context.config.otherConfigs.get("enableIcebergTest"))) {
+        logger.info("Lance prewarm requires the Iceberg MinIO fixtures")
+        return
+    }
+    String externalEnvIp = context.config.otherConfigs.get("externalEnvIp")
+    String minioPort = context.config.otherConfigs.get("iceberg_minio_port")
+    String catalogName = "test_lance_index_prewarm"
+    String tableName = "${catalogName}.doris.vs_ivf_flat_f32"
+    sql """DROP CATALOG IF EXISTS `${catalogName}`"""
+    sql """
+        CREATE CATALOG `${catalogName}` PROPERTIES (
+            "type" = "lance",
+            "lance.catalog.type" = "filesystem",
+            "warehouse" = "s3://warehouse/lance",
+            "s3.endpoint" = "http://${externalEnvIp}:${minioPort}",
+            "s3.access_key" = "admin",
+            "s3.secret_key" = "password",
+            "s3.region" = "us-east-1",
+            "use_path_style" = "true"
+        )
+    """
+    sql "SET enable_file_scanner_v2 = true"
+    def entries = sql """SELECT IndexName, DatasetVersion FROM lance_index_entries("table"="${tableName}")"""
+    assertFalse(entries.isEmpty())
+    String indexName = entries[0][0]
+    String statement = "WARM UP INDEX `${indexName}` ON ${tableName}"
+    String query = """
+        SELECT row_id, _distance FROM vector_search(
+            "table"="${tableName}", "column"="embedding",
+            "query_vector"="[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15]",
+            "top_k"="10", "metric"="l2", "nprobes"="4", "use_index"="true")
+        ORDER BY _distance, row_id
+    """
+    def before = sql(query)
+    def warm = sql(statement)
+    assertEquals(1, warm.size())
+    assertEquals(indexName, warm[0][1])
+    // Physical entries record the index build version; prewarm pins the current dataset snapshot.
+    assertTrue(warm[0][2].toLong() >= entries[0][1].toLong())
+    assertTrue(warm[0][3].toInteger() > 0)
+    assertEquals(before, sql(query), "Prewarm must preserve query results")
+    def retry = sql(statement)
+    assertEquals(warm[0][2], retry[0][2], "Prewarm must not change the dataset version")
+    assertEquals(warm[0][3], retry[0][3], "Retry must cover the same eligible backend set")
+
+    // Binary PREPARE and EXECUTE must agree on the result schema, including repeated executions.
+    // Force server preparation even when the JDBC URL defaults to client-side emulation.
+    def prepared = context.getConnection().unwrap(com.mysql.cj.jdbc.JdbcConnection)
+            .serverPrepareStatement(statement)
+    try {
+        assertTrue(prepared instanceof com.mysql.cj.jdbc.ServerPreparedStatement)
+        assertEquals(5, prepared.getMetaData().getColumnCount())
+        assertEquals("DatasetVersion", prepared.getMetaData().getColumnName(3))
+        for (int execution = 0; execution < 2; execution++) {
+            def result = exec(prepared)
+            assertEquals(1, result.size())
+            assertEquals(indexName, result[0][1])
+            assertEquals(warm[0][2].toString(), result[0][2].toString())
+            assertEquals(warm[0][3].toString(), result[0][3].toString())
+        }
+    } finally {
+        prepared.close()
+    }
+
+    test {
+        sql "WARM UP INDEX missing_index ON ${tableName}"
+        exception "does not exist in dataset version"
+    }
+    if (isCloudMode()) {
+        def groups = sql "SHOW CLUSTERS"
+        assertFalse(groups.isEmpty())
+        def explicit = sql "${statement} WITH COMPUTE GROUP `${groups[0][0]}`"
+        assertTrue(explicit[0][3].toInteger() > 0)
+    }
+
+    // Fail one target only: successful prewarm on other BEs must not hide the failure.
+    def backends = sql_return_maparray "SHOW BACKENDS"
+    def target = backends.find {

Review Comment:
   [P2] Choose the injected backend from the prewarm-eligible set. This filter accepts an alive BE with `SystemDecommissioned=false` even while it is decommissioning, but `needLoadAvailable()` excludes decommissioning BEs. If `find` picks that BE, the command can warm another BE and succeed without hitting the debug point, so the failure assertion fails independently of the cloud tag-key fix. Use the same eligibility policy as `LanceIndexPrewarm.selectBackends()`.



##########
regression-test/suites/external_table_p0/lance/test_lance_index_prewarm.groovy:
##########
@@ -0,0 +1,143 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import groovy.json.JsonSlurper
+
+import org.apache.doris.regression.util.DebugPoint
+import org.apache.doris.regression.util.NodeType
+
+suite("test_lance_index_prewarm", "p0,external") {
+    if (!"true".equalsIgnoreCase(context.config.otherConfigs.get("enableIcebergTest"))) {
+        logger.info("Lance prewarm requires the Iceberg MinIO fixtures")
+        return
+    }
+    String externalEnvIp = context.config.otherConfigs.get("externalEnvIp")
+    String minioPort = context.config.otherConfigs.get("iceberg_minio_port")
+    String catalogName = "test_lance_index_prewarm"
+    String tableName = "${catalogName}.doris.vs_ivf_flat_f32"
+    sql """DROP CATALOG IF EXISTS `${catalogName}`"""
+    sql """
+        CREATE CATALOG `${catalogName}` PROPERTIES (
+            "type" = "lance",
+            "lance.catalog.type" = "filesystem",
+            "warehouse" = "s3://warehouse/lance",
+            "s3.endpoint" = "http://${externalEnvIp}:${minioPort}",
+            "s3.access_key" = "admin",
+            "s3.secret_key" = "password",
+            "s3.region" = "us-east-1",
+            "use_path_style" = "true"
+        )
+    """
+    sql "SET enable_file_scanner_v2 = true"
+    def entries = sql """SELECT IndexName, DatasetVersion FROM lance_index_entries("table"="${tableName}")"""
+    assertFalse(entries.isEmpty())
+    String indexName = entries[0][0]
+    String statement = "WARM UP INDEX `${indexName}` ON ${tableName}"
+    String query = """
+        SELECT row_id, _distance FROM vector_search(
+            "table"="${tableName}", "column"="embedding",
+            "query_vector"="[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15]",
+            "top_k"="10", "metric"="l2", "nprobes"="4", "use_index"="true")
+        ORDER BY _distance, row_id
+    """
+    def before = sql(query)
+    def warm = sql(statement)
+    assertEquals(1, warm.size())
+    assertEquals(indexName, warm[0][1])
+    // Physical entries record the index build version; prewarm pins the current dataset snapshot.
+    assertTrue(warm[0][2].toLong() >= entries[0][1].toLong())
+    assertTrue(warm[0][3].toInteger() > 0)
+    assertEquals(before, sql(query), "Prewarm must preserve query results")
+    def retry = sql(statement)
+    assertEquals(warm[0][2], retry[0][2], "Prewarm must not change the dataset version")
+    assertEquals(warm[0][3], retry[0][3], "Retry must cover the same eligible backend set")
+
+    // Binary PREPARE and EXECUTE must agree on the result schema, including repeated executions.
+    // Force server preparation even when the JDBC URL defaults to client-side emulation.
+    def prepared = context.getConnection().unwrap(com.mysql.cj.jdbc.JdbcConnection)
+            .serverPrepareStatement(statement)
+    try {
+        assertTrue(prepared instanceof com.mysql.cj.jdbc.ServerPreparedStatement)
+        assertEquals(5, prepared.getMetaData().getColumnCount())
+        assertEquals("DatasetVersion", prepared.getMetaData().getColumnName(3))
+        for (int execution = 0; execution < 2; execution++) {
+            def result = exec(prepared)
+            assertEquals(1, result.size())
+            assertEquals(indexName, result[0][1])
+            assertEquals(warm[0][2].toString(), result[0][2].toString())
+            assertEquals(warm[0][3].toString(), result[0][3].toString())
+        }
+    } finally {
+        prepared.close()
+    }
+
+    test {
+        sql "WARM UP INDEX missing_index ON ${tableName}"
+        exception "does not exist in dataset version"
+    }
+    if (isCloudMode()) {
+        def groups = sql "SHOW CLUSTERS"
+        assertFalse(groups.isEmpty())
+        def explicit = sql "${statement} WITH COMPUTE GROUP `${groups[0][0]}`"
+        assertTrue(explicit[0][3].toInteger() > 0)
+    }
+
+    // Fail one target only: successful prewarm on other BEs must not hide the failure.
+    def backends = sql_return_maparray "SHOW BACKENDS"
+    def target = backends.find {
+        def status = new JsonSlurper().parseText(it.Status)
+        it.Alive.toString().equalsIgnoreCase("true") &&
+                !it.SystemDecommissioned.toString().equalsIgnoreCase("true") &&
+                !status.isQueryDisabled && !status.isLoadDisabled
+    }
+    assertNotNull(target)
+    String failureStatement = statement
+    if (isCloudMode()) {
+        String targetGroup = new JsonSlurper().parseText(target.Tag).cloud_cluster_name

Review Comment:
   [P2] Read the displayed compute group tag in the cloud failure test. `SHOW BACKENDS` exposes this field as `compute_group_name` (`Backend.getTagMapString()` renames it), so `cloud_cluster_name` is null here. The test sends a compute-group clause with a null name and fails before reaching the injected BE prewarm error; use `compute_group_name`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](lance) Add synchronous index prewarm SQL (branch-4.1) [doris]

Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68775:
URL: https://github.com/apache/doris/pull/68775#issuecomment-6059523638

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](lance) Add synchronous index prewarm SQL (branch-4.1) [doris]

Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68775:
URL: https://github.com/apache/doris/pull/68775#issuecomment-6054513410

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](lance) Add synchronous index prewarm SQL (branch-4.1) [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68775:
URL: https://github.com/apache/doris/pull/68775#issuecomment-6054511688

   
   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
   
   Please clearly describe your PR:
   1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
   2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
   3. What features were added. Why was this function added?
   4. Which code was refactored and why was this part of the code refactored?
   5. Which functions were optimized and what is the difference before and after the optimization?
       


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](lance) Extend WARM UP SELECT with index prewarm (branch-4.1) [doris]

Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68775:
URL: https://github.com/apache/doris/pull/68775#issuecomment-6076017814

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](lance) Add synchronous index prewarm SQL (branch-4.1) [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68775:
URL: https://github.com/apache/doris/pull/68775#discussion_r4216115261


##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/WarmUpIndexCommand.java:
##########
@@ -0,0 +1,54 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.commands;
+
+import org.apache.doris.datasource.lance.LanceIndexPrewarm;
+import org.apache.doris.info.TableNameInfo;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.StmtExecutor;
+
+/** Synchronously fills the query sessions on the selected backends. */
+public class WarmUpIndexCommand extends Command {
+    private final TableNameInfo table;
+    private final String indexName;
+    private final String computeGroup;
+    private volatile boolean cancelled;
+
+    public WarmUpIndexCommand(TableNameInfo table, String indexName, String computeGroup) {
+        super(PlanType.WARM_UP_INDEX_COMMAND);
+        this.table = table;
+        this.indexName = indexName;
+        this.computeGroup = computeGroup;
+    }
+
+    @Override
+    public void run(ConnectContext ctx, StmtExecutor executor) throws Exception {

Review Comment:
   [P2] Expose the prewarm result columns to prepared clients. `COM_STMT_PREPARE` wraps this command in `PrepareCommand`, which reads `Command.getResultSetMetaData()`; because this class inherits the empty default, the prepare response advertises zero columns. `COM_STMT_EXECUTE` then runs this method and sends the five columns in `LanceIndexPrewarm.RESULT_META`, so binary prepared clients receive a result schema different from the one they prepared. Override the metadata hook with the same five columns and cover the binary prepare/execute path.



##########
be/src/service/internal_service.cpp:
##########
@@ -811,6 +813,60 @@ void PInternalService::outfile_write_success(google::protobuf::RpcController* co
     }
 }
 
+void PInternalService::prewarm_lance_index(google::protobuf::RpcController* controller,
+                                           const PLanceIndexPrewarmRequest* request,
+                                           PLanceIndexPrewarmResponse* response,
+                                           google::protobuf::Closure* done) {
+    const auto received = std::chrono::steady_clock::now();
+    bool offered = _heavy_work_pool.try_offer([request, response, done, received]() {
+        brpc::ClosureGuard closure_guard(done);
+        Status status;
+        try {
+            auto run = [&]() -> Status {
+                DBUG_EXECUTE_IF("PInternalService.prewarm_lance_index.fail", {
+                    return Status::InternalError("Injected Lance index prewarm failure");
+                });
+                const auto queued_ms = std::chrono::duration_cast<std::chrono::milliseconds>(
+                                               std::chrono::steady_clock::now() - received)
+                                               .count();
+                if (!request->has_timeout_ms() || request->timeout_ms() <= queued_ms) {
+                    return Status::TimedOut("Lance index prewarm expired before execution");
+                }
+                // C strings cannot represent embedded NUL bytes; reject rather than silently
+                // opening a different path/index or truncating a vended credential.
+                if (request->dataset_uri().find('\0') != std::string::npos ||
+                    request->index_name().find('\0') != std::string::npos) {
+                    return Status::InvalidArgument("Invalid Lance prewarm URI or index name");
+                }
+                std::vector<const char*> options;
+                options.reserve(request->storage_options_size() * 2 + 1);
+                for (const auto& [key, value] : request->storage_options()) {
+                    if (key.find('\0') != std::string::npos ||
+                        value.find('\0') != std::string::npos) {
+                        return Status::InvalidArgument("Invalid Lance prewarm storage option");
+                    }
+                    options.push_back(key.c_str());
+                    options.push_back(value.c_str());
+                }
+                options.push_back(nullptr);
+                RETURN_IF_ERROR(format::lance::LanceSessionManager::instance().prewarm_index(

Review Comment:
   [P2] Bound prewarm work after it leaves the RPC queue. `timeout_ms` is checked only against queue time before this synchronous SDK call. If object-store open or index prewarm runs beyond the FE deadline (or the user cancels), the FE returns but this heavy-pool worker and `LanceSessionManager`'s process-wide prewarm mutex remain occupied; every retry on this BE is rejected until the old call finishes. Propagate a deadline/cancellation to the SDK work or isolate and bound abandoned work so a timed-out request cannot monopolize this path.



##########
fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceIndexPrewarm.java:
##########
@@ -0,0 +1,226 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.lance;
+
+import org.apache.doris.analysis.ResourceTypeEnum;
+import org.apache.doris.analysis.TableName;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.ScalarType;
+import org.apache.doris.catalog.TableIf;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.Config;
+import org.apache.doris.common.UserException;
+import org.apache.doris.datasource.CatalogIf;
+import org.apache.doris.datasource.lance.metadata.LanceTableMetadata;
+import org.apache.doris.info.TableNameInfo;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.proto.InternalService.PLanceIndexPrewarmRequest;
+import org.apache.doris.proto.InternalService.PLanceIndexPrewarmResponse;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.ShowResultSet;
+import org.apache.doris.qe.ShowResultSetMetaData;
+import org.apache.doris.qe.StmtExecutor;
+import org.apache.doris.resource.computegroup.ComputeGroup;
+import org.apache.doris.rpc.BackendServiceProxy;
+import org.apache.doris.system.Backend;
+import org.apache.doris.system.BeSelectionPolicy;
+import org.apache.doris.thrift.TNetworkAddress;
+import org.apache.doris.thrift.TStatusCode;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.BooleanSupplier;
+
+/** Pins table access and coordinates bounded synchronous prewarm RPCs. */
+public final class LanceIndexPrewarm {
+    static final int MAX_IN_FLIGHT = 8;
+    private static final ShowResultSetMetaData RESULT_META = ShowResultSetMetaData.builder()
+            .addColumn(new Column("Table", ScalarType.createStringType()))
+            .addColumn(new Column("Index", ScalarType.createStringType()))
+            .addColumn(new Column("DatasetVersion", ScalarType.BIGINT))
+            .addColumn(new Column("BackendCount", ScalarType.INT))
+            .addColumn(new Column("ElapsedMs", ScalarType.BIGINT)).build();
+
+    private LanceIndexPrewarm() {
+    }
+
+    public static void run(ConnectContext ctx, StmtExecutor executor, TableNameInfo tableName,
+            String indexName, String computeGroup, BooleanSupplier cancelled) throws Exception {
+        long started = System.nanoTime();
+        long elapsedMs = ctx.getStartTime() > 0 ? Math.max(0, System.currentTimeMillis() - ctx.getStartTime()) : 0;
+        long remainingMs = TimeUnit.SECONDS.toMillis(ctx.getQueryTimeoutS()) - elapsedMs;
+        long deadline = started + TimeUnit.MILLISECONDS.toNanos(Math.max(0, remainingMs));
+        tableName.analyze(ctx);
+        checkPrivileges(ctx, tableName);
+        List<Backend> targets = selectBackends(resolveComputeGroup(ctx, computeGroup));
+        CatalogIf<?> catalog = Env.getCurrentEnv().getCatalogMgr().getCatalog(tableName.getCtl());
+        if (!(catalog instanceof LanceExternalCatalog)) {
+            throw new AnalysisException("WARM UP INDEX requires a Lance catalog table");
+        }
+        TableIf table = catalog.getDbOrAnalysisException(tableName.getDb())
+                .getTableOrAnalysisException(tableName.getTbl());
+        if (!(table instanceof LanceExternalTable)) {
+            throw new AnalysisException("WARM UP INDEX requires a Lance catalog table");
+        }
+        checkActive(deadline, cancelled);
+        // This is the same snapshot/access path used by queries, including REST-vended credentials.
+        // Never independently resolve latest, index segments, or credentials on each backend.
+        LanceTableMetadata metadata = ((LanceExternalTable) table).loadMetadata();

Review Comment:
   [P2] Include metadata loading in the command's timeout and cancellation path. `loadMetadata()` synchronously resolves Lance namespace access and opens/reads the remote dataset, but the next `checkActive` is only after that call returns. With a slow object store, `KILL QUERY` or `query_timeout` leaves the FE command thread blocked well past its deadline before any BE RPC is sent. Run this phase with a bounded, cancellable deadline and stop before dispatch if it expires.



##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/WarmUpIndexCommand.java:
##########
@@ -0,0 +1,54 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.commands;
+
+import org.apache.doris.datasource.lance.LanceIndexPrewarm;
+import org.apache.doris.info.TableNameInfo;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.StmtExecutor;
+
+/** Synchronously fills the query sessions on the selected backends. */
+public class WarmUpIndexCommand extends Command {
+    private final TableNameInfo table;
+    private final String indexName;
+    private final String computeGroup;
+    private volatile boolean cancelled;
+
+    public WarmUpIndexCommand(TableNameInfo table, String indexName, String computeGroup) {
+        super(PlanType.WARM_UP_INDEX_COMMAND);
+        this.table = table;
+        this.indexName = indexName;
+        this.computeGroup = computeGroup;
+    }
+
+    @Override
+    public void run(ConnectContext ctx, StmtExecutor executor) throws Exception {
+        LanceIndexPrewarm.run(ctx, executor, table, indexName, computeGroup, () -> cancelled || ctx.isKilled());
+    }
+
+    public void cancel() {

Review Comment:
   [P2] Keep cancellation state scoped to one execution. `ExecuteCommand.run` reuses the `WarmUpIndexCommand` stored in `PrepareCommand` for each `COM_STMT_EXECUTE`; after `KILL QUERY` calls this setter, `cancelled` stays true on that retained object. A subsequent execute on the same live connection reaches `LanceIndexPrewarm.checkActive` and immediately fails as cancelled. Reset or replace per-execution state without losing cancellation of the active run.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](lance) Extend WARM UP SELECT with index prewarm (branch-4.1) [doris]

Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68775:
URL: https://github.com/apache/doris/pull/68775#issuecomment-6077114002

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](lance) Extend WARM UP SELECT with index prewarm (branch-4.1) [doris]

Posted by "ddupg (via GitHub)" <gi...@apache.org>.
ddupg commented on PR #68775:
URL: https://github.com/apache/doris/pull/68775#issuecomment-6117060782

   Could this PR also support prewarming indexes on a specific Lance branch? Queries already support branch selection, and it would be useful to prewarm the same branch before running queries.
   
   ```sql
   WARM UP SELECT embedding
   FROM lance_catalog.db.tbl@branch(dev)
   PROPERTIES ("read_index_only" = "true");
   ```
   
   It would also be fine to add this in a separate follow-up PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](lance) Extend WARM UP SELECT with index prewarm (branch-4.1) [doris]

Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 closed pull request #68775: [feature](lance) Extend WARM UP SELECT with index prewarm (branch-4.1)
URL: https://github.com/apache/doris/pull/68775


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](lance) Add synchronous index prewarm SQL (branch-4.1) [doris]

Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68775:
URL: https://github.com/apache/doris/pull/68775#issuecomment-6059523106

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](lance) Add synchronous index prewarm SQL (branch-4.1) [doris]

Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68775:
URL: https://github.com/apache/doris/pull/68775#issuecomment-6073142520

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](lance) Extend WARM UP SELECT with index prewarm (branch-4.1) [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68775:
URL: https://github.com/apache/doris/pull/68775#discussion_r4226185666


##########
fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceIndexPrewarm.java:
##########
@@ -0,0 +1,274 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.lance;
+
+import org.apache.doris.analysis.ResourceTypeEnum;
+import org.apache.doris.analysis.TableName;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.ScalarType;
+import org.apache.doris.catalog.TableIf;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.Config;
+import org.apache.doris.common.UserException;
+import org.apache.doris.datasource.CatalogIf;
+import org.apache.doris.datasource.lance.index.LanceIndexInspectionExecutor;
+import org.apache.doris.datasource.lance.index.LanceIndexSegmentGroup;
+import org.apache.doris.datasource.lance.metadata.LanceTableMetadata;
+import org.apache.doris.info.TableNameInfo;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.proto.InternalService.PLanceIndexPrewarmRequest;
+import org.apache.doris.proto.InternalService.PLanceIndexPrewarmResponse;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.ShowResultSet;
+import org.apache.doris.qe.ShowResultSetMetaData;
+import org.apache.doris.qe.StmtExecutor;
+import org.apache.doris.resource.computegroup.ComputeGroup;
+import org.apache.doris.rpc.BackendServiceProxy;
+import org.apache.doris.system.Backend;
+import org.apache.doris.system.BeSelectionPolicy;
+import org.apache.doris.thrift.TNetworkAddress;
+import org.apache.doris.thrift.TStatusCode;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.OptionalInt;
+import java.util.Set;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.BooleanSupplier;
+
+/** Pins table access and coordinates bounded synchronous prewarm RPCs. */
+public final class LanceIndexPrewarm {
+    static final int MAX_IN_FLIGHT = 8;
+    private static final ShowResultSetMetaData RESULT_META = ShowResultSetMetaData.builder()
+            .addColumn(new Column("Table", ScalarType.createStringType()))
+            .addColumn(new Column("Index", ScalarType.createStringType()))
+            .addColumn(new Column("DatasetVersion", ScalarType.BIGINT))
+            .addColumn(new Column("BackendCount", ScalarType.INT))
+            .addColumn(new Column("ElapsedMs", ScalarType.BIGINT)).build();
+
+    private LanceIndexPrewarm() {
+    }
+
+    public static ShowResultSetMetaData getResultSetMetaData() {
+        return RESULT_META;
+    }
+
+    public static void run(ConnectContext ctx, StmtExecutor executor, TableNameInfo tableName,
+            List<String> columns, BooleanSupplier cancelled) throws Exception {
+        long started = System.nanoTime();
+        long elapsedMs = ctx.getStartTime() > 0 ? Math.max(0, System.currentTimeMillis() - ctx.getStartTime()) : 0;
+        long remainingMs = TimeUnit.SECONDS.toMillis(ctx.getQueryTimeoutS()) - elapsedMs;
+        long deadline = started + TimeUnit.MILLISECONDS.toNanos(Math.max(0, remainingMs));
+        tableName.analyze(ctx);

Review Comment:
   [P2] Resolve the table name for each prepared execution. `WarmupSelectCommand` retains this `TableNameInfo`, and `analyze(ctx)` fills its missing catalog/database fields in place. A binary prepared `WARM UP SELECT * FROM items SETTINGS (read_index_only=true)` executed in Lance database A, then executed again after `USE` switches to Lance database B, still prewarms A; ordinary unbound relations resolve the current database on each execution. Copy the parsed name parts before analysis so each EXECUTE uses its own session defaults.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](lance) Extend WARM UP SELECT with index prewarm (branch-4.1) [doris]

Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68775:
URL: https://github.com/apache/doris/pull/68775#issuecomment-6077114339

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](lance) Add synchronous index prewarm SQL (branch-4.1) [doris]

Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68775:
URL: https://github.com/apache/doris/pull/68775#issuecomment-6073142939

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](lance) Extend WARM UP SELECT with index prewarm (branch-4.1) [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68775:
URL: https://github.com/apache/doris/pull/68775#issuecomment-6081889708

   # BE Regression && UT Coverage Report
   Increment line coverage `66.67% (54/81)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/68775_7ae702f5c500c6dc230053bbb1a8bf9c9a61930f_merge/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/68775_7ae702f5c500c6dc230053bbb1a8bf9c9a61930f_merge/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 74.91% (31956/42659) |
   | Line Coverage     | 59.29% (358175/604088) |
   | Region Coverage   | 55.95% (298340/533244) |
   | Branch Coverage   | 56.83% (135311/238105) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](lance) Extend WARM UP SELECT with index prewarm (branch-4.1) [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68775:
URL: https://github.com/apache/doris/pull/68775#issuecomment-6079406484

   # FE UT Coverage Report
   Increment line coverage `  66.21% (145/219)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/68775_7ae702f5c500c6dc230053bbb1a8bf9c9a61930f/fe_increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/68775_7ae702f5c500c6dc230053bbb1a8bf9c9a61930f/fe_report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](lance) Add synchronous index prewarm SQL (branch-4.1) [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68775:
URL: https://github.com/apache/doris/pull/68775#issuecomment-6059425327

   # FE UT Coverage Report
   Increment line coverage `  62.09% (95/153)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/68775_57d388521e3b1106ca9fdc99f6d813ad847b231d/fe_increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/68775_57d388521e3b1106ca9fdc99f6d813ad847b231d/fe_report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](lance) Add synchronous index prewarm SQL (branch-4.1) [doris]

Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68775:
URL: https://github.com/apache/doris/pull/68775#issuecomment-6054516492

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](lance) Extend WARM UP SELECT with index prewarm (branch-4.1) [doris]

Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68775:
URL: https://github.com/apache/doris/pull/68775#issuecomment-6076018109

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](lance) Add synchronous index prewarm SQL (branch-4.1) [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68775:
URL: https://github.com/apache/doris/pull/68775#issuecomment-6066231754

   # FE Regression Coverage Report
   Increment line coverage `  58.82% (90/153)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/68775_a6be4b70adf2f42ca6f0516eae311896333ca103_merge_fe/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/68775_a6be4b70adf2f42ca6f0516eae311896333ca103_merge_fe/report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](lance) Add synchronous index prewarm SQL (branch-4.1) [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68775:
URL: https://github.com/apache/doris/pull/68775#issuecomment-6057139922

   # Cloud UT Coverage Report
   Increment line coverage ` ` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/57d388521e3b1106ca9fdc99f6d813ad847b231d_57d388521e3b1106ca9fdc99f6d813ad847b231d_cloud/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/57d388521e3b1106ca9fdc99f6d813ad847b231d_57d388521e3b1106ca9fdc99f6d813ad847b231d_cloud/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 78.49% (1960/2497) |
   | Line Coverage     | 65.50% (35348/53964) |
   | Region Coverage   | 53.08% (33997/64043) |
   | Branch Coverage   | 55.92% (10491/18762) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](lance) Add synchronous index prewarm SQL (branch-4.1) [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68775:
URL: https://github.com/apache/doris/pull/68775#issuecomment-6058974855

   # FE Regression Coverage Report
   Increment line coverage `  36.84% (77/209)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/68775_57d388521e3b1106ca9fdc99f6d813ad847b231d_merge_fe/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/68775_57d388521e3b1106ca9fdc99f6d813ad847b231d_merge_fe/report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](lance) Add synchronous index prewarm SQL (branch-4.1) [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68775:
URL: https://github.com/apache/doris/pull/68775#discussion_r4218850983


##########
fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceIndexPrewarm.java:
##########
@@ -0,0 +1,236 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.lance;
+
+import org.apache.doris.analysis.ResourceTypeEnum;
+import org.apache.doris.analysis.TableName;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.ScalarType;
+import org.apache.doris.catalog.TableIf;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.Config;
+import org.apache.doris.common.UserException;
+import org.apache.doris.datasource.CatalogIf;
+import org.apache.doris.datasource.lance.index.LanceIndexInspectionExecutor;
+import org.apache.doris.datasource.lance.metadata.LanceTableMetadata;
+import org.apache.doris.info.TableNameInfo;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.proto.InternalService.PLanceIndexPrewarmRequest;
+import org.apache.doris.proto.InternalService.PLanceIndexPrewarmResponse;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.ShowResultSet;
+import org.apache.doris.qe.ShowResultSetMetaData;
+import org.apache.doris.qe.StmtExecutor;
+import org.apache.doris.resource.computegroup.ComputeGroup;
+import org.apache.doris.rpc.BackendServiceProxy;
+import org.apache.doris.system.Backend;
+import org.apache.doris.system.BeSelectionPolicy;
+import org.apache.doris.thrift.TNetworkAddress;
+import org.apache.doris.thrift.TStatusCode;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.BooleanSupplier;
+
+/** Pins table access and coordinates bounded synchronous prewarm RPCs. */
+public final class LanceIndexPrewarm {
+    static final int MAX_IN_FLIGHT = 8;
+    private static final ShowResultSetMetaData RESULT_META = ShowResultSetMetaData.builder()
+            .addColumn(new Column("Table", ScalarType.createStringType()))
+            .addColumn(new Column("Index", ScalarType.createStringType()))
+            .addColumn(new Column("DatasetVersion", ScalarType.BIGINT))
+            .addColumn(new Column("BackendCount", ScalarType.INT))
+            .addColumn(new Column("ElapsedMs", ScalarType.BIGINT)).build();
+
+    private LanceIndexPrewarm() {
+    }
+
+    public static ShowResultSetMetaData getResultSetMetaData() {
+        return RESULT_META;
+    }
+
+    public static void run(ConnectContext ctx, StmtExecutor executor, TableNameInfo tableName,
+            String indexName, String computeGroup, BooleanSupplier cancelled) throws Exception {
+        long started = System.nanoTime();
+        long elapsedMs = ctx.getStartTime() > 0 ? Math.max(0, System.currentTimeMillis() - ctx.getStartTime()) : 0;
+        long remainingMs = TimeUnit.SECONDS.toMillis(ctx.getQueryTimeoutS()) - elapsedMs;
+        long deadline = started + TimeUnit.MILLISECONDS.toNanos(Math.max(0, remainingMs));
+        tableName.analyze(ctx);
+        checkPrivileges(ctx, tableName);
+        List<Backend> targets = selectBackends(resolveComputeGroup(ctx, computeGroup));
+        // Resolve catalog/table initialization and snapshot IO in the bounded metadata pool.
+        // The worker owns native resources until it returns, even if this caller stops waiting.
+        String catalogName = tableName.getCtl();
+        String databaseName = tableName.getDb();
+        String name = tableName.getTbl();
+        LanceTableMetadata metadata = LanceIndexInspectionExecutor.execute(() -> {
+            CatalogIf<?> catalog = Env.getCurrentEnv().getCatalogMgr().getCatalog(catalogName);
+            if (!(catalog instanceof LanceExternalCatalog)) {
+                throw new AnalysisException("WARM UP INDEX requires a Lance catalog table");
+            }
+            TableIf table = catalog.getDbOrAnalysisException(databaseName).getTableOrAnalysisException(name);
+            if (!(table instanceof LanceExternalTable)) {
+                throw new AnalysisException("WARM UP INDEX requires a Lance catalog table");
+            }
+            checkActive(deadline, cancelled);
+            // Use the query access path, including REST-vended credentials, exactly once.
+            return ((LanceExternalTable) table).loadMetadata();
+        }, deadline, cancelled);
+        PLanceIndexPrewarmRequest request = request(metadata, indexName);
+        execute(targets, request, deadline, cancelled, (backend, rpcRequest, timeoutMs) ->
+                BackendServiceProxy.getInstance().prewarmLanceIndexAsync(
+                        new TNetworkAddress(backend.getHost(), backend.getBrpcPort()), rpcRequest, timeoutMs));
+        executor.sendResultSet(new ShowResultSet(RESULT_META, Collections.singletonList(Arrays.asList(
+                tableName.toSql(), indexName, Long.toString(metadata.getVersion()), Integer.toString(targets.size()),
+                Long.toString(TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - started))))));
+    }
+
+    static void checkPrivileges(ConnectContext ctx, TableNameInfo table) throws AnalysisException {
+        // Authorize before catalog lookup: resolving external metadata can perform remote IO.
+        if (!Env.getCurrentEnv().getAccessManager().checkGlobalPriv(ctx, PrivPredicate.ADMIN)) {
+            throw new AnalysisException("ADMIN privilege is required for WARM UP INDEX");
+        }
+        if (!Env.getCurrentEnv().getAccessManager().checkTblPriv(ctx,
+                new TableName(table.getCtl(), table.getDb(), table.getTbl()), PrivPredicate.SELECT)) {
+            throw new AnalysisException("SELECT denied for WARM UP INDEX on " + table.toSql());
+        }
+    }
+
+    static ComputeGroup resolveComputeGroup(ConnectContext ctx, String name) throws UserException {
+        if (Config.isCloudMode()) {
+            String selected = name == null ? ctx.getCloudCluster() : name;
+            if (!Env.getCurrentEnv().getAccessManager().checkCloudPriv(ctx.getCurrentUserIdentity(),
+                    selected, PrivPredicate.USAGE, ResourceTypeEnum.CLUSTER)) {
+                throw new AnalysisException("USAGE denied for compute group '" + selected + "'");
+            }
+            return Env.getCurrentEnv().getComputeGroupMgr().getComputeGroupByName(selected);
+        }
+        if (name != null) {
+            throw new AnalysisException("WITH COMPUTE GROUP requires cloud mode");
+        }
+        return ctx.getComputeGroup();
+    }
+
+    static List<Backend> selectBackends(ComputeGroup group) throws UserException {
+        if (group == ComputeGroup.INVALID_COMPUTE_GROUP) {
+            throw new AnalysisException(ComputeGroup.INVALID_COMPUTE_GROUP_ERR_MSG);
+        }
+        // Warm every eligible node, including all mixed nodes that an external scan may select
+        // to supplement compute nodes. A randomized scheduling subset is insufficient here.
+        List<Backend> targets = new BeSelectionPolicy.Builder().needQueryAvailable().needLoadAvailable()
+                .preferComputeNode(Config.prefer_compute_node_for_external_table)
+                .assignExpectBeNum(Integer.MAX_VALUE).build().getCandidateBackends(group.getBackendList());
+        if (targets.isEmpty()) {
+            throw new AnalysisException("No available backends for index prewarm in compute group '"
+                    + group.getName() + "'");
+        }
+        return new ArrayList<>(targets);
+    }
+
+    static PLanceIndexPrewarmRequest request(LanceTableMetadata metadata, String indexName)
+            throws AnalysisException {
+        if (metadata.getVersion() <= 0) {
+            throw new AnalysisException("Index prewarm requires a fixed positive dataset version");
+        }
+        if (indexName.isEmpty() || metadata.getIndexes().stream().noneMatch(i -> i.getName().equals(indexName))) {

Review Comment:
   [P2] Validate logical index existence separately from FE segment metadata. `loadMetadata()` builds `getIndexes()` from `describeUserIndexes()`, which intentionally omits legacy indexes when all physical segments lack details even though `Dataset.listIndexes()` still reports the name. `WARM UP INDEX` therefore returns “does not exist” here without attempting the BE prewarm for a listed index. Keep the pinned snapshot's logical names separate from segment-planning groups, and cover the legacy case with a result or precise unsupported error.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](lance) Add synchronous index prewarm SQL (branch-4.1) [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68775:
URL: https://github.com/apache/doris/pull/68775#issuecomment-6065978285

   # BE Regression && UT Coverage Report
   Increment line coverage `66.67% (54/81)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/68775_a6be4b70adf2f42ca6f0516eae311896333ca103_merge/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/68775_a6be4b70adf2f42ca6f0516eae311896333ca103_merge/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 74.96% (31960/42637) |
   | Line Coverage     | 59.31% (358097/603753) |
   | Region Coverage   | 55.95% (298190/532960) |
   | Branch Coverage   | 56.84% (135240/237944) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](lance) Extend WARM UP SELECT with index prewarm (branch-4.1) [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68775:
URL: https://github.com/apache/doris/pull/68775#issuecomment-6074777501

   # FE UT Coverage Report
   Increment line coverage `  66.21% (145/219)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/68775_ebb50b0b1f1116e0d9e9ec4be23e6c520e0a666b/fe_increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/68775_ebb50b0b1f1116e0d9e9ec4be23e6c520e0a666b/fe_report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](lance) Add synchronous index prewarm SQL (branch-4.1) [doris]

Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68775:
URL: https://github.com/apache/doris/pull/68775#issuecomment-6061917008

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](lance) Extend WARM UP SELECT with index prewarm (branch-4.1) [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68775:
URL: https://github.com/apache/doris/pull/68775#discussion_r4227702517


##########
fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java:
##########
@@ -9610,6 +9610,28 @@ public LogicalPlan visitWarmUpSelect(DorisParser.WarmUpSelectContext ctx) {
         }
 
         LogicalProject project = new LogicalProject(projectList, filter);
+        Map<String, String> properties = visitPropertyClause(ctx.properties);
+        for (String key : properties.keySet()) {
+            if (!"read_index_only".equals(key)) {
+                throw new AnalysisException("Unknown WARM UP SELECT property: " + key);
+            }
+        }
+        String indexOnly = properties.getOrDefault("read_index_only", "false");
+        if (!"true".equalsIgnoreCase(indexOnly) && !"false".equalsIgnoreCase(indexOnly)) {
+            throw new AnalysisException("read_index_only must be true or false");
+        }
+        if (Boolean.parseBoolean(indexOnly)) {
+            if (ctx.whereClause() != null || ctx.explain() != null) {
+                throw new AnalysisException("Index-only WARM UP SELECT does not support WHERE or EXPLAIN");
+            }
+            // Index cache fills must not execute the data scan or require the data-file cache.
+            List<String> columns = indexWarmupColumns(ctx, projectList);
+            UnboundBlackholeSink<?> sink = new UnboundBlackholeSink<>(project,
+                    new UnboundBlackholeSinkContext(true));
+            return new WarmupSelectCommand(sink,
+                    new TableNameInfo(visitMultipartIdentifier(ctx.warmUpSingleTableRef().multipartIdentifier())),

Review Comment:
   [P2] Reject overqualified table names before constructing `TableNameInfo`. `multipartIdentifier` accepts four or more components, but `TableNameInfo(List)` silently keeps only the last three. For `WARM UP SELECT * FROM ignored.lake.db.items PROPERTIES (read_index_only=true)`, this path prewarms `lake.db.items` and reports success for that different table, while normal relation binding rejects the four-part name. Validate one to three components before taking the index-only branch.



##########
fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceIndexPrewarm.java:
##########
@@ -0,0 +1,274 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.lance;
+
+import org.apache.doris.analysis.ResourceTypeEnum;
+import org.apache.doris.analysis.TableName;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.ScalarType;
+import org.apache.doris.catalog.TableIf;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.Config;
+import org.apache.doris.common.UserException;
+import org.apache.doris.datasource.CatalogIf;
+import org.apache.doris.datasource.lance.index.LanceIndexInspectionExecutor;
+import org.apache.doris.datasource.lance.index.LanceIndexSegmentGroup;
+import org.apache.doris.datasource.lance.metadata.LanceTableMetadata;
+import org.apache.doris.info.TableNameInfo;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.proto.InternalService.PLanceIndexPrewarmRequest;
+import org.apache.doris.proto.InternalService.PLanceIndexPrewarmResponse;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.ShowResultSet;
+import org.apache.doris.qe.ShowResultSetMetaData;
+import org.apache.doris.qe.StmtExecutor;
+import org.apache.doris.resource.computegroup.ComputeGroup;
+import org.apache.doris.rpc.BackendServiceProxy;
+import org.apache.doris.system.Backend;
+import org.apache.doris.system.BeSelectionPolicy;
+import org.apache.doris.thrift.TNetworkAddress;
+import org.apache.doris.thrift.TStatusCode;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.OptionalInt;
+import java.util.Set;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.BooleanSupplier;
+
+/** Pins table access and coordinates bounded synchronous prewarm RPCs. */
+public final class LanceIndexPrewarm {
+    static final int MAX_IN_FLIGHT = 8;
+    private static final ShowResultSetMetaData RESULT_META = ShowResultSetMetaData.builder()
+            .addColumn(new Column("Table", ScalarType.createStringType()))
+            .addColumn(new Column("Index", ScalarType.createStringType()))
+            .addColumn(new Column("DatasetVersion", ScalarType.BIGINT))
+            .addColumn(new Column("BackendCount", ScalarType.INT))
+            .addColumn(new Column("ElapsedMs", ScalarType.BIGINT)).build();
+
+    private LanceIndexPrewarm() {
+    }
+
+    public static ShowResultSetMetaData getResultSetMetaData() {
+        return RESULT_META;
+    }
+
+    public static void run(ConnectContext ctx, StmtExecutor executor, TableNameInfo tableName,
+            List<String> columns, BooleanSupplier cancelled) throws Exception {
+        long started = System.nanoTime();
+        long elapsedMs = ctx.getStartTime() > 0 ? Math.max(0, System.currentTimeMillis() - ctx.getStartTime()) : 0;
+        long remainingMs = TimeUnit.SECONDS.toMillis(ctx.getQueryTimeoutS()) - elapsedMs;
+        long deadline = started + TimeUnit.MILLISECONDS.toNanos(Math.max(0, remainingMs));
+        tableName.analyze(ctx);
+        checkPrivileges(ctx, tableName);
+        List<Backend> targets = selectBackends(resolveComputeGroup(ctx));
+        // Resolve catalog/table initialization and snapshot IO in the bounded metadata pool.
+        // The worker owns native resources until it returns, even if this caller stops waiting.
+        String catalogName = tableName.getCtl();
+        String databaseName = tableName.getDb();
+        String name = tableName.getTbl();
+        LanceTableMetadata metadata = LanceIndexInspectionExecutor.execute(() -> {
+            CatalogIf<?> catalog = Env.getCurrentEnv().getCatalogMgr().getCatalog(catalogName);
+            if (!(catalog instanceof LanceExternalCatalog)) {
+                throw new AnalysisException("Index-only WARM UP SELECT requires a Lance catalog table");
+            }
+            TableIf table = catalog.getDbOrAnalysisException(databaseName).getTableOrAnalysisException(name);
+            if (!(table instanceof LanceExternalTable)) {
+                throw new AnalysisException("Index-only WARM UP SELECT requires a Lance catalog table");
+            }
+            checkActive(deadline, cancelled);
+            // Use the query access path, including REST-vended credentials, exactly once.
+            return ((LanceExternalTable) table).loadMetadata();

Review Comment:
   [P2] Avoid loading every data fragment for index-only prewarm. `loadMetadata()` uses the normal scan metadata scope, and `LanceMetadataLoader.read()` calls `dataset.getFragments()` and creates one `LanceFragmentInfo` per fragment before it reads indexes. This command never uses the fragment list, so a table with many small fragments spends O(fragment count) JNI work and FE heap under the prewarm deadline before any BE cache fill. Read the pinned schema/index/access metadata without materializing scan fragments for this command.



##########
fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceIndexPrewarm.java:
##########
@@ -0,0 +1,274 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.lance;
+
+import org.apache.doris.analysis.ResourceTypeEnum;
+import org.apache.doris.analysis.TableName;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.ScalarType;
+import org.apache.doris.catalog.TableIf;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.Config;
+import org.apache.doris.common.UserException;
+import org.apache.doris.datasource.CatalogIf;
+import org.apache.doris.datasource.lance.index.LanceIndexInspectionExecutor;
+import org.apache.doris.datasource.lance.index.LanceIndexSegmentGroup;
+import org.apache.doris.datasource.lance.metadata.LanceTableMetadata;
+import org.apache.doris.info.TableNameInfo;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.proto.InternalService.PLanceIndexPrewarmRequest;
+import org.apache.doris.proto.InternalService.PLanceIndexPrewarmResponse;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.ShowResultSet;
+import org.apache.doris.qe.ShowResultSetMetaData;
+import org.apache.doris.qe.StmtExecutor;
+import org.apache.doris.resource.computegroup.ComputeGroup;
+import org.apache.doris.rpc.BackendServiceProxy;
+import org.apache.doris.system.Backend;
+import org.apache.doris.system.BeSelectionPolicy;
+import org.apache.doris.thrift.TNetworkAddress;
+import org.apache.doris.thrift.TStatusCode;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.OptionalInt;
+import java.util.Set;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.BooleanSupplier;
+
+/** Pins table access and coordinates bounded synchronous prewarm RPCs. */
+public final class LanceIndexPrewarm {
+    static final int MAX_IN_FLIGHT = 8;
+    private static final ShowResultSetMetaData RESULT_META = ShowResultSetMetaData.builder()
+            .addColumn(new Column("Table", ScalarType.createStringType()))
+            .addColumn(new Column("Index", ScalarType.createStringType()))
+            .addColumn(new Column("DatasetVersion", ScalarType.BIGINT))
+            .addColumn(new Column("BackendCount", ScalarType.INT))
+            .addColumn(new Column("ElapsedMs", ScalarType.BIGINT)).build();
+
+    private LanceIndexPrewarm() {
+    }
+
+    public static ShowResultSetMetaData getResultSetMetaData() {
+        return RESULT_META;
+    }
+
+    public static void run(ConnectContext ctx, StmtExecutor executor, TableNameInfo tableName,
+            List<String> columns, BooleanSupplier cancelled) throws Exception {
+        long started = System.nanoTime();
+        long elapsedMs = ctx.getStartTime() > 0 ? Math.max(0, System.currentTimeMillis() - ctx.getStartTime()) : 0;
+        long remainingMs = TimeUnit.SECONDS.toMillis(ctx.getQueryTimeoutS()) - elapsedMs;

Review Comment:
   [P2] Use the effective execution timeout for index prewarm. This command extends `InsertIntoTableCommand`, so `StmtExecutor.isSyncLoadKindStmt()` makes `ConnectContext.checkTimeout()` and `getExecTimeoutS()` allow `max(insert_timeout, query_timeout)` (defaults: 4 h versus 15 min). This separate deadline uses only `getQueryTimeoutS()`, so a valid index prewarm that takes over 15 minutes fails while the statement still has hours remaining. Derive the deadline from `ctx.getExecTimeoutS()` or classify this mode consistently as a query.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](lance) Extend WARM UP SELECT with index prewarm (branch-4.1) [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68775:
URL: https://github.com/apache/doris/pull/68775#issuecomment-6080574728

   # FE Regression Coverage Report
   Increment line coverage `  66.67% (146/219)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/68775_7ae702f5c500c6dc230053bbb1a8bf9c9a61930f_merge_fe/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/68775_7ae702f5c500c6dc230053bbb1a8bf9c9a61930f_merge_fe/report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](lance) Add synchronous index prewarm SQL (branch-4.1) [doris]

Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68775:
URL: https://github.com/apache/doris/pull/68775#issuecomment-6055923100

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](lance) Add synchronous index prewarm SQL (branch-4.1) [doris]

Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68775:
URL: https://github.com/apache/doris/pull/68775#issuecomment-6055925264

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](lance) Add synchronous index prewarm SQL (branch-4.1) [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68775:
URL: https://github.com/apache/doris/pull/68775#issuecomment-6054958602

   # Cloud UT Coverage Report
   Increment line coverage ` ` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/60a84c52d855cf310754e0d8e37de30944f64cd1_60a84c52d855cf310754e0d8e37de30944f64cd1_cloud/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/60a84c52d855cf310754e0d8e37de30944f64cd1_60a84c52d855cf310754e0d8e37de30944f64cd1_cloud/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 78.49% (1960/2497) |
   | Line Coverage     | 65.49% (35343/53964) |
   | Region Coverage   | 53.05% (33978/64043) |
   | Branch Coverage   | 55.91% (10489/18762) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org