You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Lchangliang (via GitHub)" <gi...@apache.org> on 2024/03/25 16:18:09 UTC

[PR] (cloud-merge) Support more multi case strategies about cache [doris]

Lchangliang opened a new pull request, #32804:
URL: https://github.com/apache/doris/pull/32804

   ## Proposed changes
   
   Issue Number: close #xxx
   
   <!--Describe your changes.-->
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]

Posted by "Lchangliang (via GitHub)" <gi...@apache.org>.
Lchangliang commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2068923514

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]

Posted by "Lchangliang (via GitHub)" <gi...@apache.org>.
Lchangliang commented on code in PR #32804:
URL: https://github.com/apache/doris/pull/32804#discussion_r1574094425


##########
fe/fe-core/src/main/java/org/apache/doris/cluster/ClusterNamespace.java:
##########
@@ -32,6 +32,10 @@ public class ClusterNamespace {
 
     private static final String CLUSTER_DELIMITER = ":";
 
+    public static String getFullName(String cluster, String name) {
+        return linkString(cluster, name);
+    }

Review Comment:
   Copy from selectdb, remove it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2069095990

   TeamCity be ut coverage result:
    Function Coverage: 35.23% (8915/25303) 
    Line Coverage: 27.00% (73323/271587)
    Region Coverage: 26.17% (37875/144729)
    Branch Coverage: 22.98% (19282/83914)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/c52664d7fd539e4f04f5a82460df2ad0eb6167cb_c52664d7fd539e4f04f5a82460df2ad0eb6167cb/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2068777997

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2068008859

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 187263 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit c4fc00513476c131946015035443c34f1fe57661, data reload: false
   
   query1	902	388	365	365
   query2	6171	2507	2422	2422
   query3	6660	210	209	209
   query4	23110	21468	21383	21383
   query5	4179	408	410	408
   query6	275	185	182	182
   query7	4583	298	289	289
   query8	251	202	185	185
   query9	8454	2309	2286	2286
   query10	418	248	256	248
   query11	14782	14144	14213	14144
   query12	141	88	86	86
   query13	1656	372	366	366
   query14	8740	8533	8193	8193
   query15	286	186	189	186
   query16	8242	261	266	261
   query17	1978	574	561	561
   query18	2100	275	266	266
   query19	330	150	149	149
   query20	91	83	83	83
   query21	198	141	135	135
   query22	5030	4879	4877	4877
   query23	33725	33219	33301	33219
   query24	10596	3094	3038	3038
   query25	571	398	399	398
   query26	709	160	164	160
   query27	2317	362	376	362
   query28	5945	2042	2080	2042
   query29	874	624	617	617
   query30	298	196	180	180
   query31	989	763	762	762
   query32	91	51	57	51
   query33	650	266	253	253
   query34	899	508	528	508
   query35	858	729	721	721
   query36	1105	964	982	964
   query37	111	78	75	75
   query38	3492	3362	3360	3360
   query39	1648	1561	1704	1561
   query40	165	139	131	131
   query41	48	42	44	42
   query42	106	99	101	99
   query43	602	559	549	549
   query44	1130	768	768	768
   query45	276	259	273	259
   query46	1115	775	756	756
   query47	2016	1925	1932	1925
   query48	395	303	317	303
   query49	841	408	398	398
   query50	786	399	409	399
   query51	6969	6898	6766	6766
   query52	101	87	86	86
   query53	354	295	281	281
   query54	297	224	228	224
   query55	79	72	71	71
   query56	235	228	225	225
   query57	1214	1149	1133	1133
   query58	217	216	211	211
   query59	3563	3133	3088	3088
   query60	251	224	225	224
   query61	87	88	85	85
   query62	613	453	443	443
   query63	311	283	283	283
   query64	4879	3963	3863	3863
   query65	3089	3066	3070	3066
   query66	764	330	338	330
   query67	15234	15037	15330	15037
   query68	5748	533	539	533
   query69	533	304	311	304
   query70	1289	1203	1182	1182
   query71	1464	1275	1283	1275
   query72	6923	2743	2574	2574
   query73	725	333	324	324
   query74	6971	6504	6504	6504
   query75	3826	2634	2652	2634
   query76	4212	1057	1032	1032
   query77	597	273	275	273
   query78	11058	10420	10233	10233
   query79	8685	531	534	531
   query80	2051	447	458	447
   query81	555	250	257	250
   query82	1632	102	97	97
   query83	323	173	176	173
   query84	265	90	89	89
   query85	1415	312	310	310
   query86	455	308	287	287
   query87	3476	3278	3261	3261
   query88	4971	2423	2448	2423
   query89	571	387	398	387
   query90	1929	187	186	186
   query91	134	112	104	104
   query92	58	48	53	48
   query93	6861	511	494	494
   query94	1186	175	179	175
   query95	388	295	305	295
   query96	603	263	261	261
   query97	3105	2919	2936	2919
   query98	244	231	230	230
   query99	1267	846	828	828
   Total cold run time: 292712 ms
   Total hot run time: 187263 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2067710314

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 30.85 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit c060c92d39ae3b91179b918804f04d6b4692c178, data reload: false
   
   query1	0.04	0.03	0.03
   query2	0.08	0.04	0.04
   query3	0.23	0.05	0.05
   query4	1.68	0.07	0.07
   query5	0.49	0.50	0.51
   query6	1.46	0.71	0.72
   query7	0.02	0.02	0.01
   query8	0.05	0.04	0.04
   query9	0.55	0.50	0.50
   query10	0.56	0.56	0.54
   query11	0.16	0.12	0.11
   query12	0.14	0.11	0.11
   query13	0.61	0.58	0.59
   query14	0.76	0.78	0.79
   query15	0.83	0.80	0.81
   query16	0.36	0.35	0.35
   query17	0.93	0.94	0.94
   query18	0.22	0.22	0.26
   query19	1.82	1.64	1.70
   query20	0.02	0.00	0.01
   query21	15.40	0.65	0.65
   query22	4.82	6.42	2.54
   query23	18.24	1.36	1.29
   query24	1.82	0.20	0.22
   query25	0.15	0.09	0.08
   query26	0.27	0.16	0.16
   query27	0.08	0.08	0.07
   query28	13.41	1.00	0.99
   query29	12.63	3.28	3.25
   query30	0.26	0.07	0.06
   query31	2.84	0.38	0.37
   query32	3.29	0.46	0.46
   query33	2.83	2.83	2.75
   query34	16.97	4.42	4.48
   query35	4.49	4.53	4.48
   query36	0.65	0.46	0.46
   query37	0.17	0.16	0.15
   query38	0.16	0.15	0.15
   query39	0.04	0.04	0.04
   query40	0.17	0.16	0.13
   query41	0.09	0.05	0.05
   query42	0.06	0.05	0.05
   query43	0.05	0.03	0.04
   Total cold run time: 109.9 s
   Total hot run time: 30.85 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]

Posted by "Lchangliang (via GitHub)" <gi...@apache.org>.
Lchangliang commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2067071464

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]

Posted by "Lchangliang (via GitHub)" <gi...@apache.org>.
Lchangliang commented on code in PR #32804:
URL: https://github.com/apache/doris/pull/32804#discussion_r1574076003


##########
be/src/cloud/cloud_backend_service.cpp:
##########
@@ -45,4 +57,137 @@ Status CloudBackendService::create_service(CloudStorageEngine& engine, ExecEnv*
     return Status::OK();
 }
 
+void CloudBackendService::sync_load_for_tablets(TSyncLoadForTabletsResponse&,
+                                                const TSyncLoadForTabletsRequest& request) {
+    auto f = [this, tablet_ids = request.tablet_ids]() {
+        std::for_each(tablet_ids.cbegin(), tablet_ids.cend(), [this](int64_t tablet_id) {
+            CloudTabletSPtr tablet;
+            auto result = _engine.tablet_mgr().get_tablet(tablet_id, true);
+            if (!result.has_value()) {
+                return;
+            }
+            Status st = result.value()->sync_rowsets(-1, true);
+            if (!st.ok()) {
+                LOG_WARNING("failed to sync load for tablet").error(st);
+            }
+        });
+    };
+    static_cast<void>(_exec_env->sync_load_for_tablets_thread_pool()->submit_func(std::move(f)));
+}
+
+void CloudBackendService::get_top_n_hot_partitions(TGetTopNHotPartitionsResponse& response,
+                                                   const TGetTopNHotPartitionsRequest& request) {
+    TabletHotspot::instance()->get_top_n_hot_partition(&response.hot_tables);
+    response.file_cache_size = io::FileCacheFactory::instance()->get_capacity();
+    response.__isset.hot_tables = !response.hot_tables.empty();
+}
+
+void CloudBackendService::warm_up_tablets(TWarmUpTabletsResponse& response,
+                                          const TWarmUpTabletsRequest& request) {
+    Status st;
+    auto* manager = CloudWarmUpManager::instance();
+    switch (request.type) {
+    case TWarmUpTabletsRequestType::SET_JOB: {
+        LOG_INFO("receive the warm up request.")
+                .tag("request_type", "SET_JOB")
+                .tag("job_id", request.job_id);
+        st = manager->check_and_set_job_id(request.job_id);
+        if (!st) {
+            LOG_WARNING("SET_JOB failed.").error(st);
+            break;
+        }
+        [[fallthrough]];

Review Comment:
   When sending the first RPC, it contains SET_JOB and SET_BATCH params.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]

Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #32804:
URL: https://github.com/apache/doris/pull/32804#discussion_r1570412055


##########
be/src/cloud/config.h:
##########
@@ -88,4 +88,6 @@ DECLARE_mInt32(schedule_sync_tablets_interval_s);
 // Cloud mow
 DECLARE_mInt32(mow_stream_load_commit_retry_times);
 
+DECLARE_mInt32(sync_load_for_tablets_thread);

Review Comment:
   add a comment



##########
be/src/cloud/cloud_tablet.cpp:
##########
@@ -191,13 +192,40 @@ void CloudTablet::add_rowsets(std::vector<RowsetSharedPtr> to_add, bool version_
 
     auto add_rowsets_directly = [=, this](std::vector<RowsetSharedPtr>& rowsets) {
         for (auto& rs : rowsets) {
+            if (version_overlap || warmup_delta_data) {
+#ifndef BE_TEST

Review Comment:
   try to use TEST_SYNC_POINT_RETURN() in stead of  ifdef



##########
be/src/cloud/cloud_backend_service.cpp:
##########
@@ -45,4 +57,137 @@ Status CloudBackendService::create_service(CloudStorageEngine& engine, ExecEnv*
     return Status::OK();
 }
 
+void CloudBackendService::sync_load_for_tablets(TSyncLoadForTabletsResponse&,
+                                                const TSyncLoadForTabletsRequest& request) {
+    auto f = [this, tablet_ids = request.tablet_ids]() {
+        std::for_each(tablet_ids.cbegin(), tablet_ids.cend(), [this](int64_t tablet_id) {
+            CloudTabletSPtr tablet;
+            auto result = _engine.tablet_mgr().get_tablet(tablet_id, true);
+            if (!result.has_value()) {
+                return;
+            }
+            Status st = result.value()->sync_rowsets(-1, true);
+            if (!st.ok()) {
+                LOG_WARNING("failed to sync load for tablet").error(st);
+            }
+        });
+    };
+    static_cast<void>(_exec_env->sync_load_for_tablets_thread_pool()->submit_func(std::move(f)));
+}
+
+void CloudBackendService::get_top_n_hot_partitions(TGetTopNHotPartitionsResponse& response,
+                                                   const TGetTopNHotPartitionsRequest& request) {
+    TabletHotspot::instance()->get_top_n_hot_partition(&response.hot_tables);
+    response.file_cache_size = io::FileCacheFactory::instance()->get_capacity();
+    response.__isset.hot_tables = !response.hot_tables.empty();
+}
+
+void CloudBackendService::warm_up_tablets(TWarmUpTabletsResponse& response,
+                                          const TWarmUpTabletsRequest& request) {
+    Status st;
+    auto* manager = CloudWarmUpManager::instance();
+    switch (request.type) {
+    case TWarmUpTabletsRequestType::SET_JOB: {
+        LOG_INFO("receive the warm up request.")
+                .tag("request_type", "SET_JOB")
+                .tag("job_id", request.job_id);
+        st = manager->check_and_set_job_id(request.job_id);
+        if (!st) {
+            LOG_WARNING("SET_JOB failed.").error(st);
+            break;
+        }
+        [[fallthrough]];
+    }
+    case TWarmUpTabletsRequestType::SET_BATCH: {
+        LOG_INFO("receive the warm up request.")
+                .tag("request_type", "SET_BATCH")
+                .tag("job_id", request.job_id)
+                .tag("batch_id", request.batch_id)
+                .tag("jobs size", request.job_metas.size());
+        bool retry = false;
+        st = manager->check_and_set_batch_id(request.job_id, request.batch_id, &retry);
+        if (!retry && st) {
+            manager->add_job(request.job_metas);
+        } else {
+            if (retry) {
+                LOG_WARNING("retry the job.")
+                        .tag("job_id", request.job_id)
+                        .tag("batch_id", request.batch_id);
+            } else {
+                LOG_WARNING("SET_BATCH failed.").error(st);
+            }
+        }
+        break;
+    }
+    case TWarmUpTabletsRequestType::GET_CURRENT_JOB_STATE_AND_LEASE: {
+        auto [job_id, batch_id, pending_job_size, finish_job_size] =
+                manager->get_current_job_state();
+        LOG_INFO("receive the warm up request.")
+                .tag("request_type", "GET_CURRENT_JOB_STATE_AND_LEASE")
+                .tag("job_id", job_id)
+                .tag("batch_id", batch_id)
+                .tag("pending_job_size", pending_job_size)
+                .tag("finish_job_size", finish_job_size);
+        response.__set_job_id(job_id);
+        response.__set_batch_id(batch_id);
+        response.__set_pending_job_size(pending_job_size);
+        response.__set_finish_job_size(finish_job_size);
+        break;
+    }
+    case TWarmUpTabletsRequestType::CLEAR_JOB: {
+        LOG_INFO("receive the warm up request.")
+                .tag("request_type", "CLEAR_JOB")
+                .tag("job_id", request.job_id);
+        st = manager->clear_job(request.job_id);
+        break;
+    }
+    default:
+        DCHECK(false);
+    };
+    st.to_thrift(&response.status);
+}
+
+void CloudBackendService::pre_cache_async(TPreCacheAsyncResponse& response,
+                                          const TPreCacheAsyncRequest& request) {
+    std::string brpc_addr = fmt::format("{}:{}", request.host, request.brpc_port);

Review Comment:
   Just use the name `warm_up_cache_async()`, `pre_cache` is hard to under stand



##########
be/src/cloud/cloud_backend_service.h:
##########
@@ -35,6 +35,22 @@ class CloudBackendService final : public BaseBackendService {
 
     // TODO(plat1ko): cloud backend functions
 
+    // If another cluster load, FE need to notify the cluster to sync the load data
+    void sync_load_for_tablets(TSyncLoadForTabletsResponse& response,
+                               const TSyncLoadForTabletsRequest& request) override;
+
+    void get_top_n_hot_partitions(TGetTopNHotPartitionsResponse& response,
+                                  const TGetTopNHotPartitionsRequest& request) override;
+
+    void warm_up_tablets(TWarmUpTabletsResponse& response,
+                         const TWarmUpTabletsRequest& request) override;
+
+    void pre_cache_async(TPreCacheAsyncResponse& response,
+                         const TPreCacheAsyncRequest& request) override;
+
+    void check_pre_cache(TCheckPreCacheResponse& response,
+                         const TCheckPreCacheRequest& request) override;
+

Review Comment:
   Add enough comments for each rpc.



##########
be/src/cloud/cloud_internal_service.h:
##########
@@ -36,6 +36,11 @@ class CloudInternalServiceImpl final : public PInternalService {
                           PAlterVaultSyncResponse* response,
                           google::protobuf::Closure* done) override;
 
+    void get_file_cache_meta_by_tablet_id(google::protobuf::RpcController* controller,

Review Comment:
   Add comment for behavior description



##########
fe/fe-core/src/main/java/org/apache/doris/cluster/ClusterNamespace.java:
##########
@@ -32,6 +32,10 @@ public class ClusterNamespace {
 
     private static final String CLUSTER_DELIMITER = ":";
 
+    public static String getFullName(String cluster, String name) {
+        return linkString(cluster, name);
+    }

Review Comment:
   the form of `cluster:db` is deprecated for new versions of doris.
   why do we need this?



##########
fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudEnv.java:
##########
@@ -438,4 +452,63 @@ public void replayUpdateCloudReplica(UpdateCloudReplicaInfo info) throws MetaNot
     public boolean getEnableStorageVault() {
         return this.enableStorageVault;
     }
+
+    public long loadCloudWarmUpJob(DataInputStream dis, long checksum) throws Exception {
+        // same as loadAlterJob
+
+        int size = dis.readInt();
+        long newChecksum = checksum ^ size;
+        if (size > 0) {
+            // There should be no old cloudWarmUp jobs, if exist throw exception, should not use this FE version
+            throw new IOException("There are [" + size + "] cloud warm up jobs."
+                    + " Please downgrade FE to an older version and handle residual jobs");
+        }
+
+        // finished or cancelled jobs
+        size = dis.readInt();
+        newChecksum ^= size;
+        if (size > 0) {
+            throw new IOException("There are [" + size + "] old finished or cancelled cloud warm up jobs."
+                    + " Please downgrade FE to an older version and handle residual jobs");
+        }
+
+        size = dis.readInt();
+        newChecksum ^= size;
+        for (int i = 0; i < size; i++) {
+            CloudWarmUpJob cloudWarmUpJob = CloudWarmUpJob.read(dis);
+            if (cloudWarmUpJob.isExpire() || cloudWarmUpJob.getJobState() == JobState.DELETED) {
+                LOG.info("cloud warm up job is expired, {}, ignore it", cloudWarmUpJob.getJobId());
+                continue;
+            }
+            this.getCacheHotspotMgr().addCloudWarmUpJob(cloudWarmUpJob);
+        }
+        LOG.info("finished replay cloud warm up job from image");
+        return newChecksum;
+    }
+
+    public long saveCloudWarmUpJob(CountingDataOutputStream dos, long checksum) throws IOException {
+        // same as saveAlterJob
+
+        Map<Long, CloudWarmUpJob> cloudWarmUpJobs;
+        cloudWarmUpJobs = this.getCacheHotspotMgr().getCloudWarmUpJobs();
+
+        int size = 0;
+        checksum ^= size;
+        dos.writeInt(size);
+
+        checksum ^= size;
+        dos.writeInt(size);

Review Comment:
   why do we write and checksum 0 twice here?
   add comment if it is by intent



##########
fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudEnv.java:
##########
@@ -438,4 +452,63 @@ public void replayUpdateCloudReplica(UpdateCloudReplicaInfo info) throws MetaNot
     public boolean getEnableStorageVault() {
         return this.enableStorageVault;
     }
+
+    public long loadCloudWarmUpJob(DataInputStream dis, long checksum) throws Exception {
+        // same as loadAlterJob

Review Comment:
   remove unnecessary comment....



##########
fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudEnv.java:
##########
@@ -438,4 +452,63 @@ public void replayUpdateCloudReplica(UpdateCloudReplicaInfo info) throws MetaNot
     public boolean getEnableStorageVault() {
         return this.enableStorageVault;
     }
+
+    public long loadCloudWarmUpJob(DataInputStream dis, long checksum) throws Exception {
+        // same as loadAlterJob
+
+        int size = dis.readInt();
+        long newChecksum = checksum ^ size;
+        if (size > 0) {
+            // There should be no old cloudWarmUp jobs, if exist throw exception, should not use this FE version
+            throw new IOException("There are [" + size + "] cloud warm up jobs."
+                    + " Please downgrade FE to an older version and handle residual jobs");
+        }
+
+        // finished or cancelled jobs
+        size = dis.readInt();
+        newChecksum ^= size;
+        if (size > 0) {
+            throw new IOException("There are [" + size + "] old finished or cancelled cloud warm up jobs."
+                    + " Please downgrade FE to an older version and handle residual jobs");
+        }
+
+        size = dis.readInt();
+        newChecksum ^= size;
+        for (int i = 0; i < size; i++) {
+            CloudWarmUpJob cloudWarmUpJob = CloudWarmUpJob.read(dis);
+            if (cloudWarmUpJob.isExpire() || cloudWarmUpJob.getJobState() == JobState.DELETED) {
+                LOG.info("cloud warm up job is expired, {}, ignore it", cloudWarmUpJob.getJobId());
+                continue;
+            }
+            this.getCacheHotspotMgr().addCloudWarmUpJob(cloudWarmUpJob);
+        }
+        LOG.info("finished replay cloud warm up job from image");

Review Comment:
   it is not replay, it is just a load from the image.



##########
fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java:
##########
@@ -2024,6 +2024,10 @@ public String getEstimatePartitionSize() {
         return "";
     }
 
+    public boolean containPartition(String partitionName) {

Review Comment:
   
   naming `containPartition` -> `containsPartition`



##########
be/src/cloud/cloud_warm_up_manager.cpp:
##########


Review Comment:
   lack of comment
   



##########
fe/fe-core/src/main/java/org/apache/doris/analysis/CancelCloudWarmUpStmt.java:
##########
@@ -0,0 +1,85 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.analysis.BinaryPredicate.Operator;
+import org.apache.doris.common.AnalysisException;
+
+public class CancelCloudWarmUpStmt extends CancelStmt {
+    private Expr whereClause;
+    private long jobId;
+
+    public CancelCloudWarmUpStmt(Expr whereClause) {
+        this.whereClause = whereClause;
+    }
+
+    public long getJobId() {
+        return jobId;
+    }
+
+    @Override
+    public void analyze(Analyzer analyzer) throws AnalysisException {

Review Comment:
   throw exception if not cloud mode
   
   



##########
be/src/cloud/cloud_internal_service.cpp:
##########
@@ -34,4 +37,68 @@ void CloudInternalServiceImpl::alter_vault_sync(google::protobuf::RpcController*
     _engine.sync_storage_vault();
 }
 
+FileCacheType cache_type_to_pb(io::FileCacheType type) {
+    switch (type) {
+    case io::FileCacheType::TTL:
+        return FileCacheType::TTL;
+        break;
+    case io::FileCacheType::INDEX:
+        return FileCacheType::INDEX;
+        break;
+    case io::FileCacheType::NORMAL:
+        return FileCacheType::NORMAL;
+        break;
+    default:
+        DCHECK(false);
+    }
+    return FileCacheType::NORMAL;
+}
+
+void CloudInternalServiceImpl::get_file_cache_meta_by_tablet_id(
+        google::protobuf::RpcController* controller [[maybe_unused]],
+        const PGetFileCacheMetaRequest* request, PGetFileCacheMetaResponse* response,
+        google::protobuf::Closure* done) {
+    brpc::ClosureGuard closure_guard(done);
+    if (!config::enable_file_cache) {
+        LOG_WARNING("try to access tablet file cache meta, but file cache not enabled");
+        return;
+    }
+    std::for_each(
+            request->tablet_ids().cbegin(), request->tablet_ids().cend(), [&](int64_t tablet_id) {
+                auto res = _engine.tablet_mgr().get_tablet(tablet_id);
+                if (!res.has_value()) {
+                    LOG(ERROR) << "failed to get tablet: " << tablet_id
+                               << " err msg: " << res.error().msg();
+                    return;
+                }
+                CloudTabletSPtr tablet = std::move(res.value());
+                auto rowsets = tablet->get_snapshot_rowset();
+                std::for_each(rowsets.cbegin(), rowsets.cend(), [&](RowsetSharedPtr rowset) {
+                    std::string rowset_id = rowset->rowset_id().to_string();
+                    for (int64_t segment_id = 0; segment_id < rowset->num_segments();
+                         segment_id++) {
+                        std::string file_name = fmt::format("{}_{}.dat", rowset_id, segment_id);
+                        auto cache_key = io::BlockFileCache::hash(file_name);
+                        auto* cache = io::FileCacheFactory::instance()->get_by_path(cache_key);
+
+                        auto segments_meta = cache->get_hot_blocks_meta(cache_key);
+                        std::for_each(

Review Comment:
   ditto



##########
be/src/cloud/cloud_internal_service.cpp:
##########
@@ -34,4 +37,68 @@ void CloudInternalServiceImpl::alter_vault_sync(google::protobuf::RpcController*
     _engine.sync_storage_vault();
 }
 
+FileCacheType cache_type_to_pb(io::FileCacheType type) {
+    switch (type) {
+    case io::FileCacheType::TTL:
+        return FileCacheType::TTL;
+        break;
+    case io::FileCacheType::INDEX:
+        return FileCacheType::INDEX;
+        break;
+    case io::FileCacheType::NORMAL:
+        return FileCacheType::NORMAL;
+        break;
+    default:
+        DCHECK(false);
+    }
+    return FileCacheType::NORMAL;
+}
+
+void CloudInternalServiceImpl::get_file_cache_meta_by_tablet_id(
+        google::protobuf::RpcController* controller [[maybe_unused]],
+        const PGetFileCacheMetaRequest* request, PGetFileCacheMetaResponse* response,
+        google::protobuf::Closure* done) {
+    brpc::ClosureGuard closure_guard(done);
+    if (!config::enable_file_cache) {
+        LOG_WARNING("try to access tablet file cache meta, but file cache not enabled");
+        return;
+    }
+    std::for_each(

Review Comment:
   Use a plain `for` expression to save indention (at least 8 spaces)
   ```
   for (auto it...; it != end();++it) {
       ...
   }
   ```



##########
gensrc/proto/internal_service.proto:
##########
@@ -796,7 +796,7 @@ message FileCacheSegmentMeta {
 }
 
 message PGetFileCacheMetaResponse {
-    repeated FileCacheSegmentMeta file_cache_segment_metas = 1;
+    repeated FileCacheBlockMeta file_cache_segment_metas = 1;

Review Comment:
   `file_cache_segment_metas` -> `file_cache_block_metas`
   



##########
be/src/cloud/cloud_internal_service.cpp:
##########
@@ -34,4 +37,68 @@ void CloudInternalServiceImpl::alter_vault_sync(google::protobuf::RpcController*
     _engine.sync_storage_vault();
 }
 
+FileCacheType cache_type_to_pb(io::FileCacheType type) {
+    switch (type) {
+    case io::FileCacheType::TTL:
+        return FileCacheType::TTL;
+        break;
+    case io::FileCacheType::INDEX:
+        return FileCacheType::INDEX;
+        break;
+    case io::FileCacheType::NORMAL:
+        return FileCacheType::NORMAL;
+        break;
+    default:

Review Comment:
   redundant `break`



##########
be/src/cloud/cloud_backend_service.cpp:
##########
@@ -45,4 +57,137 @@ Status CloudBackendService::create_service(CloudStorageEngine& engine, ExecEnv*
     return Status::OK();
 }
 
+void CloudBackendService::sync_load_for_tablets(TSyncLoadForTabletsResponse&,
+                                                const TSyncLoadForTabletsRequest& request) {
+    auto f = [this, tablet_ids = request.tablet_ids]() {
+        std::for_each(tablet_ids.cbegin(), tablet_ids.cend(), [this](int64_t tablet_id) {
+            CloudTabletSPtr tablet;
+            auto result = _engine.tablet_mgr().get_tablet(tablet_id, true);
+            if (!result.has_value()) {
+                return;
+            }
+            Status st = result.value()->sync_rowsets(-1, true);
+            if (!st.ok()) {
+                LOG_WARNING("failed to sync load for tablet").error(st);
+            }
+        });
+    };
+    static_cast<void>(_exec_env->sync_load_for_tablets_thread_pool()->submit_func(std::move(f)));
+}
+
+void CloudBackendService::get_top_n_hot_partitions(TGetTopNHotPartitionsResponse& response,
+                                                   const TGetTopNHotPartitionsRequest& request) {
+    TabletHotspot::instance()->get_top_n_hot_partition(&response.hot_tables);
+    response.file_cache_size = io::FileCacheFactory::instance()->get_capacity();
+    response.__isset.hot_tables = !response.hot_tables.empty();
+}
+
+void CloudBackendService::warm_up_tablets(TWarmUpTabletsResponse& response,
+                                          const TWarmUpTabletsRequest& request) {
+    Status st;
+    auto* manager = CloudWarmUpManager::instance();
+    switch (request.type) {
+    case TWarmUpTabletsRequestType::SET_JOB: {
+        LOG_INFO("receive the warm up request.")
+                .tag("request_type", "SET_JOB")
+                .tag("job_id", request.job_id);
+        st = manager->check_and_set_job_id(request.job_id);
+        if (!st) {
+            LOG_WARNING("SET_JOB failed.").error(st);
+            break;
+        }
+        [[fallthrough]];

Review Comment:
   Why do we need to `fallthough`?



##########
fe/fe-core/src/main/java/org/apache/doris/qe/DdlExecutor.java:
##########
@@ -374,6 +376,9 @@ public static void execute(Env env, DdlStmt ddlStmt) throws Exception {
             env.getAuth().refreshLdap((RefreshLdapStmt) ddlStmt);
         } else if (ddlStmt instanceof AlterUserStmt) {
             env.getAuth().alterUser((AlterUserStmt) ddlStmt);
+        } else if (ddlStmt instanceof CancelCloudWarmUpStmt) {
+            CancelCloudWarmUpStmt stmt = (CancelCloudWarmUpStmt) ddlStmt;
+            ((CloudEnv) env).cancelCloudWarmUp(stmt);

Review Comment:
   We should check cloud mode here 



##########
fe/fe-core/src/main/java/org/apache/doris/common/FeConstants.java:
##########
@@ -80,6 +80,7 @@ public class FeConstants {
     public static final String FS_PREFIX_FILE = "file";
 
     public static final String INTERNAL_DB_NAME = "__internal_schema";
+    public static final String INTERNAL_FILE_CACHE_HOTSPOT_TABLE_NAME = "cloud_cache_hotspot";

Review Comment:
   Is this compatible with the existing warehouses?



##########
be/src/cloud/cloud_tablet_hotspot.h:
##########


Review Comment:
   lack of comment



##########
fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudEnv.java:
##########
@@ -438,4 +452,63 @@ public void replayUpdateCloudReplica(UpdateCloudReplicaInfo info) throws MetaNot
     public boolean getEnableStorageVault() {
         return this.enableStorageVault;
     }
+
+    public long loadCloudWarmUpJob(DataInputStream dis, long checksum) throws Exception {
+        // same as loadAlterJob
+
+        int size = dis.readInt();
+        long newChecksum = checksum ^ size;
+        if (size > 0) {
+            // There should be no old cloudWarmUp jobs, if exist throw exception, should not use this FE version
+            throw new IOException("There are [" + size + "] cloud warm up jobs."
+                    + " Please downgrade FE to an older version and handle residual jobs");
+        }
+
+        // finished or cancelled jobs
+        size = dis.readInt();
+        newChecksum ^= size;
+        if (size > 0) {
+            throw new IOException("There are [" + size + "] old finished or cancelled cloud warm up jobs."
+                    + " Please downgrade FE to an older version and handle residual jobs");
+        }
+
+        size = dis.readInt();
+        newChecksum ^= size;
+        for (int i = 0; i < size; i++) {
+            CloudWarmUpJob cloudWarmUpJob = CloudWarmUpJob.read(dis);
+            if (cloudWarmUpJob.isExpire() || cloudWarmUpJob.getJobState() == JobState.DELETED) {
+                LOG.info("cloud warm up job is expired, {}, ignore it", cloudWarmUpJob.getJobId());
+                continue;
+            }
+            this.getCacheHotspotMgr().addCloudWarmUpJob(cloudWarmUpJob);
+        }
+        LOG.info("finished replay cloud warm up job from image");
+        return newChecksum;
+    }
+
+    public long saveCloudWarmUpJob(CountingDataOutputStream dos, long checksum) throws IOException {

Review Comment:
   make sure the save/load procedures are compactible with the existing warehouse



##########
be/src/cloud/cloud_tablet_hotspot.cpp:
##########
@@ -0,0 +1,195 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_tablet_hotspot.h"
+
+#include <chrono>
+#include <mutex>
+
+#include "cloud/config.h"
+#include "olap/tablet_fwd.h"
+#include "runtime/exec_env.h"
+
+namespace doris {
+
+void TabletHotspot::count(const BaseTabletSPtr& tablet) {
+    if (!config::is_cloud_mode()) return;
+    size_t slot_idx = tablet->tablet_id() % s_slot_size;
+    auto& slot = _tablets_hotspot[slot_idx];
+    std::lock_guard lock(slot.mtx);
+    HotspotCounterPtr counter;
+    if (auto iter = slot.map.find(tablet->tablet_id()); iter == slot.map.end()) {
+        counter = std::make_shared<HotspotCounter>(tablet->table_id(), tablet->index_id(),
+                                                   tablet->partition_id());
+        slot.map.insert(std::make_pair(tablet->tablet_id(), counter));
+    } else {
+        counter = iter->second;
+    }
+    counter->last_access_time = std::chrono::system_clock::now();
+    counter->cur_counter++;
+}
+
+TabletHotspot::TabletHotspot() {
+    _counter_thread = std::thread(&TabletHotspot::make_dot_point, this);
+}
+
+TabletHotspot::~TabletHotspot() {
+    {
+        std::lock_guard lock(_mtx);
+        _closed = true;
+    }
+    _cond.notify_all();
+    if (_counter_thread.joinable()) {
+        _counter_thread.join();
+    }
+}
+
+struct MapKeyHash {
+    int64_t operator()(const std::pair<int64_t, int64_t>& key) const {
+        return std::hash<int64_t> {}(key.first) + std::hash<int64_t> {}(key.second);
+    }
+};
+struct TabletHotspotMapValue {
+    uint64_t qpd = 0;
+    uint64_t qpw = 0;

Review Comment:
   Add comment for the abbreviation



##########
fe/fe-core/src/main/java/org/apache/doris/analysis/WarmUpClusterStmt.java:
##########
@@ -0,0 +1,162 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.cloud.catalog.CloudEnv;
+import org.apache.doris.cloud.system.CloudSystemInfoService;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.common.ErrorReport;
+import org.apache.doris.common.UserException;
+
+import com.google.common.base.Strings;
+import org.apache.commons.lang3.tuple.ImmutableTriple;
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+public class WarmUpClusterStmt extends StatementBase {
+    private static final Logger LOG = LogManager.getLogger(WarmUpClusterStmt.class);
+    private List<Map<TableName, String>> tableList;
+    private List<Triple<String, String, String>> tables = new ArrayList<>();
+    private String dstClusterName;
+    private String srcClusterName;
+    private boolean isWarmUpWithTable;
+    private boolean isForce;
+
+    public WarmUpClusterStmt(String dstClusterName, String srcClusterName, boolean isForce) {
+        this.dstClusterName = dstClusterName;
+        this.srcClusterName = srcClusterName;
+        this.isForce = isForce;
+        this.isWarmUpWithTable = false;
+    }
+
+    public WarmUpClusterStmt(String dstClusterName, List<Map<TableName, String>> tableList, boolean isForce) {
+        this.dstClusterName = dstClusterName;
+        this.tableList = tableList;
+        this.isForce = isForce;
+        this.isWarmUpWithTable = true;
+    }
+
+    @Override
+    public void analyze(Analyzer analyzer) throws AnalysisException, UserException {
+        super.analyze(analyzer);

Review Comment:
   throw exception if not cloud mode 



##########
fe/fe-core/src/main/java/org/apache/doris/analysis/ShowCacheHotSpotStmt.java:
##########
@@ -0,0 +1,182 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.PrimitiveType;
+import org.apache.doris.catalog.ScalarType;
+import org.apache.doris.cloud.system.CloudSystemInfoService;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.FeConstants;
+import org.apache.doris.common.UserException;
+import org.apache.doris.datasource.InternalCatalog;
+import org.apache.doris.qe.ShowResultSetMetaData;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.StringReader;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+public class ShowCacheHotSpotStmt extends ShowStmt {
+    public static final ShowResultSetMetaData[] RESULT_SET_META_DATAS = {
+        ShowResultSetMetaData.builder()
+            .addColumn(new Column("cluster_id", ScalarType.createType(PrimitiveType.BIGINT)))
+            .addColumn(new Column("cluster_name", ScalarType.createVarchar(128)))
+            .addColumn(new Column("table_id", ScalarType.createType(PrimitiveType.BIGINT)))
+            .addColumn(new Column("table_name", ScalarType.createVarchar(128)))
+            .build(),
+        ShowResultSetMetaData.builder()
+            .addColumn(new Column("table_id", ScalarType.createType(PrimitiveType.BIGINT)))
+            .addColumn(new Column("table_name", ScalarType.createVarchar(128)))
+            .addColumn(new Column("partition_id", ScalarType.createType(PrimitiveType.BIGINT)))
+            .addColumn(new Column("partition_name", ScalarType.createVarchar(65535)))
+            .build(),
+        ShowResultSetMetaData.builder()
+            .addColumn(new Column("partition_id", ScalarType.createType(PrimitiveType.BIGINT)))
+            .addColumn(new Column("partition_name", ScalarType.createVarchar(65535)))
+            .build()
+    };
+    private int metaDataPos;
+    private static final Logger LOG = LogManager.getLogger(ShowCacheHotSpotStmt.class);
+    private static final TableName TABLE_NAME = new TableName(InternalCatalog.INTERNAL_CATALOG_NAME,
+            FeConstants.INTERNAL_DB_NAME, FeConstants.INTERNAL_FILE_CACHE_HOTSPOT_TABLE_NAME);
+    private final String tablePath;
+    private List<String> whereExprVariables = Arrays.asList("cluster_name", "table_name");
+    private List<String> whereExprValues = new ArrayList<>();
+    private List<String> whereExpr = new ArrayList<>();
+    private SelectStmt selectStmt;
+
+    public ShowCacheHotSpotStmt(String url) {
+        tablePath = url;
+    }
+
+    @Override
+    public void analyze(Analyzer analyzer) throws UserException {

Review Comment:
   throw exception if not cloud mode
   
   



##########
fe/fe-core/src/main/java/org/apache/doris/analysis/ShowCloudWarmUpStmt.java:
##########
@@ -0,0 +1,129 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.analysis.BinaryPredicate.Operator;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.ScalarType;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.UserException;
+import org.apache.doris.qe.ShowResultSetMetaData;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+public class ShowCloudWarmUpStmt extends ShowStmt {
+    private static final Logger LOG = LogManager.getLogger(ShowCloudWarmUpStmt.class);
+    private Expr whereClause;
+    private boolean showAllJobs = false;
+    private long jobId = -1;
+
+    private static final ImmutableList<String> TITLE_NAMES = new ImmutableList.Builder<String>()
+            .add("JobId")
+            .add("ClusterName")
+            .add("Status")
+            .add("Type")
+            .add("CreateTime")
+            .add("FinishBatch")
+            .add("AllBatch")
+            .add("FinishTime")
+            .add("ErrMsg")
+            .build();
+
+    public ShowCloudWarmUpStmt(Expr whereClause) {
+        this.whereClause = whereClause;
+    }
+
+    public long getJobId() {
+        return jobId;
+    }
+
+    public boolean showAllJobs() {
+        return showAllJobs;
+    }
+
+    @Override
+    public void analyze(Analyzer analyzer) throws AnalysisException, UserException {

Review Comment:
   throw exception if not cloud mode
   
   



##########
be/src/cloud/cloud_warm_up_manager.h:
##########


Review Comment:
   lack of comment



##########
be/src/io/cache/block_file_cache_downloader.cpp:
##########


Review Comment:
   lack of comment



##########
be/src/olap/compaction.cpp:
##########
@@ -1122,6 +1122,8 @@ Status CloudCompactionMixin::construct_output_rowset_writer(RowsetWriterContext&
     ctx.compaction_level =
             _engine.cumu_compaction_policy(compaction_policy)->new_compaction_level(_input_rowsets);
 
+    ctx.write_file_cache = true;

Review Comment:
   make it a config option.
   base compaction may flush the cache, which cause performance issue



##########
be/src/io/cache/block_file_cache_downloader.h:
##########


Review Comment:
   lack of comment



##########
be/src/cloud/cloud_tablet_hotspot.cpp:
##########


Review Comment:
   lack of comment



##########
be/src/olap/tablet_meta.h:
##########
@@ -191,6 +191,7 @@ class TabletMeta {
     void revise_delete_bitmap_unlocked(const DeleteBitmap& delete_bitmap);
 
     const std::vector<RowsetMetaSharedPtr>& all_stale_rs_metas() const;
+    std::map<std::string, RowsetMetaSharedPtr> snapshot_rs_metas() const;

Review Comment:
   add comment what is returned, what are the key and value represent for



##########
be/src/cloud/cloud_tablet_hotspot.cpp:
##########
@@ -0,0 +1,195 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_tablet_hotspot.h"
+
+#include <chrono>
+#include <mutex>
+
+#include "cloud/config.h"
+#include "olap/tablet_fwd.h"
+#include "runtime/exec_env.h"
+
+namespace doris {
+
+void TabletHotspot::count(const BaseTabletSPtr& tablet) {
+    if (!config::is_cloud_mode()) return;
+    size_t slot_idx = tablet->tablet_id() % s_slot_size;
+    auto& slot = _tablets_hotspot[slot_idx];
+    std::lock_guard lock(slot.mtx);
+    HotspotCounterPtr counter;
+    if (auto iter = slot.map.find(tablet->tablet_id()); iter == slot.map.end()) {
+        counter = std::make_shared<HotspotCounter>(tablet->table_id(), tablet->index_id(),
+                                                   tablet->partition_id());
+        slot.map.insert(std::make_pair(tablet->tablet_id(), counter));
+    } else {
+        counter = iter->second;
+    }
+    counter->last_access_time = std::chrono::system_clock::now();
+    counter->cur_counter++;
+}
+
+TabletHotspot::TabletHotspot() {
+    _counter_thread = std::thread(&TabletHotspot::make_dot_point, this);
+}
+
+TabletHotspot::~TabletHotspot() {
+    {
+        std::lock_guard lock(_mtx);
+        _closed = true;
+    }
+    _cond.notify_all();
+    if (_counter_thread.joinable()) {
+        _counter_thread.join();
+    }
+}
+
+struct MapKeyHash {
+    int64_t operator()(const std::pair<int64_t, int64_t>& key) const {
+        return std::hash<int64_t> {}(key.first) + std::hash<int64_t> {}(key.second);
+    }
+};
+struct TabletHotspotMapValue {
+    uint64_t qpd = 0;
+    uint64_t qpw = 0;

Review Comment:
   We may also need to consider query per hour



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2068003060

   TeamCity be ut coverage result:
    Function Coverage: 35.25% (8917/25298) 
    Line Coverage: 27.00% (73309/271508)
    Region Coverage: 26.18% (37884/144714)
    Branch Coverage: 22.99% (19288/83898)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/c4fc00513476c131946015035443c34f1fe57661_c4fc00513476c131946015035443c34f1fe57661/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]

Posted by "Lchangliang (via GitHub)" <gi...@apache.org>.
Lchangliang commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2069413401

   run p0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]

Posted by "Lchangliang (via GitHub)" <gi...@apache.org>.
Lchangliang commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2067992832

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #32804:
URL: https://github.com/apache/doris/pull/32804#discussion_r1537871834


##########
be/src/cloud/cloud_backend_service.cpp:
##########
@@ -45,4 +52,28 @@ Status CloudBackendService::create_service(CloudStorageEngine& engine, ExecEnv*
     return Status::OK();
 }
 
+void CloudBackendService::sync_load_for_tablets(TSyncLoadForTabletsResponse&,
+                                                const TSyncLoadForTabletsRequest& request) {
+    auto f = [this, tablet_ids = request.tablet_ids]() {
+        std::for_each(tablet_ids.cbegin(), tablet_ids.cend(), [this](int64_t tablet_id) {
+            CloudTabletSPtr tablet;
+            auto result = _engine.tablet_mgr().get_tablet(tablet_id, true);
+            if (!result.has_value()) {
+                return;
+            }
+            Status st = result.value()->sync_rowsets(-1, true);
+            if (!st.ok()) {
+                LOG_WARNING("failed to sync load for tablet").error(st);
+            }
+        });
+    };
+}
+
+void CloudBackendService::get_top_n_hot_partitions(TGetTopNHotPartitionsResponse& response,

Review Comment:
   warning: method 'get_top_n_hot_partitions' can be made static [readability-convert-member-functions-to-static]
   
   ```suggestion
   static void CloudBackendService::get_top_n_hot_partitions(TGetTopNHotPartitionsResponse& response,
   ```
   



##########
be/src/io/cache/block_file_cache_downloader.h:
##########
@@ -0,0 +1,108 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/Interpreters/Cache/FileCacheFactory.h
+// and modified by Doris
+
+#pragma once
+
+#include <gen_cpp/internal_service.pb.h>

Review Comment:
   warning: 'gen_cpp/internal_service.pb.h' file not found [clang-diagnostic-error]
   ```cpp
   #include <gen_cpp/internal_service.pb.h>
            ^
   ```
   



##########
be/src/util/s3_util.h:
##########
@@ -19,6 +19,7 @@
 
 #include <aws/core/Aws.h>

Review Comment:
   warning: 'aws/core/Aws.h' file not found [clang-diagnostic-error]
   ```cpp
   #include <aws/core/Aws.h>
            ^
   ```
   



##########
be/src/cloud/cloud_warm_up_manager.cpp:
##########
@@ -0,0 +1,211 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_warm_up_manager.h"
+
+#include <bthread/countdown_event.h>
+
+#include <algorithm>
+#include <cstddef>
+#include <tuple>
+
+#include "cloud/cloud_tablet_mgr.h"
+#include "common/logging.h"
+#include "io/cache/block_file_cache_downloader.h"
+#include "olap/rowset/beta_rowset.h"
+#include "olap/rowset/segment_v2/inverted_index_desc.h"
+#include "olap/tablet.h"
+#include "runtime/exec_env.h"
+#include "util/time.h"
+
+namespace doris {
+
+CloudWarmUpManager* CloudWarmUpManager::instance() {
+    return ExecEnv::GetInstance()->cloud_warm_up_manager();
+}
+
+CloudWarmUpManager::CloudWarmUpManager(CloudStorageEngine& engine) : _engine(engine) {
+    _download_thread = std::thread(&CloudWarmUpManager::handle_jobs, this);
+}
+
+CloudWarmUpManager::~CloudWarmUpManager() {
+    {
+        std::lock_guard lock(_mtx);
+        _closed = true;
+    }
+    _cond.notify_all();
+    if (_download_thread.joinable()) {
+        _download_thread.join();
+    }
+}
+
+void CloudWarmUpManager::handle_jobs() {
+#ifndef BE_TEST
+    while (true) {
+        JobMeta* cur_job = nullptr;
+        {
+            std::unique_lock lock(_mtx);
+            _cond.wait(lock, [this]() { return _closed || !_pending_job_metas.empty(); });
+            if (_closed) break;
+            cur_job = &_pending_job_metas.front();
+        }
+        for (int64_t tablet_id : cur_job->tablet_ids) {
+            auto res = _engine.tablet_mgr().get_tablet(tablet_id);
+            if (!res.has_value()) {
+                LOG_WARNING("Warm up error ").tag("tablet_id", tablet_id).error(res.error());
+                continue;
+            }
+            auto tablet = res.value();
+            auto st = tablet->sync_rowsets();
+            if (!st) {
+                LOG_WARNING("Warm up error ").tag("tablet_id", tablet_id).error(st);
+                continue;
+            }
+            std::shared_ptr<bthread::CountdownEvent> wait =
+                    std::make_shared<bthread::CountdownEvent>();
+            auto callback = [=](Status st) {
+                if (!st) {
+                    LOG_WARNING("Warm up error ").error(st);
+                }
+                wait->signal();
+            };
+            auto tablet_meta = tablet->tablet_meta();
+            auto rs_metas = tablet_meta->snapshot_rs_metas();
+            for (auto& [_, rs] : rs_metas) {
+                for (int64_t seg_id = 0; seg_id < rs->num_segments(); seg_id++) {
+                    io::S3FileMeta download_file_meta;
+                    download_file_meta.file_size = 0;
+                    download_file_meta.file_system = rs->fs();
+                    std::string seg_path = BetaRowset::remote_segment_path(rs->tablet_id(),
+                                                                           rs->rowset_id(), seg_id);
+                    download_file_meta.path = seg_path;
+                    // download_file_meta.expiration_time =
+                    //         tablet_meta->ttl_seconds() == 0
+                    //                 ? 0
+                    //                 : rs->newest_write_timestamp() + tablet_meta->ttl_seconds();
+                    if (download_file_meta.expiration_time <= UnixSeconds()) {
+                        download_file_meta.expiration_time = 0;
+                    }
+                    download_file_meta.download_callback = callback;
+                    wait->add_count();
+                    io::FileCacheBlockDownloader::instance()->submit_download_task(
+                            std::move(download_file_meta));
+                }
+            }
+            // TODO(liuchangliang): flow control
+            if (!wait->wait()) {
+                LOG_WARNING("Warm up tablet {} take a long time", tablet_meta->tablet_id());
+            }
+        }
+        {
+            std::unique_lock lock(_mtx);
+            _finish_job.push_back(std::move(*cur_job));
+            _pending_job_metas.pop_front();
+        }
+    }
+#endif
+}
+
+JobMeta::JobMeta(const TJobMeta& meta)
+        : be_ip(meta.be_ip), brpc_port(meta.brpc_port), tablet_ids(meta.tablet_ids) {
+    switch (meta.download_type) {
+    case TDownloadType::BE:
+        download_type = DownloadType::BE;
+        break;
+    case TDownloadType::S3:
+        download_type = DownloadType::S3;
+        break;
+    }
+}
+
+Status CloudWarmUpManager::check_and_set_job_id(int64_t job_id) {
+    std::lock_guard lock(_mtx);
+    if (_cur_job_id == 0) {
+        _cur_job_id = job_id;
+    }
+    Status st = Status::OK();
+    if (_cur_job_id != job_id) {
+        st = Status::InternalError("The job {} is running", _cur_job_id);
+    }
+    return st;
+}
+
+Status CloudWarmUpManager::check_and_set_batch_id(int64_t job_id, int64_t batch_id, bool* retry) {
+    std::lock_guard lock(_mtx);
+    Status st = Status::OK();
+    if (_cur_job_id != 0 && _cur_job_id != job_id) {
+        st = Status::InternalError("The job {} is not current job, current job is {}", job_id,
+                                   _cur_job_id);
+        return st;
+    }
+    if (_cur_job_id == 0) {
+        _cur_job_id = job_id;
+    }
+    if (_cur_batch_id == batch_id) {
+        *retry = true;
+        return st;
+    }
+    if (_pending_job_metas.empty()) {
+        _cur_batch_id = batch_id;
+    } else {
+        st = Status::InternalError("The batch {} is not finish", _cur_batch_id);
+    }
+    return st;
+}
+
+void CloudWarmUpManager::add_job(const std::vector<TJobMeta>& job_metas) {
+    {
+        std::lock_guard lock(_mtx);
+        std::for_each(job_metas.begin(), job_metas.end(),
+                      [this](const TJobMeta& meta) { _pending_job_metas.emplace_back(meta); });
+    }
+    _cond.notify_all();
+}
+
+#ifdef BE_TEST
+void CloudWarmUpManager::consumer_job() {
+    {
+        std::unique_lock lock(_mtx);
+        _finish_job.push_back(_pending_job_metas.front());
+        _pending_job_metas.pop_front();
+    }
+}
+
+#endif
+
+std::tuple<int64_t, int64_t, int64_t, int64_t> CloudWarmUpManager::get_current_job_state() {

Review Comment:
   warning: method 'get_current_job_state' can be made static [readability-convert-member-functions-to-static]
   
   be/src/cloud/cloud_warm_up_manager.h:62:
   ```diff
   -     std::tuple<int64_t, int64_t, int64_t, int64_t> get_current_job_state();
   +     static std::tuple<int64_t, int64_t, int64_t, int64_t> get_current_job_state();
   ```
   



##########
be/src/cloud/cloud_tablet_hotspot.cpp:
##########
@@ -0,0 +1,196 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_tablet_hotspot.h"
+
+#include <chrono>
+#include <mutex>
+
+#include "cloud/config.h"
+#include "olap/tablet_fwd.h"
+#include "runtime/exec_env.h"
+
+namespace doris {
+
+void TabletHotspot::count(const BaseTabletSPtr& tablet) {
+    if (!config::is_cloud_mode()) return;

Review Comment:
   warning: statement should be inside braces [readability-braces-around-statements]
   
   ```suggestion
       if (!config::is_cloud_mode()) { return;
   }
   ```
   



##########
be/src/util/s3_rate_limiter.h:
##########
@@ -0,0 +1,79 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <bvar/bvar.h>

Review Comment:
   warning: 'bvar/bvar.h' file not found [clang-diagnostic-error]
   ```cpp
   #include <bvar/bvar.h>
            ^
   ```
   



##########
be/src/cloud/cloud_tablet_hotspot.cpp:
##########
@@ -0,0 +1,196 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_tablet_hotspot.h"
+
+#include <chrono>
+#include <mutex>
+
+#include "cloud/config.h"
+#include "olap/tablet_fwd.h"
+#include "runtime/exec_env.h"
+
+namespace doris {
+
+void TabletHotspot::count(const BaseTabletSPtr& tablet) {

Review Comment:
   warning: method 'count' can be made static [readability-convert-member-functions-to-static]
   
   be/src/cloud/cloud_tablet_hotspot.h:55:
   ```diff
   -     void count(const BaseTabletSPtr& tablet);
   +     static void count(const BaseTabletSPtr& tablet);
   ```
   



##########
be/src/cloud/cloud_tablet_hotspot.h:
##########
@@ -0,0 +1,74 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <atomic>
+#include <chrono>
+#include <condition_variable>
+#include <memory>
+
+#include "gen_cpp/BackendService.h"

Review Comment:
   warning: 'gen_cpp/BackendService.h' file not found [clang-diagnostic-error]
   ```cpp
   #include "gen_cpp/BackendService.h"
            ^
   ```
   



##########
be/src/cloud/cloud_warm_up_manager.cpp:
##########
@@ -0,0 +1,211 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_warm_up_manager.h"
+
+#include <bthread/countdown_event.h>
+
+#include <algorithm>
+#include <cstddef>
+#include <tuple>
+
+#include "cloud/cloud_tablet_mgr.h"
+#include "common/logging.h"
+#include "io/cache/block_file_cache_downloader.h"
+#include "olap/rowset/beta_rowset.h"
+#include "olap/rowset/segment_v2/inverted_index_desc.h"
+#include "olap/tablet.h"
+#include "runtime/exec_env.h"
+#include "util/time.h"
+
+namespace doris {
+
+CloudWarmUpManager* CloudWarmUpManager::instance() {
+    return ExecEnv::GetInstance()->cloud_warm_up_manager();
+}
+
+CloudWarmUpManager::CloudWarmUpManager(CloudStorageEngine& engine) : _engine(engine) {
+    _download_thread = std::thread(&CloudWarmUpManager::handle_jobs, this);
+}
+
+CloudWarmUpManager::~CloudWarmUpManager() {
+    {
+        std::lock_guard lock(_mtx);
+        _closed = true;
+    }
+    _cond.notify_all();
+    if (_download_thread.joinable()) {
+        _download_thread.join();
+    }
+}
+
+void CloudWarmUpManager::handle_jobs() {
+#ifndef BE_TEST
+    while (true) {
+        JobMeta* cur_job = nullptr;
+        {
+            std::unique_lock lock(_mtx);
+            _cond.wait(lock, [this]() { return _closed || !_pending_job_metas.empty(); });
+            if (_closed) break;

Review Comment:
   warning: statement should be inside braces [readability-braces-around-statements]
   
   ```suggestion
               if (_closed) { break;
   }
   ```
   



##########
be/src/io/cache/block_file_cache_downloader.cpp:
##########
@@ -0,0 +1,411 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/Interpreters/Cache/FileCacheFactory.h
+// and modified by Doris
+
+#include "io/cache/block_file_cache_downloader.h"
+
+#include <aws/transfer/TransferHandle.h>
+#include <aws/transfer/TransferManager.h>
+#include <bthread/countdown_event.h>
+#include <bvar/bvar.h>
+#include <fmt/core.h>
+#include <gen_cpp/internal_service.pb.h>
+
+#include <memory>
+#include <mutex>
+#include <variant>
+
+#include "cloud/cloud_tablet_mgr.h"
+#include "common/config.h"
+#include "common/logging.h"
+#include "common/sync_point.h"
+#include "io/cache/block_file_cache.h"
+#include "io/cache/block_file_cache_factory.h"
+#include "io/cache/file_block.h"
+#include "io/cache/file_cache_common.h"
+#include "io/fs/err_utils.h"
+#include "io/fs/s3_common.h"
+#include "io/fs/s3_file_bufferpool.h"
+#include "io/fs/s3_file_system.h"
+#include "olap/rowset/beta_rowset.h"
+#include "olap/tablet.h"
+// #include "runtime/exec_env.h"
+#include "util/bvar_helper.h"
+#include "util/s3_util.h"
+
+using Aws::S3::Model::GetObjectRequest;
+
+namespace doris::io {
+
+namespace {
+bvar::Adder<uint64_t> file_cache_downloader_counter("file_cache_downloader", "size");
+
+std::string s3_path(std::string_view bucket, std::string_view key) {
+    return fmt::format("s3://{}/{}", bucket, key);
+}
+
+Status _download_part(std::shared_ptr<::Aws::S3::S3Client> client, std::string key_name,
+                      std::string bucket, size_t offset, size_t size, Slice& s) {
+    GetObjectRequest request;
+    request.WithBucket(bucket).WithKey(key_name);
+    request.SetRange(fmt::format("bytes={}-{}", offset, offset + size - 1));
+    request.SetResponseStreamFactory(AwsWriteableStreamFactory((void*)s.get_data(), size));
+    SCOPED_BVAR_LATENCY(s3_bvar::s3_get_latency);
+    auto outcome = SYNC_POINT_HOOK_RETURN_VALUE(DO_S3_GET_RATE_LIMIT(client->GetObject(request)),
+                                                "io::_download_part", std::cref(request).get(), &s);
+
+    TEST_SYNC_POINT_CALLBACK("io::_download_part::error", &outcome);
+    if (!outcome.IsSuccess()) {
+        return s3fs_error(outcome.GetError(),
+                          fmt::format("failed to read from {}", s3_path(bucket, key_name)));
+    }
+    auto bytes_read = outcome.GetResult().GetContentLength();
+    if (bytes_read != size) {
+        return Status::InternalError("failed to read from {}(bytes read: {}, bytes req: {})",
+                                     s3_path(bucket, key_name), bytes_read, size);
+    }
+    s.size = bytes_read;
+
+    return Status::OK();
+}
+
+// maybe we should move this logic inside s3 file bufferpool.cpp
+void _append_data_to_file_cache(FileBlocksHolderPtr holder, Slice data) {
+    size_t offset = 0;
+    std::for_each(
+            holder->file_blocks.begin(), holder->file_blocks.end(), [&](FileBlockSPtr& file_block) {
+                if (file_block->is_downloader() && offset < data.size) {
+                    size_t append_size = std::min(data.size - offset, file_block->range().size());
+                    Slice append_data(data.data + offset, append_size);
+                    Status st;
+                    st = file_block->append(append_data);
+                    if (st.ok()) {
+                        st = file_block->finalize();
+                    }
+                    if (!st.ok()) {
+                        LOG_WARNING("failed to append data to file cache").error(st);
+                    }
+                }
+                offset += file_block->range().size();
+            });
+}
+
+} // namespace
+
+struct DownloadTaskExecutor {
+    DownloadTaskExecutor() = default;
+    ~DownloadTaskExecutor() = default;
+
+    void execute(std::shared_ptr<Aws::S3::S3Client> client, std::string key_name, size_t offset,

Review Comment:
   warning: function 'execute' exceeds recommended size/complexity thresholds [readability-function-size]
   ```cpp
       void execute(std::shared_ptr<Aws::S3::S3Client> client, std::string key_name, size_t offset,
            ^
   ```
   <details>
   <summary>Additional context</summary>
   
   **be/src/io/cache/block_file_cache_downloader.cpp:114:** 83 lines including whitespace and comments (threshold 80)
   ```cpp
       void execute(std::shared_ptr<Aws::S3::S3Client> client, std::string key_name, size_t offset,
            ^
   ```
   
   </details>
   



##########
be/src/cloud/cloud_tablet_hotspot.cpp:
##########
@@ -0,0 +1,196 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_tablet_hotspot.h"
+
+#include <chrono>
+#include <mutex>
+
+#include "cloud/config.h"
+#include "olap/tablet_fwd.h"
+#include "runtime/exec_env.h"
+
+namespace doris {
+
+void TabletHotspot::count(const BaseTabletSPtr& tablet) {
+    if (!config::is_cloud_mode()) return;
+    size_t slot_idx = tablet->tablet_id() % s_slot_size;
+    auto& slot = _tablets_hotspot[slot_idx];
+    std::lock_guard lock(slot.mtx);
+    HotspotCounterPtr counter;
+    if (auto iter = slot.map.find(tablet->tablet_id()); iter == slot.map.end()) {
+        counter = std::make_shared<HotspotCounter>(tablet->table_id(), tablet->index_id(),
+                                                   tablet->partition_id());
+        slot.map.insert(std::make_pair(tablet->tablet_id(), counter));
+    } else {
+        counter = iter->second;
+    }
+    counter->last_access_time = std::chrono::system_clock::now();
+    counter->cur_counter++;
+}
+
+TabletHotspot::TabletHotspot() {
+    _counter_thread = std::thread(&TabletHotspot::make_dot_point, this);
+}
+
+TabletHotspot::~TabletHotspot() {
+    {
+        std::lock_guard lock(_mtx);
+        _closed = true;
+    }
+    _cond.notify_all();
+    if (_counter_thread.joinable()) {
+        _counter_thread.join();
+    }
+}
+
+struct MapKeyHash {
+    int64_t operator()(const std::pair<int64_t, int64_t>& key) const {
+        return std::hash<int64_t> {}(key.first) + std::hash<int64_t> {}(key.second);
+    }
+};
+struct TabletHotspotMapValue {
+    uint64_t qpd = 0;
+    uint64_t qpw = 0;
+    int64_t last_access_time;
+};
+
+using TabletHotspotMapKey = std::pair<int64_t, int64_t>;
+
+TabletHotspot* TabletHotspot::instance() {
+    // return ExecEnv::GetInstance()->tabletHotspot();
+    return nullptr;
+}
+
+void TabletHotspot::get_top_n_hot_partition(std::vector<THotTableMessage>* hot_tables) {

Review Comment:
   warning: method 'get_top_n_hot_partition' can be made static [readability-convert-member-functions-to-static]
   
   ```suggestion
   static void TabletHotspot::get_top_n_hot_partition(std::vector<THotTableMessage>* hot_tables) {
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]

Posted by "deardeng (via GitHub)" <gi...@apache.org>.
deardeng commented on code in PR #32804:
URL: https://github.com/apache/doris/pull/32804#discussion_r1570080600


##########
be/src/cloud/cloud_warm_up_manager.cpp:
##########
@@ -0,0 +1,208 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_warm_up_manager.h"
+
+#include <bthread/countdown_event.h>
+
+#include <algorithm>
+#include <cstddef>
+#include <tuple>
+
+#include "cloud/cloud_tablet_mgr.h"
+#include "common/logging.h"
+#include "io/cache/block_file_cache_downloader.h"
+#include "olap/rowset/beta_rowset.h"
+#include "olap/rowset/segment_v2/inverted_index_desc.h"
+#include "olap/tablet.h"
+#include "runtime/exec_env.h"
+#include "util/time.h"
+
+namespace doris {
+
+CloudWarmUpManager* CloudWarmUpManager::instance() {
+    return ExecEnv::GetInstance()->cloud_warm_up_manager();
+}
+
+CloudWarmUpManager::CloudWarmUpManager(CloudStorageEngine& engine) : _engine(engine) {
+    _download_thread = std::thread(&CloudWarmUpManager::handle_jobs, this);
+}
+
+CloudWarmUpManager::~CloudWarmUpManager() {
+    {
+        std::lock_guard lock(_mtx);
+        _closed = true;
+    }
+    _cond.notify_all();
+    if (_download_thread.joinable()) {
+        _download_thread.join();
+    }
+}
+
+void CloudWarmUpManager::handle_jobs() {
+#ifndef BE_TEST
+    while (true) {
+        JobMeta* cur_job = nullptr;
+        {
+            std::unique_lock lock(_mtx);
+            _cond.wait(lock, [this]() { return _closed || !_pending_job_metas.empty(); });
+            if (_closed) break;

Review Comment:
   这里be close了,还要继续做66行的任务么?看着没有return啊



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2067705675

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 38666 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit c060c92d39ae3b91179b918804f04d6b4692c178, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17613	4530	4315	4315
   q2	2013	196	181	181
   q3	10425	1164	1161	1161
   q4	10192	770	848	770
   q5	7541	2712	2686	2686
   q6	218	131	132	131
   q7	993	593	585	585
   q8	9226	2094	2054	2054
   q9	7482	6676	6569	6569
   q10	8603	3522	3615	3522
   q11	446	230	234	230
   q12	455	222	211	211
   q13	17768	3013	2971	2971
   q14	264	239	231	231
   q15	525	496	473	473
   q16	524	372	378	372
   q17	980	639	705	639
   q18	7459	6784	6665	6665
   q19	6946	1494	1512	1494
   q20	656	310	311	310
   q21	3616	2790	2827	2790
   q22	362	310	306	306
   Total cold run time: 114307 ms
   Total hot run time: 38666 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4342	4203	4299	4203
   q2	377	278	269	269
   q3	3069	2814	2836	2814
   q4	1945	1649	1663	1649
   q5	5341	5330	5310	5310
   q6	213	124	124	124
   q7	2195	1893	1875	1875
   q8	3223	3382	3359	3359
   q9	8635	8612	8721	8612
   q10	4108	3931	4003	3931
   q11	605	489	492	489
   q12	814	613	607	607
   q13	16575	3209	3128	3128
   q14	315	270	278	270
   q15	536	490	481	481
   q16	494	450	456	450
   q17	1833	1595	1520	1520
   q18	8124	8019	7853	7853
   q19	1670	1586	1564	1564
   q20	2061	1890	1831	1831
   q21	8311	5084	4992	4992
   q22	538	468	455	455
   Total cold run time: 75324 ms
   Total hot run time: 55786 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2068005775

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 38596 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit c4fc00513476c131946015035443c34f1fe57661, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17594	4321	4219	4219
   q2	2010	188	181	181
   q3	10457	1202	1210	1202
   q4	10203	813	745	745
   q5	7571	2727	2663	2663
   q6	216	139	137	137
   q7	1008	617	573	573
   q8	9222	2065	2054	2054
   q9	7217	6626	6563	6563
   q10	8468	3548	3485	3485
   q11	452	241	228	228
   q12	491	225	213	213
   q13	18144	2944	2956	2944
   q14	276	232	240	232
   q15	522	478	498	478
   q16	523	404	376	376
   q17	972	729	640	640
   q18	7413	6975	6830	6830
   q19	5709	1538	1497	1497
   q20	635	326	302	302
   q21	3517	2722	2875	2722
   q22	364	312	312	312
   Total cold run time: 112984 ms
   Total hot run time: 38596 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4352	4325	4201	4201
   q2	374	275	267	267
   q3	2965	2714	2689	2689
   q4	1859	1548	1595	1548
   q5	5359	5313	5313	5313
   q6	209	126	128	126
   q7	2218	1860	1900	1860
   q8	3221	3329	3339	3329
   q9	8549	8578	8681	8578
   q10	4115	3883	4003	3883
   q11	597	509	505	505
   q12	812	615	672	615
   q13	17142	3177	3152	3152
   q14	317	298	309	298
   q15	536	494	473	473
   q16	496	455	454	454
   q17	1882	1526	1517	1517
   q18	8156	8073	7789	7789
   q19	1692	1555	1576	1555
   q20	2059	1880	1819	1819
   q21	8040	4998	4886	4886
   q22	584	461	466	461
   Total cold run time: 75534 ms
   Total hot run time: 55318 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]

Posted by "deardeng (via GitHub)" <gi...@apache.org>.
deardeng commented on code in PR #32804:
URL: https://github.com/apache/doris/pull/32804#discussion_r1570080600


##########
be/src/cloud/cloud_warm_up_manager.cpp:
##########
@@ -0,0 +1,208 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_warm_up_manager.h"
+
+#include <bthread/countdown_event.h>
+
+#include <algorithm>
+#include <cstddef>
+#include <tuple>
+
+#include "cloud/cloud_tablet_mgr.h"
+#include "common/logging.h"
+#include "io/cache/block_file_cache_downloader.h"
+#include "olap/rowset/beta_rowset.h"
+#include "olap/rowset/segment_v2/inverted_index_desc.h"
+#include "olap/tablet.h"
+#include "runtime/exec_env.h"
+#include "util/time.h"
+
+namespace doris {
+
+CloudWarmUpManager* CloudWarmUpManager::instance() {
+    return ExecEnv::GetInstance()->cloud_warm_up_manager();
+}
+
+CloudWarmUpManager::CloudWarmUpManager(CloudStorageEngine& engine) : _engine(engine) {
+    _download_thread = std::thread(&CloudWarmUpManager::handle_jobs, this);
+}
+
+CloudWarmUpManager::~CloudWarmUpManager() {
+    {
+        std::lock_guard lock(_mtx);
+        _closed = true;
+    }
+    _cond.notify_all();
+    if (_download_thread.joinable()) {
+        _download_thread.join();
+    }
+}
+
+void CloudWarmUpManager::handle_jobs() {
+#ifndef BE_TEST
+    while (true) {
+        JobMeta* cur_job = nullptr;
+        {
+            std::unique_lock lock(_mtx);
+            _cond.wait(lock, [this]() { return _closed || !_pending_job_metas.empty(); });
+            if (_closed) break;

Review Comment:
   这里be close了,还要继续做66行的任务么?看着没有return啊



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]

Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #32804:
URL: https://github.com/apache/doris/pull/32804#discussion_r1574317098


##########
be/src/cloud/config.h:
##########
@@ -88,4 +88,6 @@ DECLARE_mInt32(schedule_sync_tablets_interval_s);
 // Cloud mow
 DECLARE_mInt32(mow_stream_load_commit_retry_times);
 
+DECLARE_mInt32(sync_load_for_tablets_thread);

Review Comment:
   sync_load_for_tablets_num_threads



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2067571986

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 184218 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit fb510c126b36d4d5669ed574c9304e5f09895977, data reload: false
   
   query1	912	374	376	374
   query2	6524	2548	2465	2465
   query3	6922	218	218	218
   query4	23742	21401	21244	21244
   query5	4160	413	417	413
   query6	264	182	167	167
   query7	4592	288	286	286
   query8	247	186	189	186
   query9	8466	2263	2260	2260
   query10	609	250	244	244
   query11	14687	14286	14155	14155
   query12	142	91	86	86
   query13	1646	358	366	358
   query14	8833	7119	8040	7119
   query15	266	182	190	182
   query16	8252	268	272	268
   query17	1912	583	539	539
   query18	2101	272	269	269
   query19	280	148	147	147
   query20	92	85	86	85
   query21	203	128	125	125
   query22	4978	4851	4853	4851
   query23	33946	32770	33391	32770
   query24	10826	2974	2909	2909
   query25	634	382	406	382
   query26	1601	145	147	145
   query27	3040	311	314	311
   query28	7369	1964	1961	1961
   query29	958	610	584	584
   query30	302	174	171	171
   query31	960	732	746	732
   query32	94	55	53	53
   query33	747	246	237	237
   query34	1057	482	478	478
   query35	847	682	697	682
   query36	1076	890	882	882
   query37	138	71	71	71
   query38	3364	3192	3198	3192
   query39	1574	1541	1536	1536
   query40	285	134	130	130
   query41	49	47	44	44
   query42	104	95	98	95
   query43	604	531	563	531
   query44	1204	725	728	725
   query45	281	266	275	266
   query46	1074	764	737	737
   query47	1925	1903	1867	1867
   query48	367	303	294	294
   query49	1162	398	399	398
   query50	756	379	391	379
   query51	7006	6743	6732	6732
   query52	104	90	91	90
   query53	364	282	286	282
   query54	304	253	253	253
   query55	81	74	75	74
   query56	253	240	236	236
   query57	1193	1165	1122	1122
   query58	232	199	198	198
   query59	3237	3114	3131	3114
   query60	258	231	235	231
   query61	108	107	111	107
   query62	661	452	447	447
   query63	306	284	284	284
   query64	6065	3993	3988	3988
   query65	3556	3032	3039	3032
   query66	1413	349	365	349
   query67	15340	15006	14892	14892
   query68	6708	538	541	538
   query69	535	315	320	315
   query70	1292	1230	1153	1153
   query71	1488	1266	1268	1266
   query72	6670	2743	2572	2572
   query73	727	320	319	319
   query74	6807	6388	6428	6388
   query75	4022	2675	2671	2671
   query76	4907	1024	912	912
   query77	610	267	279	267
   query78	11070	10087	10212	10087
   query79	12916	532	518	518
   query80	2669	463	429	429
   query81	520	248	239	239
   query82	808	100	95	95
   query83	219	163	164	163
   query84	268	84	86	84
   query85	1559	280	264	264
   query86	415	310	303	303
   query87	3465	3286	3271	3271
   query88	5628	2412	2418	2412
   query89	531	371	370	370
   query90	2008	179	176	176
   query91	121	95	97	95
   query92	56	44	48	44
   query93	8104	513	497	497
   query94	1267	178	178	178
   query95	398	302	289	289
   query96	618	263	268	263
   query97	3141	2949	2968	2949
   query98	243	225	211	211
   query99	1295	862	846	846
   Total cold run time: 307740 ms
   Total hot run time: 184218 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2067708724

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 184684 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit c060c92d39ae3b91179b918804f04d6b4692c178, data reload: false
   
   query1	894	374	360	360
   query2	6230	2595	2329	2329
   query3	6652	222	214	214
   query4	24493	21304	21625	21304
   query5	4129	406	418	406
   query6	277	187	184	184
   query7	4589	293	289	289
   query8	246	195	194	194
   query9	8413	2304	2269	2269
   query10	417	246	242	242
   query11	14882	14100	14191	14100
   query12	133	89	84	84
   query13	1636	361	361	361
   query14	9763	7286	7519	7286
   query15	274	188	179	179
   query16	8154	259	258	258
   query17	1925	557	535	535
   query18	2082	268	262	262
   query19	202	148	151	148
   query20	90	83	86	83
   query21	199	126	122	122
   query22	4989	4831	4831	4831
   query23	34431	32906	33249	32906
   query24	10535	3062	2914	2914
   query25	605	387	386	386
   query26	703	153	158	153
   query27	2407	368	387	368
   query28	6072	2051	2048	2048
   query29	867	619	604	604
   query30	311	171	176	171
   query31	1072	747	771	747
   query32	91	54	53	53
   query33	673	247	254	247
   query34	975	502	485	485
   query35	883	710	721	710
   query36	1107	939	945	939
   query37	111	67	68	67
   query38	3436	3304	3325	3304
   query39	1045	1029	1004	1004
   query40	171	127	124	124
   query41	43	44	44	44
   query42	108	99	101	99
   query43	575	529	546	529
   query44	1141	746	740	740
   query45	278	265	270	265
   query46	1115	758	729	729
   query47	1993	1985	1993	1985
   query48	378	302	306	302
   query49	902	400	394	394
   query50	785	422	400	400
   query51	6930	6884	6905	6884
   query52	96	86	88	86
   query53	338	279	277	277
   query54	299	234	220	220
   query55	79	72	74	72
   query56	236	223	225	223
   query57	1204	1131	1118	1118
   query58	208	192	194	192
   query59	3286	3256	3191	3191
   query60	254	231	229	229
   query61	87	87	85	85
   query62	592	457	437	437
   query63	301	284	281	281
   query64	4836	3962	4053	3962
   query65	3061	3069	3045	3045
   query66	755	334	340	334
   query67	15463	15151	15195	15151
   query68	6724	554	545	545
   query69	555	306	314	306
   query70	1201	1152	1182	1152
   query71	1498	1276	1273	1273
   query72	6662	2743	2568	2568
   query73	731	319	324	319
   query74	6841	6366	6379	6366
   query75	3975	2581	2601	2581
   query76	5048	970	996	970
   query77	610	261	266	261
   query78	10981	10183	10250	10183
   query79	8765	521	523	521
   query80	1672	433	436	433
   query81	514	249	235	235
   query82	867	94	92	92
   query83	195	166	164	164
   query84	261	86	83	83
   query85	1221	255	317	255
   query86	442	277	294	277
   query87	3465	3247	3310	3247
   query88	4990	2428	2418	2418
   query89	536	376	380	376
   query90	1957	182	184	182
   query91	123	94	94	94
   query92	58	48	45	45
   query93	6935	518	509	509
   query94	980	178	175	175
   query95	390	295	296	295
   query96	615	268	261	261
   query97	3139	2961	2931	2931
   query98	223	215	210	210
   query99	1265	813	847	813
   Total cold run time: 294946 ms
   Total hot run time: 184684 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]

Posted by "Lchangliang (via GitHub)" <gi...@apache.org>.
Lchangliang commented on code in PR #32804:
URL: https://github.com/apache/doris/pull/32804#discussion_r1574091111


##########
fe/fe-core/src/main/java/org/apache/doris/common/FeConstants.java:
##########
@@ -80,6 +80,7 @@ public class FeConstants {
     public static final String FS_PREFIX_FILE = "file";
 
     public static final String INTERNAL_DB_NAME = "__internal_schema";
+    public static final String INTERNAL_FILE_CACHE_HOTSPOT_TABLE_NAME = "cloud_cache_hotspot";

Review Comment:
   The table name is not same as selectdb. But it no problem for compatibility.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2067568924

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 39896 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit fb510c126b36d4d5669ed574c9304e5f09895977, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17739	4791	4392	4392
   q2	2726	191	187	187
   q3	11472	1165	1173	1165
   q4	10557	782	855	782
   q5	7597	2791	2722	2722
   q6	222	139	136	136
   q7	1044	626	639	626
   q8	9328	2122	2093	2093
   q9	8188	7351	7266	7266
   q10	9801	3697	3588	3588
   q11	478	229	235	229
   q12	450	227	225	225
   q13	18068	2918	2928	2918
   q14	278	225	241	225
   q15	519	483	463	463
   q16	536	394	393	393
   q17	989	758	733	733
   q18	7454	6860	6846	6846
   q19	4759	1538	1516	1516
   q20	669	314	303	303
   q21	3573	2775	2812	2775
   q22	369	313	332	313
   Total cold run time: 116816 ms
   Total hot run time: 39896 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4318	4206	4252	4206
   q2	382	286	266	266
   q3	2993	2729	2787	2729
   q4	1897	1632	1598	1598
   q5	5351	5345	5354	5345
   q6	204	121	123	121
   q7	2252	1871	1845	1845
   q8	3225	3354	3364	3354
   q9	9283	9225	9282	9225
   q10	3924	3679	3691	3679
   q11	578	476	484	476
   q12	742	578	591	578
   q13	16714	2944	2961	2944
   q14	318	280	290	280
   q15	507	472	466	466
   q16	480	418	429	418
   q17	1769	1563	1458	1458
   q18	7650	7552	7352	7352
   q19	1648	1515	1566	1515
   q20	1948	1736	1747	1736
   q21	4914	4822	4889	4822
   q22	552	468	456	456
   Total cold run time: 71649 ms
   Total hot run time: 54869 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]

Posted by "Lchangliang (via GitHub)" <gi...@apache.org>.
Lchangliang commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2067684922

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]

Posted by "Lchangliang (via GitHub)" <gi...@apache.org>.
Lchangliang commented on code in PR #32804:
URL: https://github.com/apache/doris/pull/32804#discussion_r1574181399


##########
fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudEnv.java:
##########
@@ -438,4 +452,63 @@ public void replayUpdateCloudReplica(UpdateCloudReplicaInfo info) throws MetaNot
     public boolean getEnableStorageVault() {
         return this.enableStorageVault;
     }
+
+    public long loadCloudWarmUpJob(DataInputStream dis, long checksum) throws Exception {
+        // same as loadAlterJob
+
+        int size = dis.readInt();
+        long newChecksum = checksum ^ size;
+        if (size > 0) {
+            // There should be no old cloudWarmUp jobs, if exist throw exception, should not use this FE version
+            throw new IOException("There are [" + size + "] cloud warm up jobs."
+                    + " Please downgrade FE to an older version and handle residual jobs");
+        }
+
+        // finished or cancelled jobs
+        size = dis.readInt();
+        newChecksum ^= size;
+        if (size > 0) {
+            throw new IOException("There are [" + size + "] old finished or cancelled cloud warm up jobs."
+                    + " Please downgrade FE to an older version and handle residual jobs");
+        }
+
+        size = dis.readInt();
+        newChecksum ^= size;
+        for (int i = 0; i < size; i++) {
+            CloudWarmUpJob cloudWarmUpJob = CloudWarmUpJob.read(dis);
+            if (cloudWarmUpJob.isExpire() || cloudWarmUpJob.getJobState() == JobState.DELETED) {
+                LOG.info("cloud warm up job is expired, {}, ignore it", cloudWarmUpJob.getJobId());
+                continue;
+            }
+            this.getCacheHotspotMgr().addCloudWarmUpJob(cloudWarmUpJob);
+        }
+        LOG.info("finished replay cloud warm up job from image");
+        return newChecksum;
+    }
+
+    public long saveCloudWarmUpJob(CountingDataOutputStream dos, long checksum) throws IOException {

Review Comment:
   The persistence is same.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]

Posted by "deardeng (via GitHub)" <gi...@apache.org>.
deardeng commented on code in PR #32804:
URL: https://github.com/apache/doris/pull/32804#discussion_r1574696678


##########
fe/fe-core/src/main/java/org/apache/doris/analysis/CancelCloudWarmUpStmt.java:
##########
@@ -0,0 +1,85 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.analysis.BinaryPredicate.Operator;
+import org.apache.doris.common.AnalysisException;
+
+public class CancelCloudWarmUpStmt extends CancelStmt {
+    private Expr whereClause;
+    private long jobId;
+
+    public CancelCloudWarmUpStmt(Expr whereClause) {
+        this.whereClause = whereClause;
+    }
+
+    public long getJobId() {
+        return jobId;
+    }
+
+    @Override
+    public void analyze(Analyzer analyzer) throws AnalysisException {

Review Comment:
   这些文件 不要放到这个包目录下?
   org/apache/doris/cloud/analysis 可以么?
   ```suggestion
       public void analyze(Analyzer analyzer) throws AnalysisException {
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]

Posted by "Lchangliang (via GitHub)" <gi...@apache.org>.
Lchangliang commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2063485087

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #32804:
URL: https://github.com/apache/doris/pull/32804#discussion_r1559327877


##########
be/src/cloud/cloud_backend_service.cpp:
##########
@@ -45,4 +53,94 @@ Status CloudBackendService::create_service(CloudStorageEngine& engine, ExecEnv*
     return Status::OK();
 }
 
+void CloudBackendService::sync_load_for_tablets(TSyncLoadForTabletsResponse&,
+                                                const TSyncLoadForTabletsRequest& request) {
+    auto f = [this, tablet_ids = request.tablet_ids]() {
+        std::for_each(tablet_ids.cbegin(), tablet_ids.cend(), [this](int64_t tablet_id) {
+            CloudTabletSPtr tablet;
+            auto result = _engine.tablet_mgr().get_tablet(tablet_id, true);
+            if (!result.has_value()) {
+                return;
+            }
+            Status st = result.value()->sync_rowsets(-1, true);
+            if (!st.ok()) {
+                LOG_WARNING("failed to sync load for tablet").error(st);
+            }
+        });
+    };
+    static_cast<void>(_exec_env->sync_load_for_tablets_thread_pool()->submit_func(std::move(f)));
+}
+
+void CloudBackendService::get_top_n_hot_partitions(TGetTopNHotPartitionsResponse& response,
+                                                   const TGetTopNHotPartitionsRequest& request) {
+    TabletHotspot::instance()->get_top_n_hot_partition(&response.hot_tables);
+    response.file_cache_size = io::FileCacheFactory::instance()->get_capacity();
+    response.__isset.hot_tables = !response.hot_tables.empty();
+}
+
+void CloudBackendService::warm_up_tablets(TWarmUpTabletsResponse& response,

Review Comment:
   warning: method 'warm_up_tablets' can be made static [readability-convert-member-functions-to-static]
   
   ```suggestion
   static void CloudBackendService::warm_up_tablets(TWarmUpTabletsResponse& response,
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]

Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #32804:
URL: https://github.com/apache/doris/pull/32804#discussion_r1560703197


##########
fe/fe-common/src/main/java/org/apache/doris/common/Config.java:
##########
@@ -2685,6 +2685,24 @@ public static boolean isNotCloudMode() {
 
     @ConfField(mutable = true, masterOnly = false)
     public static String security_checker_class_name = "";
+    public static int history_cloud_warm_up_job_keep_max_second = 7 * 24 * 3600;

Review Comment:
   miss  @ConfField(?
   BTW, we should use global session variable as far as possible



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #32804:
URL: https://github.com/apache/doris/pull/32804#discussion_r1565143956


##########
be/src/cloud/cloud_backend_service.cpp:
##########
@@ -45,4 +57,137 @@
     return Status::OK();
 }
 
+void CloudBackendService::sync_load_for_tablets(TSyncLoadForTabletsResponse&,
+                                                const TSyncLoadForTabletsRequest& request) {
+    auto f = [this, tablet_ids = request.tablet_ids]() {
+        std::for_each(tablet_ids.cbegin(), tablet_ids.cend(), [this](int64_t tablet_id) {
+            CloudTabletSPtr tablet;
+            auto result = _engine.tablet_mgr().get_tablet(tablet_id, true);
+            if (!result.has_value()) {
+                return;
+            }
+            Status st = result.value()->sync_rowsets(-1, true);
+            if (!st.ok()) {
+                LOG_WARNING("failed to sync load for tablet").error(st);
+            }
+        });
+    };
+    static_cast<void>(_exec_env->sync_load_for_tablets_thread_pool()->submit_func(std::move(f)));
+}
+
+void CloudBackendService::get_top_n_hot_partitions(TGetTopNHotPartitionsResponse& response,
+                                                   const TGetTopNHotPartitionsRequest& request) {
+    TabletHotspot::instance()->get_top_n_hot_partition(&response.hot_tables);
+    response.file_cache_size = io::FileCacheFactory::instance()->get_capacity();
+    response.__isset.hot_tables = !response.hot_tables.empty();
+}
+
+void CloudBackendService::warm_up_tablets(TWarmUpTabletsResponse& response,
+                                          const TWarmUpTabletsRequest& request) {
+    Status st;
+    auto* manager = CloudWarmUpManager::instance();
+    switch (request.type) {
+    case TWarmUpTabletsRequestType::SET_JOB: {
+        LOG_INFO("receive the warm up request.")
+                .tag("request_type", "SET_JOB")
+                .tag("job_id", request.job_id);
+        st = manager->check_and_set_job_id(request.job_id);
+        if (!st) {
+            LOG_WARNING("SET_JOB failed.").error(st);
+            break;
+        }
+        [[fallthrough]];
+    }
+    case TWarmUpTabletsRequestType::SET_BATCH: {
+        LOG_INFO("receive the warm up request.")
+                .tag("request_type", "SET_BATCH")
+                .tag("job_id", request.job_id)
+                .tag("batch_id", request.batch_id)
+                .tag("jobs size", request.job_metas.size());
+        bool retry = false;
+        st = manager->check_and_set_batch_id(request.job_id, request.batch_id, &retry);
+        if (!retry && st) {
+            manager->add_job(request.job_metas);
+        } else {
+            if (retry) {
+                LOG_WARNING("retry the job.")
+                        .tag("job_id", request.job_id)
+                        .tag("batch_id", request.batch_id);
+            } else {
+                LOG_WARNING("SET_BATCH failed.").error(st);
+            }
+        }
+        break;
+    }
+    case TWarmUpTabletsRequestType::GET_CURRENT_JOB_STATE_AND_LEASE: {
+        auto [job_id, batch_id, pending_job_size, finish_job_size] =
+                manager->get_current_job_state();
+        LOG_INFO("receive the warm up request.")
+                .tag("request_type", "GET_CURRENT_JOB_STATE_AND_LEASE")
+                .tag("job_id", job_id)
+                .tag("batch_id", batch_id)
+                .tag("pending_job_size", pending_job_size)
+                .tag("finish_job_size", finish_job_size);
+        response.__set_job_id(job_id);
+        response.__set_batch_id(batch_id);
+        response.__set_pending_job_size(pending_job_size);
+        response.__set_finish_job_size(finish_job_size);
+        break;
+    }
+    case TWarmUpTabletsRequestType::CLEAR_JOB: {
+        LOG_INFO("receive the warm up request.")
+                .tag("request_type", "CLEAR_JOB")
+                .tag("job_id", request.job_id);
+        st = manager->clear_job(request.job_id);
+        break;
+    }
+    default:
+        DCHECK(false);
+    };
+    st.to_thrift(&response.status);
+}
+
+void CloudBackendService::pre_cache_async(TPreCacheAsyncResponse& response,
+                                          const TPreCacheAsyncRequest& request) {
+    std::string brpc_addr = fmt::format("{}:{}", request.host, request.brpc_port);
+    Status st = Status::OK();
+    TStatus t_status;
+    std::shared_ptr<PBackendService_Stub> brpc_stub =
+            _exec_env->brpc_internal_client_cache()->get_new_client_no_cache(brpc_addr);
+    if (!brpc_stub) {
+        st = Status::RpcError("Address {} is wrong", brpc_addr);
+        return;
+    }
+    brpc::Controller cntl;
+    PGetFileCacheMetaRequest brpc_request;
+    std::for_each(request.tablet_ids.cbegin(), request.tablet_ids.cend(),
+                  [&](int64_t tablet_id) { brpc_request.add_tablet_ids(tablet_id); });
+    PGetFileCacheMetaResponse brpc_response;
+    brpc_stub->get_file_cache_meta_by_tablet_id(&cntl, &brpc_request, &brpc_response, nullptr);
+    if (!cntl.Failed()) {
+        std::vector<FileCacheBlockMeta> metas;
+        std::transform(brpc_response.file_cache_segment_metas().cbegin(),
+                       brpc_response.file_cache_segment_metas().cend(), std::back_inserter(metas),
+                       [](const FileCacheBlockMeta& meta) { return meta; });
+        io::DownloadTask download_task(std::move(metas));
+        io::FileCacheBlockDownloader::instance()->submit_download_task(download_task);
+    } else {
+        st = Status::RpcError("{} isn't connected", brpc_addr);
+    }
+    st.to_thrift(&t_status);
+    response.status = t_status;
+}
+
+void CloudBackendService::check_pre_cache(TCheckPreCacheResponse& response,

Review Comment:
   warning: method 'check_pre_cache' can be made static [readability-convert-member-functions-to-static]
   
   ```suggestion
   static void CloudBackendService::check_pre_cache(TCheckPreCacheResponse& response,
   ```
   



##########
be/src/cloud/cloud_backend_service.cpp:
##########
@@ -45,4 +57,137 @@ Status CloudBackendService::create_service(CloudStorageEngine& engine, ExecEnv*
     return Status::OK();
 }
 
+void CloudBackendService::sync_load_for_tablets(TSyncLoadForTabletsResponse&,
+                                                const TSyncLoadForTabletsRequest& request) {
+    auto f = [this, tablet_ids = request.tablet_ids]() {
+        std::for_each(tablet_ids.cbegin(), tablet_ids.cend(), [this](int64_t tablet_id) {
+            CloudTabletSPtr tablet;
+            auto result = _engine.tablet_mgr().get_tablet(tablet_id, true);
+            if (!result.has_value()) {
+                return;
+            }
+            Status st = result.value()->sync_rowsets(-1, true);
+            if (!st.ok()) {
+                LOG_WARNING("failed to sync load for tablet").error(st);
+            }
+        });
+    };
+    static_cast<void>(_exec_env->sync_load_for_tablets_thread_pool()->submit_func(std::move(f)));
+}
+
+void CloudBackendService::get_top_n_hot_partitions(TGetTopNHotPartitionsResponse& response,
+                                                   const TGetTopNHotPartitionsRequest& request) {
+    TabletHotspot::instance()->get_top_n_hot_partition(&response.hot_tables);
+    response.file_cache_size = io::FileCacheFactory::instance()->get_capacity();
+    response.__isset.hot_tables = !response.hot_tables.empty();
+}
+
+void CloudBackendService::warm_up_tablets(TWarmUpTabletsResponse& response,
+                                          const TWarmUpTabletsRequest& request) {
+    Status st;
+    auto* manager = CloudWarmUpManager::instance();
+    switch (request.type) {
+    case TWarmUpTabletsRequestType::SET_JOB: {
+        LOG_INFO("receive the warm up request.")
+                .tag("request_type", "SET_JOB")
+                .tag("job_id", request.job_id);
+        st = manager->check_and_set_job_id(request.job_id);
+        if (!st) {
+            LOG_WARNING("SET_JOB failed.").error(st);
+            break;
+        }
+        [[fallthrough]];
+    }
+    case TWarmUpTabletsRequestType::SET_BATCH: {
+        LOG_INFO("receive the warm up request.")
+                .tag("request_type", "SET_BATCH")
+                .tag("job_id", request.job_id)
+                .tag("batch_id", request.batch_id)
+                .tag("jobs size", request.job_metas.size());
+        bool retry = false;
+        st = manager->check_and_set_batch_id(request.job_id, request.batch_id, &retry);
+        if (!retry && st) {
+            manager->add_job(request.job_metas);
+        } else {
+            if (retry) {
+                LOG_WARNING("retry the job.")
+                        .tag("job_id", request.job_id)
+                        .tag("batch_id", request.batch_id);
+            } else {
+                LOG_WARNING("SET_BATCH failed.").error(st);
+            }
+        }
+        break;
+    }
+    case TWarmUpTabletsRequestType::GET_CURRENT_JOB_STATE_AND_LEASE: {
+        auto [job_id, batch_id, pending_job_size, finish_job_size] =
+                manager->get_current_job_state();
+        LOG_INFO("receive the warm up request.")
+                .tag("request_type", "GET_CURRENT_JOB_STATE_AND_LEASE")
+                .tag("job_id", job_id)
+                .tag("batch_id", batch_id)
+                .tag("pending_job_size", pending_job_size)
+                .tag("finish_job_size", finish_job_size);
+        response.__set_job_id(job_id);
+        response.__set_batch_id(batch_id);
+        response.__set_pending_job_size(pending_job_size);
+        response.__set_finish_job_size(finish_job_size);
+        break;
+    }
+    case TWarmUpTabletsRequestType::CLEAR_JOB: {
+        LOG_INFO("receive the warm up request.")
+                .tag("request_type", "CLEAR_JOB")
+                .tag("job_id", request.job_id);
+        st = manager->clear_job(request.job_id);
+        break;
+    }
+    default:
+        DCHECK(false);
+    };
+    st.to_thrift(&response.status);
+}
+
+void CloudBackendService::pre_cache_async(TPreCacheAsyncResponse& response,

Review Comment:
   warning: method 'pre_cache_async' can be made static [readability-convert-member-functions-to-static]
   
   ```suggestion
   static void CloudBackendService::pre_cache_async(TPreCacheAsyncResponse& response,
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]

Posted by "Lchangliang (via GitHub)" <gi...@apache.org>.
Lchangliang commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2067556199

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2067573101

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 29.91 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit fb510c126b36d4d5669ed574c9304e5f09895977, data reload: false
   
   query1	0.04	0.03	0.02
   query2	0.09	0.04	0.04
   query3	0.23	0.05	0.05
   query4	1.69	0.07	0.06
   query5	0.50	0.50	0.50
   query6	1.47	0.72	0.73
   query7	0.02	0.01	0.01
   query8	0.05	0.04	0.04
   query9	0.55	0.49	0.50
   query10	0.54	0.54	0.54
   query11	0.15	0.11	0.11
   query12	0.14	0.11	0.12
   query13	0.60	0.59	0.58
   query14	0.76	0.76	0.78
   query15	0.82	0.81	0.81
   query16	0.37	0.38	0.35
   query17	0.99	0.97	0.99
   query18	0.20	0.27	0.24
   query19	1.76	1.69	1.79
   query20	0.01	0.02	0.01
   query21	15.43	0.64	0.63
   query22	4.53	7.80	1.50
   query23	18.24	1.42	1.19
   query24	1.50	0.30	0.26
   query25	0.14	0.08	0.09
   query26	0.24	0.16	0.16
   query27	0.08	0.08	0.08
   query28	13.34	1.02	0.98
   query29	12.57	3.35	3.33
   query30	0.27	0.06	0.06
   query31	2.85	0.38	0.36
   query32	3.29	0.46	0.46
   query33	2.80	2.85	2.88
   query34	17.16	4.39	4.41
   query35	4.51	4.44	4.48
   query36	0.66	0.46	0.45
   query37	0.19	0.15	0.15
   query38	0.15	0.14	0.14
   query39	0.05	0.04	0.04
   query40	0.17	0.13	0.15
   query41	0.09	0.04	0.04
   query42	0.05	0.06	0.05
   query43	0.04	0.04	0.04
   Total cold run time: 109.33 s
   Total hot run time: 29.91 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2068010238

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 30.65 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit c4fc00513476c131946015035443c34f1fe57661, data reload: false
   
   query1	0.04	0.03	0.03
   query2	0.09	0.04	0.04
   query3	0.23	0.04	0.04
   query4	1.69	0.08	0.07
   query5	0.48	0.48	0.49
   query6	1.47	0.73	0.72
   query7	0.02	0.01	0.01
   query8	0.05	0.04	0.04
   query9	0.55	0.50	0.50
   query10	0.56	0.55	0.55
   query11	0.16	0.12	0.12
   query12	0.14	0.12	0.12
   query13	0.59	0.58	0.57
   query14	0.76	0.77	0.77
   query15	0.81	0.81	0.81
   query16	0.36	0.36	0.38
   query17	0.95	0.99	0.95
   query18	0.23	0.22	0.24
   query19	1.87	1.79	1.70
   query20	0.01	0.01	0.01
   query21	15.41	0.66	0.65
   query22	4.30	6.55	2.26
   query23	18.33	1.45	1.21
   query24	1.35	0.26	0.31
   query25	0.14	0.08	0.08
   query26	0.27	0.16	0.16
   query27	0.07	0.08	0.08
   query28	13.52	0.99	0.99
   query29	12.62	3.27	3.25
   query30	0.26	0.06	0.06
   query31	2.85	0.36	0.37
   query32	3.32	0.46	0.46
   query33	2.76	2.83	2.85
   query34	17.21	4.42	4.39
   query35	4.50	4.46	4.52
   query36	0.66	0.50	0.46
   query37	0.18	0.15	0.16
   query38	0.15	0.14	0.14
   query39	0.04	0.03	0.04
   query40	0.17	0.16	0.13
   query41	0.09	0.04	0.04
   query42	0.06	0.05	0.05
   query43	0.05	0.04	0.04
   Total cold run time: 109.37 s
   Total hot run time: 30.65 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]

Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #32804:
URL: https://github.com/apache/doris/pull/32804


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2071389060

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org