You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Lchangliang (via GitHub)" <gi...@apache.org> on 2024/03/25 16:18:09 UTC
[PR] (cloud-merge) Support more multi case strategies about cache [doris]
Lchangliang opened a new pull request, #32804:
URL: https://github.com/apache/doris/pull/32804
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]
Posted by "Lchangliang (via GitHub)" <gi...@apache.org>.
Lchangliang commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2068923514
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]
Posted by "Lchangliang (via GitHub)" <gi...@apache.org>.
Lchangliang commented on code in PR #32804:
URL: https://github.com/apache/doris/pull/32804#discussion_r1574094425
##########
fe/fe-core/src/main/java/org/apache/doris/cluster/ClusterNamespace.java:
##########
@@ -32,6 +32,10 @@ public class ClusterNamespace {
private static final String CLUSTER_DELIMITER = ":";
+ public static String getFullName(String cluster, String name) {
+ return linkString(cluster, name);
+ }
Review Comment:
Copy from selectdb, remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2069095990
TeamCity be ut coverage result:
Function Coverage: 35.23% (8915/25303)
Line Coverage: 27.00% (73323/271587)
Region Coverage: 26.17% (37875/144729)
Branch Coverage: 22.98% (19282/83914)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c52664d7fd539e4f04f5a82460df2ad0eb6167cb_c52664d7fd539e4f04f5a82460df2ad0eb6167cb/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2068777997
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2068008859
<details>
<summary>TPC-DS: <b>Total hot run time: 187263 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c4fc00513476c131946015035443c34f1fe57661, data reload: false
query1 902 388 365 365
query2 6171 2507 2422 2422
query3 6660 210 209 209
query4 23110 21468 21383 21383
query5 4179 408 410 408
query6 275 185 182 182
query7 4583 298 289 289
query8 251 202 185 185
query9 8454 2309 2286 2286
query10 418 248 256 248
query11 14782 14144 14213 14144
query12 141 88 86 86
query13 1656 372 366 366
query14 8740 8533 8193 8193
query15 286 186 189 186
query16 8242 261 266 261
query17 1978 574 561 561
query18 2100 275 266 266
query19 330 150 149 149
query20 91 83 83 83
query21 198 141 135 135
query22 5030 4879 4877 4877
query23 33725 33219 33301 33219
query24 10596 3094 3038 3038
query25 571 398 399 398
query26 709 160 164 160
query27 2317 362 376 362
query28 5945 2042 2080 2042
query29 874 624 617 617
query30 298 196 180 180
query31 989 763 762 762
query32 91 51 57 51
query33 650 266 253 253
query34 899 508 528 508
query35 858 729 721 721
query36 1105 964 982 964
query37 111 78 75 75
query38 3492 3362 3360 3360
query39 1648 1561 1704 1561
query40 165 139 131 131
query41 48 42 44 42
query42 106 99 101 99
query43 602 559 549 549
query44 1130 768 768 768
query45 276 259 273 259
query46 1115 775 756 756
query47 2016 1925 1932 1925
query48 395 303 317 303
query49 841 408 398 398
query50 786 399 409 399
query51 6969 6898 6766 6766
query52 101 87 86 86
query53 354 295 281 281
query54 297 224 228 224
query55 79 72 71 71
query56 235 228 225 225
query57 1214 1149 1133 1133
query58 217 216 211 211
query59 3563 3133 3088 3088
query60 251 224 225 224
query61 87 88 85 85
query62 613 453 443 443
query63 311 283 283 283
query64 4879 3963 3863 3863
query65 3089 3066 3070 3066
query66 764 330 338 330
query67 15234 15037 15330 15037
query68 5748 533 539 533
query69 533 304 311 304
query70 1289 1203 1182 1182
query71 1464 1275 1283 1275
query72 6923 2743 2574 2574
query73 725 333 324 324
query74 6971 6504 6504 6504
query75 3826 2634 2652 2634
query76 4212 1057 1032 1032
query77 597 273 275 273
query78 11058 10420 10233 10233
query79 8685 531 534 531
query80 2051 447 458 447
query81 555 250 257 250
query82 1632 102 97 97
query83 323 173 176 173
query84 265 90 89 89
query85 1415 312 310 310
query86 455 308 287 287
query87 3476 3278 3261 3261
query88 4971 2423 2448 2423
query89 571 387 398 387
query90 1929 187 186 186
query91 134 112 104 104
query92 58 48 53 48
query93 6861 511 494 494
query94 1186 175 179 175
query95 388 295 305 295
query96 603 263 261 261
query97 3105 2919 2936 2919
query98 244 231 230 230
query99 1267 846 828 828
Total cold run time: 292712 ms
Total hot run time: 187263 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2067710314
<details>
<summary>ClickBench: <b>Total hot run time: 30.85 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c060c92d39ae3b91179b918804f04d6b4692c178, data reload: false
query1 0.04 0.03 0.03
query2 0.08 0.04 0.04
query3 0.23 0.05 0.05
query4 1.68 0.07 0.07
query5 0.49 0.50 0.51
query6 1.46 0.71 0.72
query7 0.02 0.02 0.01
query8 0.05 0.04 0.04
query9 0.55 0.50 0.50
query10 0.56 0.56 0.54
query11 0.16 0.12 0.11
query12 0.14 0.11 0.11
query13 0.61 0.58 0.59
query14 0.76 0.78 0.79
query15 0.83 0.80 0.81
query16 0.36 0.35 0.35
query17 0.93 0.94 0.94
query18 0.22 0.22 0.26
query19 1.82 1.64 1.70
query20 0.02 0.00 0.01
query21 15.40 0.65 0.65
query22 4.82 6.42 2.54
query23 18.24 1.36 1.29
query24 1.82 0.20 0.22
query25 0.15 0.09 0.08
query26 0.27 0.16 0.16
query27 0.08 0.08 0.07
query28 13.41 1.00 0.99
query29 12.63 3.28 3.25
query30 0.26 0.07 0.06
query31 2.84 0.38 0.37
query32 3.29 0.46 0.46
query33 2.83 2.83 2.75
query34 16.97 4.42 4.48
query35 4.49 4.53 4.48
query36 0.65 0.46 0.46
query37 0.17 0.16 0.15
query38 0.16 0.15 0.15
query39 0.04 0.04 0.04
query40 0.17 0.16 0.13
query41 0.09 0.05 0.05
query42 0.06 0.05 0.05
query43 0.05 0.03 0.04
Total cold run time: 109.9 s
Total hot run time: 30.85 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]
Posted by "Lchangliang (via GitHub)" <gi...@apache.org>.
Lchangliang commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2067071464
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]
Posted by "Lchangliang (via GitHub)" <gi...@apache.org>.
Lchangliang commented on code in PR #32804:
URL: https://github.com/apache/doris/pull/32804#discussion_r1574076003
##########
be/src/cloud/cloud_backend_service.cpp:
##########
@@ -45,4 +57,137 @@ Status CloudBackendService::create_service(CloudStorageEngine& engine, ExecEnv*
return Status::OK();
}
+void CloudBackendService::sync_load_for_tablets(TSyncLoadForTabletsResponse&,
+ const TSyncLoadForTabletsRequest& request) {
+ auto f = [this, tablet_ids = request.tablet_ids]() {
+ std::for_each(tablet_ids.cbegin(), tablet_ids.cend(), [this](int64_t tablet_id) {
+ CloudTabletSPtr tablet;
+ auto result = _engine.tablet_mgr().get_tablet(tablet_id, true);
+ if (!result.has_value()) {
+ return;
+ }
+ Status st = result.value()->sync_rowsets(-1, true);
+ if (!st.ok()) {
+ LOG_WARNING("failed to sync load for tablet").error(st);
+ }
+ });
+ };
+ static_cast<void>(_exec_env->sync_load_for_tablets_thread_pool()->submit_func(std::move(f)));
+}
+
+void CloudBackendService::get_top_n_hot_partitions(TGetTopNHotPartitionsResponse& response,
+ const TGetTopNHotPartitionsRequest& request) {
+ TabletHotspot::instance()->get_top_n_hot_partition(&response.hot_tables);
+ response.file_cache_size = io::FileCacheFactory::instance()->get_capacity();
+ response.__isset.hot_tables = !response.hot_tables.empty();
+}
+
+void CloudBackendService::warm_up_tablets(TWarmUpTabletsResponse& response,
+ const TWarmUpTabletsRequest& request) {
+ Status st;
+ auto* manager = CloudWarmUpManager::instance();
+ switch (request.type) {
+ case TWarmUpTabletsRequestType::SET_JOB: {
+ LOG_INFO("receive the warm up request.")
+ .tag("request_type", "SET_JOB")
+ .tag("job_id", request.job_id);
+ st = manager->check_and_set_job_id(request.job_id);
+ if (!st) {
+ LOG_WARNING("SET_JOB failed.").error(st);
+ break;
+ }
+ [[fallthrough]];
Review Comment:
When sending the first RPC, it contains SET_JOB and SET_BATCH params.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #32804:
URL: https://github.com/apache/doris/pull/32804#discussion_r1570412055
##########
be/src/cloud/config.h:
##########
@@ -88,4 +88,6 @@ DECLARE_mInt32(schedule_sync_tablets_interval_s);
// Cloud mow
DECLARE_mInt32(mow_stream_load_commit_retry_times);
+DECLARE_mInt32(sync_load_for_tablets_thread);
Review Comment:
add a comment
##########
be/src/cloud/cloud_tablet.cpp:
##########
@@ -191,13 +192,40 @@ void CloudTablet::add_rowsets(std::vector<RowsetSharedPtr> to_add, bool version_
auto add_rowsets_directly = [=, this](std::vector<RowsetSharedPtr>& rowsets) {
for (auto& rs : rowsets) {
+ if (version_overlap || warmup_delta_data) {
+#ifndef BE_TEST
Review Comment:
try to use TEST_SYNC_POINT_RETURN() in stead of ifdef
##########
be/src/cloud/cloud_backend_service.cpp:
##########
@@ -45,4 +57,137 @@ Status CloudBackendService::create_service(CloudStorageEngine& engine, ExecEnv*
return Status::OK();
}
+void CloudBackendService::sync_load_for_tablets(TSyncLoadForTabletsResponse&,
+ const TSyncLoadForTabletsRequest& request) {
+ auto f = [this, tablet_ids = request.tablet_ids]() {
+ std::for_each(tablet_ids.cbegin(), tablet_ids.cend(), [this](int64_t tablet_id) {
+ CloudTabletSPtr tablet;
+ auto result = _engine.tablet_mgr().get_tablet(tablet_id, true);
+ if (!result.has_value()) {
+ return;
+ }
+ Status st = result.value()->sync_rowsets(-1, true);
+ if (!st.ok()) {
+ LOG_WARNING("failed to sync load for tablet").error(st);
+ }
+ });
+ };
+ static_cast<void>(_exec_env->sync_load_for_tablets_thread_pool()->submit_func(std::move(f)));
+}
+
+void CloudBackendService::get_top_n_hot_partitions(TGetTopNHotPartitionsResponse& response,
+ const TGetTopNHotPartitionsRequest& request) {
+ TabletHotspot::instance()->get_top_n_hot_partition(&response.hot_tables);
+ response.file_cache_size = io::FileCacheFactory::instance()->get_capacity();
+ response.__isset.hot_tables = !response.hot_tables.empty();
+}
+
+void CloudBackendService::warm_up_tablets(TWarmUpTabletsResponse& response,
+ const TWarmUpTabletsRequest& request) {
+ Status st;
+ auto* manager = CloudWarmUpManager::instance();
+ switch (request.type) {
+ case TWarmUpTabletsRequestType::SET_JOB: {
+ LOG_INFO("receive the warm up request.")
+ .tag("request_type", "SET_JOB")
+ .tag("job_id", request.job_id);
+ st = manager->check_and_set_job_id(request.job_id);
+ if (!st) {
+ LOG_WARNING("SET_JOB failed.").error(st);
+ break;
+ }
+ [[fallthrough]];
+ }
+ case TWarmUpTabletsRequestType::SET_BATCH: {
+ LOG_INFO("receive the warm up request.")
+ .tag("request_type", "SET_BATCH")
+ .tag("job_id", request.job_id)
+ .tag("batch_id", request.batch_id)
+ .tag("jobs size", request.job_metas.size());
+ bool retry = false;
+ st = manager->check_and_set_batch_id(request.job_id, request.batch_id, &retry);
+ if (!retry && st) {
+ manager->add_job(request.job_metas);
+ } else {
+ if (retry) {
+ LOG_WARNING("retry the job.")
+ .tag("job_id", request.job_id)
+ .tag("batch_id", request.batch_id);
+ } else {
+ LOG_WARNING("SET_BATCH failed.").error(st);
+ }
+ }
+ break;
+ }
+ case TWarmUpTabletsRequestType::GET_CURRENT_JOB_STATE_AND_LEASE: {
+ auto [job_id, batch_id, pending_job_size, finish_job_size] =
+ manager->get_current_job_state();
+ LOG_INFO("receive the warm up request.")
+ .tag("request_type", "GET_CURRENT_JOB_STATE_AND_LEASE")
+ .tag("job_id", job_id)
+ .tag("batch_id", batch_id)
+ .tag("pending_job_size", pending_job_size)
+ .tag("finish_job_size", finish_job_size);
+ response.__set_job_id(job_id);
+ response.__set_batch_id(batch_id);
+ response.__set_pending_job_size(pending_job_size);
+ response.__set_finish_job_size(finish_job_size);
+ break;
+ }
+ case TWarmUpTabletsRequestType::CLEAR_JOB: {
+ LOG_INFO("receive the warm up request.")
+ .tag("request_type", "CLEAR_JOB")
+ .tag("job_id", request.job_id);
+ st = manager->clear_job(request.job_id);
+ break;
+ }
+ default:
+ DCHECK(false);
+ };
+ st.to_thrift(&response.status);
+}
+
+void CloudBackendService::pre_cache_async(TPreCacheAsyncResponse& response,
+ const TPreCacheAsyncRequest& request) {
+ std::string brpc_addr = fmt::format("{}:{}", request.host, request.brpc_port);
Review Comment:
Just use the name `warm_up_cache_async()`, `pre_cache` is hard to under stand
##########
be/src/cloud/cloud_backend_service.h:
##########
@@ -35,6 +35,22 @@ class CloudBackendService final : public BaseBackendService {
// TODO(plat1ko): cloud backend functions
+ // If another cluster load, FE need to notify the cluster to sync the load data
+ void sync_load_for_tablets(TSyncLoadForTabletsResponse& response,
+ const TSyncLoadForTabletsRequest& request) override;
+
+ void get_top_n_hot_partitions(TGetTopNHotPartitionsResponse& response,
+ const TGetTopNHotPartitionsRequest& request) override;
+
+ void warm_up_tablets(TWarmUpTabletsResponse& response,
+ const TWarmUpTabletsRequest& request) override;
+
+ void pre_cache_async(TPreCacheAsyncResponse& response,
+ const TPreCacheAsyncRequest& request) override;
+
+ void check_pre_cache(TCheckPreCacheResponse& response,
+ const TCheckPreCacheRequest& request) override;
+
Review Comment:
Add enough comments for each rpc.
##########
be/src/cloud/cloud_internal_service.h:
##########
@@ -36,6 +36,11 @@ class CloudInternalServiceImpl final : public PInternalService {
PAlterVaultSyncResponse* response,
google::protobuf::Closure* done) override;
+ void get_file_cache_meta_by_tablet_id(google::protobuf::RpcController* controller,
Review Comment:
Add comment for behavior description
##########
fe/fe-core/src/main/java/org/apache/doris/cluster/ClusterNamespace.java:
##########
@@ -32,6 +32,10 @@ public class ClusterNamespace {
private static final String CLUSTER_DELIMITER = ":";
+ public static String getFullName(String cluster, String name) {
+ return linkString(cluster, name);
+ }
Review Comment:
the form of `cluster:db` is deprecated for new versions of doris.
why do we need this?
##########
fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudEnv.java:
##########
@@ -438,4 +452,63 @@ public void replayUpdateCloudReplica(UpdateCloudReplicaInfo info) throws MetaNot
public boolean getEnableStorageVault() {
return this.enableStorageVault;
}
+
+ public long loadCloudWarmUpJob(DataInputStream dis, long checksum) throws Exception {
+ // same as loadAlterJob
+
+ int size = dis.readInt();
+ long newChecksum = checksum ^ size;
+ if (size > 0) {
+ // There should be no old cloudWarmUp jobs, if exist throw exception, should not use this FE version
+ throw new IOException("There are [" + size + "] cloud warm up jobs."
+ + " Please downgrade FE to an older version and handle residual jobs");
+ }
+
+ // finished or cancelled jobs
+ size = dis.readInt();
+ newChecksum ^= size;
+ if (size > 0) {
+ throw new IOException("There are [" + size + "] old finished or cancelled cloud warm up jobs."
+ + " Please downgrade FE to an older version and handle residual jobs");
+ }
+
+ size = dis.readInt();
+ newChecksum ^= size;
+ for (int i = 0; i < size; i++) {
+ CloudWarmUpJob cloudWarmUpJob = CloudWarmUpJob.read(dis);
+ if (cloudWarmUpJob.isExpire() || cloudWarmUpJob.getJobState() == JobState.DELETED) {
+ LOG.info("cloud warm up job is expired, {}, ignore it", cloudWarmUpJob.getJobId());
+ continue;
+ }
+ this.getCacheHotspotMgr().addCloudWarmUpJob(cloudWarmUpJob);
+ }
+ LOG.info("finished replay cloud warm up job from image");
+ return newChecksum;
+ }
+
+ public long saveCloudWarmUpJob(CountingDataOutputStream dos, long checksum) throws IOException {
+ // same as saveAlterJob
+
+ Map<Long, CloudWarmUpJob> cloudWarmUpJobs;
+ cloudWarmUpJobs = this.getCacheHotspotMgr().getCloudWarmUpJobs();
+
+ int size = 0;
+ checksum ^= size;
+ dos.writeInt(size);
+
+ checksum ^= size;
+ dos.writeInt(size);
Review Comment:
why do we write and checksum 0 twice here?
add comment if it is by intent
##########
fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudEnv.java:
##########
@@ -438,4 +452,63 @@ public void replayUpdateCloudReplica(UpdateCloudReplicaInfo info) throws MetaNot
public boolean getEnableStorageVault() {
return this.enableStorageVault;
}
+
+ public long loadCloudWarmUpJob(DataInputStream dis, long checksum) throws Exception {
+ // same as loadAlterJob
Review Comment:
remove unnecessary comment....
##########
fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudEnv.java:
##########
@@ -438,4 +452,63 @@ public void replayUpdateCloudReplica(UpdateCloudReplicaInfo info) throws MetaNot
public boolean getEnableStorageVault() {
return this.enableStorageVault;
}
+
+ public long loadCloudWarmUpJob(DataInputStream dis, long checksum) throws Exception {
+ // same as loadAlterJob
+
+ int size = dis.readInt();
+ long newChecksum = checksum ^ size;
+ if (size > 0) {
+ // There should be no old cloudWarmUp jobs, if exist throw exception, should not use this FE version
+ throw new IOException("There are [" + size + "] cloud warm up jobs."
+ + " Please downgrade FE to an older version and handle residual jobs");
+ }
+
+ // finished or cancelled jobs
+ size = dis.readInt();
+ newChecksum ^= size;
+ if (size > 0) {
+ throw new IOException("There are [" + size + "] old finished or cancelled cloud warm up jobs."
+ + " Please downgrade FE to an older version and handle residual jobs");
+ }
+
+ size = dis.readInt();
+ newChecksum ^= size;
+ for (int i = 0; i < size; i++) {
+ CloudWarmUpJob cloudWarmUpJob = CloudWarmUpJob.read(dis);
+ if (cloudWarmUpJob.isExpire() || cloudWarmUpJob.getJobState() == JobState.DELETED) {
+ LOG.info("cloud warm up job is expired, {}, ignore it", cloudWarmUpJob.getJobId());
+ continue;
+ }
+ this.getCacheHotspotMgr().addCloudWarmUpJob(cloudWarmUpJob);
+ }
+ LOG.info("finished replay cloud warm up job from image");
Review Comment:
it is not replay, it is just a load from the image.
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java:
##########
@@ -2024,6 +2024,10 @@ public String getEstimatePartitionSize() {
return "";
}
+ public boolean containPartition(String partitionName) {
Review Comment:
naming `containPartition` -> `containsPartition`
##########
be/src/cloud/cloud_warm_up_manager.cpp:
##########
Review Comment:
lack of comment
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/CancelCloudWarmUpStmt.java:
##########
@@ -0,0 +1,85 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.analysis.BinaryPredicate.Operator;
+import org.apache.doris.common.AnalysisException;
+
+public class CancelCloudWarmUpStmt extends CancelStmt {
+ private Expr whereClause;
+ private long jobId;
+
+ public CancelCloudWarmUpStmt(Expr whereClause) {
+ this.whereClause = whereClause;
+ }
+
+ public long getJobId() {
+ return jobId;
+ }
+
+ @Override
+ public void analyze(Analyzer analyzer) throws AnalysisException {
Review Comment:
throw exception if not cloud mode
##########
be/src/cloud/cloud_internal_service.cpp:
##########
@@ -34,4 +37,68 @@ void CloudInternalServiceImpl::alter_vault_sync(google::protobuf::RpcController*
_engine.sync_storage_vault();
}
+FileCacheType cache_type_to_pb(io::FileCacheType type) {
+ switch (type) {
+ case io::FileCacheType::TTL:
+ return FileCacheType::TTL;
+ break;
+ case io::FileCacheType::INDEX:
+ return FileCacheType::INDEX;
+ break;
+ case io::FileCacheType::NORMAL:
+ return FileCacheType::NORMAL;
+ break;
+ default:
+ DCHECK(false);
+ }
+ return FileCacheType::NORMAL;
+}
+
+void CloudInternalServiceImpl::get_file_cache_meta_by_tablet_id(
+ google::protobuf::RpcController* controller [[maybe_unused]],
+ const PGetFileCacheMetaRequest* request, PGetFileCacheMetaResponse* response,
+ google::protobuf::Closure* done) {
+ brpc::ClosureGuard closure_guard(done);
+ if (!config::enable_file_cache) {
+ LOG_WARNING("try to access tablet file cache meta, but file cache not enabled");
+ return;
+ }
+ std::for_each(
+ request->tablet_ids().cbegin(), request->tablet_ids().cend(), [&](int64_t tablet_id) {
+ auto res = _engine.tablet_mgr().get_tablet(tablet_id);
+ if (!res.has_value()) {
+ LOG(ERROR) << "failed to get tablet: " << tablet_id
+ << " err msg: " << res.error().msg();
+ return;
+ }
+ CloudTabletSPtr tablet = std::move(res.value());
+ auto rowsets = tablet->get_snapshot_rowset();
+ std::for_each(rowsets.cbegin(), rowsets.cend(), [&](RowsetSharedPtr rowset) {
+ std::string rowset_id = rowset->rowset_id().to_string();
+ for (int64_t segment_id = 0; segment_id < rowset->num_segments();
+ segment_id++) {
+ std::string file_name = fmt::format("{}_{}.dat", rowset_id, segment_id);
+ auto cache_key = io::BlockFileCache::hash(file_name);
+ auto* cache = io::FileCacheFactory::instance()->get_by_path(cache_key);
+
+ auto segments_meta = cache->get_hot_blocks_meta(cache_key);
+ std::for_each(
Review Comment:
ditto
##########
be/src/cloud/cloud_internal_service.cpp:
##########
@@ -34,4 +37,68 @@ void CloudInternalServiceImpl::alter_vault_sync(google::protobuf::RpcController*
_engine.sync_storage_vault();
}
+FileCacheType cache_type_to_pb(io::FileCacheType type) {
+ switch (type) {
+ case io::FileCacheType::TTL:
+ return FileCacheType::TTL;
+ break;
+ case io::FileCacheType::INDEX:
+ return FileCacheType::INDEX;
+ break;
+ case io::FileCacheType::NORMAL:
+ return FileCacheType::NORMAL;
+ break;
+ default:
+ DCHECK(false);
+ }
+ return FileCacheType::NORMAL;
+}
+
+void CloudInternalServiceImpl::get_file_cache_meta_by_tablet_id(
+ google::protobuf::RpcController* controller [[maybe_unused]],
+ const PGetFileCacheMetaRequest* request, PGetFileCacheMetaResponse* response,
+ google::protobuf::Closure* done) {
+ brpc::ClosureGuard closure_guard(done);
+ if (!config::enable_file_cache) {
+ LOG_WARNING("try to access tablet file cache meta, but file cache not enabled");
+ return;
+ }
+ std::for_each(
Review Comment:
Use a plain `for` expression to save indention (at least 8 spaces)
```
for (auto it...; it != end();++it) {
...
}
```
##########
gensrc/proto/internal_service.proto:
##########
@@ -796,7 +796,7 @@ message FileCacheSegmentMeta {
}
message PGetFileCacheMetaResponse {
- repeated FileCacheSegmentMeta file_cache_segment_metas = 1;
+ repeated FileCacheBlockMeta file_cache_segment_metas = 1;
Review Comment:
`file_cache_segment_metas` -> `file_cache_block_metas`
##########
be/src/cloud/cloud_internal_service.cpp:
##########
@@ -34,4 +37,68 @@ void CloudInternalServiceImpl::alter_vault_sync(google::protobuf::RpcController*
_engine.sync_storage_vault();
}
+FileCacheType cache_type_to_pb(io::FileCacheType type) {
+ switch (type) {
+ case io::FileCacheType::TTL:
+ return FileCacheType::TTL;
+ break;
+ case io::FileCacheType::INDEX:
+ return FileCacheType::INDEX;
+ break;
+ case io::FileCacheType::NORMAL:
+ return FileCacheType::NORMAL;
+ break;
+ default:
Review Comment:
redundant `break`
##########
be/src/cloud/cloud_backend_service.cpp:
##########
@@ -45,4 +57,137 @@ Status CloudBackendService::create_service(CloudStorageEngine& engine, ExecEnv*
return Status::OK();
}
+void CloudBackendService::sync_load_for_tablets(TSyncLoadForTabletsResponse&,
+ const TSyncLoadForTabletsRequest& request) {
+ auto f = [this, tablet_ids = request.tablet_ids]() {
+ std::for_each(tablet_ids.cbegin(), tablet_ids.cend(), [this](int64_t tablet_id) {
+ CloudTabletSPtr tablet;
+ auto result = _engine.tablet_mgr().get_tablet(tablet_id, true);
+ if (!result.has_value()) {
+ return;
+ }
+ Status st = result.value()->sync_rowsets(-1, true);
+ if (!st.ok()) {
+ LOG_WARNING("failed to sync load for tablet").error(st);
+ }
+ });
+ };
+ static_cast<void>(_exec_env->sync_load_for_tablets_thread_pool()->submit_func(std::move(f)));
+}
+
+void CloudBackendService::get_top_n_hot_partitions(TGetTopNHotPartitionsResponse& response,
+ const TGetTopNHotPartitionsRequest& request) {
+ TabletHotspot::instance()->get_top_n_hot_partition(&response.hot_tables);
+ response.file_cache_size = io::FileCacheFactory::instance()->get_capacity();
+ response.__isset.hot_tables = !response.hot_tables.empty();
+}
+
+void CloudBackendService::warm_up_tablets(TWarmUpTabletsResponse& response,
+ const TWarmUpTabletsRequest& request) {
+ Status st;
+ auto* manager = CloudWarmUpManager::instance();
+ switch (request.type) {
+ case TWarmUpTabletsRequestType::SET_JOB: {
+ LOG_INFO("receive the warm up request.")
+ .tag("request_type", "SET_JOB")
+ .tag("job_id", request.job_id);
+ st = manager->check_and_set_job_id(request.job_id);
+ if (!st) {
+ LOG_WARNING("SET_JOB failed.").error(st);
+ break;
+ }
+ [[fallthrough]];
Review Comment:
Why do we need to `fallthough`?
##########
fe/fe-core/src/main/java/org/apache/doris/qe/DdlExecutor.java:
##########
@@ -374,6 +376,9 @@ public static void execute(Env env, DdlStmt ddlStmt) throws Exception {
env.getAuth().refreshLdap((RefreshLdapStmt) ddlStmt);
} else if (ddlStmt instanceof AlterUserStmt) {
env.getAuth().alterUser((AlterUserStmt) ddlStmt);
+ } else if (ddlStmt instanceof CancelCloudWarmUpStmt) {
+ CancelCloudWarmUpStmt stmt = (CancelCloudWarmUpStmt) ddlStmt;
+ ((CloudEnv) env).cancelCloudWarmUp(stmt);
Review Comment:
We should check cloud mode here
##########
fe/fe-core/src/main/java/org/apache/doris/common/FeConstants.java:
##########
@@ -80,6 +80,7 @@ public class FeConstants {
public static final String FS_PREFIX_FILE = "file";
public static final String INTERNAL_DB_NAME = "__internal_schema";
+ public static final String INTERNAL_FILE_CACHE_HOTSPOT_TABLE_NAME = "cloud_cache_hotspot";
Review Comment:
Is this compatible with the existing warehouses?
##########
be/src/cloud/cloud_tablet_hotspot.h:
##########
Review Comment:
lack of comment
##########
fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudEnv.java:
##########
@@ -438,4 +452,63 @@ public void replayUpdateCloudReplica(UpdateCloudReplicaInfo info) throws MetaNot
public boolean getEnableStorageVault() {
return this.enableStorageVault;
}
+
+ public long loadCloudWarmUpJob(DataInputStream dis, long checksum) throws Exception {
+ // same as loadAlterJob
+
+ int size = dis.readInt();
+ long newChecksum = checksum ^ size;
+ if (size > 0) {
+ // There should be no old cloudWarmUp jobs, if exist throw exception, should not use this FE version
+ throw new IOException("There are [" + size + "] cloud warm up jobs."
+ + " Please downgrade FE to an older version and handle residual jobs");
+ }
+
+ // finished or cancelled jobs
+ size = dis.readInt();
+ newChecksum ^= size;
+ if (size > 0) {
+ throw new IOException("There are [" + size + "] old finished or cancelled cloud warm up jobs."
+ + " Please downgrade FE to an older version and handle residual jobs");
+ }
+
+ size = dis.readInt();
+ newChecksum ^= size;
+ for (int i = 0; i < size; i++) {
+ CloudWarmUpJob cloudWarmUpJob = CloudWarmUpJob.read(dis);
+ if (cloudWarmUpJob.isExpire() || cloudWarmUpJob.getJobState() == JobState.DELETED) {
+ LOG.info("cloud warm up job is expired, {}, ignore it", cloudWarmUpJob.getJobId());
+ continue;
+ }
+ this.getCacheHotspotMgr().addCloudWarmUpJob(cloudWarmUpJob);
+ }
+ LOG.info("finished replay cloud warm up job from image");
+ return newChecksum;
+ }
+
+ public long saveCloudWarmUpJob(CountingDataOutputStream dos, long checksum) throws IOException {
Review Comment:
make sure the save/load procedures are compactible with the existing warehouse
##########
be/src/cloud/cloud_tablet_hotspot.cpp:
##########
@@ -0,0 +1,195 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_tablet_hotspot.h"
+
+#include <chrono>
+#include <mutex>
+
+#include "cloud/config.h"
+#include "olap/tablet_fwd.h"
+#include "runtime/exec_env.h"
+
+namespace doris {
+
+void TabletHotspot::count(const BaseTabletSPtr& tablet) {
+ if (!config::is_cloud_mode()) return;
+ size_t slot_idx = tablet->tablet_id() % s_slot_size;
+ auto& slot = _tablets_hotspot[slot_idx];
+ std::lock_guard lock(slot.mtx);
+ HotspotCounterPtr counter;
+ if (auto iter = slot.map.find(tablet->tablet_id()); iter == slot.map.end()) {
+ counter = std::make_shared<HotspotCounter>(tablet->table_id(), tablet->index_id(),
+ tablet->partition_id());
+ slot.map.insert(std::make_pair(tablet->tablet_id(), counter));
+ } else {
+ counter = iter->second;
+ }
+ counter->last_access_time = std::chrono::system_clock::now();
+ counter->cur_counter++;
+}
+
+TabletHotspot::TabletHotspot() {
+ _counter_thread = std::thread(&TabletHotspot::make_dot_point, this);
+}
+
+TabletHotspot::~TabletHotspot() {
+ {
+ std::lock_guard lock(_mtx);
+ _closed = true;
+ }
+ _cond.notify_all();
+ if (_counter_thread.joinable()) {
+ _counter_thread.join();
+ }
+}
+
+struct MapKeyHash {
+ int64_t operator()(const std::pair<int64_t, int64_t>& key) const {
+ return std::hash<int64_t> {}(key.first) + std::hash<int64_t> {}(key.second);
+ }
+};
+struct TabletHotspotMapValue {
+ uint64_t qpd = 0;
+ uint64_t qpw = 0;
Review Comment:
Add comment for the abbreviation
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/WarmUpClusterStmt.java:
##########
@@ -0,0 +1,162 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.cloud.catalog.CloudEnv;
+import org.apache.doris.cloud.system.CloudSystemInfoService;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.common.ErrorReport;
+import org.apache.doris.common.UserException;
+
+import com.google.common.base.Strings;
+import org.apache.commons.lang3.tuple.ImmutableTriple;
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+public class WarmUpClusterStmt extends StatementBase {
+ private static final Logger LOG = LogManager.getLogger(WarmUpClusterStmt.class);
+ private List<Map<TableName, String>> tableList;
+ private List<Triple<String, String, String>> tables = new ArrayList<>();
+ private String dstClusterName;
+ private String srcClusterName;
+ private boolean isWarmUpWithTable;
+ private boolean isForce;
+
+ public WarmUpClusterStmt(String dstClusterName, String srcClusterName, boolean isForce) {
+ this.dstClusterName = dstClusterName;
+ this.srcClusterName = srcClusterName;
+ this.isForce = isForce;
+ this.isWarmUpWithTable = false;
+ }
+
+ public WarmUpClusterStmt(String dstClusterName, List<Map<TableName, String>> tableList, boolean isForce) {
+ this.dstClusterName = dstClusterName;
+ this.tableList = tableList;
+ this.isForce = isForce;
+ this.isWarmUpWithTable = true;
+ }
+
+ @Override
+ public void analyze(Analyzer analyzer) throws AnalysisException, UserException {
+ super.analyze(analyzer);
Review Comment:
throw exception if not cloud mode
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/ShowCacheHotSpotStmt.java:
##########
@@ -0,0 +1,182 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.PrimitiveType;
+import org.apache.doris.catalog.ScalarType;
+import org.apache.doris.cloud.system.CloudSystemInfoService;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.FeConstants;
+import org.apache.doris.common.UserException;
+import org.apache.doris.datasource.InternalCatalog;
+import org.apache.doris.qe.ShowResultSetMetaData;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.StringReader;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+public class ShowCacheHotSpotStmt extends ShowStmt {
+ public static final ShowResultSetMetaData[] RESULT_SET_META_DATAS = {
+ ShowResultSetMetaData.builder()
+ .addColumn(new Column("cluster_id", ScalarType.createType(PrimitiveType.BIGINT)))
+ .addColumn(new Column("cluster_name", ScalarType.createVarchar(128)))
+ .addColumn(new Column("table_id", ScalarType.createType(PrimitiveType.BIGINT)))
+ .addColumn(new Column("table_name", ScalarType.createVarchar(128)))
+ .build(),
+ ShowResultSetMetaData.builder()
+ .addColumn(new Column("table_id", ScalarType.createType(PrimitiveType.BIGINT)))
+ .addColumn(new Column("table_name", ScalarType.createVarchar(128)))
+ .addColumn(new Column("partition_id", ScalarType.createType(PrimitiveType.BIGINT)))
+ .addColumn(new Column("partition_name", ScalarType.createVarchar(65535)))
+ .build(),
+ ShowResultSetMetaData.builder()
+ .addColumn(new Column("partition_id", ScalarType.createType(PrimitiveType.BIGINT)))
+ .addColumn(new Column("partition_name", ScalarType.createVarchar(65535)))
+ .build()
+ };
+ private int metaDataPos;
+ private static final Logger LOG = LogManager.getLogger(ShowCacheHotSpotStmt.class);
+ private static final TableName TABLE_NAME = new TableName(InternalCatalog.INTERNAL_CATALOG_NAME,
+ FeConstants.INTERNAL_DB_NAME, FeConstants.INTERNAL_FILE_CACHE_HOTSPOT_TABLE_NAME);
+ private final String tablePath;
+ private List<String> whereExprVariables = Arrays.asList("cluster_name", "table_name");
+ private List<String> whereExprValues = new ArrayList<>();
+ private List<String> whereExpr = new ArrayList<>();
+ private SelectStmt selectStmt;
+
+ public ShowCacheHotSpotStmt(String url) {
+ tablePath = url;
+ }
+
+ @Override
+ public void analyze(Analyzer analyzer) throws UserException {
Review Comment:
throw exception if not cloud mode
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/ShowCloudWarmUpStmt.java:
##########
@@ -0,0 +1,129 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.analysis.BinaryPredicate.Operator;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.ScalarType;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.UserException;
+import org.apache.doris.qe.ShowResultSetMetaData;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+public class ShowCloudWarmUpStmt extends ShowStmt {
+ private static final Logger LOG = LogManager.getLogger(ShowCloudWarmUpStmt.class);
+ private Expr whereClause;
+ private boolean showAllJobs = false;
+ private long jobId = -1;
+
+ private static final ImmutableList<String> TITLE_NAMES = new ImmutableList.Builder<String>()
+ .add("JobId")
+ .add("ClusterName")
+ .add("Status")
+ .add("Type")
+ .add("CreateTime")
+ .add("FinishBatch")
+ .add("AllBatch")
+ .add("FinishTime")
+ .add("ErrMsg")
+ .build();
+
+ public ShowCloudWarmUpStmt(Expr whereClause) {
+ this.whereClause = whereClause;
+ }
+
+ public long getJobId() {
+ return jobId;
+ }
+
+ public boolean showAllJobs() {
+ return showAllJobs;
+ }
+
+ @Override
+ public void analyze(Analyzer analyzer) throws AnalysisException, UserException {
Review Comment:
throw exception if not cloud mode
##########
be/src/cloud/cloud_warm_up_manager.h:
##########
Review Comment:
lack of comment
##########
be/src/io/cache/block_file_cache_downloader.cpp:
##########
Review Comment:
lack of comment
##########
be/src/olap/compaction.cpp:
##########
@@ -1122,6 +1122,8 @@ Status CloudCompactionMixin::construct_output_rowset_writer(RowsetWriterContext&
ctx.compaction_level =
_engine.cumu_compaction_policy(compaction_policy)->new_compaction_level(_input_rowsets);
+ ctx.write_file_cache = true;
Review Comment:
make it a config option.
base compaction may flush the cache, which cause performance issue
##########
be/src/io/cache/block_file_cache_downloader.h:
##########
Review Comment:
lack of comment
##########
be/src/cloud/cloud_tablet_hotspot.cpp:
##########
Review Comment:
lack of comment
##########
be/src/olap/tablet_meta.h:
##########
@@ -191,6 +191,7 @@ class TabletMeta {
void revise_delete_bitmap_unlocked(const DeleteBitmap& delete_bitmap);
const std::vector<RowsetMetaSharedPtr>& all_stale_rs_metas() const;
+ std::map<std::string, RowsetMetaSharedPtr> snapshot_rs_metas() const;
Review Comment:
add comment what is returned, what are the key and value represent for
##########
be/src/cloud/cloud_tablet_hotspot.cpp:
##########
@@ -0,0 +1,195 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_tablet_hotspot.h"
+
+#include <chrono>
+#include <mutex>
+
+#include "cloud/config.h"
+#include "olap/tablet_fwd.h"
+#include "runtime/exec_env.h"
+
+namespace doris {
+
+void TabletHotspot::count(const BaseTabletSPtr& tablet) {
+ if (!config::is_cloud_mode()) return;
+ size_t slot_idx = tablet->tablet_id() % s_slot_size;
+ auto& slot = _tablets_hotspot[slot_idx];
+ std::lock_guard lock(slot.mtx);
+ HotspotCounterPtr counter;
+ if (auto iter = slot.map.find(tablet->tablet_id()); iter == slot.map.end()) {
+ counter = std::make_shared<HotspotCounter>(tablet->table_id(), tablet->index_id(),
+ tablet->partition_id());
+ slot.map.insert(std::make_pair(tablet->tablet_id(), counter));
+ } else {
+ counter = iter->second;
+ }
+ counter->last_access_time = std::chrono::system_clock::now();
+ counter->cur_counter++;
+}
+
+TabletHotspot::TabletHotspot() {
+ _counter_thread = std::thread(&TabletHotspot::make_dot_point, this);
+}
+
+TabletHotspot::~TabletHotspot() {
+ {
+ std::lock_guard lock(_mtx);
+ _closed = true;
+ }
+ _cond.notify_all();
+ if (_counter_thread.joinable()) {
+ _counter_thread.join();
+ }
+}
+
+struct MapKeyHash {
+ int64_t operator()(const std::pair<int64_t, int64_t>& key) const {
+ return std::hash<int64_t> {}(key.first) + std::hash<int64_t> {}(key.second);
+ }
+};
+struct TabletHotspotMapValue {
+ uint64_t qpd = 0;
+ uint64_t qpw = 0;
Review Comment:
We may also need to consider query per hour
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2068003060
TeamCity be ut coverage result:
Function Coverage: 35.25% (8917/25298)
Line Coverage: 27.00% (73309/271508)
Region Coverage: 26.18% (37884/144714)
Branch Coverage: 22.99% (19288/83898)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c4fc00513476c131946015035443c34f1fe57661_c4fc00513476c131946015035443c34f1fe57661/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]
Posted by "Lchangliang (via GitHub)" <gi...@apache.org>.
Lchangliang commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2069413401
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]
Posted by "Lchangliang (via GitHub)" <gi...@apache.org>.
Lchangliang commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2067992832
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #32804:
URL: https://github.com/apache/doris/pull/32804#discussion_r1537871834
##########
be/src/cloud/cloud_backend_service.cpp:
##########
@@ -45,4 +52,28 @@ Status CloudBackendService::create_service(CloudStorageEngine& engine, ExecEnv*
return Status::OK();
}
+void CloudBackendService::sync_load_for_tablets(TSyncLoadForTabletsResponse&,
+ const TSyncLoadForTabletsRequest& request) {
+ auto f = [this, tablet_ids = request.tablet_ids]() {
+ std::for_each(tablet_ids.cbegin(), tablet_ids.cend(), [this](int64_t tablet_id) {
+ CloudTabletSPtr tablet;
+ auto result = _engine.tablet_mgr().get_tablet(tablet_id, true);
+ if (!result.has_value()) {
+ return;
+ }
+ Status st = result.value()->sync_rowsets(-1, true);
+ if (!st.ok()) {
+ LOG_WARNING("failed to sync load for tablet").error(st);
+ }
+ });
+ };
+}
+
+void CloudBackendService::get_top_n_hot_partitions(TGetTopNHotPartitionsResponse& response,
Review Comment:
warning: method 'get_top_n_hot_partitions' can be made static [readability-convert-member-functions-to-static]
```suggestion
static void CloudBackendService::get_top_n_hot_partitions(TGetTopNHotPartitionsResponse& response,
```
##########
be/src/io/cache/block_file_cache_downloader.h:
##########
@@ -0,0 +1,108 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/Interpreters/Cache/FileCacheFactory.h
+// and modified by Doris
+
+#pragma once
+
+#include <gen_cpp/internal_service.pb.h>
Review Comment:
warning: 'gen_cpp/internal_service.pb.h' file not found [clang-diagnostic-error]
```cpp
#include <gen_cpp/internal_service.pb.h>
^
```
##########
be/src/util/s3_util.h:
##########
@@ -19,6 +19,7 @@
#include <aws/core/Aws.h>
Review Comment:
warning: 'aws/core/Aws.h' file not found [clang-diagnostic-error]
```cpp
#include <aws/core/Aws.h>
^
```
##########
be/src/cloud/cloud_warm_up_manager.cpp:
##########
@@ -0,0 +1,211 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_warm_up_manager.h"
+
+#include <bthread/countdown_event.h>
+
+#include <algorithm>
+#include <cstddef>
+#include <tuple>
+
+#include "cloud/cloud_tablet_mgr.h"
+#include "common/logging.h"
+#include "io/cache/block_file_cache_downloader.h"
+#include "olap/rowset/beta_rowset.h"
+#include "olap/rowset/segment_v2/inverted_index_desc.h"
+#include "olap/tablet.h"
+#include "runtime/exec_env.h"
+#include "util/time.h"
+
+namespace doris {
+
+CloudWarmUpManager* CloudWarmUpManager::instance() {
+ return ExecEnv::GetInstance()->cloud_warm_up_manager();
+}
+
+CloudWarmUpManager::CloudWarmUpManager(CloudStorageEngine& engine) : _engine(engine) {
+ _download_thread = std::thread(&CloudWarmUpManager::handle_jobs, this);
+}
+
+CloudWarmUpManager::~CloudWarmUpManager() {
+ {
+ std::lock_guard lock(_mtx);
+ _closed = true;
+ }
+ _cond.notify_all();
+ if (_download_thread.joinable()) {
+ _download_thread.join();
+ }
+}
+
+void CloudWarmUpManager::handle_jobs() {
+#ifndef BE_TEST
+ while (true) {
+ JobMeta* cur_job = nullptr;
+ {
+ std::unique_lock lock(_mtx);
+ _cond.wait(lock, [this]() { return _closed || !_pending_job_metas.empty(); });
+ if (_closed) break;
+ cur_job = &_pending_job_metas.front();
+ }
+ for (int64_t tablet_id : cur_job->tablet_ids) {
+ auto res = _engine.tablet_mgr().get_tablet(tablet_id);
+ if (!res.has_value()) {
+ LOG_WARNING("Warm up error ").tag("tablet_id", tablet_id).error(res.error());
+ continue;
+ }
+ auto tablet = res.value();
+ auto st = tablet->sync_rowsets();
+ if (!st) {
+ LOG_WARNING("Warm up error ").tag("tablet_id", tablet_id).error(st);
+ continue;
+ }
+ std::shared_ptr<bthread::CountdownEvent> wait =
+ std::make_shared<bthread::CountdownEvent>();
+ auto callback = [=](Status st) {
+ if (!st) {
+ LOG_WARNING("Warm up error ").error(st);
+ }
+ wait->signal();
+ };
+ auto tablet_meta = tablet->tablet_meta();
+ auto rs_metas = tablet_meta->snapshot_rs_metas();
+ for (auto& [_, rs] : rs_metas) {
+ for (int64_t seg_id = 0; seg_id < rs->num_segments(); seg_id++) {
+ io::S3FileMeta download_file_meta;
+ download_file_meta.file_size = 0;
+ download_file_meta.file_system = rs->fs();
+ std::string seg_path = BetaRowset::remote_segment_path(rs->tablet_id(),
+ rs->rowset_id(), seg_id);
+ download_file_meta.path = seg_path;
+ // download_file_meta.expiration_time =
+ // tablet_meta->ttl_seconds() == 0
+ // ? 0
+ // : rs->newest_write_timestamp() + tablet_meta->ttl_seconds();
+ if (download_file_meta.expiration_time <= UnixSeconds()) {
+ download_file_meta.expiration_time = 0;
+ }
+ download_file_meta.download_callback = callback;
+ wait->add_count();
+ io::FileCacheBlockDownloader::instance()->submit_download_task(
+ std::move(download_file_meta));
+ }
+ }
+ // TODO(liuchangliang): flow control
+ if (!wait->wait()) {
+ LOG_WARNING("Warm up tablet {} take a long time", tablet_meta->tablet_id());
+ }
+ }
+ {
+ std::unique_lock lock(_mtx);
+ _finish_job.push_back(std::move(*cur_job));
+ _pending_job_metas.pop_front();
+ }
+ }
+#endif
+}
+
+JobMeta::JobMeta(const TJobMeta& meta)
+ : be_ip(meta.be_ip), brpc_port(meta.brpc_port), tablet_ids(meta.tablet_ids) {
+ switch (meta.download_type) {
+ case TDownloadType::BE:
+ download_type = DownloadType::BE;
+ break;
+ case TDownloadType::S3:
+ download_type = DownloadType::S3;
+ break;
+ }
+}
+
+Status CloudWarmUpManager::check_and_set_job_id(int64_t job_id) {
+ std::lock_guard lock(_mtx);
+ if (_cur_job_id == 0) {
+ _cur_job_id = job_id;
+ }
+ Status st = Status::OK();
+ if (_cur_job_id != job_id) {
+ st = Status::InternalError("The job {} is running", _cur_job_id);
+ }
+ return st;
+}
+
+Status CloudWarmUpManager::check_and_set_batch_id(int64_t job_id, int64_t batch_id, bool* retry) {
+ std::lock_guard lock(_mtx);
+ Status st = Status::OK();
+ if (_cur_job_id != 0 && _cur_job_id != job_id) {
+ st = Status::InternalError("The job {} is not current job, current job is {}", job_id,
+ _cur_job_id);
+ return st;
+ }
+ if (_cur_job_id == 0) {
+ _cur_job_id = job_id;
+ }
+ if (_cur_batch_id == batch_id) {
+ *retry = true;
+ return st;
+ }
+ if (_pending_job_metas.empty()) {
+ _cur_batch_id = batch_id;
+ } else {
+ st = Status::InternalError("The batch {} is not finish", _cur_batch_id);
+ }
+ return st;
+}
+
+void CloudWarmUpManager::add_job(const std::vector<TJobMeta>& job_metas) {
+ {
+ std::lock_guard lock(_mtx);
+ std::for_each(job_metas.begin(), job_metas.end(),
+ [this](const TJobMeta& meta) { _pending_job_metas.emplace_back(meta); });
+ }
+ _cond.notify_all();
+}
+
+#ifdef BE_TEST
+void CloudWarmUpManager::consumer_job() {
+ {
+ std::unique_lock lock(_mtx);
+ _finish_job.push_back(_pending_job_metas.front());
+ _pending_job_metas.pop_front();
+ }
+}
+
+#endif
+
+std::tuple<int64_t, int64_t, int64_t, int64_t> CloudWarmUpManager::get_current_job_state() {
Review Comment:
warning: method 'get_current_job_state' can be made static [readability-convert-member-functions-to-static]
be/src/cloud/cloud_warm_up_manager.h:62:
```diff
- std::tuple<int64_t, int64_t, int64_t, int64_t> get_current_job_state();
+ static std::tuple<int64_t, int64_t, int64_t, int64_t> get_current_job_state();
```
##########
be/src/cloud/cloud_tablet_hotspot.cpp:
##########
@@ -0,0 +1,196 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_tablet_hotspot.h"
+
+#include <chrono>
+#include <mutex>
+
+#include "cloud/config.h"
+#include "olap/tablet_fwd.h"
+#include "runtime/exec_env.h"
+
+namespace doris {
+
+void TabletHotspot::count(const BaseTabletSPtr& tablet) {
+ if (!config::is_cloud_mode()) return;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (!config::is_cloud_mode()) { return;
}
```
##########
be/src/util/s3_rate_limiter.h:
##########
@@ -0,0 +1,79 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <bvar/bvar.h>
Review Comment:
warning: 'bvar/bvar.h' file not found [clang-diagnostic-error]
```cpp
#include <bvar/bvar.h>
^
```
##########
be/src/cloud/cloud_tablet_hotspot.cpp:
##########
@@ -0,0 +1,196 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_tablet_hotspot.h"
+
+#include <chrono>
+#include <mutex>
+
+#include "cloud/config.h"
+#include "olap/tablet_fwd.h"
+#include "runtime/exec_env.h"
+
+namespace doris {
+
+void TabletHotspot::count(const BaseTabletSPtr& tablet) {
Review Comment:
warning: method 'count' can be made static [readability-convert-member-functions-to-static]
be/src/cloud/cloud_tablet_hotspot.h:55:
```diff
- void count(const BaseTabletSPtr& tablet);
+ static void count(const BaseTabletSPtr& tablet);
```
##########
be/src/cloud/cloud_tablet_hotspot.h:
##########
@@ -0,0 +1,74 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <atomic>
+#include <chrono>
+#include <condition_variable>
+#include <memory>
+
+#include "gen_cpp/BackendService.h"
Review Comment:
warning: 'gen_cpp/BackendService.h' file not found [clang-diagnostic-error]
```cpp
#include "gen_cpp/BackendService.h"
^
```
##########
be/src/cloud/cloud_warm_up_manager.cpp:
##########
@@ -0,0 +1,211 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_warm_up_manager.h"
+
+#include <bthread/countdown_event.h>
+
+#include <algorithm>
+#include <cstddef>
+#include <tuple>
+
+#include "cloud/cloud_tablet_mgr.h"
+#include "common/logging.h"
+#include "io/cache/block_file_cache_downloader.h"
+#include "olap/rowset/beta_rowset.h"
+#include "olap/rowset/segment_v2/inverted_index_desc.h"
+#include "olap/tablet.h"
+#include "runtime/exec_env.h"
+#include "util/time.h"
+
+namespace doris {
+
+CloudWarmUpManager* CloudWarmUpManager::instance() {
+ return ExecEnv::GetInstance()->cloud_warm_up_manager();
+}
+
+CloudWarmUpManager::CloudWarmUpManager(CloudStorageEngine& engine) : _engine(engine) {
+ _download_thread = std::thread(&CloudWarmUpManager::handle_jobs, this);
+}
+
+CloudWarmUpManager::~CloudWarmUpManager() {
+ {
+ std::lock_guard lock(_mtx);
+ _closed = true;
+ }
+ _cond.notify_all();
+ if (_download_thread.joinable()) {
+ _download_thread.join();
+ }
+}
+
+void CloudWarmUpManager::handle_jobs() {
+#ifndef BE_TEST
+ while (true) {
+ JobMeta* cur_job = nullptr;
+ {
+ std::unique_lock lock(_mtx);
+ _cond.wait(lock, [this]() { return _closed || !_pending_job_metas.empty(); });
+ if (_closed) break;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (_closed) { break;
}
```
##########
be/src/io/cache/block_file_cache_downloader.cpp:
##########
@@ -0,0 +1,411 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/Interpreters/Cache/FileCacheFactory.h
+// and modified by Doris
+
+#include "io/cache/block_file_cache_downloader.h"
+
+#include <aws/transfer/TransferHandle.h>
+#include <aws/transfer/TransferManager.h>
+#include <bthread/countdown_event.h>
+#include <bvar/bvar.h>
+#include <fmt/core.h>
+#include <gen_cpp/internal_service.pb.h>
+
+#include <memory>
+#include <mutex>
+#include <variant>
+
+#include "cloud/cloud_tablet_mgr.h"
+#include "common/config.h"
+#include "common/logging.h"
+#include "common/sync_point.h"
+#include "io/cache/block_file_cache.h"
+#include "io/cache/block_file_cache_factory.h"
+#include "io/cache/file_block.h"
+#include "io/cache/file_cache_common.h"
+#include "io/fs/err_utils.h"
+#include "io/fs/s3_common.h"
+#include "io/fs/s3_file_bufferpool.h"
+#include "io/fs/s3_file_system.h"
+#include "olap/rowset/beta_rowset.h"
+#include "olap/tablet.h"
+// #include "runtime/exec_env.h"
+#include "util/bvar_helper.h"
+#include "util/s3_util.h"
+
+using Aws::S3::Model::GetObjectRequest;
+
+namespace doris::io {
+
+namespace {
+bvar::Adder<uint64_t> file_cache_downloader_counter("file_cache_downloader", "size");
+
+std::string s3_path(std::string_view bucket, std::string_view key) {
+ return fmt::format("s3://{}/{}", bucket, key);
+}
+
+Status _download_part(std::shared_ptr<::Aws::S3::S3Client> client, std::string key_name,
+ std::string bucket, size_t offset, size_t size, Slice& s) {
+ GetObjectRequest request;
+ request.WithBucket(bucket).WithKey(key_name);
+ request.SetRange(fmt::format("bytes={}-{}", offset, offset + size - 1));
+ request.SetResponseStreamFactory(AwsWriteableStreamFactory((void*)s.get_data(), size));
+ SCOPED_BVAR_LATENCY(s3_bvar::s3_get_latency);
+ auto outcome = SYNC_POINT_HOOK_RETURN_VALUE(DO_S3_GET_RATE_LIMIT(client->GetObject(request)),
+ "io::_download_part", std::cref(request).get(), &s);
+
+ TEST_SYNC_POINT_CALLBACK("io::_download_part::error", &outcome);
+ if (!outcome.IsSuccess()) {
+ return s3fs_error(outcome.GetError(),
+ fmt::format("failed to read from {}", s3_path(bucket, key_name)));
+ }
+ auto bytes_read = outcome.GetResult().GetContentLength();
+ if (bytes_read != size) {
+ return Status::InternalError("failed to read from {}(bytes read: {}, bytes req: {})",
+ s3_path(bucket, key_name), bytes_read, size);
+ }
+ s.size = bytes_read;
+
+ return Status::OK();
+}
+
+// maybe we should move this logic inside s3 file bufferpool.cpp
+void _append_data_to_file_cache(FileBlocksHolderPtr holder, Slice data) {
+ size_t offset = 0;
+ std::for_each(
+ holder->file_blocks.begin(), holder->file_blocks.end(), [&](FileBlockSPtr& file_block) {
+ if (file_block->is_downloader() && offset < data.size) {
+ size_t append_size = std::min(data.size - offset, file_block->range().size());
+ Slice append_data(data.data + offset, append_size);
+ Status st;
+ st = file_block->append(append_data);
+ if (st.ok()) {
+ st = file_block->finalize();
+ }
+ if (!st.ok()) {
+ LOG_WARNING("failed to append data to file cache").error(st);
+ }
+ }
+ offset += file_block->range().size();
+ });
+}
+
+} // namespace
+
+struct DownloadTaskExecutor {
+ DownloadTaskExecutor() = default;
+ ~DownloadTaskExecutor() = default;
+
+ void execute(std::shared_ptr<Aws::S3::S3Client> client, std::string key_name, size_t offset,
Review Comment:
warning: function 'execute' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
void execute(std::shared_ptr<Aws::S3::S3Client> client, std::string key_name, size_t offset,
^
```
<details>
<summary>Additional context</summary>
**be/src/io/cache/block_file_cache_downloader.cpp:114:** 83 lines including whitespace and comments (threshold 80)
```cpp
void execute(std::shared_ptr<Aws::S3::S3Client> client, std::string key_name, size_t offset,
^
```
</details>
##########
be/src/cloud/cloud_tablet_hotspot.cpp:
##########
@@ -0,0 +1,196 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_tablet_hotspot.h"
+
+#include <chrono>
+#include <mutex>
+
+#include "cloud/config.h"
+#include "olap/tablet_fwd.h"
+#include "runtime/exec_env.h"
+
+namespace doris {
+
+void TabletHotspot::count(const BaseTabletSPtr& tablet) {
+ if (!config::is_cloud_mode()) return;
+ size_t slot_idx = tablet->tablet_id() % s_slot_size;
+ auto& slot = _tablets_hotspot[slot_idx];
+ std::lock_guard lock(slot.mtx);
+ HotspotCounterPtr counter;
+ if (auto iter = slot.map.find(tablet->tablet_id()); iter == slot.map.end()) {
+ counter = std::make_shared<HotspotCounter>(tablet->table_id(), tablet->index_id(),
+ tablet->partition_id());
+ slot.map.insert(std::make_pair(tablet->tablet_id(), counter));
+ } else {
+ counter = iter->second;
+ }
+ counter->last_access_time = std::chrono::system_clock::now();
+ counter->cur_counter++;
+}
+
+TabletHotspot::TabletHotspot() {
+ _counter_thread = std::thread(&TabletHotspot::make_dot_point, this);
+}
+
+TabletHotspot::~TabletHotspot() {
+ {
+ std::lock_guard lock(_mtx);
+ _closed = true;
+ }
+ _cond.notify_all();
+ if (_counter_thread.joinable()) {
+ _counter_thread.join();
+ }
+}
+
+struct MapKeyHash {
+ int64_t operator()(const std::pair<int64_t, int64_t>& key) const {
+ return std::hash<int64_t> {}(key.first) + std::hash<int64_t> {}(key.second);
+ }
+};
+struct TabletHotspotMapValue {
+ uint64_t qpd = 0;
+ uint64_t qpw = 0;
+ int64_t last_access_time;
+};
+
+using TabletHotspotMapKey = std::pair<int64_t, int64_t>;
+
+TabletHotspot* TabletHotspot::instance() {
+ // return ExecEnv::GetInstance()->tabletHotspot();
+ return nullptr;
+}
+
+void TabletHotspot::get_top_n_hot_partition(std::vector<THotTableMessage>* hot_tables) {
Review Comment:
warning: method 'get_top_n_hot_partition' can be made static [readability-convert-member-functions-to-static]
```suggestion
static void TabletHotspot::get_top_n_hot_partition(std::vector<THotTableMessage>* hot_tables) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]
Posted by "deardeng (via GitHub)" <gi...@apache.org>.
deardeng commented on code in PR #32804:
URL: https://github.com/apache/doris/pull/32804#discussion_r1570080600
##########
be/src/cloud/cloud_warm_up_manager.cpp:
##########
@@ -0,0 +1,208 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_warm_up_manager.h"
+
+#include <bthread/countdown_event.h>
+
+#include <algorithm>
+#include <cstddef>
+#include <tuple>
+
+#include "cloud/cloud_tablet_mgr.h"
+#include "common/logging.h"
+#include "io/cache/block_file_cache_downloader.h"
+#include "olap/rowset/beta_rowset.h"
+#include "olap/rowset/segment_v2/inverted_index_desc.h"
+#include "olap/tablet.h"
+#include "runtime/exec_env.h"
+#include "util/time.h"
+
+namespace doris {
+
+CloudWarmUpManager* CloudWarmUpManager::instance() {
+ return ExecEnv::GetInstance()->cloud_warm_up_manager();
+}
+
+CloudWarmUpManager::CloudWarmUpManager(CloudStorageEngine& engine) : _engine(engine) {
+ _download_thread = std::thread(&CloudWarmUpManager::handle_jobs, this);
+}
+
+CloudWarmUpManager::~CloudWarmUpManager() {
+ {
+ std::lock_guard lock(_mtx);
+ _closed = true;
+ }
+ _cond.notify_all();
+ if (_download_thread.joinable()) {
+ _download_thread.join();
+ }
+}
+
+void CloudWarmUpManager::handle_jobs() {
+#ifndef BE_TEST
+ while (true) {
+ JobMeta* cur_job = nullptr;
+ {
+ std::unique_lock lock(_mtx);
+ _cond.wait(lock, [this]() { return _closed || !_pending_job_metas.empty(); });
+ if (_closed) break;
Review Comment:
这里be close了,还要继续做66行的任务么?看着没有return啊
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2067705675
<details>
<summary>TPC-H: <b>Total hot run time: 38666 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c060c92d39ae3b91179b918804f04d6b4692c178, data reload: false
------ Round 1 ----------------------------------
q1 17613 4530 4315 4315
q2 2013 196 181 181
q3 10425 1164 1161 1161
q4 10192 770 848 770
q5 7541 2712 2686 2686
q6 218 131 132 131
q7 993 593 585 585
q8 9226 2094 2054 2054
q9 7482 6676 6569 6569
q10 8603 3522 3615 3522
q11 446 230 234 230
q12 455 222 211 211
q13 17768 3013 2971 2971
q14 264 239 231 231
q15 525 496 473 473
q16 524 372 378 372
q17 980 639 705 639
q18 7459 6784 6665 6665
q19 6946 1494 1512 1494
q20 656 310 311 310
q21 3616 2790 2827 2790
q22 362 310 306 306
Total cold run time: 114307 ms
Total hot run time: 38666 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4342 4203 4299 4203
q2 377 278 269 269
q3 3069 2814 2836 2814
q4 1945 1649 1663 1649
q5 5341 5330 5310 5310
q6 213 124 124 124
q7 2195 1893 1875 1875
q8 3223 3382 3359 3359
q9 8635 8612 8721 8612
q10 4108 3931 4003 3931
q11 605 489 492 489
q12 814 613 607 607
q13 16575 3209 3128 3128
q14 315 270 278 270
q15 536 490 481 481
q16 494 450 456 450
q17 1833 1595 1520 1520
q18 8124 8019 7853 7853
q19 1670 1586 1564 1564
q20 2061 1890 1831 1831
q21 8311 5084 4992 4992
q22 538 468 455 455
Total cold run time: 75324 ms
Total hot run time: 55786 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2068005775
<details>
<summary>TPC-H: <b>Total hot run time: 38596 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c4fc00513476c131946015035443c34f1fe57661, data reload: false
------ Round 1 ----------------------------------
q1 17594 4321 4219 4219
q2 2010 188 181 181
q3 10457 1202 1210 1202
q4 10203 813 745 745
q5 7571 2727 2663 2663
q6 216 139 137 137
q7 1008 617 573 573
q8 9222 2065 2054 2054
q9 7217 6626 6563 6563
q10 8468 3548 3485 3485
q11 452 241 228 228
q12 491 225 213 213
q13 18144 2944 2956 2944
q14 276 232 240 232
q15 522 478 498 478
q16 523 404 376 376
q17 972 729 640 640
q18 7413 6975 6830 6830
q19 5709 1538 1497 1497
q20 635 326 302 302
q21 3517 2722 2875 2722
q22 364 312 312 312
Total cold run time: 112984 ms
Total hot run time: 38596 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4352 4325 4201 4201
q2 374 275 267 267
q3 2965 2714 2689 2689
q4 1859 1548 1595 1548
q5 5359 5313 5313 5313
q6 209 126 128 126
q7 2218 1860 1900 1860
q8 3221 3329 3339 3329
q9 8549 8578 8681 8578
q10 4115 3883 4003 3883
q11 597 509 505 505
q12 812 615 672 615
q13 17142 3177 3152 3152
q14 317 298 309 298
q15 536 494 473 473
q16 496 455 454 454
q17 1882 1526 1517 1517
q18 8156 8073 7789 7789
q19 1692 1555 1576 1555
q20 2059 1880 1819 1819
q21 8040 4998 4886 4886
q22 584 461 466 461
Total cold run time: 75534 ms
Total hot run time: 55318 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]
Posted by "deardeng (via GitHub)" <gi...@apache.org>.
deardeng commented on code in PR #32804:
URL: https://github.com/apache/doris/pull/32804#discussion_r1570080600
##########
be/src/cloud/cloud_warm_up_manager.cpp:
##########
@@ -0,0 +1,208 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_warm_up_manager.h"
+
+#include <bthread/countdown_event.h>
+
+#include <algorithm>
+#include <cstddef>
+#include <tuple>
+
+#include "cloud/cloud_tablet_mgr.h"
+#include "common/logging.h"
+#include "io/cache/block_file_cache_downloader.h"
+#include "olap/rowset/beta_rowset.h"
+#include "olap/rowset/segment_v2/inverted_index_desc.h"
+#include "olap/tablet.h"
+#include "runtime/exec_env.h"
+#include "util/time.h"
+
+namespace doris {
+
+CloudWarmUpManager* CloudWarmUpManager::instance() {
+ return ExecEnv::GetInstance()->cloud_warm_up_manager();
+}
+
+CloudWarmUpManager::CloudWarmUpManager(CloudStorageEngine& engine) : _engine(engine) {
+ _download_thread = std::thread(&CloudWarmUpManager::handle_jobs, this);
+}
+
+CloudWarmUpManager::~CloudWarmUpManager() {
+ {
+ std::lock_guard lock(_mtx);
+ _closed = true;
+ }
+ _cond.notify_all();
+ if (_download_thread.joinable()) {
+ _download_thread.join();
+ }
+}
+
+void CloudWarmUpManager::handle_jobs() {
+#ifndef BE_TEST
+ while (true) {
+ JobMeta* cur_job = nullptr;
+ {
+ std::unique_lock lock(_mtx);
+ _cond.wait(lock, [this]() { return _closed || !_pending_job_metas.empty(); });
+ if (_closed) break;
Review Comment:
这里be close了,还要继续做66行的任务么?看着没有return啊
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #32804:
URL: https://github.com/apache/doris/pull/32804#discussion_r1574317098
##########
be/src/cloud/config.h:
##########
@@ -88,4 +88,6 @@ DECLARE_mInt32(schedule_sync_tablets_interval_s);
// Cloud mow
DECLARE_mInt32(mow_stream_load_commit_retry_times);
+DECLARE_mInt32(sync_load_for_tablets_thread);
Review Comment:
sync_load_for_tablets_num_threads
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2067571986
<details>
<summary>TPC-DS: <b>Total hot run time: 184218 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fb510c126b36d4d5669ed574c9304e5f09895977, data reload: false
query1 912 374 376 374
query2 6524 2548 2465 2465
query3 6922 218 218 218
query4 23742 21401 21244 21244
query5 4160 413 417 413
query6 264 182 167 167
query7 4592 288 286 286
query8 247 186 189 186
query9 8466 2263 2260 2260
query10 609 250 244 244
query11 14687 14286 14155 14155
query12 142 91 86 86
query13 1646 358 366 358
query14 8833 7119 8040 7119
query15 266 182 190 182
query16 8252 268 272 268
query17 1912 583 539 539
query18 2101 272 269 269
query19 280 148 147 147
query20 92 85 86 85
query21 203 128 125 125
query22 4978 4851 4853 4851
query23 33946 32770 33391 32770
query24 10826 2974 2909 2909
query25 634 382 406 382
query26 1601 145 147 145
query27 3040 311 314 311
query28 7369 1964 1961 1961
query29 958 610 584 584
query30 302 174 171 171
query31 960 732 746 732
query32 94 55 53 53
query33 747 246 237 237
query34 1057 482 478 478
query35 847 682 697 682
query36 1076 890 882 882
query37 138 71 71 71
query38 3364 3192 3198 3192
query39 1574 1541 1536 1536
query40 285 134 130 130
query41 49 47 44 44
query42 104 95 98 95
query43 604 531 563 531
query44 1204 725 728 725
query45 281 266 275 266
query46 1074 764 737 737
query47 1925 1903 1867 1867
query48 367 303 294 294
query49 1162 398 399 398
query50 756 379 391 379
query51 7006 6743 6732 6732
query52 104 90 91 90
query53 364 282 286 282
query54 304 253 253 253
query55 81 74 75 74
query56 253 240 236 236
query57 1193 1165 1122 1122
query58 232 199 198 198
query59 3237 3114 3131 3114
query60 258 231 235 231
query61 108 107 111 107
query62 661 452 447 447
query63 306 284 284 284
query64 6065 3993 3988 3988
query65 3556 3032 3039 3032
query66 1413 349 365 349
query67 15340 15006 14892 14892
query68 6708 538 541 538
query69 535 315 320 315
query70 1292 1230 1153 1153
query71 1488 1266 1268 1266
query72 6670 2743 2572 2572
query73 727 320 319 319
query74 6807 6388 6428 6388
query75 4022 2675 2671 2671
query76 4907 1024 912 912
query77 610 267 279 267
query78 11070 10087 10212 10087
query79 12916 532 518 518
query80 2669 463 429 429
query81 520 248 239 239
query82 808 100 95 95
query83 219 163 164 163
query84 268 84 86 84
query85 1559 280 264 264
query86 415 310 303 303
query87 3465 3286 3271 3271
query88 5628 2412 2418 2412
query89 531 371 370 370
query90 2008 179 176 176
query91 121 95 97 95
query92 56 44 48 44
query93 8104 513 497 497
query94 1267 178 178 178
query95 398 302 289 289
query96 618 263 268 263
query97 3141 2949 2968 2949
query98 243 225 211 211
query99 1295 862 846 846
Total cold run time: 307740 ms
Total hot run time: 184218 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2067708724
<details>
<summary>TPC-DS: <b>Total hot run time: 184684 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c060c92d39ae3b91179b918804f04d6b4692c178, data reload: false
query1 894 374 360 360
query2 6230 2595 2329 2329
query3 6652 222 214 214
query4 24493 21304 21625 21304
query5 4129 406 418 406
query6 277 187 184 184
query7 4589 293 289 289
query8 246 195 194 194
query9 8413 2304 2269 2269
query10 417 246 242 242
query11 14882 14100 14191 14100
query12 133 89 84 84
query13 1636 361 361 361
query14 9763 7286 7519 7286
query15 274 188 179 179
query16 8154 259 258 258
query17 1925 557 535 535
query18 2082 268 262 262
query19 202 148 151 148
query20 90 83 86 83
query21 199 126 122 122
query22 4989 4831 4831 4831
query23 34431 32906 33249 32906
query24 10535 3062 2914 2914
query25 605 387 386 386
query26 703 153 158 153
query27 2407 368 387 368
query28 6072 2051 2048 2048
query29 867 619 604 604
query30 311 171 176 171
query31 1072 747 771 747
query32 91 54 53 53
query33 673 247 254 247
query34 975 502 485 485
query35 883 710 721 710
query36 1107 939 945 939
query37 111 67 68 67
query38 3436 3304 3325 3304
query39 1045 1029 1004 1004
query40 171 127 124 124
query41 43 44 44 44
query42 108 99 101 99
query43 575 529 546 529
query44 1141 746 740 740
query45 278 265 270 265
query46 1115 758 729 729
query47 1993 1985 1993 1985
query48 378 302 306 302
query49 902 400 394 394
query50 785 422 400 400
query51 6930 6884 6905 6884
query52 96 86 88 86
query53 338 279 277 277
query54 299 234 220 220
query55 79 72 74 72
query56 236 223 225 223
query57 1204 1131 1118 1118
query58 208 192 194 192
query59 3286 3256 3191 3191
query60 254 231 229 229
query61 87 87 85 85
query62 592 457 437 437
query63 301 284 281 281
query64 4836 3962 4053 3962
query65 3061 3069 3045 3045
query66 755 334 340 334
query67 15463 15151 15195 15151
query68 6724 554 545 545
query69 555 306 314 306
query70 1201 1152 1182 1152
query71 1498 1276 1273 1273
query72 6662 2743 2568 2568
query73 731 319 324 319
query74 6841 6366 6379 6366
query75 3975 2581 2601 2581
query76 5048 970 996 970
query77 610 261 266 261
query78 10981 10183 10250 10183
query79 8765 521 523 521
query80 1672 433 436 433
query81 514 249 235 235
query82 867 94 92 92
query83 195 166 164 164
query84 261 86 83 83
query85 1221 255 317 255
query86 442 277 294 277
query87 3465 3247 3310 3247
query88 4990 2428 2418 2418
query89 536 376 380 376
query90 1957 182 184 182
query91 123 94 94 94
query92 58 48 45 45
query93 6935 518 509 509
query94 980 178 175 175
query95 390 295 296 295
query96 615 268 261 261
query97 3139 2961 2931 2931
query98 223 215 210 210
query99 1265 813 847 813
Total cold run time: 294946 ms
Total hot run time: 184684 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]
Posted by "Lchangliang (via GitHub)" <gi...@apache.org>.
Lchangliang commented on code in PR #32804:
URL: https://github.com/apache/doris/pull/32804#discussion_r1574091111
##########
fe/fe-core/src/main/java/org/apache/doris/common/FeConstants.java:
##########
@@ -80,6 +80,7 @@ public class FeConstants {
public static final String FS_PREFIX_FILE = "file";
public static final String INTERNAL_DB_NAME = "__internal_schema";
+ public static final String INTERNAL_FILE_CACHE_HOTSPOT_TABLE_NAME = "cloud_cache_hotspot";
Review Comment:
The table name is not same as selectdb. But it no problem for compatibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2067568924
<details>
<summary>TPC-H: <b>Total hot run time: 39896 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fb510c126b36d4d5669ed574c9304e5f09895977, data reload: false
------ Round 1 ----------------------------------
q1 17739 4791 4392 4392
q2 2726 191 187 187
q3 11472 1165 1173 1165
q4 10557 782 855 782
q5 7597 2791 2722 2722
q6 222 139 136 136
q7 1044 626 639 626
q8 9328 2122 2093 2093
q9 8188 7351 7266 7266
q10 9801 3697 3588 3588
q11 478 229 235 229
q12 450 227 225 225
q13 18068 2918 2928 2918
q14 278 225 241 225
q15 519 483 463 463
q16 536 394 393 393
q17 989 758 733 733
q18 7454 6860 6846 6846
q19 4759 1538 1516 1516
q20 669 314 303 303
q21 3573 2775 2812 2775
q22 369 313 332 313
Total cold run time: 116816 ms
Total hot run time: 39896 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4318 4206 4252 4206
q2 382 286 266 266
q3 2993 2729 2787 2729
q4 1897 1632 1598 1598
q5 5351 5345 5354 5345
q6 204 121 123 121
q7 2252 1871 1845 1845
q8 3225 3354 3364 3354
q9 9283 9225 9282 9225
q10 3924 3679 3691 3679
q11 578 476 484 476
q12 742 578 591 578
q13 16714 2944 2961 2944
q14 318 280 290 280
q15 507 472 466 466
q16 480 418 429 418
q17 1769 1563 1458 1458
q18 7650 7552 7352 7352
q19 1648 1515 1566 1515
q20 1948 1736 1747 1736
q21 4914 4822 4889 4822
q22 552 468 456 456
Total cold run time: 71649 ms
Total hot run time: 54869 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]
Posted by "Lchangliang (via GitHub)" <gi...@apache.org>.
Lchangliang commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2067684922
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]
Posted by "Lchangliang (via GitHub)" <gi...@apache.org>.
Lchangliang commented on code in PR #32804:
URL: https://github.com/apache/doris/pull/32804#discussion_r1574181399
##########
fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudEnv.java:
##########
@@ -438,4 +452,63 @@ public void replayUpdateCloudReplica(UpdateCloudReplicaInfo info) throws MetaNot
public boolean getEnableStorageVault() {
return this.enableStorageVault;
}
+
+ public long loadCloudWarmUpJob(DataInputStream dis, long checksum) throws Exception {
+ // same as loadAlterJob
+
+ int size = dis.readInt();
+ long newChecksum = checksum ^ size;
+ if (size > 0) {
+ // There should be no old cloudWarmUp jobs, if exist throw exception, should not use this FE version
+ throw new IOException("There are [" + size + "] cloud warm up jobs."
+ + " Please downgrade FE to an older version and handle residual jobs");
+ }
+
+ // finished or cancelled jobs
+ size = dis.readInt();
+ newChecksum ^= size;
+ if (size > 0) {
+ throw new IOException("There are [" + size + "] old finished or cancelled cloud warm up jobs."
+ + " Please downgrade FE to an older version and handle residual jobs");
+ }
+
+ size = dis.readInt();
+ newChecksum ^= size;
+ for (int i = 0; i < size; i++) {
+ CloudWarmUpJob cloudWarmUpJob = CloudWarmUpJob.read(dis);
+ if (cloudWarmUpJob.isExpire() || cloudWarmUpJob.getJobState() == JobState.DELETED) {
+ LOG.info("cloud warm up job is expired, {}, ignore it", cloudWarmUpJob.getJobId());
+ continue;
+ }
+ this.getCacheHotspotMgr().addCloudWarmUpJob(cloudWarmUpJob);
+ }
+ LOG.info("finished replay cloud warm up job from image");
+ return newChecksum;
+ }
+
+ public long saveCloudWarmUpJob(CountingDataOutputStream dos, long checksum) throws IOException {
Review Comment:
The persistence is same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]
Posted by "deardeng (via GitHub)" <gi...@apache.org>.
deardeng commented on code in PR #32804:
URL: https://github.com/apache/doris/pull/32804#discussion_r1574696678
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/CancelCloudWarmUpStmt.java:
##########
@@ -0,0 +1,85 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.analysis.BinaryPredicate.Operator;
+import org.apache.doris.common.AnalysisException;
+
+public class CancelCloudWarmUpStmt extends CancelStmt {
+ private Expr whereClause;
+ private long jobId;
+
+ public CancelCloudWarmUpStmt(Expr whereClause) {
+ this.whereClause = whereClause;
+ }
+
+ public long getJobId() {
+ return jobId;
+ }
+
+ @Override
+ public void analyze(Analyzer analyzer) throws AnalysisException {
Review Comment:
这些文件 不要放到这个包目录下?
org/apache/doris/cloud/analysis 可以么?
```suggestion
public void analyze(Analyzer analyzer) throws AnalysisException {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]
Posted by "Lchangliang (via GitHub)" <gi...@apache.org>.
Lchangliang commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2063485087
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #32804:
URL: https://github.com/apache/doris/pull/32804#discussion_r1559327877
##########
be/src/cloud/cloud_backend_service.cpp:
##########
@@ -45,4 +53,94 @@ Status CloudBackendService::create_service(CloudStorageEngine& engine, ExecEnv*
return Status::OK();
}
+void CloudBackendService::sync_load_for_tablets(TSyncLoadForTabletsResponse&,
+ const TSyncLoadForTabletsRequest& request) {
+ auto f = [this, tablet_ids = request.tablet_ids]() {
+ std::for_each(tablet_ids.cbegin(), tablet_ids.cend(), [this](int64_t tablet_id) {
+ CloudTabletSPtr tablet;
+ auto result = _engine.tablet_mgr().get_tablet(tablet_id, true);
+ if (!result.has_value()) {
+ return;
+ }
+ Status st = result.value()->sync_rowsets(-1, true);
+ if (!st.ok()) {
+ LOG_WARNING("failed to sync load for tablet").error(st);
+ }
+ });
+ };
+ static_cast<void>(_exec_env->sync_load_for_tablets_thread_pool()->submit_func(std::move(f)));
+}
+
+void CloudBackendService::get_top_n_hot_partitions(TGetTopNHotPartitionsResponse& response,
+ const TGetTopNHotPartitionsRequest& request) {
+ TabletHotspot::instance()->get_top_n_hot_partition(&response.hot_tables);
+ response.file_cache_size = io::FileCacheFactory::instance()->get_capacity();
+ response.__isset.hot_tables = !response.hot_tables.empty();
+}
+
+void CloudBackendService::warm_up_tablets(TWarmUpTabletsResponse& response,
Review Comment:
warning: method 'warm_up_tablets' can be made static [readability-convert-member-functions-to-static]
```suggestion
static void CloudBackendService::warm_up_tablets(TWarmUpTabletsResponse& response,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #32804:
URL: https://github.com/apache/doris/pull/32804#discussion_r1560703197
##########
fe/fe-common/src/main/java/org/apache/doris/common/Config.java:
##########
@@ -2685,6 +2685,24 @@ public static boolean isNotCloudMode() {
@ConfField(mutable = true, masterOnly = false)
public static String security_checker_class_name = "";
+ public static int history_cloud_warm_up_job_keep_max_second = 7 * 24 * 3600;
Review Comment:
miss @ConfField(?
BTW, we should use global session variable as far as possible
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #32804:
URL: https://github.com/apache/doris/pull/32804#discussion_r1565143956
##########
be/src/cloud/cloud_backend_service.cpp:
##########
@@ -45,4 +57,137 @@
return Status::OK();
}
+void CloudBackendService::sync_load_for_tablets(TSyncLoadForTabletsResponse&,
+ const TSyncLoadForTabletsRequest& request) {
+ auto f = [this, tablet_ids = request.tablet_ids]() {
+ std::for_each(tablet_ids.cbegin(), tablet_ids.cend(), [this](int64_t tablet_id) {
+ CloudTabletSPtr tablet;
+ auto result = _engine.tablet_mgr().get_tablet(tablet_id, true);
+ if (!result.has_value()) {
+ return;
+ }
+ Status st = result.value()->sync_rowsets(-1, true);
+ if (!st.ok()) {
+ LOG_WARNING("failed to sync load for tablet").error(st);
+ }
+ });
+ };
+ static_cast<void>(_exec_env->sync_load_for_tablets_thread_pool()->submit_func(std::move(f)));
+}
+
+void CloudBackendService::get_top_n_hot_partitions(TGetTopNHotPartitionsResponse& response,
+ const TGetTopNHotPartitionsRequest& request) {
+ TabletHotspot::instance()->get_top_n_hot_partition(&response.hot_tables);
+ response.file_cache_size = io::FileCacheFactory::instance()->get_capacity();
+ response.__isset.hot_tables = !response.hot_tables.empty();
+}
+
+void CloudBackendService::warm_up_tablets(TWarmUpTabletsResponse& response,
+ const TWarmUpTabletsRequest& request) {
+ Status st;
+ auto* manager = CloudWarmUpManager::instance();
+ switch (request.type) {
+ case TWarmUpTabletsRequestType::SET_JOB: {
+ LOG_INFO("receive the warm up request.")
+ .tag("request_type", "SET_JOB")
+ .tag("job_id", request.job_id);
+ st = manager->check_and_set_job_id(request.job_id);
+ if (!st) {
+ LOG_WARNING("SET_JOB failed.").error(st);
+ break;
+ }
+ [[fallthrough]];
+ }
+ case TWarmUpTabletsRequestType::SET_BATCH: {
+ LOG_INFO("receive the warm up request.")
+ .tag("request_type", "SET_BATCH")
+ .tag("job_id", request.job_id)
+ .tag("batch_id", request.batch_id)
+ .tag("jobs size", request.job_metas.size());
+ bool retry = false;
+ st = manager->check_and_set_batch_id(request.job_id, request.batch_id, &retry);
+ if (!retry && st) {
+ manager->add_job(request.job_metas);
+ } else {
+ if (retry) {
+ LOG_WARNING("retry the job.")
+ .tag("job_id", request.job_id)
+ .tag("batch_id", request.batch_id);
+ } else {
+ LOG_WARNING("SET_BATCH failed.").error(st);
+ }
+ }
+ break;
+ }
+ case TWarmUpTabletsRequestType::GET_CURRENT_JOB_STATE_AND_LEASE: {
+ auto [job_id, batch_id, pending_job_size, finish_job_size] =
+ manager->get_current_job_state();
+ LOG_INFO("receive the warm up request.")
+ .tag("request_type", "GET_CURRENT_JOB_STATE_AND_LEASE")
+ .tag("job_id", job_id)
+ .tag("batch_id", batch_id)
+ .tag("pending_job_size", pending_job_size)
+ .tag("finish_job_size", finish_job_size);
+ response.__set_job_id(job_id);
+ response.__set_batch_id(batch_id);
+ response.__set_pending_job_size(pending_job_size);
+ response.__set_finish_job_size(finish_job_size);
+ break;
+ }
+ case TWarmUpTabletsRequestType::CLEAR_JOB: {
+ LOG_INFO("receive the warm up request.")
+ .tag("request_type", "CLEAR_JOB")
+ .tag("job_id", request.job_id);
+ st = manager->clear_job(request.job_id);
+ break;
+ }
+ default:
+ DCHECK(false);
+ };
+ st.to_thrift(&response.status);
+}
+
+void CloudBackendService::pre_cache_async(TPreCacheAsyncResponse& response,
+ const TPreCacheAsyncRequest& request) {
+ std::string brpc_addr = fmt::format("{}:{}", request.host, request.brpc_port);
+ Status st = Status::OK();
+ TStatus t_status;
+ std::shared_ptr<PBackendService_Stub> brpc_stub =
+ _exec_env->brpc_internal_client_cache()->get_new_client_no_cache(brpc_addr);
+ if (!brpc_stub) {
+ st = Status::RpcError("Address {} is wrong", brpc_addr);
+ return;
+ }
+ brpc::Controller cntl;
+ PGetFileCacheMetaRequest brpc_request;
+ std::for_each(request.tablet_ids.cbegin(), request.tablet_ids.cend(),
+ [&](int64_t tablet_id) { brpc_request.add_tablet_ids(tablet_id); });
+ PGetFileCacheMetaResponse brpc_response;
+ brpc_stub->get_file_cache_meta_by_tablet_id(&cntl, &brpc_request, &brpc_response, nullptr);
+ if (!cntl.Failed()) {
+ std::vector<FileCacheBlockMeta> metas;
+ std::transform(brpc_response.file_cache_segment_metas().cbegin(),
+ brpc_response.file_cache_segment_metas().cend(), std::back_inserter(metas),
+ [](const FileCacheBlockMeta& meta) { return meta; });
+ io::DownloadTask download_task(std::move(metas));
+ io::FileCacheBlockDownloader::instance()->submit_download_task(download_task);
+ } else {
+ st = Status::RpcError("{} isn't connected", brpc_addr);
+ }
+ st.to_thrift(&t_status);
+ response.status = t_status;
+}
+
+void CloudBackendService::check_pre_cache(TCheckPreCacheResponse& response,
Review Comment:
warning: method 'check_pre_cache' can be made static [readability-convert-member-functions-to-static]
```suggestion
static void CloudBackendService::check_pre_cache(TCheckPreCacheResponse& response,
```
##########
be/src/cloud/cloud_backend_service.cpp:
##########
@@ -45,4 +57,137 @@ Status CloudBackendService::create_service(CloudStorageEngine& engine, ExecEnv*
return Status::OK();
}
+void CloudBackendService::sync_load_for_tablets(TSyncLoadForTabletsResponse&,
+ const TSyncLoadForTabletsRequest& request) {
+ auto f = [this, tablet_ids = request.tablet_ids]() {
+ std::for_each(tablet_ids.cbegin(), tablet_ids.cend(), [this](int64_t tablet_id) {
+ CloudTabletSPtr tablet;
+ auto result = _engine.tablet_mgr().get_tablet(tablet_id, true);
+ if (!result.has_value()) {
+ return;
+ }
+ Status st = result.value()->sync_rowsets(-1, true);
+ if (!st.ok()) {
+ LOG_WARNING("failed to sync load for tablet").error(st);
+ }
+ });
+ };
+ static_cast<void>(_exec_env->sync_load_for_tablets_thread_pool()->submit_func(std::move(f)));
+}
+
+void CloudBackendService::get_top_n_hot_partitions(TGetTopNHotPartitionsResponse& response,
+ const TGetTopNHotPartitionsRequest& request) {
+ TabletHotspot::instance()->get_top_n_hot_partition(&response.hot_tables);
+ response.file_cache_size = io::FileCacheFactory::instance()->get_capacity();
+ response.__isset.hot_tables = !response.hot_tables.empty();
+}
+
+void CloudBackendService::warm_up_tablets(TWarmUpTabletsResponse& response,
+ const TWarmUpTabletsRequest& request) {
+ Status st;
+ auto* manager = CloudWarmUpManager::instance();
+ switch (request.type) {
+ case TWarmUpTabletsRequestType::SET_JOB: {
+ LOG_INFO("receive the warm up request.")
+ .tag("request_type", "SET_JOB")
+ .tag("job_id", request.job_id);
+ st = manager->check_and_set_job_id(request.job_id);
+ if (!st) {
+ LOG_WARNING("SET_JOB failed.").error(st);
+ break;
+ }
+ [[fallthrough]];
+ }
+ case TWarmUpTabletsRequestType::SET_BATCH: {
+ LOG_INFO("receive the warm up request.")
+ .tag("request_type", "SET_BATCH")
+ .tag("job_id", request.job_id)
+ .tag("batch_id", request.batch_id)
+ .tag("jobs size", request.job_metas.size());
+ bool retry = false;
+ st = manager->check_and_set_batch_id(request.job_id, request.batch_id, &retry);
+ if (!retry && st) {
+ manager->add_job(request.job_metas);
+ } else {
+ if (retry) {
+ LOG_WARNING("retry the job.")
+ .tag("job_id", request.job_id)
+ .tag("batch_id", request.batch_id);
+ } else {
+ LOG_WARNING("SET_BATCH failed.").error(st);
+ }
+ }
+ break;
+ }
+ case TWarmUpTabletsRequestType::GET_CURRENT_JOB_STATE_AND_LEASE: {
+ auto [job_id, batch_id, pending_job_size, finish_job_size] =
+ manager->get_current_job_state();
+ LOG_INFO("receive the warm up request.")
+ .tag("request_type", "GET_CURRENT_JOB_STATE_AND_LEASE")
+ .tag("job_id", job_id)
+ .tag("batch_id", batch_id)
+ .tag("pending_job_size", pending_job_size)
+ .tag("finish_job_size", finish_job_size);
+ response.__set_job_id(job_id);
+ response.__set_batch_id(batch_id);
+ response.__set_pending_job_size(pending_job_size);
+ response.__set_finish_job_size(finish_job_size);
+ break;
+ }
+ case TWarmUpTabletsRequestType::CLEAR_JOB: {
+ LOG_INFO("receive the warm up request.")
+ .tag("request_type", "CLEAR_JOB")
+ .tag("job_id", request.job_id);
+ st = manager->clear_job(request.job_id);
+ break;
+ }
+ default:
+ DCHECK(false);
+ };
+ st.to_thrift(&response.status);
+}
+
+void CloudBackendService::pre_cache_async(TPreCacheAsyncResponse& response,
Review Comment:
warning: method 'pre_cache_async' can be made static [readability-convert-member-functions-to-static]
```suggestion
static void CloudBackendService::pre_cache_async(TPreCacheAsyncResponse& response,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]
Posted by "Lchangliang (via GitHub)" <gi...@apache.org>.
Lchangliang commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2067556199
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2067573101
<details>
<summary>ClickBench: <b>Total hot run time: 29.91 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fb510c126b36d4d5669ed574c9304e5f09895977, data reload: false
query1 0.04 0.03 0.02
query2 0.09 0.04 0.04
query3 0.23 0.05 0.05
query4 1.69 0.07 0.06
query5 0.50 0.50 0.50
query6 1.47 0.72 0.73
query7 0.02 0.01 0.01
query8 0.05 0.04 0.04
query9 0.55 0.49 0.50
query10 0.54 0.54 0.54
query11 0.15 0.11 0.11
query12 0.14 0.11 0.12
query13 0.60 0.59 0.58
query14 0.76 0.76 0.78
query15 0.82 0.81 0.81
query16 0.37 0.38 0.35
query17 0.99 0.97 0.99
query18 0.20 0.27 0.24
query19 1.76 1.69 1.79
query20 0.01 0.02 0.01
query21 15.43 0.64 0.63
query22 4.53 7.80 1.50
query23 18.24 1.42 1.19
query24 1.50 0.30 0.26
query25 0.14 0.08 0.09
query26 0.24 0.16 0.16
query27 0.08 0.08 0.08
query28 13.34 1.02 0.98
query29 12.57 3.35 3.33
query30 0.27 0.06 0.06
query31 2.85 0.38 0.36
query32 3.29 0.46 0.46
query33 2.80 2.85 2.88
query34 17.16 4.39 4.41
query35 4.51 4.44 4.48
query36 0.66 0.46 0.45
query37 0.19 0.15 0.15
query38 0.15 0.14 0.14
query39 0.05 0.04 0.04
query40 0.17 0.13 0.15
query41 0.09 0.04 0.04
query42 0.05 0.06 0.05
query43 0.04 0.04 0.04
Total cold run time: 109.33 s
Total hot run time: 29.91 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2068010238
<details>
<summary>ClickBench: <b>Total hot run time: 30.65 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c4fc00513476c131946015035443c34f1fe57661, data reload: false
query1 0.04 0.03 0.03
query2 0.09 0.04 0.04
query3 0.23 0.04 0.04
query4 1.69 0.08 0.07
query5 0.48 0.48 0.49
query6 1.47 0.73 0.72
query7 0.02 0.01 0.01
query8 0.05 0.04 0.04
query9 0.55 0.50 0.50
query10 0.56 0.55 0.55
query11 0.16 0.12 0.12
query12 0.14 0.12 0.12
query13 0.59 0.58 0.57
query14 0.76 0.77 0.77
query15 0.81 0.81 0.81
query16 0.36 0.36 0.38
query17 0.95 0.99 0.95
query18 0.23 0.22 0.24
query19 1.87 1.79 1.70
query20 0.01 0.01 0.01
query21 15.41 0.66 0.65
query22 4.30 6.55 2.26
query23 18.33 1.45 1.21
query24 1.35 0.26 0.31
query25 0.14 0.08 0.08
query26 0.27 0.16 0.16
query27 0.07 0.08 0.08
query28 13.52 0.99 0.99
query29 12.62 3.27 3.25
query30 0.26 0.06 0.06
query31 2.85 0.36 0.37
query32 3.32 0.46 0.46
query33 2.76 2.83 2.85
query34 17.21 4.42 4.39
query35 4.50 4.46 4.52
query36 0.66 0.50 0.46
query37 0.18 0.15 0.16
query38 0.15 0.14 0.14
query39 0.04 0.03 0.04
query40 0.17 0.16 0.13
query41 0.09 0.04 0.04
query42 0.06 0.05 0.05
query43 0.05 0.04 0.04
Total cold run time: 109.37 s
Total hot run time: 30.65 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #32804:
URL: https://github.com/apache/doris/pull/32804
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (cloud-merge) Support more multi case strategies about cache [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32804:
URL: https://github.com/apache/doris/pull/32804#issuecomment-2071389060
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org