You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "deardeng (via GitHub)" <gi...@apache.org> on 2024/10/18 03:13:49 UTC
[PR] [Feature](cloud) Add cloud report for clean up expired tablets [doris]
deardeng opened a new pull request, #42066:
URL: https://github.com/apache/doris/pull/42066
## Proposed changes
1. add tablet report to fe in cloud
2. clean up expired tablet in be
Issue Number: close #xxx
<!--Describe your changes.-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](cloud) Add cloud report for clean up expired tablets [doris]
Posted by "deardeng (via GitHub)" <gi...@apache.org>.
deardeng commented on code in PR #42066:
URL: https://github.com/apache/doris/pull/42066#discussion_r1812141857
##########
be/src/agent/task_worker_pool.cpp:
##########
@@ -1134,6 +1140,46 @@ void report_tablet_callback(StorageEngine& engine, const TMasterInfo& master_inf
}
}
+void report_tablet_callback(CloudStorageEngine& engine, const TMasterInfo& master_info) {
+ // Random sleep 1~5 seconds before doing report.
+ // In order to avoid the problem that the FE receives many report requests at the same time
+ // and can not be processed.
+ if (config::report_random_wait) {
+ random_sleep(5);
+ }
+
+ TReportRequest request;
+ request.__set_backend(BackendOptions::get_local_backend());
+ request.__isset.tablets = true;
+
+ increase_report_version();
+ uint64_t report_version;
+ uint64_t tablet_num = 0;
+ for (int i = 0; i < 5; i++) {
+ request.tablets.clear();
+ report_version = s_report_version;
+ engine.tablet_mgr().build_all_report_tablets_info(&request.tablets, &tablet_num);
+ if (report_version == s_report_version) {
+ break;
+ }
+ }
+
+ if (report_version < s_report_version) {
Review Comment:
目前没用到report version，留着看看以后会不会用？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](cloud) Add cloud report for clean up expired tablets [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42066:
URL: https://github.com/apache/doris/pull/42066#issuecomment-2429787248
TeamCity be ut coverage result:
Function Coverage: 37.45% (9706/25916)
Line Coverage: 28.71% (80512/280406)
Region Coverage: 28.14% (41625/147929)
Branch Coverage: 24.71% (21148/85588)
Coverage Report: http://coverage.selectdb-in.cc/coverage/bf1dd9d524b3e4a56da7cd70ed30ca7c0f85e84f_bf1dd9d524b3e4a56da7cd70ed30ca7c0f85e84f/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](cloud) Add cloud report for clean up expired tablets [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou merged PR #42066:
URL: https://github.com/apache/doris/pull/42066
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](cloud) Add cloud report for clean up expired tablets [doris]
Posted by "deardeng (via GitHub)" <gi...@apache.org>.
deardeng commented on PR #42066:
URL: https://github.com/apache/doris/pull/42066#issuecomment-2421207411
wait regression case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](cloud) Add cloud report for clean up expired tablets [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #42066:
URL: https://github.com/apache/doris/pull/42066#discussion_r1812139881
##########
be/src/agent/task_worker_pool.cpp:
##########
@@ -1134,6 +1140,46 @@ void report_tablet_callback(StorageEngine& engine, const TMasterInfo& master_inf
}
}
+void report_tablet_callback(CloudStorageEngine& engine, const TMasterInfo& master_info) {
+ // Random sleep 1~5 seconds before doing report.
+ // In order to avoid the problem that the FE receives many report requests at the same time
+ // and can not be processed.
+ if (config::report_random_wait) {
+ random_sleep(5);
+ }
+
+ TReportRequest request;
+ request.__set_backend(BackendOptions::get_local_backend());
+ request.__isset.tablets = true;
+
+ increase_report_version();
+ uint64_t report_version;
+ uint64_t tablet_num = 0;
+ for (int i = 0; i < 5; i++) {
+ request.tablets.clear();
+ report_version = s_report_version;
+ engine.tablet_mgr().build_all_report_tablets_info(&request.tablets, &tablet_num);
+ if (report_version == s_report_version) {
+ break;
+ }
+ }
+
+ if (report_version < s_report_version) {
Review Comment:
do cloud tablet report need check report_version ? maybe a stale report is ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](cloud) Add cloud report for clean up expired tablets [doris]
Posted by "deardeng (via GitHub)" <gi...@apache.org>.
deardeng commented on PR #42066:
URL: https://github.com/apache/doris/pull/42066#issuecomment-2429529452
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](cloud) Add cloud report for clean up expired tablets [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42066:
URL: https://github.com/apache/doris/pull/42066#issuecomment-2421202426
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](cloud) Add cloud report for clean up expired tablets [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #42066:
URL: https://github.com/apache/doris/pull/42066#discussion_r1812122408
##########
be/src/agent/agent_server.cpp:
##########
@@ -193,7 +194,7 @@ void AgentServer::start_workers(StorageEngine& engine, ExecEnv* exec_env) {
"REPORT_DISK_STATE", _master_info, config::report_disk_state_interval_seconds, [&engine, &master_info = _master_info] { report_disk_callback(engine, master_info); }));
_report_workers.push_back(std::make_unique<ReportWorker>(
- "REPORT_OLAP_TABLE", _master_info, config::report_tablet_interval_seconds,[&engine, &master_info = _master_info] { report_tablet_callback(engine, master_info); }));
Review Comment:
ReportAction use report worker's name. service/http_service.cpp need modify REPORT_OLAP_TABLE too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](cloud) Add cloud report for clean up expired tablets [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #42066:
URL: https://github.com/apache/doris/pull/42066#discussion_r1805782968
##########
be/src/agent/task_worker_pool.cpp:
##########
@@ -1610,6 +1656,16 @@ void drop_tablet_callback(StorageEngine& engine, const TAgentTaskRequest& req) {
remove_task_info(req.task_type, req.signature);
}
+void drop_tablet_callback(CloudStorageEngine& engine, const TAgentTaskRequest& req) {
+ const auto& drop_tablet_req = req.drop_tablet_req;
+ // 1. erase lru from tablet mgr
+ // TODO(dx) clean tablet file cache
+ // get tablet's info(such as cachekey, tablet id, rsid)
+ engine.tablet_mgr().erase_tablet(drop_tablet_req.tablet_id);
+ // 2. gen clean file cache task
+ return;
Review Comment:
warning: redundant return statement at the end of a function with a void return type [readability-redundant-control-flow]
be/src/agent/task_worker_pool.cpp:1664:
```diff
- // 2. gen clean file cache task
- return;
- }
+ }
```
##########
be/src/cloud/cloud_tablet_mgr.h:
##########
@@ -17,6 +17,9 @@
#pragma once
+#include <gen_cpp/MasterService_types.h>
Review Comment:
warning: 'gen_cpp/MasterService_types.h' file not found [clang-diagnostic-error]
```cpp
#include <gen_cpp/MasterService_types.h>
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](cloud) Add cloud report for clean up expired tablets [doris]
Posted by "deardeng (via GitHub)" <gi...@apache.org>.
deardeng commented on PR #42066:
URL: https://github.com/apache/doris/pull/42066#issuecomment-2440883611
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](cloud) Add cloud report for clean up expired tablets [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #42066:
URL: https://github.com/apache/doris/pull/42066#discussion_r1812133246
##########
be/src/agent/agent_server.cpp:
##########
@@ -193,7 +194,7 @@ void AgentServer::start_workers(StorageEngine& engine, ExecEnv* exec_env) {
"REPORT_DISK_STATE", _master_info, config::report_disk_state_interval_seconds, [&engine, &master_info = _master_info] { report_disk_callback(engine, master_info); }));
_report_workers.push_back(std::make_unique<ReportWorker>(
- "REPORT_OLAP_TABLE", _master_info, config::report_tablet_interval_seconds,[&engine, &master_info = _master_info] { report_tablet_callback(engine, master_info); }));
+ "REPORT_OLAP_TABLET", _master_info, config::report_tablet_interval_seconds,[&engine, &master_info = _master_info] { report_tablet_callback(engine, master_info); }));
Review Comment:
ReportAction use report worker's name. service/http_service.cpp need modify REPORT_OLAP_TABLE too.
what's more, service/http_service.cpp contains a api /api/report/tablet for not-cloud mode, also add this api to cloud mode ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](cloud) Add cloud report for clean up expired tablets [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42066:
URL: https://github.com/apache/doris/pull/42066#issuecomment-2431789956
TeamCity be ut coverage result:
Function Coverage: 37.47% (9709/25908)
Line Coverage: 28.72% (80504/280325)
Region Coverage: 28.16% (41640/147877)
Branch Coverage: 24.71% (21152/85588)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9a370cb2af6e370f26c8c94828aec6ddc179d9ff_9a370cb2af6e370f26c8c94828aec6ddc179d9ff/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](cloud) Add cloud report for clean up expired tablets [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #42066:
URL: https://github.com/apache/doris/pull/42066#discussion_r1814879427
##########
fe/fe-core/src/main/java/org/apache/doris/cloud/master/CloudReportHandler.java:
##########
@@ -0,0 +1,92 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.cloud.master;
+
+import org.apache.doris.catalog.Env;
+import org.apache.doris.cloud.catalog.CloudEnv;
+import org.apache.doris.master.ReportHandler;
+import org.apache.doris.system.Backend;
+import org.apache.doris.task.AgentBatchTask;
+import org.apache.doris.task.AgentTaskExecutor;
+import org.apache.doris.task.DropReplicaTask;
+import org.apache.doris.thrift.TTablet;
+
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+public class CloudReportHandler extends ReportHandler {
+ private static final Logger LOG = LogManager.getLogger(CloudReportHandler.class);
+
+ @Override
+ public void tabletReport(long backendId, Map<Long, TTablet> backendTablets,
+ Map<Long, Long> backendPartitionsVersion, long backendReportVersion, long numTablets) {
+ long start = System.currentTimeMillis();
+ LOG.info("backend[{}] have {} tablet(s), {} need deal tablet(s). report version: {}",
+ backendId, numTablets, backendTablets.size(), backendReportVersion);
+ // current be useful
+ Set<Long> tabletIdsInFe = ((CloudEnv) Env.getCurrentEnv()).getCloudTabletRebalancer()
+ .getSnapshotTabletsInPrimaryAndSecondaryByBeId(backendId);
+
+ Set<Long> tabletIdsInBe = backendTablets.keySet();
+ // handle (be - meta)
+ Set<Long> tabletIdsNeedDrop = diffTablets(tabletIdsInFe, tabletIdsInBe);
+ // drop agent task
+ deleteFromBackend(backendId, tabletIdsNeedDrop);
+
+ Backend be = Env.getCurrentSystemInfo().getBackend(backendId);
+ LOG.info("finished to handle task report from backend {}-{}, "
+ + "diff task num: {}, cost: {} ms.",
+ backendId, be != null ? be.getHost() : "",
+ tabletIdsNeedDrop.size(),
+ (System.currentTimeMillis() - start));
+ }
+
+ // tabletIdsInFe, tablet is used in Primary or Secondary
+ // tabletIdsInBe, tablet report exceed time, need to check
+ // returns tabletIds need to drop
+ private Set<Long> diffTablets(Set<Long> tabletIdsInFe, Set<Long> tabletIdsInBe) {
+ // tabletsInBe - tabletsInFe
+ Set<Long> result = new HashSet<>(tabletIdsInBe);
+ result.removeAll(tabletIdsInFe);
+ return result;
+ }
+
+ private static void deleteFromBackend(long backendId, Set<Long> tabletIdsWillDrop) {
+ int deleteFromBackendCounter = 0;
+ AgentBatchTask batchTask = new AgentBatchTask();
+ for (Long tabletId : tabletIdsWillDrop) {
+ if (LOG.isDebugEnabled()) {
Review Comment:
redundant log
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](cloud) Add cloud report for clean up expired tablets [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #42066:
URL: https://github.com/apache/doris/pull/42066#discussion_r1814888549
##########
be/src/cloud/cloud_tablet_mgr.h:
##########
@@ -65,6 +68,11 @@ class CloudTabletMgr {
std::vector<std::shared_ptr<CloudTablet>>* tablets,
int64_t* max_score);
+ void build_all_report_tablets_info(std::map<TTabletId, TTablet>* tablets_info,
Review Comment:
add comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](cloud) Add cloud report for clean up expired tablets [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #42066:
URL: https://github.com/apache/doris/pull/42066#discussion_r1814857729
##########
fe/fe-core/src/main/java/org/apache/doris/cloud/master/CloudReportHandler.java:
##########
@@ -0,0 +1,92 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.cloud.master;
+
+import org.apache.doris.catalog.Env;
+import org.apache.doris.cloud.catalog.CloudEnv;
+import org.apache.doris.master.ReportHandler;
+import org.apache.doris.system.Backend;
+import org.apache.doris.task.AgentBatchTask;
+import org.apache.doris.task.AgentTaskExecutor;
+import org.apache.doris.task.DropReplicaTask;
+import org.apache.doris.thrift.TTablet;
+
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+public class CloudReportHandler extends ReportHandler {
+ private static final Logger LOG = LogManager.getLogger(CloudReportHandler.class);
+
+ @Override
+ public void tabletReport(long backendId, Map<Long, TTablet> backendTablets,
+ Map<Long, Long> backendPartitionsVersion, long backendReportVersion, long numTablets) {
+ long start = System.currentTimeMillis();
+ LOG.info("backend[{}] have {} tablet(s), {} need deal tablet(s). report version: {}",
+ backendId, numTablets, backendTablets.size(), backendReportVersion);
+ // current be useful
+ Set<Long> tabletIdsInFe = ((CloudEnv) Env.getCurrentEnv()).getCloudTabletRebalancer()
+ .getSnapshotTabletsInPrimaryAndSecondaryByBeId(backendId);
+
+ Set<Long> tabletIdsInBe = backendTablets.keySet();
+ // handle (be - meta)
+ Set<Long> tabletIdsNeedDrop = diffTablets(tabletIdsInFe, tabletIdsInBe);
+ // drop agent task
+ deleteFromBackend(backendId, tabletIdsNeedDrop);
+
+ Backend be = Env.getCurrentSystemInfo().getBackend(backendId);
+ LOG.info("finished to handle task report from backend {}-{}, "
Review Comment:
we can also print the first N, say 100, tablet ids in the log.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](cloud) Add cloud report for clean up expired tablets [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #42066:
URL: https://github.com/apache/doris/pull/42066#discussion_r1812127761
##########
be/src/agent/agent_server.cpp:
##########
@@ -211,13 +212,25 @@ void AgentServer::cloud_start_workers(CloudStorageEngine& engine, ExecEnv* exec_
"CALC_DBM_TASK", config::calc_delete_bitmap_worker_count,
[&engine](auto&& task) { return calc_delete_bitmap_callback(engine, task); });
+ // cloud, drop tablet just clean clear_cache, so just one thread do it
+ _workers[TTaskType::DROP] = std::make_unique<TaskWorkerPool>(
+ "DROP_TABLE", 1, [&engine](auto&& task) { return drop_tablet_callback(engine, task); });
+
_report_workers.push_back(std::make_unique<ReportWorker>(
"REPORT_TASK", _master_info, config::report_task_interval_seconds,
[&master_info = _master_info] { report_task_callback(master_info); }));
_report_workers.push_back(std::make_unique<ReportWorker>(
"REPORT_DISK_STATE", _master_info, config::report_disk_state_interval_seconds,
[&engine, &master_info = _master_info] { report_disk_callback(engine, master_info); }));
+
+ if (config::enable_cloud_tablet_report) {
Review Comment:
if modify this param, need restart be to take effect ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](cloud) Add cloud report for clean up expired tablets [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #42066:
URL: https://github.com/apache/doris/pull/42066#discussion_r1810909133
##########
be/src/agent/task_worker_pool.cpp:
##########
@@ -1610,6 +1656,21 @@ void drop_tablet_callback(StorageEngine& engine, const TAgentTaskRequest& req) {
remove_task_info(req.task_type, req.signature);
}
+void drop_tablet_callback(CloudStorageEngine& engine, const TAgentTaskRequest& req) {
+ const auto& drop_tablet_req = req.drop_tablet_req;
+ DBUG_EXECUTE_IF("WorkPoolCloudDropTablet.drop_tablet_callback.failed", {
+ LOG_WARNING("WorkPoolCloudDropTablet.drop_tablet_callback.failed")
+ .tag("tablet_id", drop_tablet_req.tablet_id);
+ return;
+ });
+ // 1. erase lru from tablet mgr
+ // TODO(dx) clean tablet file cache
+ // get tablet's info(such as cachekey, tablet id, rsid)
+ engine.tablet_mgr().erase_tablet(drop_tablet_req.tablet_id);
+ // 2. gen clean file cache task
+ return;
Review Comment:
warning: redundant return statement at the end of a function with a void return type [readability-redundant-control-flow]
be/src/agent/task_worker_pool.cpp:1669:
```diff
- // 2. gen clean file cache task
- return;
- }
+ }
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](cloud) Add cloud report for clean up expired tablets [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #42066:
URL: https://github.com/apache/doris/pull/42066#issuecomment-2449642384
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](cloud) Add cloud report for clean up expired tablets [doris]
Posted by "deardeng (via GitHub)" <gi...@apache.org>.
deardeng commented on PR #42066:
URL: https://github.com/apache/doris/pull/42066#issuecomment-2431552991
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](cloud) Add cloud report for clean up expired tablets [doris]
Posted by "deardeng (via GitHub)" <gi...@apache.org>.
deardeng commented on code in PR #42066:
URL: https://github.com/apache/doris/pull/42066#discussion_r1812285455
##########
be/src/agent/agent_server.cpp:
##########
@@ -211,13 +212,25 @@ void AgentServer::cloud_start_workers(CloudStorageEngine& engine, ExecEnv* exec_
"CALC_DBM_TASK", config::calc_delete_bitmap_worker_count,
[&engine](auto&& task) { return calc_delete_bitmap_callback(engine, task); });
+ // cloud, drop tablet just clean clear_cache, so just one thread do it
+ _workers[TTaskType::DROP] = std::make_unique<TaskWorkerPool>(
+ "DROP_TABLE", 1, [&engine](auto&& task) { return drop_tablet_callback(engine, task); });
+
_report_workers.push_back(std::make_unique<ReportWorker>(
"REPORT_TASK", _master_info, config::report_task_interval_seconds,
[&master_info = _master_info] { report_task_callback(master_info); }));
_report_workers.push_back(std::make_unique<ReportWorker>(
"REPORT_DISK_STATE", _master_info, config::report_disk_state_interval_seconds,
[&engine, &master_info = _master_info] { report_disk_callback(engine, master_info); }));
+
+ if (config::enable_cloud_tablet_report) {
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](cloud) Add cloud report for clean up expired tablets [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #42066:
URL: https://github.com/apache/doris/pull/42066#discussion_r1812122408
##########
be/src/agent/agent_server.cpp:
##########
@@ -193,7 +194,7 @@ void AgentServer::start_workers(StorageEngine& engine, ExecEnv* exec_env) {
"REPORT_DISK_STATE", _master_info, config::report_disk_state_interval_seconds, [&engine, &master_info = _master_info] { report_disk_callback(engine, master_info); }));
_report_workers.push_back(std::make_unique<ReportWorker>(
- "REPORT_OLAP_TABLE", _master_info, config::report_tablet_interval_seconds,[&engine, &master_info = _master_info] { report_tablet_callback(engine, master_info); }));
Review Comment:
ReportAction use report worker's name. service/http_service.cpp need modify REPORT_OLAP_TABLE too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](cloud) Add cloud report for clean up expired tablets [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #42066:
URL: https://github.com/apache/doris/pull/42066#discussion_r1814843747
##########
gensrc/thrift/HeartbeatService.thrift:
##########
@@ -41,6 +41,8 @@ struct TMasterInfo {
9: optional list<TFrontendInfo> frontend_infos
10: optional string meta_service_endpoint;
11: optional string cloud_unique_id;
+ // See configuration item Config.java rehash_tablet_after_be_dead_seconds for meaning
+ 12: optional i64 cloud_tablet_report_exceed_time_limit;
Review Comment:
does it mean "timeout" or "interval"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](cloud) Add cloud report for clean up expired tablets [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #42066:
URL: https://github.com/apache/doris/pull/42066#discussion_r1814881958
##########
be/src/olap/txn_manager.cpp:
##########
@@ -34,6 +34,7 @@
#include "common/config.h"
#include "common/logging.h"
#include "common/status.h"
+#include "olap/base_tablet.h"
Review Comment:
?
##########
be/src/olap/tablet_manager.cpp:
##########
@@ -41,6 +41,7 @@
#include "gutil/strings/strcat.h"
#include "gutil/strings/substitute.h"
#include "io/fs/local_file_system.h"
+#include "olap/base_tablet.h"
Review Comment:
?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](cloud) Add cloud report for clean up expired tablets [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42066:
URL: https://github.com/apache/doris/pull/42066#issuecomment-2441170776
TeamCity be ut coverage result:
Function Coverage: 37.46% (9718/25943)
Line Coverage: 28.73% (80634/280621)
Region Coverage: 28.16% (41685/148046)
Branch Coverage: 24.71% (21173/85688)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f49234c345a523e9a168a91919fa4a865b8f7942_f49234c345a523e9a168a91919fa4a865b8f7942/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](cloud) Add cloud report for clean up expired tablets [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #42066:
URL: https://github.com/apache/doris/pull/42066#discussion_r1814896216
##########
be/src/cloud/cloud_tablet_mgr.cpp:
##########
@@ -181,8 +181,14 @@ Result<std::shared_ptr<CloudTablet>> CloudTabletMgr::get_tablet(int64_t tablet_i
auto* handle = _cache->insert(key, value.release(), 1, sizeof(CloudTablet),
CachePriority::NORMAL);
- auto ret = std::shared_ptr<CloudTablet>(
- tablet.get(), [this, handle](...) { _cache->release(handle); });
+ auto ret =
+ std::shared_ptr<CloudTablet>(tablet.get(), [this, handle](CloudTablet* tablet) {
+ int64_t now = duration_cast<std::chrono::milliseconds>(
+ std::chrono::system_clock::now().time_since_epoch())
+ .count();
+ tablet->last_cache_release_ms = now;
Review Comment:
both before and after release to refresh
last_access_time_ms
##########
be/src/cloud/cloud_tablet.h:
##########
@@ -196,10 +196,13 @@ class CloudTablet final : public BaseTablet {
int64_t last_base_compaction_success_time_ms = 0;
int64_t last_cumu_compaction_success_time_ms = 0;
int64_t last_cumu_no_suitable_version_ms = 0;
+ int64_t last_cache_release_ms = 0;
Review Comment:
last_access_time_ms
##########
be/src/cloud/cloud_tablet_mgr.h:
##########
@@ -65,6 +68,11 @@ class CloudTabletMgr {
std::vector<std::shared_ptr<CloudTablet>>* tablets,
int64_t* max_score);
+ void build_all_report_tablets_info(std::map<TTabletId, TTablet>* tablets_info,
+ uint64_t* tablet_num);
+
+ void obtain_specific_quantity_tablets(std::vector<TabletInfo>& tablets_info, int64_t num);
Review Comment:
get_tablet_info(int64_t num_tablets, std::vector<TabletInfo>* tablets_info)
##########
be/src/agent/task_worker_pool.cpp:
##########
@@ -1134,6 +1140,46 @@ void report_tablet_callback(StorageEngine& engine, const TMasterInfo& master_inf
}
}
+void report_tablet_callback(CloudStorageEngine& engine, const TMasterInfo& master_info) {
+ // Random sleep 1~5 seconds before doing report.
+ // In order to avoid the problem that the FE receives many report requests at the same time
+ // and can not be processed.
+ if (config::report_random_wait) {
+ random_sleep(5);
+ }
+
+ TReportRequest request;
+ request.__set_backend(BackendOptions::get_local_backend());
+ request.__isset.tablets = true;
+
+ increase_report_version();
+ uint64_t report_version;
+ uint64_t tablet_num = 0;
Review Comment:
total_num_tablets
##########
be/src/agent/task_worker_pool.cpp:
##########
@@ -1134,6 +1140,46 @@ void report_tablet_callback(StorageEngine& engine, const TMasterInfo& master_inf
}
}
+void report_tablet_callback(CloudStorageEngine& engine, const TMasterInfo& master_info) {
+ // Random sleep 1~5 seconds before doing report.
+ // In order to avoid the problem that the FE receives many report requests at the same time
+ // and can not be processed.
+ if (config::report_random_wait) {
+ random_sleep(5);
+ }
+
+ TReportRequest request;
+ request.__set_backend(BackendOptions::get_local_backend());
+ request.__isset.tablets = true;
+
+ increase_report_version();
+ uint64_t report_version;
+ uint64_t tablet_num = 0;
+ for (int i = 0; i < 5; i++) {
Review Comment:
why for 5 times?
##########
be/src/cloud/cloud_tablet_mgr.cpp:
##########
@@ -357,4 +368,57 @@ Status CloudTabletMgr::get_topn_tablets_to_compact(
return Status::OK();
}
+void CloudTabletMgr::build_all_report_tablets_info(std::map<TTabletId, TTablet>* tablets_info,
+ uint64_t* tablet_num) {
+ DCHECK(tablets_info != nullptr);
+ VLOG_NOTICE << "begin to build all report cloud tablets info";
+
+ HistogramStat tablet_version_num_hist;
+
+ auto handler = [&](const std::weak_ptr<CloudTablet>& tablet_wk) {
+ auto tablet = tablet_wk.lock();
+ if (!tablet) return;
+ (*tablet_num)++;
+ TTabletInfo tablet_info;
+ tablet->build_tablet_report_info(&tablet_info);
+ int64_t now = duration_cast<std::chrono::milliseconds>(
+ std::chrono::system_clock::now().time_since_epoch())
+ .count();
+ if (now - config::cloud_tablet_report_exceed_time_limit * 1000 <
Review Comment:
tablet_report_inactive_duration_ms
##########
be/src/cloud/cloud_tablet_mgr.cpp:
##########
@@ -357,4 +368,57 @@ Status CloudTabletMgr::get_topn_tablets_to_compact(
return Status::OK();
}
+void CloudTabletMgr::build_all_report_tablets_info(std::map<TTabletId, TTablet>* tablets_info,
+ uint64_t* tablet_num) {
+ DCHECK(tablets_info != nullptr);
+ VLOG_NOTICE << "begin to build all report cloud tablets info";
+
+ HistogramStat tablet_version_num_hist;
+
+ auto handler = [&](const std::weak_ptr<CloudTablet>& tablet_wk) {
+ auto tablet = tablet_wk.lock();
+ if (!tablet) return;
+ (*tablet_num)++;
+ TTabletInfo tablet_info;
+ tablet->build_tablet_report_info(&tablet_info);
+ int64_t now = duration_cast<std::chrono::milliseconds>(
+ std::chrono::system_clock::now().time_since_epoch())
+ .count();
+ if (now - config::cloud_tablet_report_exceed_time_limit * 1000 <
+ tablet->last_cache_release_ms) {
+ // the tablet is still being accessed and used in recently, so not report it
+ return;
+ }
+ auto& t_tablet = (*tablets_info)[tablet->tablet_id()];
+ // On the cloud, a specific BE has only one tablet replica;
+ // there are no multiple replicas for a specific BE.
+ // This is only to reuse the non-cloud report protocol.
+ tablet_version_num_hist.add(tablet_info.total_version_count);
+ t_tablet.tablet_infos.emplace_back(std::move(tablet_info));
+ };
+
+ auto weak_tablets = get_weak_tablets();
+ std::for_each(weak_tablets.begin(), weak_tablets.end(), handler);
+
+ DorisMetrics::instance()->tablet_version_num_distribution->set_histogram(
+ tablet_version_num_hist);
+ LOG(INFO) << "success to build all cloud report tablets info. all_tablet_count=" << *tablet_num
Review Comment:
all_tablet_count -> num_tablets
number inactive tablets
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](cloud) Add cloud report for clean up expired tablets [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #42066:
URL: https://github.com/apache/doris/pull/42066#issuecomment-2431854131
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org