You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "luwei16 (via GitHub)" <gi...@apache.org> on 2024/02/21 08:44:40 UTC
[PR] [feature](merge-cloud) Implement cloud compaction [doris]
luwei16 opened a new pull request, #31215:
URL: https://github.com/apache/doris/pull/31215
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Implement cloud compaction [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31215:
URL: https://github.com/apache/doris/pull/31215#issuecomment-1969384238
TeamCity be ut coverage result:
Function Coverage: 35.54% (8553/24064)
Line Coverage: 27.29% (69402/254300)
Region Coverage: 26.48% (35988/135921)
Branch Coverage: 23.30% (18387/78904)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5e79761f76ee16ed729bd481acdcd791fe1ca5ba_5e79761f76ee16ed729bd481acdcd791fe1ca5ba/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Implement cloud compaction [doris]
Posted by "platoneko (via GitHub)" <gi...@apache.org>.
platoneko commented on code in PR #31215:
URL: https://github.com/apache/doris/pull/31215#discussion_r1508810111
##########
be/src/olap/compaction.cpp:
##########
@@ -846,4 +851,80 @@ void Compaction::_load_segment_to_cache() {
}
}
+void CloudCompactionMixin::build_basic_info() {
+ _output_version =
+ Version(_input_rowsets.front()->start_version(), _input_rowsets.back()->end_version());
+
+ _newest_write_timestamp = _input_rowsets.back()->newest_write_timestamp();
+
+ std::vector<RowsetMetaSharedPtr> rowset_metas(_input_rowsets.size());
+ std::transform(_input_rowsets.begin(), _input_rowsets.end(), rowset_metas.begin(),
+ [](const RowsetSharedPtr& rowset) { return rowset->rowset_meta(); });
+ _cur_tablet_schema = _tablet->tablet_schema_with_merged_max_schema_version(rowset_metas);
+}
+
+int64_t CloudCompactionMixin::get_compaction_permits() {
+ int64_t permits = 0;
+ for (auto&& rowset : _input_rowsets) {
+ permits += rowset->rowset_meta()->get_compaction_score();
+ }
+ return permits;
+}
+
+CloudCompactionMixin::CloudCompactionMixin(CloudStorageEngine& engine, CloudTabletSPtr tablet,
+ const std::string& label)
+ : Compaction(tablet, label), _engine(engine) {}
+
+Status CloudCompactionMixin::execute_compact_impl(int64_t permits) {
+ OlapStopWatch watch;
+
+ build_basic_info();
+
+ LOG(INFO) << "start " << compaction_name() << ". tablet=" << _tablet->tablet_id()
+ << ", output_version=" << _output_version << ", permits: " << permits;
+
+ RETURN_IF_ERROR(merge_input_rowsets());
+
+ RETURN_IF_ERROR(_engine.meta_mgr().commit_rowset(*_output_rowset->rowset_meta().get(), true));
+
+ // 4. modify rowsets in memory
+ RETURN_IF_ERROR(modify_rowsets());
+
+ return Status::OK();
+}
+
+Status CloudCompactionMixin::execute_compact() {
+ TEST_INJECTION_POINT("Compaction::do_compaction");
+ int64_t permits = get_compaction_permits();
+ Status st = execute_compact_impl(permits);
+ if (!st.ok()) {
+ garbage_collection();
+ return st;
+ }
+ _load_segment_to_cache();
+ return st;
+}
+
+Status CloudCompactionMixin::modify_rowsets() {
Review Comment:
Consider use pure virtual function
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Implement cloud compaction [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #31215:
URL: https://github.com/apache/doris/pull/31215#discussion_r1507207577
##########
be/src/cloud/cloud_storage_engine.cpp:
##########
@@ -190,4 +240,454 @@ void CloudStorageEngine::_sync_tablets_thread_callback() {
}
}
+void CloudStorageEngine::get_cumu_compaction(
+ int64_t tablet_id, std::vector<std::shared_ptr<CloudCumulativeCompaction>>& res) {
+ std::lock_guard lock(_compaction_mtx);
+ if (auto it = _submitted_cumu_compactions.find(tablet_id);
+ it != _submitted_cumu_compactions.end()) {
+ res = it->second;
+ }
+}
+
+void CloudStorageEngine::_adjust_compaction_thread_num() {
+ int base_thread_num = get_base_thread_num();
+ if (_base_compaction_thread_pool->max_threads() != base_thread_num) {
+ int old_max_threads = _base_compaction_thread_pool->max_threads();
+ Status status = _base_compaction_thread_pool->set_max_threads(base_thread_num);
+ if (status.ok()) {
+ VLOG_NOTICE << "update base compaction thread pool max_threads from " << old_max_threads
+ << " to " << base_thread_num;
+ }
+ }
+ if (_base_compaction_thread_pool->min_threads() != base_thread_num) {
+ int old_min_threads = _base_compaction_thread_pool->min_threads();
+ Status status = _base_compaction_thread_pool->set_min_threads(base_thread_num);
+ if (status.ok()) {
+ VLOG_NOTICE << "update base compaction thread pool min_threads from " << old_min_threads
+ << " to " << base_thread_num;
+ }
+ }
+
+ int cumu_thread_num = get_cumu_thread_num();
+ if (_cumu_compaction_thread_pool->max_threads() != cumu_thread_num) {
+ int old_max_threads = _cumu_compaction_thread_pool->max_threads();
+ Status status = _cumu_compaction_thread_pool->set_max_threads(cumu_thread_num);
+ if (status.ok()) {
+ VLOG_NOTICE << "update cumu compaction thread pool max_threads from " << old_max_threads
+ << " to " << cumu_thread_num;
+ }
+ }
+ if (_cumu_compaction_thread_pool->min_threads() != cumu_thread_num) {
+ int old_min_threads = _cumu_compaction_thread_pool->min_threads();
+ Status status = _cumu_compaction_thread_pool->set_min_threads(cumu_thread_num);
+ if (status.ok()) {
+ VLOG_NOTICE << "update cumu compaction thread pool min_threads from " << old_min_threads
+ << " to " << cumu_thread_num;
+ }
+ }
+}
+
+void CloudStorageEngine::_compaction_tasks_producer_callback() {
+ LOG(INFO) << "try to start compaction producer process!";
+
+ int round = 0;
+ CompactionType compaction_type;
+
+ // Used to record the time when the score metric was last updated.
+ // The update of the score metric is accompanied by the logic of selecting the tablet.
+ // If there is no slot available, the logic of selecting the tablet will be terminated,
+ // which causes the score metric update to be terminated.
+ // In order to avoid this situation, we need to update the score regularly.
+ int64_t last_cumulative_score_update_time = 0;
+ int64_t last_base_score_update_time = 0;
+ static const int64_t check_score_interval_ms = 5000; // 5 secs
+
+ int64_t interval = config::generate_compaction_tasks_interval_ms;
+ do {
+ if (!config::disable_auto_compaction) {
+ _adjust_compaction_thread_num();
+
+ bool check_score = false;
+ int64_t cur_time = UnixMillis();
+ if (round < config::cumulative_compaction_rounds_for_each_base_compaction_round) {
+ compaction_type = CompactionType::CUMULATIVE_COMPACTION;
+ round++;
+ if (cur_time - last_cumulative_score_update_time >= check_score_interval_ms) {
+ check_score = true;
+ last_cumulative_score_update_time = cur_time;
+ }
+ } else {
+ compaction_type = CompactionType::BASE_COMPACTION;
+ round = 0;
+ if (cur_time - last_base_score_update_time >= check_score_interval_ms) {
+ check_score = true;
+ last_base_score_update_time = cur_time;
+ }
+ }
+ std::unique_ptr<ThreadPool>& thread_pool =
+ (compaction_type == CompactionType::CUMULATIVE_COMPACTION)
+ ? _cumu_compaction_thread_pool
+ : _base_compaction_thread_pool;
+ VLOG_CRITICAL << "compaction thread pool. type: "
+ << (compaction_type == CompactionType::CUMULATIVE_COMPACTION ? "CUMU"
+ : "BASE")
+ << ", num_threads: " << thread_pool->num_threads()
+ << ", num_threads_pending_start: "
+ << thread_pool->num_threads_pending_start()
+ << ", num_active_threads: " << thread_pool->num_active_threads()
+ << ", max_threads: " << thread_pool->max_threads()
+ << ", min_threads: " << thread_pool->min_threads()
+ << ", num_total_queued_tasks: " << thread_pool->get_queue_size();
+ std::vector<CloudTabletSPtr> tablets_compaction =
+ _generate_cloud_compaction_tasks(compaction_type, check_score);
+
+ /// Regardless of whether the tablet is submitted for compaction or not,
+ /// we need to call 'reset_compaction' to clean up the base_compaction or cumulative_compaction objects
+ /// in the tablet, because these two objects store the tablet's own shared_ptr.
+ /// If it is not cleaned up, the reference count of the tablet will always be greater than 1,
+ /// thus cannot be collected by the garbage collector. (TabletManager::start_trash_sweep)
+ for (const auto& tablet : tablets_compaction) {
+ Status st = submit_compaction_task(tablet, compaction_type);
+ if (st.ok()) continue;
+ if ((!st.is<ErrorCode::BE_NO_SUITABLE_VERSION>() &&
+ !st.is<ErrorCode::CUMULATIVE_NO_SUITABLE_VERSION>()) ||
+ VLOG_DEBUG_IS_ON) {
+ LOG(WARNING) << "failed to submit compaction task for tablet: "
+ << tablet->tablet_id() << ", err: " << st;
+ }
+ }
+ interval = config::generate_compaction_tasks_interval_ms;
+ } else {
+ interval = config::check_auto_compaction_interval_seconds * 1000;
+ }
+ } while (!_stop_background_threads_latch.wait_for(std::chrono::milliseconds(interval)));
+}
+
+std::vector<CloudTabletSPtr> CloudStorageEngine::_generate_cloud_compaction_tasks(
Review Comment:
warning: function '_generate_cloud_compaction_tasks' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
std::vector<CloudTabletSPtr> CloudStorageEngine::_generate_cloud_compaction_tasks(
^
```
<details>
<summary>Additional context</summary>
**be/src/cloud/cloud_storage_engine.cpp:365:** 86 lines including whitespace and comments (threshold 80)
```cpp
std::vector<CloudTabletSPtr> CloudStorageEngine::_generate_cloud_compaction_tasks(
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Implement cloud compaction [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31215:
URL: https://github.com/apache/doris/pull/31215#issuecomment-1970636513
TeamCity be ut coverage result:
Function Coverage: 35.53% (8554/24076)
Line Coverage: 27.29% (69405/254327)
Region Coverage: 26.48% (35994/135948)
Branch Coverage: 23.30% (18389/78922)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e9def044fe349e018f6a4058efa70fbcbc9aadbd_e9def044fe349e018f6a4058efa70fbcbc9aadbd/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Implement cloud compaction [doris]
Posted by "luwei16 (via GitHub)" <gi...@apache.org>.
luwei16 commented on PR #31215:
URL: https://github.com/apache/doris/pull/31215#issuecomment-1970912983
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Implement cloud compaction [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #31215:
URL: https://github.com/apache/doris/pull/31215#discussion_r1497168073
##########
be/src/cloud/cloud_base_compaction.cpp:
##########
@@ -0,0 +1,370 @@
+#include "cloud/cloud_base_compaction.h"
+
+#include <boost/container_hash/hash.hpp>
+
+#include "cloud/cloud_meta_mgr.h"
+#include "cloud/config.h"
+#include "common/config.h"
+#include "common/sync_point.h"
+#include "gen_cpp/cloud.pb.h"
+#include "olap/compaction.h"
+#include "olap/task/engine_checksum_task.h"
+#include "service/backend_options.h"
+#include "util/thread.h"
+#include "util/uuid_generator.h"
+
+namespace doris {
+using namespace ErrorCode;
+
+bvar::Adder<uint64_t> base_output_size("base_compaction", "output_size");
+
+CloudBaseCompaction::CloudBaseCompaction(CloudStorageEngine& engine, CloudTabletSPtr tablet)
+ : CloudCompactionMixin(engine, std::move(tablet),
+ "BaseCompaction:" + std::to_string(tablet->tablet_id())) {
+ auto uuid = UUIDGenerator::instance()->next_uuid();
+ std::stringstream ss;
+ ss << uuid;
+ _uuid = ss.str();
+}
+
+CloudBaseCompaction::~CloudBaseCompaction() = default;
+
+Status CloudBaseCompaction::prepare_compact() {
+ if (_tablet->tablet_state() != TABLET_RUNNING) {
+ return Status::InternalError("invalid tablet state. tablet_id={}", _tablet->tablet_id());
+ }
+
+ bool need_sync_tablet = true;
+ {
+ std::shared_lock rlock(_tablet->get_header_lock());
+ // If number of rowsets is equal to approximate_num_rowsets, it is very likely that this tablet has been
+ // synchronized with meta-service.
+ if (_tablet->tablet_meta()->all_rs_metas().size() >=
+ cloud_tablet()->fetch_add_approximate_num_rowsets(0) &&
+ cloud_tablet()->last_sync_time_s > 0) {
+ need_sync_tablet = false;
+ }
+ }
+ if (need_sync_tablet) {
+ RETURN_IF_ERROR(cloud_tablet()->sync_rowsets());
+ }
+
+ RETURN_IF_ERROR(pick_rowsets_to_compact());
+
+ // prepare compaction job
+ cloud::TabletJobInfoPB job;
+ auto idx = job.mutable_idx();
+ idx->set_tablet_id(_tablet->tablet_id());
+ idx->set_table_id(_tablet->table_id());
+ idx->set_index_id(_tablet->index_id());
+ idx->set_partition_id(_tablet->partition_id());
+ auto compaction_job = job.add_compaction();
+ compaction_job->set_id(_uuid);
+ compaction_job->set_initiator(BackendOptions::get_localhost() + ':' +
+ std::to_string(config::heartbeat_service_port));
+ compaction_job->set_type(cloud::TabletCompactionJobPB::BASE);
+ compaction_job->set_base_compaction_cnt(_base_compaction_cnt);
+ compaction_job->set_cumulative_compaction_cnt(_cumulative_compaction_cnt);
+ using namespace std::chrono;
+ int64_t now = duration_cast<seconds>(system_clock::now().time_since_epoch()).count();
+ _expiration = now + config::compaction_timeout_seconds;
+ compaction_job->set_expiration(_expiration);
+ compaction_job->set_lease(now + config::lease_compaction_interval_seconds * 4);
+ cloud::StartTabletJobResponse resp;
+ //auto st = cloud::meta_mgr()->prepare_tablet_job(job, &resp);
+ auto st = _engine.meta_mgr().prepare_tablet_job(job, &resp);
+ if (!st.ok()) {
+ if (resp.status().code() == cloud::STALE_TABLET_CACHE) {
+ // set last_sync_time to 0 to force sync tablet next time
+ cloud_tablet()->last_sync_time_s = 0;
+ } else if (resp.status().code() == cloud::TABLET_NOT_FOUND) {
+ // tablet not found
+ cloud_tablet()->recycle_cached_data();
+ }
+ return st;
+ }
+
+ for (auto& rs : _input_rowsets) {
+ _input_row_num += rs->num_rows();
+ _input_segments += rs->num_segments();
+ _input_rowsets_size += rs->data_disk_size();
+ }
+ LOG_INFO("start CloudBaseCompaction, tablet_id={}, range=[{}-{}]", _tablet->tablet_id(),
+ _input_rowsets.front()->start_version(), _input_rowsets.back()->end_version())
+ .tag("job_id", _uuid)
+ .tag("input_rowsets", _input_rowsets.size())
+ .tag("input_rows", _input_row_num)
+ .tag("input_segments", _input_segments)
+ .tag("input_data_size", _input_rowsets_size);
+ return st;
+}
+
+void CloudBaseCompaction::_filter_input_rowset() {
+ // if dup_key and no delete predicate
+ // we skip big files to save resources
+ if (_tablet->keys_type() != KeysType::DUP_KEYS) {
+ return;
+ }
+ for (auto& rs : _input_rowsets) {
+ if (rs->rowset_meta()->has_delete_predicate()) {
+ return;
+ }
+ }
+ int64_t max_size = config::base_compaction_dup_key_max_file_size_mbytes * 1024 * 1024;
+ // first find a proper rowset for start
+ auto rs_iter = _input_rowsets.begin();
+ while (rs_iter != _input_rowsets.end()) {
+ if ((*rs_iter)->rowset_meta()->total_disk_size() >= max_size) {
+ rs_iter = _input_rowsets.erase(rs_iter);
+ } else {
+ break;
+ }
+ }
+}
+
+Status CloudBaseCompaction::pick_rowsets_to_compact() {
+ _input_rowsets.clear();
+ {
+ std::shared_lock rlock(_tablet->get_header_lock());
+ _base_compaction_cnt = cloud_tablet()->base_compaction_cnt();
+ _cumulative_compaction_cnt = cloud_tablet()->cumulative_compaction_cnt();
+ _input_rowsets = cloud_tablet()->pick_candidate_rowsets_to_base_compaction();
+ }
+ if (auto st = check_version_continuity(_input_rowsets); !st.ok()) {
+ DCHECK(false) << st;
+ return st;
+ }
+ _filter_input_rowset();
+ if (_input_rowsets.size() <= 1) {
+ return Status::Error<BE_NO_SUITABLE_VERSION>(
+ "insuffient compation input rowset, #rowsets={}", _input_rowsets.size());
+ }
+
+ if (_input_rowsets.size() == 2 && _input_rowsets[0]->end_version() == 1) {
+ // the tablet is with rowset: [0-1], [2-y]
+ // and [0-1] has no data. in this situation, no need to do base compaction.
+ return Status::Error<BE_NO_SUITABLE_VERSION>("no suitable versions for compaction");
+ }
+
+ // 1. cumulative rowset must reach base_compaction_min_rowset_num threshold
+ if (_input_rowsets.size() > config::base_compaction_min_rowset_num) {
+ VLOG_NOTICE << "satisfy the base compaction policy. tablet=" << _tablet->tablet_id()
+ << ", num_cumulative_rowsets=" << _input_rowsets.size() - 1
+ << ", base_compaction_num_cumulative_rowsets="
+ << config::base_compaction_min_rowset_num;
+ return Status::OK();
+ }
+
+ // 2. the ratio between base rowset and all input cumulative rowsets reaches the threshold
+ // `_input_rowsets` has been sorted by end version, so we consider `_input_rowsets[0]` is the base rowset.
+ int64_t base_size = _input_rowsets.front()->data_disk_size();
+ int64_t cumulative_total_size = 0;
+ for (auto it = _input_rowsets.begin() + 1; it != _input_rowsets.end(); ++it) {
+ cumulative_total_size += (*it)->data_disk_size();
+ }
+
+ double base_cumulative_delta_ratio = config::base_compaction_min_data_ratio;
+ if (base_size == 0) {
+ // base_size == 0 means this may be a base version [0-1], which has no data.
+ // set to 1 to void divide by zero
+ base_size = 1;
+ }
+ double cumulative_base_ratio = static_cast<double>(cumulative_total_size) / base_size;
+
+ if (cumulative_base_ratio > base_cumulative_delta_ratio) {
+ VLOG_NOTICE << "satisfy the base compaction policy. tablet=" << _tablet->tablet_id()
+ << ", cumulative_total_size=" << cumulative_total_size
+ << ", base_size=" << base_size
+ << ", cumulative_base_ratio=" << cumulative_base_ratio
+ << ", policy_ratio=" << base_cumulative_delta_ratio;
+ return Status::OK();
+ }
+
+ // 3. the interval since last base compaction reaches the threshold
+ int64_t base_creation_time = _input_rowsets[0]->creation_time();
+ int64_t interval_threshold = config::base_compaction_interval_seconds_since_last_operation;
+ int64_t interval_since_last_base_compaction = time(nullptr) - base_creation_time;
+ if (interval_since_last_base_compaction > interval_threshold) {
+ VLOG_NOTICE << "satisfy the base compaction policy. tablet=" << _tablet->tablet_id()
+ << ", interval_since_last_base_compaction="
+ << interval_since_last_base_compaction
+ << ", interval_threshold=" << interval_threshold;
+ return Status::OK();
+ }
+
+ VLOG_NOTICE << "don't satisfy the base compaction policy. tablet=" << _tablet->tablet_id()
+ << ", num_cumulative_rowsets=" << _input_rowsets.size() - 1
+ << ", cumulative_base_ratio=" << cumulative_base_ratio
+ << ", interval_since_last_base_compaction=" << interval_since_last_base_compaction;
+ return Status::Error<BE_NO_SUITABLE_VERSION>("no suitable versions for compaction");
+}
+
+Status CloudBaseCompaction::execute_compact() {
+ if (config::enable_base_compaction_idle_sched) {
+ Thread::set_idle_sched();
+ }
+
+ SCOPED_ATTACH_TASK(_mem_tracker);
+
+ using namespace std::chrono;
+ auto start = steady_clock::now();
+ RETURN_IF_ERROR(CloudCompactionMixin::execute_compact());
+ LOG_INFO("finish CloudBaseCompaction, tablet_id={}, cost={}ms", _tablet->tablet_id(),
+ duration_cast<milliseconds>(steady_clock::now() - start).count())
+ .tag("job_id", _uuid)
+ .tag("input_rowsets", _input_rowsets.size())
+ .tag("input_rows", _input_row_num)
+ .tag("input_segments", _input_segments)
+ .tag("input_data_size", _input_rowsets_size)
+ .tag("output_rows", _output_rowset->num_rows())
+ .tag("output_segments", _output_rowset->num_segments())
+ .tag("output_data_size", _output_rowset->data_disk_size());
+
+ //_compaction_succeed = true;
+ _state = CompactionState::SUCCESS;
+
+ DorisMetrics::instance()->base_compaction_deltas_total->increment(_input_rowsets.size());
+ DorisMetrics::instance()->base_compaction_bytes_total->increment(_input_rowsets_size);
+ base_output_size << _output_rowset->data_disk_size();
+
+ return Status::OK();
+}
+
+Status CloudBaseCompaction::modify_rowsets() {
Review Comment:
warning: function 'modify_rowsets' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status CloudBaseCompaction::modify_rowsets() {
^
```
<details>
<summary>Additional context</summary>
**be/src/cloud/cloud_base_compaction.cpp:232:** 82 lines including whitespace and comments (threshold 80)
```cpp
Status CloudBaseCompaction::modify_rowsets() {
^
```
</details>
##########
be/src/cloud/cloud_cumulative_compaction.cpp:
##########
@@ -0,0 +1,460 @@
+#include "cloud/cloud_cumulative_compaction.h"
+
+#include "cloud/cloud_meta_mgr.h"
+#include "cloud/config.h"
+#include "common/config.h"
+#include "common/logging.h"
+#include "common/status.h"
+#include "common/sync_point.h"
+#include "gen_cpp/cloud.pb.h"
+#include "olap/compaction.h"
+#include "olap/cumulative_compaction_policy.h"
+#include "service/backend_options.h"
+#include "util/trace.h"
+#include "util/uuid_generator.h"
+
+namespace doris {
+using namespace ErrorCode;
+
+bvar::Adder<uint64_t> cumu_output_size("cumu_compaction", "output_size");
+
+CloudCumulativeCompaction::CloudCumulativeCompaction(CloudStorageEngine& engine,
+ CloudTabletSPtr tablet)
+ : CloudCompactionMixin(engine, std::move(tablet),
+ "BaseCompaction:" + std::to_string(tablet->tablet_id())) {
+ auto uuid = UUIDGenerator::instance()->next_uuid();
+ std::stringstream ss;
+ ss << uuid;
+ _uuid = ss.str();
+}
+
+CloudCumulativeCompaction::~CloudCumulativeCompaction() = default;
+
+Status CloudCumulativeCompaction::prepare_compact() {
+ if (_tablet->tablet_state() != TABLET_RUNNING) {
+ return Status::InternalError("invalid tablet state. tablet_id={}", _tablet->tablet_id());
+ }
+
+ std::vector<std::shared_ptr<CloudCumulativeCompaction>> cumu_compactions;
+ _engine.get_cumu_compaction(_tablet->tablet_id(), cumu_compactions);
+ if (!cumu_compactions.empty()) {
+ for (auto& cumu : cumu_compactions) {
+ _max_conflict_version =
+ std::max(_max_conflict_version, cumu->_input_rowsets.back()->end_version());
+ }
+ }
+
+ int tried = 0;
+PREPARE_TRY_AGAIN:
+
+ bool need_sync_tablet = true;
+ {
+ std::shared_lock rlock(_tablet->get_header_lock());
+ // If number of rowsets is equal to approximate_num_rowsets, it is very likely that this tablet has been
+ // synchronized with meta-service.
+ if (_tablet->tablet_meta()->all_rs_metas().size() >=
+ cloud_tablet()->fetch_add_approximate_num_rowsets(0) &&
+ cloud_tablet()->last_sync_time_s > 0) {
+ need_sync_tablet = false;
+ }
+ }
+ if (need_sync_tablet) {
+ RETURN_IF_ERROR(cloud_tablet()->sync_rowsets());
+ }
+
+ // pick rowsets to compact
+ auto st = pick_rowsets_to_compact();
+ if (!st.ok()) {
+ if (tried == 0 && _last_delete_version.first != -1) {
+ // we meet a delete version, should increase the cumulative point to let base compaction handle the delete version.
+ // plus 1 to skip the delete version.
+ // NOTICE: after that, the cumulative point may be larger than max version of this tablet, but it doesn't matter.
+ update_cumulative_point();
+ }
+ return st;
+ }
+
+ // prepare compaction job
+ cloud::TabletJobInfoPB job;
+ auto idx = job.mutable_idx();
+ idx->set_tablet_id(_tablet->tablet_id());
+ idx->set_table_id(_tablet->table_id());
+ idx->set_index_id(_tablet->index_id());
+ idx->set_partition_id(_tablet->partition_id());
+ auto compaction_job = job.add_compaction();
+ compaction_job->set_id(_uuid);
+ compaction_job->set_initiator(BackendOptions::get_localhost() + ':' +
+ std::to_string(config::heartbeat_service_port));
+ compaction_job->set_type(cloud::TabletCompactionJobPB::CUMULATIVE);
+ compaction_job->set_base_compaction_cnt(_base_compaction_cnt);
+ compaction_job->set_cumulative_compaction_cnt(_cumulative_compaction_cnt);
+ using namespace std::chrono;
+ int64_t now = duration_cast<seconds>(system_clock::now().time_since_epoch()).count();
+ _expiration = now + config::compaction_timeout_seconds;
+ compaction_job->set_expiration(_expiration);
+ compaction_job->set_lease(now + config::lease_compaction_interval_seconds * 4);
+ if (config::enable_parallel_cumu_compaction) {
+ // Set input version range to let meta-service judge version range conflict
+ compaction_job->add_input_versions(_input_rowsets.front()->start_version());
+ compaction_job->add_input_versions(_input_rowsets.back()->end_version());
+ }
+ cloud::StartTabletJobResponse resp;
+ st = _engine.meta_mgr().prepare_tablet_job(job, &resp);
+ if (!st.ok()) {
+ if (resp.status().code() == cloud::STALE_TABLET_CACHE) {
+ // set last_sync_time to 0 to force sync tablet next time
+ cloud_tablet()->last_sync_time_s = 0;
+ } else if (resp.status().code() == cloud::TABLET_NOT_FOUND) {
+ // tablet not found
+ cloud_tablet()->recycle_cached_data();
+ } else if (resp.status().code() == cloud::JOB_TABLET_BUSY) {
+ if (config::enable_parallel_cumu_compaction && resp.version_in_compaction_size() > 0 &&
+ ++tried <= 2) {
+ _max_conflict_version = *std::max_element(resp.version_in_compaction().begin(),
+ resp.version_in_compaction().end());
+ LOG_INFO("retry pick input rowsets")
+ .tag("job_id", _uuid)
+ .tag("max_conflict_version", _max_conflict_version)
+ .tag("tried", tried)
+ .tag("msg", resp.status().msg());
+ goto PREPARE_TRY_AGAIN;
+ } else {
+ LOG_WARNING("failed to prepare cumu compaction")
+ .tag("job_id", _uuid)
+ .tag("msg", resp.status().msg());
+ return Status::Error<CUMULATIVE_NO_SUITABLE_VERSION>("no suitable versions");
+ }
+ }
+ return st;
+ }
+
+ for (auto& rs : _input_rowsets) {
+ _input_row_num += rs->num_rows();
+ _input_segments += rs->num_segments();
+ _input_rowsets_size += rs->data_disk_size();
+ }
+ LOG_INFO("start CloudCumulativeCompaction, tablet_id={}, range=[{}-{}]", _tablet->tablet_id(),
+ _input_rowsets.front()->start_version(), _input_rowsets.back()->end_version())
+ .tag("job_id", _uuid)
+ .tag("input_rowsets", _input_rowsets.size())
+ .tag("input_rows", _input_row_num)
+ .tag("input_segments", _input_segments)
+ .tag("input_data_size", _input_rowsets_size)
+ .tag("tablet_max_version", cloud_tablet()->max_version_unlocked())
+ .tag("cumulative_point", cloud_tablet()->cumulative_layer_point())
+ .tag("num_rowsets", cloud_tablet()->fetch_add_approximate_num_rowsets(0))
+ .tag("cumu_num_rowsets", cloud_tablet()->fetch_add_approximate_cumu_num_rowsets(0));
+ return st;
+}
+
+Status CloudCumulativeCompaction::execute_compact() {
+ TEST_SYNC_POINT_RETURN_WITH_VALUE("CloudCumulativeCompaction::execute_compact_impl",
+ Status::OK(), this);
+ using namespace std::chrono;
+ auto start = steady_clock::now();
+ RETURN_IF_ERROR(CloudCompactionMixin::execute_compact());
+ LOG_INFO("finish CloudCumulativeCompaction, tablet_id={}, cost={}ms", _tablet->tablet_id(),
+ duration_cast<milliseconds>(steady_clock::now() - start).count())
+ .tag("job_id", _uuid)
+ .tag("input_rowsets", _input_rowsets.size())
+ .tag("input_rows", _input_row_num)
+ .tag("input_segments", _input_segments)
+ .tag("input_data_size", _input_rowsets_size)
+ .tag("output_rows", _output_rowset->num_rows())
+ .tag("output_segments", _output_rowset->num_segments())
+ .tag("output_data_size", _output_rowset->data_disk_size())
+ .tag("tablet_max_version", _tablet->max_version_unlocked())
+ .tag("cumulative_point", cloud_tablet()->cumulative_layer_point())
+ .tag("num_rowsets", cloud_tablet()->fetch_add_approximate_num_rowsets(0))
+ .tag("cumu_num_rowsets", cloud_tablet()->fetch_add_approximate_cumu_num_rowsets(0));
+
+ _state = CompactionState::SUCCESS;
+
+ DorisMetrics::instance()->cumulative_compaction_deltas_total->increment(_input_rowsets.size());
+ DorisMetrics::instance()->cumulative_compaction_bytes_total->increment(_input_rowsets_size);
+ cumu_output_size << _output_rowset->data_disk_size();
+
+ return Status::OK();
+}
+
+Status CloudCumulativeCompaction::modify_rowsets() {
Review Comment:
warning: function 'modify_rowsets' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status CloudCumulativeCompaction::modify_rowsets() {
^
```
<details>
<summary>Additional context</summary>
**be/src/cloud/cloud_cumulative_compaction.cpp:179:** 102 lines including whitespace and comments (threshold 80)
```cpp
Status CloudCumulativeCompaction::modify_rowsets() {
^
```
</details>
##########
be/src/cloud/cloud_cumulative_compaction_policy.cpp:
##########
@@ -0,0 +1,206 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_cumulative_compaction_policy.h"
+
+#include <algorithm>
+#include <list>
+#include <ostream>
+#include <string>
+
+#include "cloud/config.h"
+#include "common/config.h"
+#include "common/logging.h"
+#include "common/sync_point.h"
+#include "olap/olap_common.h"
+#include "olap/tablet.h"
+#include "olap/tablet_meta.h"
+
+namespace doris {
+
+CloudSizeBasedCumulativeCompactionPolicy::CloudSizeBasedCumulativeCompactionPolicy(
+ int64_t promotion_size, double promotion_ratio, int64_t promotion_min_size,
+ int64_t compaction_min_size)
+ : _promotion_size(promotion_size),
+ _promotion_ratio(promotion_ratio),
+ _promotion_min_size(promotion_min_size),
+ _compaction_min_size(compaction_min_size) {}
+
+int64_t CloudSizeBasedCumulativeCompactionPolicy::_level_size(const int64_t size) {
+ if (size < 1024) return 0;
+ int64_t max_level = (int64_t)1
+ << (sizeof(_promotion_size) * 8 - 1 - __builtin_clzl(_promotion_size / 2));
+ if (size >= max_level) return max_level;
+ return (int64_t)1 << (sizeof(size) * 8 - 1 - __builtin_clzl(size));
+}
+
+int CloudSizeBasedCumulativeCompactionPolicy::pick_input_rowsets(
Review Comment:
warning: function 'pick_input_rowsets' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
int CloudSizeBasedCumulativeCompactionPolicy::pick_input_rowsets(
^
```
<details>
<summary>Additional context</summary>
**be/src/cloud/cloud_cumulative_compaction_policy.cpp:50:** 127 lines including whitespace and comments (threshold 80)
```cpp
int CloudSizeBasedCumulativeCompactionPolicy::pick_input_rowsets(
^
```
</details>
##########
be/src/cloud/cloud_tablet.h:
##########
@@ -109,6 +109,66 @@ class CloudTablet final : public BaseTablet {
void set_cumulative_compaction_cnt(int64_t cnt) { _cumulative_compaction_cnt = cnt; }
void set_cumulative_layer_point(int64_t new_point);
+ int64_t last_cumu_compaction_failure_time() { return _last_cumu_compaction_failure_millis; }
+ void set_last_cumu_compaction_failure_time(int64_t millis) {
+ _last_cumu_compaction_failure_millis = millis;
+ }
+
+ int64_t last_base_compaction_failure_time() { return _last_base_compaction_failure_millis; }
+ void set_last_base_compaction_failure_time(int64_t millis) {
+ _last_base_compaction_failure_millis = millis;
+ }
+
+ int64_t last_full_compaction_failure_time() { return _last_full_compaction_failure_millis; }
+ void set_last_full_compaction_failure_time(int64_t millis) {
+ _last_full_compaction_failure_millis = millis;
+ }
+
+ int64_t last_cumu_compaction_success_time() { return _last_cumu_compaction_success_millis; }
+ void set_last_cumu_compaction_success_time(int64_t millis) {
+ _last_cumu_compaction_success_millis = millis;
+ }
+
+ int64_t last_base_compaction_success_time() { return _last_base_compaction_success_millis; }
+ void set_last_base_compaction_success_time(int64_t millis) {
+ _last_base_compaction_success_millis = millis;
+ }
+
+ int64_t last_full_compaction_success_time() { return _last_full_compaction_success_millis; }
+ void set_last_full_compaction_success_time(int64_t millis) {
+ _last_full_compaction_success_millis = millis;
+ }
+
+ int64_t last_base_compaction_schedule_time() { return _last_base_compaction_schedule_millis; }
+ void set_last_base_compaction_schedule_time(int64_t millis) {
+ _last_base_compaction_schedule_millis = millis;
+ }
+
+ std::vector<RowsetSharedPtr> pick_candidate_rowsets_to_base_compaction();
+
+ void traverse_rowsets(std::function<void(const RowsetSharedPtr&)> visitor,
+ bool include_stale = false) {
+ std::shared_lock rlock(_meta_lock);
+ for (auto& [v, rs] : _rs_version_map) {
+ visitor(rs);
+ }
+ if (!include_stale) return;
+ for (auto& [v, rs] : _stale_rs_version_map) {
+ visitor(rs);
+ }
Review Comment:
warning: method 'traverse_rowsets' can be made static [readability-convert-member-functions-to-static]
```suggestion
static void traverse_rowsets(std::function<void(const RowsetSharedPtr&)> visitor,
```
##########
be/src/cloud/cloud_storage_engine.cpp:
##########
@@ -190,4 +235,419 @@
}
}
+void CloudStorageEngine::get_cumu_compaction(
+ int64_t tablet_id, std::vector<std::shared_ptr<CloudCumulativeCompaction>>& res) {
+ std::lock_guard lock(_compaction_mtx);
+ if (auto it = _submitted_cumu_compactions.find(tablet_id);
+ it != _submitted_cumu_compactions.end()) {
+ res = it->second;
+ }
+}
+
+void CloudStorageEngine::_adjust_compaction_thread_num() {
+ int base_thread_num = get_base_thread_num();
+ if (_base_compaction_thread_pool->max_threads() != base_thread_num) {
+ int old_max_threads = _base_compaction_thread_pool->max_threads();
+ Status status = _base_compaction_thread_pool->set_max_threads(base_thread_num);
+ if (status.ok()) {
+ VLOG_NOTICE << "update base compaction thread pool max_threads from " << old_max_threads
+ << " to " << base_thread_num;
+ }
+ }
+ if (_base_compaction_thread_pool->min_threads() != base_thread_num) {
+ int old_min_threads = _base_compaction_thread_pool->min_threads();
+ Status status = _base_compaction_thread_pool->set_min_threads(base_thread_num);
+ if (status.ok()) {
+ VLOG_NOTICE << "update base compaction thread pool min_threads from " << old_min_threads
+ << " to " << base_thread_num;
+ }
+ }
+
+ int cumu_thread_num = get_cumu_thread_num();
+ if (_cumu_compaction_thread_pool->max_threads() != cumu_thread_num) {
+ int old_max_threads = _cumu_compaction_thread_pool->max_threads();
+ Status status = _cumu_compaction_thread_pool->set_max_threads(cumu_thread_num);
+ if (status.ok()) {
+ VLOG_NOTICE << "update cumu compaction thread pool max_threads from " << old_max_threads
+ << " to " << cumu_thread_num;
+ }
+ }
+ if (_cumu_compaction_thread_pool->min_threads() != cumu_thread_num) {
+ int old_min_threads = _cumu_compaction_thread_pool->min_threads();
+ Status status = _cumu_compaction_thread_pool->set_min_threads(cumu_thread_num);
+ if (status.ok()) {
+ VLOG_NOTICE << "update cumu compaction thread pool min_threads from " << old_min_threads
+ << " to " << cumu_thread_num;
+ }
+ }
+}
+
+void CloudStorageEngine::_compaction_tasks_producer_callback() {
+ LOG(INFO) << "try to start compaction producer process!";
+
+ int round = 0;
+ CompactionType compaction_type;
+
+ // Used to record the time when the score metric was last updated.
+ // The update of the score metric is accompanied by the logic of selecting the tablet.
+ // If there is no slot available, the logic of selecting the tablet will be terminated,
+ // which causes the score metric update to be terminated.
+ // In order to avoid this situation, we need to update the score regularly.
+ int64_t last_cumulative_score_update_time = 0;
+ int64_t last_base_score_update_time = 0;
+ static const int64_t check_score_interval_ms = 5000; // 5 secs
+
+ int64_t interval = config::generate_compaction_tasks_interval_ms;
+ do {
+ if (!config::disable_auto_compaction) {
+ _adjust_compaction_thread_num();
+
+ bool check_score = false;
+ int64_t cur_time = UnixMillis();
+ if (round < config::cumulative_compaction_rounds_for_each_base_compaction_round) {
+ compaction_type = CompactionType::CUMULATIVE_COMPACTION;
+ round++;
+ if (cur_time - last_cumulative_score_update_time >= check_score_interval_ms) {
+ check_score = true;
+ last_cumulative_score_update_time = cur_time;
+ }
+ } else {
+ compaction_type = CompactionType::BASE_COMPACTION;
+ round = 0;
+ if (cur_time - last_base_score_update_time >= check_score_interval_ms) {
+ check_score = true;
+ last_base_score_update_time = cur_time;
+ }
+ }
+ std::unique_ptr<ThreadPool>& thread_pool =
+ (compaction_type == CompactionType::CUMULATIVE_COMPACTION)
+ ? _cumu_compaction_thread_pool
+ : _base_compaction_thread_pool;
+ VLOG_CRITICAL << "compaction thread pool. type: "
+ << (compaction_type == CompactionType::CUMULATIVE_COMPACTION ? "CUMU"
+ : "BASE")
+ << ", num_threads: " << thread_pool->num_threads()
+ << ", num_threads_pending_start: "
+ << thread_pool->num_threads_pending_start()
+ << ", num_active_threads: " << thread_pool->num_active_threads()
+ << ", max_threads: " << thread_pool->max_threads()
+ << ", min_threads: " << thread_pool->min_threads()
+ << ", num_total_queued_tasks: " << thread_pool->get_queue_size();
+ std::vector<CloudTabletSPtr> tablets_compaction =
+ _generate_cloud_compaction_tasks(compaction_type, check_score);
+
+ /// Regardless of whether the tablet is submitted for compaction or not,
+ /// we need to call 'reset_compaction' to clean up the base_compaction or cumulative_compaction objects
+ /// in the tablet, because these two objects store the tablet's own shared_ptr.
+ /// If it is not cleaned up, the reference count of the tablet will always be greater than 1,
+ /// thus cannot be collected by the garbage collector. (TabletManager::start_trash_sweep)
+ for (const auto& tablet : tablets_compaction) {
+ Status st = submit_compaction_task(tablet, compaction_type);
+ if (st.ok()) continue;
+ if ((!st.is<ErrorCode::BE_NO_SUITABLE_VERSION>() &&
+ !st.is<ErrorCode::CUMULATIVE_NO_SUITABLE_VERSION>()) ||
+ VLOG_DEBUG_IS_ON) {
+ LOG(WARNING) << "failed to submit compaction task for tablet: "
+ << tablet->tablet_id() << ", err: " << st;
+ }
+ }
+ interval = config::generate_compaction_tasks_interval_ms;
+ } else {
+ interval = config::check_auto_compaction_interval_seconds * 1000;
+ }
+ } while (!_stop_background_threads_latch.wait_for(std::chrono::milliseconds(interval)));
+}
+
+std::vector<CloudTabletSPtr> CloudStorageEngine::_generate_cloud_compaction_tasks(
Review Comment:
warning: function '_generate_cloud_compaction_tasks' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
std::vector<CloudTabletSPtr> CloudStorageEngine::_generate_cloud_compaction_tasks(
^
```
<details>
<summary>Additional context</summary>
**be/src/cloud/cloud_storage_engine.cpp:360:** 86 lines including whitespace and comments (threshold 80)
```cpp
std::vector<CloudTabletSPtr> CloudStorageEngine::_generate_cloud_compaction_tasks(
^
```
</details>
##########
be/src/cloud/cloud_cumulative_compaction.cpp:
##########
@@ -0,0 +1,460 @@
+#include "cloud/cloud_cumulative_compaction.h"
+
+#include "cloud/cloud_meta_mgr.h"
+#include "cloud/config.h"
+#include "common/config.h"
+#include "common/logging.h"
+#include "common/status.h"
+#include "common/sync_point.h"
+#include "gen_cpp/cloud.pb.h"
+#include "olap/compaction.h"
+#include "olap/cumulative_compaction_policy.h"
+#include "service/backend_options.h"
+#include "util/trace.h"
+#include "util/uuid_generator.h"
+
+namespace doris {
+using namespace ErrorCode;
+
+bvar::Adder<uint64_t> cumu_output_size("cumu_compaction", "output_size");
+
+CloudCumulativeCompaction::CloudCumulativeCompaction(CloudStorageEngine& engine,
+ CloudTabletSPtr tablet)
+ : CloudCompactionMixin(engine, std::move(tablet),
+ "BaseCompaction:" + std::to_string(tablet->tablet_id())) {
+ auto uuid = UUIDGenerator::instance()->next_uuid();
+ std::stringstream ss;
+ ss << uuid;
+ _uuid = ss.str();
+}
+
+CloudCumulativeCompaction::~CloudCumulativeCompaction() = default;
+
+Status CloudCumulativeCompaction::prepare_compact() {
Review Comment:
warning: function 'prepare_compact' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status CloudCumulativeCompaction::prepare_compact() {
^
```
<details>
<summary>Additional context</summary>
**be/src/cloud/cloud_cumulative_compaction.cpp:32:** 115 lines including whitespace and comments (threshold 80)
```cpp
Status CloudCumulativeCompaction::prepare_compact() {
^
```
</details>
##########
be/src/olap/compaction.cpp:
##########
@@ -844,4 +847,80 @@
}
}
+void CloudCompactionMixin::build_basic_info() {
+ _output_version =
+ Version(_input_rowsets.front()->start_version(), _input_rowsets.back()->end_version());
+
+ _newest_write_timestamp = _input_rowsets.back()->newest_write_timestamp();
+
+ std::vector<RowsetMetaSharedPtr> rowset_metas(_input_rowsets.size());
+ std::transform(_input_rowsets.begin(), _input_rowsets.end(), rowset_metas.begin(),
+ [](const RowsetSharedPtr& rowset) { return rowset->rowset_meta(); });
+ _cur_tablet_schema = _tablet->tablet_schema_with_merged_max_schema_version(rowset_metas);
+}
+
+int64_t CloudCompactionMixin::get_compaction_permits() {
Review Comment:
warning: method 'get_compaction_permits' can be made static [readability-convert-member-functions-to-static]
be/src/olap/compaction.h:187:
```diff
- int64_t get_compaction_permits();
+ static int64_t get_compaction_permits();
```
##########
be/src/olap/compaction.cpp:
##########
@@ -844,4 +847,80 @@ void Compaction::_load_segment_to_cache() {
}
}
+void CloudCompactionMixin::build_basic_info() {
Review Comment:
warning: method 'build_basic_info' can be made static [readability-convert-member-functions-to-static]
be/src/olap/compaction.h:183:
```diff
- void build_basic_info();
+ static void build_basic_info();
```
##########
be/src/cloud/cloud_tablet.h:
##########
@@ -109,6 +109,66 @@
void set_cumulative_compaction_cnt(int64_t cnt) { _cumulative_compaction_cnt = cnt; }
void set_cumulative_layer_point(int64_t new_point);
+ int64_t last_cumu_compaction_failure_time() { return _last_cumu_compaction_failure_millis; }
+ void set_last_cumu_compaction_failure_time(int64_t millis) {
+ _last_cumu_compaction_failure_millis = millis;
+ }
+
+ int64_t last_base_compaction_failure_time() { return _last_base_compaction_failure_millis; }
+ void set_last_base_compaction_failure_time(int64_t millis) {
+ _last_base_compaction_failure_millis = millis;
+ }
+
+ int64_t last_full_compaction_failure_time() { return _last_full_compaction_failure_millis; }
+ void set_last_full_compaction_failure_time(int64_t millis) {
+ _last_full_compaction_failure_millis = millis;
+ }
+
+ int64_t last_cumu_compaction_success_time() { return _last_cumu_compaction_success_millis; }
+ void set_last_cumu_compaction_success_time(int64_t millis) {
+ _last_cumu_compaction_success_millis = millis;
+ }
+
+ int64_t last_base_compaction_success_time() { return _last_base_compaction_success_millis; }
+ void set_last_base_compaction_success_time(int64_t millis) {
+ _last_base_compaction_success_millis = millis;
+ }
+
+ int64_t last_full_compaction_success_time() { return _last_full_compaction_success_millis; }
+ void set_last_full_compaction_success_time(int64_t millis) {
+ _last_full_compaction_success_millis = millis;
+ }
+
+ int64_t last_base_compaction_schedule_time() { return _last_base_compaction_schedule_millis; }
+ void set_last_base_compaction_schedule_time(int64_t millis) {
+ _last_base_compaction_schedule_millis = millis;
+ }
+
+ std::vector<RowsetSharedPtr> pick_candidate_rowsets_to_base_compaction();
+
+ void traverse_rowsets(std::function<void(const RowsetSharedPtr&)> visitor,
+ bool include_stale = false) {
+ std::shared_lock rlock(_meta_lock);
+ for (auto& [v, rs] : _rs_version_map) {
+ visitor(rs);
+ }
+ if (!include_stale) return;
+ for (auto& [v, rs] : _stale_rs_version_map) {
+ visitor(rs);
+ }
+ }
+
+ inline Version max_version() const {
+ std::shared_lock rdlock(_meta_lock);
+ return _tablet_meta->max_version();
+ }
+
+ int64_t base_size() const { return _base_size; }
+
+ std::vector<RowsetSharedPtr> pick_candidate_rowsets_to_single_replica_compaction();
+
+ std::vector<RowsetSharedPtr> pick_candidate_rowsets_to_full_compaction();
Review Comment:
warning: method 'max_version' can be made static [readability-convert-member-functions-to-static]
```suggestion
static inline Version max_version() {
```
##########
be/src/cloud/cloud_tablet.h:
##########
@@ -109,6 +109,66 @@
void set_cumulative_compaction_cnt(int64_t cnt) { _cumulative_compaction_cnt = cnt; }
void set_cumulative_layer_point(int64_t new_point);
+ int64_t last_cumu_compaction_failure_time() { return _last_cumu_compaction_failure_millis; }
+ void set_last_cumu_compaction_failure_time(int64_t millis) {
+ _last_cumu_compaction_failure_millis = millis;
+ }
+
+ int64_t last_base_compaction_failure_time() { return _last_base_compaction_failure_millis; }
+ void set_last_base_compaction_failure_time(int64_t millis) {
+ _last_base_compaction_failure_millis = millis;
+ }
+
+ int64_t last_full_compaction_failure_time() { return _last_full_compaction_failure_millis; }
+ void set_last_full_compaction_failure_time(int64_t millis) {
+ _last_full_compaction_failure_millis = millis;
+ }
+
+ int64_t last_cumu_compaction_success_time() { return _last_cumu_compaction_success_millis; }
+ void set_last_cumu_compaction_success_time(int64_t millis) {
+ _last_cumu_compaction_success_millis = millis;
+ }
+
+ int64_t last_base_compaction_success_time() { return _last_base_compaction_success_millis; }
+ void set_last_base_compaction_success_time(int64_t millis) {
+ _last_base_compaction_success_millis = millis;
+ }
+
+ int64_t last_full_compaction_success_time() { return _last_full_compaction_success_millis; }
+ void set_last_full_compaction_success_time(int64_t millis) {
+ _last_full_compaction_success_millis = millis;
+ }
+
+ int64_t last_base_compaction_schedule_time() { return _last_base_compaction_schedule_millis; }
+ void set_last_base_compaction_schedule_time(int64_t millis) {
+ _last_base_compaction_schedule_millis = millis;
+ }
+
+ std::vector<RowsetSharedPtr> pick_candidate_rowsets_to_base_compaction();
+
+ void traverse_rowsets(std::function<void(const RowsetSharedPtr&)> visitor,
+ bool include_stale = false) {
+ std::shared_lock rlock(_meta_lock);
+ for (auto& [v, rs] : _rs_version_map) {
+ visitor(rs);
+ }
+ if (!include_stale) return;
+ for (auto& [v, rs] : _stale_rs_version_map) {
+ visitor(rs);
+ }
+ }
+
+ inline Version max_version() const {
+ std::shared_lock rdlock(_meta_lock);
+ return _tablet_meta->max_version();
+ }
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (!include_stale) { return;
}
```
##########
be/src/cloud/cloud_storage_engine.cpp:
##########
@@ -190,4 +235,419 @@ void CloudStorageEngine::_sync_tablets_thread_callback() {
}
}
+void CloudStorageEngine::get_cumu_compaction(
+ int64_t tablet_id, std::vector<std::shared_ptr<CloudCumulativeCompaction>>& res) {
+ std::lock_guard lock(_compaction_mtx);
+ if (auto it = _submitted_cumu_compactions.find(tablet_id);
+ it != _submitted_cumu_compactions.end()) {
+ res = it->second;
+ }
+}
+
+void CloudStorageEngine::_adjust_compaction_thread_num() {
Review Comment:
warning: method '_adjust_compaction_thread_num' can be made static [readability-convert-member-functions-to-static]
be/src/cloud/cloud_storage_engine.h:85:
```diff
- void _adjust_compaction_thread_num();
+ static void _adjust_compaction_thread_num();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Implement cloud compaction [doris]
Posted by "luwei16 (via GitHub)" <gi...@apache.org>.
luwei16 commented on PR #31215:
URL: https://github.com/apache/doris/pull/31215#issuecomment-1970276856
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Implement cloud compaction [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31215:
URL: https://github.com/apache/doris/pull/31215#issuecomment-1972948603
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Implement cloud compaction [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #31215:
URL: https://github.com/apache/doris/pull/31215
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Implement cloud compaction [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #31215:
URL: https://github.com/apache/doris/pull/31215#discussion_r1497123801
##########
be/src/cloud/cloud_cumulative_compaction_policy.cpp:
##########
@@ -0,0 +1,205 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_cumulative_compaction_policy.h"
+
+#include <algorithm>
+#include <list>
+#include <ostream>
+#include <string>
+
+#include "common/config.h"
+#include "common/logging.h"
+#include "common/sync_point.h"
+#include "cloud/config.h"
+#include "olap/olap_common.h"
+#include "olap/tablet.h"
+#include "olap/tablet_meta.h"
+
+namespace doris {
+
+CloudSizeBasedCumulativeCompactionPolicy::CloudSizeBasedCumulativeCompactionPolicy(
+ int64_t promotion_size, double promotion_ratio, int64_t promotion_min_size,
+ int64_t compaction_min_size)
+ : _promotion_size(promotion_size),
+ _promotion_ratio(promotion_ratio),
+ _promotion_min_size(promotion_min_size),
+ _compaction_min_size(compaction_min_size) {}
+
+int64_t CloudSizeBasedCumulativeCompactionPolicy::_level_size(const int64_t size) {
+ if (size < 1024) return 0;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (size < 1024) { return 0;
}
```
##########
be/src/cloud/cloud_cumulative_compaction.cpp:
##########
@@ -0,0 +1,457 @@
+#include "cloud/cloud_cumulative_compaction.h"
+
+#include "cloud/config.h"
+#include "cloud/cloud_meta_mgr.h"
+#include "common/config.h"
+#include "common/logging.h"
+#include "common/status.h"
+#include "common/sync_point.h"
+#include "gen_cpp/cloud.pb.h"
+#include "olap/compaction.h"
+#include "olap/cumulative_compaction_policy.h"
+#include "service/backend_options.h"
+#include "util/trace.h"
+#include "util/uuid_generator.h"
+
+namespace doris {
+using namespace ErrorCode;
+
+bvar::Adder<uint64_t> cumu_output_size("cumu_compaction", "output_size");
+
+CloudCumulativeCompaction::CloudCumulativeCompaction(CloudStorageEngine& engine, CloudTabletSPtr tablet)
+ : CloudCompactionMixin(engine, std::move(tablet), "BaseCompaction:" + std::to_string(tablet->tablet_id())) {
+ auto uuid = UUIDGenerator::instance()->next_uuid();
+ std::stringstream ss;
+ ss << uuid;
+ _uuid = ss.str();
+}
+
+CloudCumulativeCompaction::~CloudCumulativeCompaction() = default;
+
+Status CloudCumulativeCompaction::prepare_compact() {
Review Comment:
warning: function 'prepare_compact' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status CloudCumulativeCompaction::prepare_compact() {
^
```
<details>
<summary>Additional context</summary>
**be/src/cloud/cloud_cumulative_compaction.cpp:30:** 115 lines including whitespace and comments (threshold 80)
```cpp
Status CloudCumulativeCompaction::prepare_compact() {
^
```
</details>
##########
be/src/cloud/cloud_cumulative_compaction_policy.cpp:
##########
@@ -0,0 +1,205 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_cumulative_compaction_policy.h"
+
+#include <algorithm>
+#include <list>
+#include <ostream>
+#include <string>
+
+#include "common/config.h"
+#include "common/logging.h"
+#include "common/sync_point.h"
+#include "cloud/config.h"
+#include "olap/olap_common.h"
+#include "olap/tablet.h"
+#include "olap/tablet_meta.h"
+
+namespace doris {
+
+CloudSizeBasedCumulativeCompactionPolicy::CloudSizeBasedCumulativeCompactionPolicy(
+ int64_t promotion_size, double promotion_ratio, int64_t promotion_min_size,
+ int64_t compaction_min_size)
+ : _promotion_size(promotion_size),
+ _promotion_ratio(promotion_ratio),
+ _promotion_min_size(promotion_min_size),
+ _compaction_min_size(compaction_min_size) {}
+
+int64_t CloudSizeBasedCumulativeCompactionPolicy::_level_size(const int64_t size) {
+ if (size < 1024) return 0;
+ int64_t max_level = (int64_t)1
+ << (sizeof(_promotion_size) * 8 - 1 - __builtin_clzl(_promotion_size / 2));
+ if (size >= max_level) return max_level;
+ return (int64_t)1 << (sizeof(size) * 8 - 1 - __builtin_clzl(size));
+}
+
+int CloudSizeBasedCumulativeCompactionPolicy::pick_input_rowsets(
Review Comment:
warning: function 'pick_input_rowsets' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
int CloudSizeBasedCumulativeCompactionPolicy::pick_input_rowsets(
^
```
<details>
<summary>Additional context</summary>
**be/src/cloud/cloud_cumulative_compaction_policy.cpp:50:** 126 lines including whitespace and comments (threshold 80)
```cpp
int CloudSizeBasedCumulativeCompactionPolicy::pick_input_rowsets(
^
```
</details>
##########
be/src/cloud/cloud_cumulative_compaction_policy.h:
##########
@@ -0,0 +1,73 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <stddef.h>
+#include <stdint.h>
+
+#include <memory>
+#include <string>
+#include <vector>
+
+#include "common/config.h"
+#include "cloud/cloud_tablet.h"
+#include "olap/rowset/rowset.h"
+#include "olap/rowset/rowset_meta.h"
+
+namespace doris {
+
+class Tablet;
+struct Version;
+
+class CloudSizeBasedCumulativeCompactionPolicy {
+public:
+ CloudSizeBasedCumulativeCompactionPolicy(
+ int64_t promotion_size = config::compaction_promotion_size_mbytes * 1024 * 1024,
+ double promotion_ratio = config::compaction_promotion_ratio,
+ int64_t promotion_min_size = config::compaction_promotion_min_size_mbytes * 1024 * 1024,
+ int64_t compaction_min_size = config::compaction_min_size_mbytes * 1024 * 1024);
+
+ ~CloudSizeBasedCumulativeCompactionPolicy() {}
+
+ int64_t new_cumulative_point(CloudTablet* tablet, const RowsetSharedPtr& output_rowset,
+ Version& last_delete_version,
+ int64_t last_cumulative_point);
+
+ int pick_input_rowsets(CloudTablet* tablet, const std::vector<RowsetSharedPtr>& candidate_rowsets,
+ const int64_t max_compaction_score, const int64_t min_compaction_score,
+ std::vector<RowsetSharedPtr>* input_rowsets,
+ Version* last_delete_version, size_t* compaction_score,
+ bool allow_delete = false);
+
+private:
+ int64_t _level_size(const int64_t size);
+
+ int64_t cloud_promotion_size(CloudTablet* tablet) const;
+
+private:
Review Comment:
warning: redundant access specifier has the same accessibility as the previous access specifier [readability-redundant-access-specifiers]
```suggestion
```
<details>
<summary>Additional context</summary>
**be/src/cloud/cloud_cumulative_compaction_policy.h:56:** previously declared here
```cpp
private:
^
```
</details>
##########
be/src/cloud/cloud_cumulative_compaction.cpp:
##########
@@ -0,0 +1,457 @@
+#include "cloud/cloud_cumulative_compaction.h"
+
+#include "cloud/config.h"
+#include "cloud/cloud_meta_mgr.h"
+#include "common/config.h"
+#include "common/logging.h"
+#include "common/status.h"
+#include "common/sync_point.h"
+#include "gen_cpp/cloud.pb.h"
+#include "olap/compaction.h"
+#include "olap/cumulative_compaction_policy.h"
+#include "service/backend_options.h"
+#include "util/trace.h"
+#include "util/uuid_generator.h"
+
+namespace doris {
+using namespace ErrorCode;
+
+bvar::Adder<uint64_t> cumu_output_size("cumu_compaction", "output_size");
+
+CloudCumulativeCompaction::CloudCumulativeCompaction(CloudStorageEngine& engine, CloudTabletSPtr tablet)
+ : CloudCompactionMixin(engine, std::move(tablet), "BaseCompaction:" + std::to_string(tablet->tablet_id())) {
+ auto uuid = UUIDGenerator::instance()->next_uuid();
+ std::stringstream ss;
+ ss << uuid;
+ _uuid = ss.str();
+}
+
+CloudCumulativeCompaction::~CloudCumulativeCompaction() = default;
+
+Status CloudCumulativeCompaction::prepare_compact() {
+ if (_tablet->tablet_state() != TABLET_RUNNING) {
+ return Status::InternalError("invalid tablet state. tablet_id={}", _tablet->tablet_id());
+ }
+
+ std::vector<std::shared_ptr<CloudCumulativeCompaction>> cumu_compactions;
+ _engine.get_cumu_compaction(_tablet->tablet_id(), cumu_compactions);
+ if (!cumu_compactions.empty()) {
+ for (auto& cumu : cumu_compactions) {
+ _max_conflict_version =
+ std::max(_max_conflict_version, cumu->_input_rowsets.back()->end_version());
+ }
+ }
+
+ int tried = 0;
+PREPARE_TRY_AGAIN:
+
+ bool need_sync_tablet = true;
+ {
+ std::shared_lock rlock(_tablet->get_header_lock());
+ // If number of rowsets is equal to approximate_num_rowsets, it is very likely that this tablet has been
+ // synchronized with meta-service.
+ if (_tablet->tablet_meta()->all_rs_metas().size() >=
+ cloud_tablet()->fetch_add_approximate_num_rowsets(0) &&
+ cloud_tablet()->last_sync_time_s > 0) {
+ need_sync_tablet = false;
+ }
+ }
+ if (need_sync_tablet) {
+ RETURN_IF_ERROR(cloud_tablet()->sync_rowsets());
+ }
+
+ // pick rowsets to compact
+ auto st = pick_rowsets_to_compact();
+ if (!st.ok()) {
+ if (tried == 0 && _last_delete_version.first != -1) {
+ // we meet a delete version, should increase the cumulative point to let base compaction handle the delete version.
+ // plus 1 to skip the delete version.
+ // NOTICE: after that, the cumulative point may be larger than max version of this tablet, but it doesn't matter.
+ update_cumulative_point();
+ }
+ return st;
+ }
+
+ // prepare compaction job
+ cloud::TabletJobInfoPB job;
+ auto idx = job.mutable_idx();
+ idx->set_tablet_id(_tablet->tablet_id());
+ idx->set_table_id(_tablet->table_id());
+ idx->set_index_id(_tablet->index_id());
+ idx->set_partition_id(_tablet->partition_id());
+ auto compaction_job = job.add_compaction();
+ compaction_job->set_id(_uuid);
+ compaction_job->set_initiator(BackendOptions::get_localhost() + ':' +
+ std::to_string(config::heartbeat_service_port));
+ compaction_job->set_type(cloud::TabletCompactionJobPB::CUMULATIVE);
+ compaction_job->set_base_compaction_cnt(_base_compaction_cnt);
+ compaction_job->set_cumulative_compaction_cnt(_cumulative_compaction_cnt);
+ using namespace std::chrono;
+ int64_t now = duration_cast<seconds>(system_clock::now().time_since_epoch()).count();
+ _expiration = now + config::compaction_timeout_seconds;
+ compaction_job->set_expiration(_expiration);
+ compaction_job->set_lease(now + config::lease_compaction_interval_seconds * 4);
+ if (config::enable_parallel_cumu_compaction) {
+ // Set input version range to let meta-service judge version range conflict
+ compaction_job->add_input_versions(_input_rowsets.front()->start_version());
+ compaction_job->add_input_versions(_input_rowsets.back()->end_version());
+ }
+ cloud::StartTabletJobResponse resp;
+ st = _engine.meta_mgr().prepare_tablet_job(job, &resp);
+ if (!st.ok()) {
+ if (resp.status().code() == cloud::STALE_TABLET_CACHE) {
+ // set last_sync_time to 0 to force sync tablet next time
+ cloud_tablet()->last_sync_time_s = 0;
+ } else if (resp.status().code() == cloud::TABLET_NOT_FOUND) {
+ // tablet not found
+ cloud_tablet()->recycle_cached_data();
+ } else if (resp.status().code() == cloud::JOB_TABLET_BUSY) {
+ if (config::enable_parallel_cumu_compaction && resp.version_in_compaction_size() > 0 &&
+ ++tried <= 2) {
+ _max_conflict_version = *std::max_element(resp.version_in_compaction().begin(),
+ resp.version_in_compaction().end());
+ LOG_INFO("retry pick input rowsets")
+ .tag("job_id", _uuid)
+ .tag("max_conflict_version", _max_conflict_version)
+ .tag("tried", tried)
+ .tag("msg", resp.status().msg());
+ goto PREPARE_TRY_AGAIN;
+ } else {
+ LOG_WARNING("failed to prepare cumu compaction")
+ .tag("job_id", _uuid)
+ .tag("msg", resp.status().msg());
+ return Status::Error<CUMULATIVE_NO_SUITABLE_VERSION>("no suitable versions");
+ }
+ }
+ return st;
+ }
+
+ for (auto& rs : _input_rowsets) {
+ _input_row_num += rs->num_rows();
+ _input_segments += rs->num_segments();
+ _input_rowsets_size += rs->data_disk_size();
+ }
+ LOG_INFO("start CloudCumulativeCompaction, tablet_id={}, range=[{}-{}]", _tablet->tablet_id(),
+ _input_rowsets.front()->start_version(), _input_rowsets.back()->end_version())
+ .tag("job_id", _uuid)
+ .tag("input_rowsets", _input_rowsets.size())
+ .tag("input_rows", _input_row_num)
+ .tag("input_segments", _input_segments)
+ .tag("input_data_size", _input_rowsets_size)
+ .tag("tablet_max_version", cloud_tablet()->max_version_unlocked())
+ .tag("cumulative_point", cloud_tablet()->cumulative_layer_point())
+ .tag("num_rowsets", cloud_tablet()->fetch_add_approximate_num_rowsets(0))
+ .tag("cumu_num_rowsets", cloud_tablet()->fetch_add_approximate_cumu_num_rowsets(0));
+ return st;
+}
+
+Status CloudCumulativeCompaction::execute_compact() {
+ TEST_SYNC_POINT_RETURN_WITH_VALUE("CloudCumulativeCompaction::execute_compact_impl",
+ Status::OK(), this);
+ using namespace std::chrono;
+ auto start = steady_clock::now();
+ RETURN_IF_ERROR(CloudCompactionMixin::execute_compact());
+ LOG_INFO("finish CloudCumulativeCompaction, tablet_id={}, cost={}ms", _tablet->tablet_id(),
+ duration_cast<milliseconds>(steady_clock::now() - start).count())
+ .tag("job_id", _uuid)
+ .tag("input_rowsets", _input_rowsets.size())
+ .tag("input_rows", _input_row_num)
+ .tag("input_segments", _input_segments)
+ .tag("input_data_size", _input_rowsets_size)
+ .tag("output_rows", _output_rowset->num_rows())
+ .tag("output_segments", _output_rowset->num_segments())
+ .tag("output_data_size", _output_rowset->data_disk_size())
+ .tag("tablet_max_version", _tablet->max_version_unlocked())
+ .tag("cumulative_point", cloud_tablet()->cumulative_layer_point())
+ .tag("num_rowsets", cloud_tablet()->fetch_add_approximate_num_rowsets(0))
+ .tag("cumu_num_rowsets", cloud_tablet()->fetch_add_approximate_cumu_num_rowsets(0));
+
+ _state = CompactionState::SUCCESS;
+
+ DorisMetrics::instance()->cumulative_compaction_deltas_total->increment(_input_rowsets.size());
+ DorisMetrics::instance()->cumulative_compaction_bytes_total->increment(_input_rowsets_size);
+ cumu_output_size << _output_rowset->data_disk_size();
+
+ return Status::OK();
+}
+
+Status CloudCumulativeCompaction::modify_rowsets() {
Review Comment:
warning: function 'modify_rowsets' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status CloudCumulativeCompaction::modify_rowsets() {
^
```
<details>
<summary>Additional context</summary>
**be/src/cloud/cloud_cumulative_compaction.cpp:177:** 102 lines including whitespace and comments (threshold 80)
```cpp
Status CloudCumulativeCompaction::modify_rowsets() {
^
```
</details>
##########
be/src/cloud/cloud_cumulative_compaction_policy.h:
##########
@@ -0,0 +1,73 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <stddef.h>
+#include <stdint.h>
+
+#include <memory>
+#include <string>
+#include <vector>
+
+#include "common/config.h"
+#include "cloud/cloud_tablet.h"
+#include "olap/rowset/rowset.h"
+#include "olap/rowset/rowset_meta.h"
+
+namespace doris {
+
+class Tablet;
+struct Version;
+
+class CloudSizeBasedCumulativeCompactionPolicy {
+public:
+ CloudSizeBasedCumulativeCompactionPolicy(
+ int64_t promotion_size = config::compaction_promotion_size_mbytes * 1024 * 1024,
+ double promotion_ratio = config::compaction_promotion_ratio,
+ int64_t promotion_min_size = config::compaction_promotion_min_size_mbytes * 1024 * 1024,
+ int64_t compaction_min_size = config::compaction_min_size_mbytes * 1024 * 1024);
+
+ ~CloudSizeBasedCumulativeCompactionPolicy() {}
+
+ int64_t new_cumulative_point(CloudTablet* tablet, const RowsetSharedPtr& output_rowset,
+ Version& last_delete_version,
+ int64_t last_cumulative_point);
+
+ int pick_input_rowsets(CloudTablet* tablet, const std::vector<RowsetSharedPtr>& candidate_rowsets,
+ const int64_t max_compaction_score, const int64_t min_compaction_score,
Review Comment:
warning: parameter 'max_compaction_score' is const-qualified in the function declaration; const-qualification of parameters only has an effect in function definitions [readability-avoid-const-params-in-decls]
```suggestion
int64_t max_compaction_score, const int64_t min_compaction_score,
```
##########
be/src/cloud/cloud_tablet.h:
##########
@@ -108,6 +108,75 @@ class CloudTablet final : public BaseTablet {
void set_base_compaction_cnt(int64_t cnt) { _base_compaction_cnt = cnt; }
void set_cumulative_compaction_cnt(int64_t cnt) { _cumulative_compaction_cnt = cnt; }
void set_cumulative_layer_point(int64_t new_point);
+
+ /*
+ int64_t base_compaction_cnt() const { return _base_compaction_cnt; }
+ void set_base_compaction_cnt(int64_t cnt) { _base_compaction_cnt = cnt; }
+ int64_t cumulative_compaction_cnt() const { return _cumulative_compaction_cnt; }
+ void set_cumulative_compaction_cnt(int64_t cnt) { _cumulative_compaction_cnt = cnt; }
+ int64_t full_compaction_cnt() const { return _full_compaction_cnt; }
+ void set_full_compaction_cnt(int64_t cnt) { _full_compaction_cnt = cnt; }
+ */
+
+ int64_t last_cumu_compaction_failure_time() { return _last_cumu_compaction_failure_millis; }
+ void set_last_cumu_compaction_failure_time(int64_t millis) {
+ _last_cumu_compaction_failure_millis = millis;
+ }
+
+ int64_t last_base_compaction_failure_time() { return _last_base_compaction_failure_millis; }
+ void set_last_base_compaction_failure_time(int64_t millis) {
+ _last_base_compaction_failure_millis = millis;
+ }
+
+ int64_t last_full_compaction_failure_time() { return _last_full_compaction_failure_millis; }
+ void set_last_full_compaction_failure_time(int64_t millis) {
+ _last_full_compaction_failure_millis = millis;
+ }
+
+ int64_t last_cumu_compaction_success_time() { return _last_cumu_compaction_success_millis; }
+ void set_last_cumu_compaction_success_time(int64_t millis) {
+ _last_cumu_compaction_success_millis = millis;
+ }
+
+ int64_t last_base_compaction_success_time() { return _last_base_compaction_success_millis; }
+ void set_last_base_compaction_success_time(int64_t millis) {
+ _last_base_compaction_success_millis = millis;
+ }
+
+ int64_t last_full_compaction_success_time() { return _last_full_compaction_success_millis; }
+ void set_last_full_compaction_success_time(int64_t millis) {
+ _last_full_compaction_success_millis = millis;
+ }
+
+ int64_t last_base_compaction_schedule_time() { return _last_base_compaction_schedule_millis; }
+ void set_last_base_compaction_schedule_time(int64_t millis) {
+ _last_base_compaction_schedule_millis = millis;
+ }
+
+ std::vector<RowsetSharedPtr> pick_candidate_rowsets_to_base_compaction();
+
+ void traverse_rowsets(std::function<void(const RowsetSharedPtr&)> visitor,
Review Comment:
warning: method 'traverse_rowsets' can be made static [readability-convert-member-functions-to-static]
```suggestion
static void traverse_rowsets(std::function<void(const RowsetSharedPtr&)> visitor,
```
##########
be/src/cloud/cloud_tablet.h:
##########
@@ -108,6 +108,75 @@
void set_base_compaction_cnt(int64_t cnt) { _base_compaction_cnt = cnt; }
void set_cumulative_compaction_cnt(int64_t cnt) { _cumulative_compaction_cnt = cnt; }
void set_cumulative_layer_point(int64_t new_point);
+
+ /*
+ int64_t base_compaction_cnt() const { return _base_compaction_cnt; }
+ void set_base_compaction_cnt(int64_t cnt) { _base_compaction_cnt = cnt; }
+ int64_t cumulative_compaction_cnt() const { return _cumulative_compaction_cnt; }
+ void set_cumulative_compaction_cnt(int64_t cnt) { _cumulative_compaction_cnt = cnt; }
+ int64_t full_compaction_cnt() const { return _full_compaction_cnt; }
+ void set_full_compaction_cnt(int64_t cnt) { _full_compaction_cnt = cnt; }
+ */
+
+ int64_t last_cumu_compaction_failure_time() { return _last_cumu_compaction_failure_millis; }
+ void set_last_cumu_compaction_failure_time(int64_t millis) {
+ _last_cumu_compaction_failure_millis = millis;
+ }
+
+ int64_t last_base_compaction_failure_time() { return _last_base_compaction_failure_millis; }
+ void set_last_base_compaction_failure_time(int64_t millis) {
+ _last_base_compaction_failure_millis = millis;
+ }
+
+ int64_t last_full_compaction_failure_time() { return _last_full_compaction_failure_millis; }
+ void set_last_full_compaction_failure_time(int64_t millis) {
+ _last_full_compaction_failure_millis = millis;
+ }
+
+ int64_t last_cumu_compaction_success_time() { return _last_cumu_compaction_success_millis; }
+ void set_last_cumu_compaction_success_time(int64_t millis) {
+ _last_cumu_compaction_success_millis = millis;
+ }
+
+ int64_t last_base_compaction_success_time() { return _last_base_compaction_success_millis; }
+ void set_last_base_compaction_success_time(int64_t millis) {
+ _last_base_compaction_success_millis = millis;
+ }
+
+ int64_t last_full_compaction_success_time() { return _last_full_compaction_success_millis; }
+ void set_last_full_compaction_success_time(int64_t millis) {
+ _last_full_compaction_success_millis = millis;
+ }
+
+ int64_t last_base_compaction_schedule_time() { return _last_base_compaction_schedule_millis; }
+ void set_last_base_compaction_schedule_time(int64_t millis) {
+ _last_base_compaction_schedule_millis = millis;
+ }
+
+ std::vector<RowsetSharedPtr> pick_candidate_rowsets_to_base_compaction();
+
+ void traverse_rowsets(std::function<void(const RowsetSharedPtr&)> visitor,
+ bool include_stale = false) {
+ std::shared_lock rlock(_meta_lock);
+ for (auto& [v, rs] : _rs_version_map) {
+ visitor(rs);
+ }
+ if (!include_stale) return;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (!include_stale) { return;
}
```
##########
be/src/cloud/cloud_storage_engine.cpp:
##########
@@ -190,4 +234,418 @@
}
}
+void CloudStorageEngine::get_cumu_compaction(
+ int64_t tablet_id, std::vector<std::shared_ptr<CloudCumulativeCompaction>>& res) {
+ std::lock_guard lock(_compaction_mtx);
+ if (auto it = _submitted_cumu_compactions.find(tablet_id);
+ it != _submitted_cumu_compactions.end()) {
+ res = it->second;
+ }
+}
+
+void CloudStorageEngine::_adjust_compaction_thread_num() {
+ int base_thread_num = get_base_thread_num();
+ if (_base_compaction_thread_pool->max_threads() != base_thread_num) {
+ int old_max_threads = _base_compaction_thread_pool->max_threads();
+ Status status = _base_compaction_thread_pool->set_max_threads(base_thread_num);
+ if (status.ok()) {
+ VLOG_NOTICE << "update base compaction thread pool max_threads from " << old_max_threads
+ << " to " << base_thread_num;
+ }
+ }
+ if (_base_compaction_thread_pool->min_threads() != base_thread_num) {
+ int old_min_threads = _base_compaction_thread_pool->min_threads();
+ Status status = _base_compaction_thread_pool->set_min_threads(base_thread_num);
+ if (status.ok()) {
+ VLOG_NOTICE << "update base compaction thread pool min_threads from " << old_min_threads
+ << " to " << base_thread_num;
+ }
+ }
+
+ int cumu_thread_num = get_cumu_thread_num();
+ if (_cumu_compaction_thread_pool->max_threads() != cumu_thread_num) {
+ int old_max_threads = _cumu_compaction_thread_pool->max_threads();
+ Status status = _cumu_compaction_thread_pool->set_max_threads(cumu_thread_num);
+ if (status.ok()) {
+ VLOG_NOTICE << "update cumu compaction thread pool max_threads from " << old_max_threads
+ << " to " << cumu_thread_num;
+ }
+ }
+ if (_cumu_compaction_thread_pool->min_threads() != cumu_thread_num) {
+ int old_min_threads = _cumu_compaction_thread_pool->min_threads();
+ Status status = _cumu_compaction_thread_pool->set_min_threads(cumu_thread_num);
+ if (status.ok()) {
+ VLOG_NOTICE << "update cumu compaction thread pool min_threads from " << old_min_threads
+ << " to " << cumu_thread_num;
+ }
+ }
+}
+
+void CloudStorageEngine::_compaction_tasks_producer_callback() {
+ LOG(INFO) << "try to start compaction producer process!";
+
+ int round = 0;
+ CompactionType compaction_type;
+
+ // Used to record the time when the score metric was last updated.
+ // The update of the score metric is accompanied by the logic of selecting the tablet.
+ // If there is no slot available, the logic of selecting the tablet will be terminated,
+ // which causes the score metric update to be terminated.
+ // In order to avoid this situation, we need to update the score regularly.
+ int64_t last_cumulative_score_update_time = 0;
+ int64_t last_base_score_update_time = 0;
+ static const int64_t check_score_interval_ms = 5000; // 5 secs
+
+ int64_t interval = config::generate_compaction_tasks_interval_ms;
+ do {
+ if (!config::disable_auto_compaction) {
+ _adjust_compaction_thread_num();
+
+ bool check_score = false;
+ int64_t cur_time = UnixMillis();
+ if (round < config::cumulative_compaction_rounds_for_each_base_compaction_round) {
+ compaction_type = CompactionType::CUMULATIVE_COMPACTION;
+ round++;
+ if (cur_time - last_cumulative_score_update_time >= check_score_interval_ms) {
+ check_score = true;
+ last_cumulative_score_update_time = cur_time;
+ }
+ } else {
+ compaction_type = CompactionType::BASE_COMPACTION;
+ round = 0;
+ if (cur_time - last_base_score_update_time >= check_score_interval_ms) {
+ check_score = true;
+ last_base_score_update_time = cur_time;
+ }
+ }
+ std::unique_ptr<ThreadPool>& thread_pool =
+ (compaction_type == CompactionType::CUMULATIVE_COMPACTION)
+ ? _cumu_compaction_thread_pool
+ : _base_compaction_thread_pool;
+ VLOG_CRITICAL << "compaction thread pool. type: "
+ << (compaction_type == CompactionType::CUMULATIVE_COMPACTION ? "CUMU"
+ : "BASE")
+ << ", num_threads: " << thread_pool->num_threads()
+ << ", num_threads_pending_start: "
+ << thread_pool->num_threads_pending_start()
+ << ", num_active_threads: " << thread_pool->num_active_threads()
+ << ", max_threads: " << thread_pool->max_threads()
+ << ", min_threads: " << thread_pool->min_threads()
+ << ", num_total_queued_tasks: " << thread_pool->get_queue_size();
+ std::vector<CloudTabletSPtr> tablets_compaction =
+ _generate_cloud_compaction_tasks(compaction_type, check_score);
+
+ /// Regardless of whether the tablet is submitted for compaction or not,
+ /// we need to call 'reset_compaction' to clean up the base_compaction or cumulative_compaction objects
+ /// in the tablet, because these two objects store the tablet's own shared_ptr.
+ /// If it is not cleaned up, the reference count of the tablet will always be greater than 1,
+ /// thus cannot be collected by the garbage collector. (TabletManager::start_trash_sweep)
+ for (const auto& tablet : tablets_compaction) {
+ Status st = submit_compaction_task(tablet, compaction_type);
+ if (st.ok()) continue;
+ if ((!st.is<ErrorCode::BE_NO_SUITABLE_VERSION>() &&
+ !st.is<ErrorCode::CUMULATIVE_NO_SUITABLE_VERSION>()) ||
+ VLOG_DEBUG_IS_ON) {
+ LOG(WARNING) << "failed to submit compaction task for tablet: "
+ << tablet->tablet_id() << ", err: " << st;
+ }
+ }
+ interval = config::generate_compaction_tasks_interval_ms;
+ } else {
+ interval = config::check_auto_compaction_interval_seconds * 1000;
+ }
+ } while (!_stop_background_threads_latch.wait_for(std::chrono::milliseconds(interval)));
+}
+
+std::vector<CloudTabletSPtr> CloudStorageEngine::_generate_cloud_compaction_tasks(
+ CompactionType compaction_type, bool check_score) {
+ std::vector<std::shared_ptr<CloudTablet>> tablets_compaction;
+
+ int64_t max_compaction_score = 0;
+ std::unordered_set<int64_t> tablet_preparing_cumu_compaction;
+ std::unordered_map<int64_t, std::vector<std::shared_ptr<CloudCumulativeCompaction>>>
+ submitted_cumu_compactions;
+ std::unordered_map<int64_t, std::shared_ptr<CloudBaseCompaction>> submitted_base_compactions;
+ std::unordered_map<int64_t, std::shared_ptr<CloudFullCompaction>> submitted_full_compactions;
+ {
+ std::lock_guard lock(_compaction_mtx);
+ tablet_preparing_cumu_compaction = _tablet_preparing_cumu_compaction;
+ submitted_cumu_compactions = _submitted_cumu_compactions;
+ submitted_base_compactions = _submitted_base_compactions;
+ submitted_full_compactions = _submitted_full_compactions;
+ }
+
+ bool need_pick_tablet = true;
+ int thread_per_disk =
+ config::compaction_task_num_per_fast_disk; // all disks are fast in cloud mode
+ int num_cumu =
+ std::accumulate(submitted_cumu_compactions.begin(), submitted_cumu_compactions.end(), 0,
+ [](int a, auto& b) { return a + b.second.size(); });
+ int num_base = submitted_base_compactions.size() + submitted_full_compactions.size();
+ int n = thread_per_disk - num_cumu - num_base;
+ if (compaction_type == CompactionType::BASE_COMPACTION) {
+ // We need to reserve at least one thread for cumulative compaction,
+ // because base compactions may take too long to complete, which may
+ // leads to "too many rowsets" error.
+ int base_n = std::min(config::max_base_compaction_task_num_per_disk, thread_per_disk - 1) -
+ num_base;
+ n = std::min(base_n, n);
+ }
+ if (n <= 0) { // No threads available
+ if (!check_score) return tablets_compaction;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (!check_score) { return tablets_compaction;
}
```
##########
be/src/cloud/cloud_tablet.h:
##########
@@ -108,6 +108,75 @@
void set_base_compaction_cnt(int64_t cnt) { _base_compaction_cnt = cnt; }
void set_cumulative_compaction_cnt(int64_t cnt) { _cumulative_compaction_cnt = cnt; }
void set_cumulative_layer_point(int64_t new_point);
+
+ /*
+ int64_t base_compaction_cnt() const { return _base_compaction_cnt; }
+ void set_base_compaction_cnt(int64_t cnt) { _base_compaction_cnt = cnt; }
+ int64_t cumulative_compaction_cnt() const { return _cumulative_compaction_cnt; }
+ void set_cumulative_compaction_cnt(int64_t cnt) { _cumulative_compaction_cnt = cnt; }
+ int64_t full_compaction_cnt() const { return _full_compaction_cnt; }
+ void set_full_compaction_cnt(int64_t cnt) { _full_compaction_cnt = cnt; }
+ */
+
+ int64_t last_cumu_compaction_failure_time() { return _last_cumu_compaction_failure_millis; }
+ void set_last_cumu_compaction_failure_time(int64_t millis) {
+ _last_cumu_compaction_failure_millis = millis;
+ }
+
+ int64_t last_base_compaction_failure_time() { return _last_base_compaction_failure_millis; }
+ void set_last_base_compaction_failure_time(int64_t millis) {
+ _last_base_compaction_failure_millis = millis;
+ }
+
+ int64_t last_full_compaction_failure_time() { return _last_full_compaction_failure_millis; }
+ void set_last_full_compaction_failure_time(int64_t millis) {
+ _last_full_compaction_failure_millis = millis;
+ }
+
+ int64_t last_cumu_compaction_success_time() { return _last_cumu_compaction_success_millis; }
+ void set_last_cumu_compaction_success_time(int64_t millis) {
+ _last_cumu_compaction_success_millis = millis;
+ }
+
+ int64_t last_base_compaction_success_time() { return _last_base_compaction_success_millis; }
+ void set_last_base_compaction_success_time(int64_t millis) {
+ _last_base_compaction_success_millis = millis;
+ }
+
+ int64_t last_full_compaction_success_time() { return _last_full_compaction_success_millis; }
+ void set_last_full_compaction_success_time(int64_t millis) {
+ _last_full_compaction_success_millis = millis;
+ }
+
+ int64_t last_base_compaction_schedule_time() { return _last_base_compaction_schedule_millis; }
+ void set_last_base_compaction_schedule_time(int64_t millis) {
+ _last_base_compaction_schedule_millis = millis;
+ }
+
+ std::vector<RowsetSharedPtr> pick_candidate_rowsets_to_base_compaction();
+
+ void traverse_rowsets(std::function<void(const RowsetSharedPtr&)> visitor,
+ bool include_stale = false) {
+ std::shared_lock rlock(_meta_lock);
+ for (auto& [v, rs] : _rs_version_map) {
+ visitor(rs);
+ }
+ if (!include_stale) return;
+ for (auto& [v, rs] : _stale_rs_version_map) {
+ visitor(rs);
+ }
+ }
+
+ inline Version max_version() const {
Review Comment:
warning: method 'max_version' can be made static [readability-convert-member-functions-to-static]
```suggestion
static inline Version max_version() {
```
##########
be/src/cloud/cloud_cumulative_compaction_policy.cpp:
##########
@@ -0,0 +1,205 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_cumulative_compaction_policy.h"
+
+#include <algorithm>
+#include <list>
+#include <ostream>
+#include <string>
+
+#include "common/config.h"
+#include "common/logging.h"
+#include "common/sync_point.h"
+#include "cloud/config.h"
+#include "olap/olap_common.h"
+#include "olap/tablet.h"
+#include "olap/tablet_meta.h"
+
+namespace doris {
+
+CloudSizeBasedCumulativeCompactionPolicy::CloudSizeBasedCumulativeCompactionPolicy(
+ int64_t promotion_size, double promotion_ratio, int64_t promotion_min_size,
+ int64_t compaction_min_size)
+ : _promotion_size(promotion_size),
+ _promotion_ratio(promotion_ratio),
+ _promotion_min_size(promotion_min_size),
+ _compaction_min_size(compaction_min_size) {}
+
+int64_t CloudSizeBasedCumulativeCompactionPolicy::_level_size(const int64_t size) {
+ if (size < 1024) return 0;
+ int64_t max_level = (int64_t)1
+ << (sizeof(_promotion_size) * 8 - 1 - __builtin_clzl(_promotion_size / 2));
+ if (size >= max_level) return max_level;
+ return (int64_t)1 << (sizeof(size) * 8 - 1 - __builtin_clzl(size));
+}
+
+int CloudSizeBasedCumulativeCompactionPolicy::pick_input_rowsets(
+ CloudTablet* tablet, const std::vector<RowsetSharedPtr>& candidate_rowsets,
+ const int64_t max_compaction_score, const int64_t min_compaction_score,
+ std::vector<RowsetSharedPtr>* input_rowsets, Version* last_delete_version,
+ size_t* compaction_score, bool allow_delete) {
+ //size_t promotion_size = tablet->cumulative_promotion_size();
+ auto max_version = tablet->max_version().first;
+ int transient_size = 0;
+ *compaction_score = 0;
+ int64_t total_size = 0;
+ for (auto& rowset : candidate_rowsets) {
+ // check whether this rowset is delete version
+ if (!allow_delete && rowset->rowset_meta()->has_delete_predicate()) {
+ *last_delete_version = rowset->version();
+ if (!input_rowsets->empty()) {
+ // we meet a delete version, and there were other versions before.
+ // we should compact those version before handling them over to base compaction
+ break;
+ } else {
+ // we meet a delete version, and no other versions before, skip it and continue
+ input_rowsets->clear();
+ *compaction_score = 0;
+ transient_size = 0;
+ continue;
+ }
+ }
+ if (tablet->tablet_state() == TABLET_NOTREADY) {
+ // If tablet under alter, keep latest 10 version so that base tablet max version
+ // not merged in new tablet, and then we can copy data from base tablet
+ if (rowset->version().second < max_version - 10) {
+ continue;
+ }
+ }
+ if (*compaction_score >= max_compaction_score) {
+ // got enough segments
+ break;
+ }
+ *compaction_score += rowset->rowset_meta()->get_compaction_score();
+ total_size += rowset->rowset_meta()->total_disk_size();
+
+ transient_size += 1;
+ input_rowsets->push_back(rowset);
+ }
+
+ // if there is delete version, do compaction directly
+ if (last_delete_version->first != -1) {
+ if (input_rowsets->size() == 1) {
+ auto rs_meta = input_rowsets->front()->rowset_meta();
+ // if there is only one rowset and not overlapping,
+ // we do not need to do cumulative compaction
+ if (!rs_meta->is_segments_overlapping()) {
+ input_rowsets->clear();
+ *compaction_score = 0;
+ }
+ }
+ return transient_size;
+ }
+
+ auto rs_begin = input_rowsets->begin();
+ size_t new_compaction_score = *compaction_score;
+ while (rs_begin != input_rowsets->end()) {
+ auto& rs_meta = (*rs_begin)->rowset_meta();
+ int current_level = _level_size(rs_meta->total_disk_size());
+ int remain_level = _level_size(total_size - rs_meta->total_disk_size());
+ // if current level less then remain level, input rowsets contain current rowset
+ // and process return; otherwise, input rowsets do not contain current rowset.
+ if (current_level <= remain_level) {
+ break;
+ }
+ total_size -= rs_meta->total_disk_size();
+ new_compaction_score -= rs_meta->get_compaction_score();
+ ++rs_begin;
+ }
+ if (rs_begin == input_rowsets->end()) { // No suitable level size found in `input_rowsets`
+ if (config::prioritize_query_perf_in_compaction && tablet->keys_type() != DUP_KEYS) {
+ // While tablet's key type is not `DUP_KEYS`, compacting rowset in such tablets has a significant
+ // positive impact on queries and reduces space amplification, so we ignore level limitation and
+ // pick candidate rowsets as input rowsets.
+ return transient_size;
+ } else if (*compaction_score >= max_compaction_score) {
+ // Score of `input_rowsets` exceed max compaction score, which means `input_rowsets` will never change and
+ // this tablet will never execute cumulative compaction. MUST execute compaction on these `input_rowsets`
+ // to reduce compaction score.
+ RowsetSharedPtr rs_with_max_score;
+ uint32_t max_score = 1;
+ for (auto& rs : *input_rowsets) {
+ if (rs->rowset_meta()->get_compaction_score() > max_score) {
+ max_score = rs->rowset_meta()->get_compaction_score();
+ rs_with_max_score = rs;
+ }
+ }
+ if (rs_with_max_score) {
+ input_rowsets->clear();
+ input_rowsets->push_back(std::move(rs_with_max_score));
+ *compaction_score = max_score;
+ return transient_size;
+ }
+ // Exceeding max compaction score, do compaction on all candidate rowsets anyway
+ return transient_size;
+ }
+ }
+ input_rowsets->erase(input_rowsets->begin(), rs_begin);
+ *compaction_score = new_compaction_score;
+
+ VLOG_CRITICAL << "cumulative compaction size_based policy, compaction_score = "
+ << *compaction_score << ", total_size = " << total_size
+ //<< ", calc promotion size value = " << promotion_size
+ << ", tablet = " << tablet->tablet_id() << ", input_rowset size "
+ << input_rowsets->size();
+
+ // empty return
+ if (input_rowsets->empty()) {
+ return transient_size;
+ }
+
+ // if we have a sufficient number of segments, we should process the compaction.
+ // otherwise, we check number of segments and total_size whether can do compaction.
+ if (total_size < _compaction_min_size && *compaction_score < min_compaction_score) {
+ input_rowsets->clear();
+ *compaction_score = 0;
+ } else if (total_size >= _compaction_min_size && input_rowsets->size() == 1) {
+ auto rs_meta = input_rowsets->front()->rowset_meta();
+ // if there is only one rowset and not overlapping,
+ // we do not need to do compaction
+ if (!rs_meta->is_segments_overlapping()) {
+ input_rowsets->clear();
+ *compaction_score = 0;
+ }
+ }
+ return transient_size;
+}
+
+int64_t CloudSizeBasedCumulativeCompactionPolicy::cloud_promotion_size(CloudTablet* t) const {
+ int64_t promotion_size = t->base_size() * _promotion_ratio;
+ // promotion_size is between _size_based_promotion_size and _size_based_promotion_min_size
+ return promotion_size > _promotion_size ? _promotion_size
+ : promotion_size < _promotion_min_size ? _promotion_min_size
+ : promotion_size;
+}
+
+int64_t CloudSizeBasedCumulativeCompactionPolicy::new_cumulative_point(
Review Comment:
warning: method 'new_cumulative_point' can be made static [readability-convert-member-functions-to-static]
be/src/cloud/cloud_cumulative_compaction_policy.h:46:
```diff
- int64_t new_cumulative_point(CloudTablet* tablet, const RowsetSharedPtr& output_rowset,
+ static int64_t new_cumulative_point(CloudTablet* tablet, const RowsetSharedPtr& output_rowset,
```
##########
be/src/cloud/cloud_cumulative_compaction_policy.h:
##########
@@ -0,0 +1,73 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <stddef.h>
+#include <stdint.h>
Review Comment:
warning: inclusion of deprecated C++ header 'stdint.h'; consider using 'cstdint' instead [modernize-deprecated-headers]
```suggestion
#include <cstdint>
```
##########
be/src/cloud/cloud_cumulative_compaction_policy.cpp:
##########
@@ -0,0 +1,205 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_cumulative_compaction_policy.h"
+
+#include <algorithm>
+#include <list>
+#include <ostream>
+#include <string>
+
+#include "common/config.h"
+#include "common/logging.h"
+#include "common/sync_point.h"
+#include "cloud/config.h"
+#include "olap/olap_common.h"
+#include "olap/tablet.h"
+#include "olap/tablet_meta.h"
+
+namespace doris {
+
+CloudSizeBasedCumulativeCompactionPolicy::CloudSizeBasedCumulativeCompactionPolicy(
+ int64_t promotion_size, double promotion_ratio, int64_t promotion_min_size,
+ int64_t compaction_min_size)
+ : _promotion_size(promotion_size),
+ _promotion_ratio(promotion_ratio),
+ _promotion_min_size(promotion_min_size),
+ _compaction_min_size(compaction_min_size) {}
+
+int64_t CloudSizeBasedCumulativeCompactionPolicy::_level_size(const int64_t size) {
+ if (size < 1024) return 0;
+ int64_t max_level = (int64_t)1
+ << (sizeof(_promotion_size) * 8 - 1 - __builtin_clzl(_promotion_size / 2));
+ if (size >= max_level) return max_level;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (size >= max_level) { return max_level;
}
```
##########
be/src/cloud/cloud_base_compaction.cpp:
##########
@@ -0,0 +1,369 @@
+#include "cloud/cloud_base_compaction.h"
+
+#include <boost/container_hash/hash.hpp>
+
+#include "cloud/cloud_meta_mgr.h"
+#include "cloud/config.h"
+#include "common/config.h"
+#include "common/sync_point.h"
+#include "gen_cpp/cloud.pb.h"
+#include "olap/compaction.h"
+#include "olap/task/engine_checksum_task.h"
+#include "service/backend_options.h"
+#include "util/thread.h"
+#include "util/uuid_generator.h"
+
+namespace doris {
+using namespace ErrorCode;
+
+bvar::Adder<uint64_t> base_output_size("base_compaction", "output_size");
+
+CloudBaseCompaction::CloudBaseCompaction(CloudStorageEngine& engine, CloudTabletSPtr tablet)
+ : CloudCompactionMixin(engine, std::move(tablet), "BaseCompaction:" + std::to_string(tablet->tablet_id())) {
+ auto uuid = UUIDGenerator::instance()->next_uuid();
+ std::stringstream ss;
+ ss << uuid;
+ _uuid = ss.str();
+}
+
+CloudBaseCompaction::~CloudBaseCompaction() = default;
+
+Status CloudBaseCompaction::prepare_compact() {
+ if (_tablet->tablet_state() != TABLET_RUNNING) {
+ return Status::InternalError("invalid tablet state. tablet_id={}", _tablet->tablet_id());
+ }
+
+ bool need_sync_tablet = true;
+ {
+ std::shared_lock rlock(_tablet->get_header_lock());
+ // If number of rowsets is equal to approximate_num_rowsets, it is very likely that this tablet has been
+ // synchronized with meta-service.
+ if (_tablet->tablet_meta()->all_rs_metas().size() >=
+ cloud_tablet()->fetch_add_approximate_num_rowsets(0) &&
+ cloud_tablet()->last_sync_time_s > 0) {
+ need_sync_tablet = false;
+ }
+ }
+ if (need_sync_tablet) {
+ RETURN_IF_ERROR(cloud_tablet()->sync_rowsets());
+ }
+
+ RETURN_IF_ERROR(pick_rowsets_to_compact());
+
+ // prepare compaction job
+ cloud::TabletJobInfoPB job;
+ auto idx = job.mutable_idx();
+ idx->set_tablet_id(_tablet->tablet_id());
+ idx->set_table_id(_tablet->table_id());
+ idx->set_index_id(_tablet->index_id());
+ idx->set_partition_id(_tablet->partition_id());
+ auto compaction_job = job.add_compaction();
+ compaction_job->set_id(_uuid);
+ compaction_job->set_initiator(BackendOptions::get_localhost() + ':' +
+ std::to_string(config::heartbeat_service_port));
+ compaction_job->set_type(cloud::TabletCompactionJobPB::BASE);
+ compaction_job->set_base_compaction_cnt(_base_compaction_cnt);
+ compaction_job->set_cumulative_compaction_cnt(_cumulative_compaction_cnt);
+ using namespace std::chrono;
+ int64_t now = duration_cast<seconds>(system_clock::now().time_since_epoch()).count();
+ _expiration = now + config::compaction_timeout_seconds;
+ compaction_job->set_expiration(_expiration);
+ compaction_job->set_lease(now + config::lease_compaction_interval_seconds * 4);
+ cloud::StartTabletJobResponse resp;
+ //auto st = cloud::meta_mgr()->prepare_tablet_job(job, &resp);
+ auto st = _engine.meta_mgr().prepare_tablet_job(job, &resp);
+ if (!st.ok()) {
+ if (resp.status().code() == cloud::STALE_TABLET_CACHE) {
+ // set last_sync_time to 0 to force sync tablet next time
+ cloud_tablet()->last_sync_time_s = 0;
+ } else if (resp.status().code() == cloud::TABLET_NOT_FOUND) {
+ // tablet not found
+ cloud_tablet()->recycle_cached_data();
+ }
+ return st;
+ }
+
+ for (auto& rs : _input_rowsets) {
+ _input_row_num += rs->num_rows();
+ _input_segments += rs->num_segments();
+ _input_rowsets_size += rs->data_disk_size();
+ }
+ LOG_INFO("start CloudBaseCompaction, tablet_id={}, range=[{}-{}]", _tablet->tablet_id(),
+ _input_rowsets.front()->start_version(), _input_rowsets.back()->end_version())
+ .tag("job_id", _uuid)
+ .tag("input_rowsets", _input_rowsets.size())
+ .tag("input_rows", _input_row_num)
+ .tag("input_segments", _input_segments)
+ .tag("input_data_size", _input_rowsets_size);
+ return st;
+}
+
+void CloudBaseCompaction::_filter_input_rowset() {
+ // if dup_key and no delete predicate
+ // we skip big files to save resources
+ if (_tablet->keys_type() != KeysType::DUP_KEYS) {
+ return;
+ }
+ for (auto& rs : _input_rowsets) {
+ if (rs->rowset_meta()->has_delete_predicate()) {
+ return;
+ }
+ }
+ int64_t max_size = config::base_compaction_dup_key_max_file_size_mbytes * 1024 * 1024;
+ // first find a proper rowset for start
+ auto rs_iter = _input_rowsets.begin();
+ while (rs_iter != _input_rowsets.end()) {
+ if ((*rs_iter)->rowset_meta()->total_disk_size() >= max_size) {
+ rs_iter = _input_rowsets.erase(rs_iter);
+ } else {
+ break;
+ }
+ }
+}
+
+Status CloudBaseCompaction::pick_rowsets_to_compact() {
+ _input_rowsets.clear();
+ {
+ std::shared_lock rlock(_tablet->get_header_lock());
+ _base_compaction_cnt = cloud_tablet()->base_compaction_cnt();
+ _cumulative_compaction_cnt = cloud_tablet()->cumulative_compaction_cnt();
+ _input_rowsets = cloud_tablet()->pick_candidate_rowsets_to_base_compaction();
+ }
+ if (auto st = check_version_continuity(_input_rowsets); !st.ok()) {
+ DCHECK(false) << st;
+ return st;
+ }
+ _filter_input_rowset();
+ if (_input_rowsets.size() <= 1) {
+ return Status::Error<BE_NO_SUITABLE_VERSION>(
+ "insuffient compation input rowset, #rowsets={}", _input_rowsets.size());
+ }
+
+ if (_input_rowsets.size() == 2 && _input_rowsets[0]->end_version() == 1) {
+ // the tablet is with rowset: [0-1], [2-y]
+ // and [0-1] has no data. in this situation, no need to do base compaction.
+ return Status::Error<BE_NO_SUITABLE_VERSION>("no suitable versions for compaction");
+ }
+
+ // 1. cumulative rowset must reach base_compaction_min_rowset_num threshold
+ if (_input_rowsets.size() > config::base_compaction_min_rowset_num) {
+ VLOG_NOTICE << "satisfy the base compaction policy. tablet=" << _tablet->tablet_id()
+ << ", num_cumulative_rowsets=" << _input_rowsets.size() - 1
+ << ", base_compaction_num_cumulative_rowsets="
+ << config::base_compaction_min_rowset_num;
+ return Status::OK();
+ }
+
+ // 2. the ratio between base rowset and all input cumulative rowsets reaches the threshold
+ // `_input_rowsets` has been sorted by end version, so we consider `_input_rowsets[0]` is the base rowset.
+ int64_t base_size = _input_rowsets.front()->data_disk_size();
+ int64_t cumulative_total_size = 0;
+ for (auto it = _input_rowsets.begin() + 1; it != _input_rowsets.end(); ++it) {
+ cumulative_total_size += (*it)->data_disk_size();
+ }
+
+ double base_cumulative_delta_ratio = config::base_compaction_min_data_ratio;
+ if (base_size == 0) {
+ // base_size == 0 means this may be a base version [0-1], which has no data.
+ // set to 1 to void divide by zero
+ base_size = 1;
+ }
+ double cumulative_base_ratio = static_cast<double>(cumulative_total_size) / base_size;
+
+ if (cumulative_base_ratio > base_cumulative_delta_ratio) {
+ VLOG_NOTICE << "satisfy the base compaction policy. tablet=" << _tablet->tablet_id()
+ << ", cumulative_total_size=" << cumulative_total_size
+ << ", base_size=" << base_size
+ << ", cumulative_base_ratio=" << cumulative_base_ratio
+ << ", policy_ratio=" << base_cumulative_delta_ratio;
+ return Status::OK();
+ }
+
+ // 3. the interval since last base compaction reaches the threshold
+ int64_t base_creation_time = _input_rowsets[0]->creation_time();
+ int64_t interval_threshold = config::base_compaction_interval_seconds_since_last_operation;
+ int64_t interval_since_last_base_compaction = time(nullptr) - base_creation_time;
+ if (interval_since_last_base_compaction > interval_threshold) {
+ VLOG_NOTICE << "satisfy the base compaction policy. tablet=" << _tablet->tablet_id()
+ << ", interval_since_last_base_compaction="
+ << interval_since_last_base_compaction
+ << ", interval_threshold=" << interval_threshold;
+ return Status::OK();
+ }
+
+ VLOG_NOTICE << "don't satisfy the base compaction policy. tablet=" << _tablet->tablet_id()
+ << ", num_cumulative_rowsets=" << _input_rowsets.size() - 1
+ << ", cumulative_base_ratio=" << cumulative_base_ratio
+ << ", interval_since_last_base_compaction=" << interval_since_last_base_compaction;
+ return Status::Error<BE_NO_SUITABLE_VERSION>("no suitable versions for compaction");
+}
+
+Status CloudBaseCompaction::execute_compact() {
+ if (config::enable_base_compaction_idle_sched) {
+ Thread::set_idle_sched();
+ }
+
+ SCOPED_ATTACH_TASK(_mem_tracker);
+
+ using namespace std::chrono;
+ auto start = steady_clock::now();
+ RETURN_IF_ERROR(CloudCompactionMixin::execute_compact());
+ LOG_INFO("finish CloudBaseCompaction, tablet_id={}, cost={}ms", _tablet->tablet_id(),
+ duration_cast<milliseconds>(steady_clock::now() - start).count())
+ .tag("job_id", _uuid)
+ .tag("input_rowsets", _input_rowsets.size())
+ .tag("input_rows", _input_row_num)
+ .tag("input_segments", _input_segments)
+ .tag("input_data_size", _input_rowsets_size)
+ .tag("output_rows", _output_rowset->num_rows())
+ .tag("output_segments", _output_rowset->num_segments())
+ .tag("output_data_size", _output_rowset->data_disk_size());
+
+ //_compaction_succeed = true;
+ _state = CompactionState::SUCCESS;
+
+ DorisMetrics::instance()->base_compaction_deltas_total->increment(_input_rowsets.size());
+ DorisMetrics::instance()->base_compaction_bytes_total->increment(_input_rowsets_size);
+ base_output_size << _output_rowset->data_disk_size();
+
+ return Status::OK();
+}
+
+Status CloudBaseCompaction::modify_rowsets() {
Review Comment:
warning: function 'modify_rowsets' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status CloudBaseCompaction::modify_rowsets() {
^
```
<details>
<summary>Additional context</summary>
**be/src/cloud/cloud_base_compaction.cpp:231:** 82 lines including whitespace and comments (threshold 80)
```cpp
Status CloudBaseCompaction::modify_rowsets() {
^
```
</details>
##########
be/src/cloud/cloud_base_compaction.cpp:
##########
@@ -0,0 +1,369 @@
+#include "cloud/cloud_base_compaction.h"
+
+#include <boost/container_hash/hash.hpp>
+
+#include "cloud/cloud_meta_mgr.h"
+#include "cloud/config.h"
+#include "common/config.h"
+#include "common/sync_point.h"
+#include "gen_cpp/cloud.pb.h"
+#include "olap/compaction.h"
+#include "olap/task/engine_checksum_task.h"
+#include "service/backend_options.h"
+#include "util/thread.h"
+#include "util/uuid_generator.h"
+
+namespace doris {
+using namespace ErrorCode;
+
+bvar::Adder<uint64_t> base_output_size("base_compaction", "output_size");
+
+CloudBaseCompaction::CloudBaseCompaction(CloudStorageEngine& engine, CloudTabletSPtr tablet)
+ : CloudCompactionMixin(engine, std::move(tablet), "BaseCompaction:" + std::to_string(tablet->tablet_id())) {
+ auto uuid = UUIDGenerator::instance()->next_uuid();
+ std::stringstream ss;
+ ss << uuid;
+ _uuid = ss.str();
+}
+
+CloudBaseCompaction::~CloudBaseCompaction() = default;
+
+Status CloudBaseCompaction::prepare_compact() {
+ if (_tablet->tablet_state() != TABLET_RUNNING) {
+ return Status::InternalError("invalid tablet state. tablet_id={}", _tablet->tablet_id());
+ }
+
+ bool need_sync_tablet = true;
+ {
+ std::shared_lock rlock(_tablet->get_header_lock());
+ // If number of rowsets is equal to approximate_num_rowsets, it is very likely that this tablet has been
+ // synchronized with meta-service.
+ if (_tablet->tablet_meta()->all_rs_metas().size() >=
+ cloud_tablet()->fetch_add_approximate_num_rowsets(0) &&
+ cloud_tablet()->last_sync_time_s > 0) {
+ need_sync_tablet = false;
+ }
+ }
+ if (need_sync_tablet) {
+ RETURN_IF_ERROR(cloud_tablet()->sync_rowsets());
+ }
+
+ RETURN_IF_ERROR(pick_rowsets_to_compact());
+
+ // prepare compaction job
+ cloud::TabletJobInfoPB job;
+ auto idx = job.mutable_idx();
+ idx->set_tablet_id(_tablet->tablet_id());
+ idx->set_table_id(_tablet->table_id());
+ idx->set_index_id(_tablet->index_id());
+ idx->set_partition_id(_tablet->partition_id());
+ auto compaction_job = job.add_compaction();
+ compaction_job->set_id(_uuid);
+ compaction_job->set_initiator(BackendOptions::get_localhost() + ':' +
+ std::to_string(config::heartbeat_service_port));
+ compaction_job->set_type(cloud::TabletCompactionJobPB::BASE);
+ compaction_job->set_base_compaction_cnt(_base_compaction_cnt);
+ compaction_job->set_cumulative_compaction_cnt(_cumulative_compaction_cnt);
+ using namespace std::chrono;
+ int64_t now = duration_cast<seconds>(system_clock::now().time_since_epoch()).count();
+ _expiration = now + config::compaction_timeout_seconds;
+ compaction_job->set_expiration(_expiration);
+ compaction_job->set_lease(now + config::lease_compaction_interval_seconds * 4);
+ cloud::StartTabletJobResponse resp;
+ //auto st = cloud::meta_mgr()->prepare_tablet_job(job, &resp);
+ auto st = _engine.meta_mgr().prepare_tablet_job(job, &resp);
+ if (!st.ok()) {
+ if (resp.status().code() == cloud::STALE_TABLET_CACHE) {
+ // set last_sync_time to 0 to force sync tablet next time
+ cloud_tablet()->last_sync_time_s = 0;
+ } else if (resp.status().code() == cloud::TABLET_NOT_FOUND) {
+ // tablet not found
+ cloud_tablet()->recycle_cached_data();
+ }
+ return st;
+ }
+
+ for (auto& rs : _input_rowsets) {
+ _input_row_num += rs->num_rows();
+ _input_segments += rs->num_segments();
+ _input_rowsets_size += rs->data_disk_size();
+ }
+ LOG_INFO("start CloudBaseCompaction, tablet_id={}, range=[{}-{}]", _tablet->tablet_id(),
+ _input_rowsets.front()->start_version(), _input_rowsets.back()->end_version())
+ .tag("job_id", _uuid)
+ .tag("input_rowsets", _input_rowsets.size())
+ .tag("input_rows", _input_row_num)
+ .tag("input_segments", _input_segments)
+ .tag("input_data_size", _input_rowsets_size);
+ return st;
+}
+
+void CloudBaseCompaction::_filter_input_rowset() {
Review Comment:
warning: method '_filter_input_rowset' can be made static [readability-convert-member-functions-to-static]
be/src/cloud/cloud_base_compaction.h:29:
```diff
- void _filter_input_rowset();
+ static void _filter_input_rowset();
```
##########
be/src/cloud/cloud_cumulative_compaction_policy.h:
##########
@@ -0,0 +1,73 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <stddef.h>
+#include <stdint.h>
+
+#include <memory>
+#include <string>
+#include <vector>
+
+#include "common/config.h"
+#include "cloud/cloud_tablet.h"
+#include "olap/rowset/rowset.h"
+#include "olap/rowset/rowset_meta.h"
+
+namespace doris {
+
+class Tablet;
+struct Version;
+
+class CloudSizeBasedCumulativeCompactionPolicy {
+public:
+ CloudSizeBasedCumulativeCompactionPolicy(
+ int64_t promotion_size = config::compaction_promotion_size_mbytes * 1024 * 1024,
+ double promotion_ratio = config::compaction_promotion_ratio,
+ int64_t promotion_min_size = config::compaction_promotion_min_size_mbytes * 1024 * 1024,
+ int64_t compaction_min_size = config::compaction_min_size_mbytes * 1024 * 1024);
+
+ ~CloudSizeBasedCumulativeCompactionPolicy() {}
+
+ int64_t new_cumulative_point(CloudTablet* tablet, const RowsetSharedPtr& output_rowset,
+ Version& last_delete_version,
+ int64_t last_cumulative_point);
+
+ int pick_input_rowsets(CloudTablet* tablet, const std::vector<RowsetSharedPtr>& candidate_rowsets,
+ const int64_t max_compaction_score, const int64_t min_compaction_score,
Review Comment:
warning: parameter 'min_compaction_score' is const-qualified in the function declaration; const-qualification of parameters only has an effect in function definitions [readability-avoid-const-params-in-decls]
```suggestion
const int64_t max_compaction_score, int64_t min_compaction_score,
```
##########
be/src/cloud/cloud_storage_engine.cpp:
##########
@@ -190,4 +234,418 @@
}
}
+void CloudStorageEngine::get_cumu_compaction(
+ int64_t tablet_id, std::vector<std::shared_ptr<CloudCumulativeCompaction>>& res) {
+ std::lock_guard lock(_compaction_mtx);
+ if (auto it = _submitted_cumu_compactions.find(tablet_id);
+ it != _submitted_cumu_compactions.end()) {
+ res = it->second;
+ }
+}
+
+void CloudStorageEngine::_adjust_compaction_thread_num() {
Review Comment:
warning: method '_adjust_compaction_thread_num' can be made static [readability-convert-member-functions-to-static]
be/src/cloud/cloud_storage_engine.h:83:
```diff
- void _adjust_compaction_thread_num();
+ static void _adjust_compaction_thread_num();
```
##########
be/src/cloud/cloud_cumulative_compaction_policy.h:
##########
@@ -0,0 +1,73 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <stddef.h>
Review Comment:
warning: inclusion of deprecated C++ header 'stddef.h'; consider using 'cstddef' instead [modernize-deprecated-headers]
```suggestion
#include <cstddef>
```
##########
be/src/cloud/cloud_cumulative_compaction_policy.h:
##########
@@ -0,0 +1,73 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <stddef.h>
+#include <stdint.h>
+
+#include <memory>
+#include <string>
+#include <vector>
+
+#include "common/config.h"
+#include "cloud/cloud_tablet.h"
+#include "olap/rowset/rowset.h"
+#include "olap/rowset/rowset_meta.h"
+
+namespace doris {
+
+class Tablet;
+struct Version;
+
+class CloudSizeBasedCumulativeCompactionPolicy {
+public:
+ CloudSizeBasedCumulativeCompactionPolicy(
+ int64_t promotion_size = config::compaction_promotion_size_mbytes * 1024 * 1024,
+ double promotion_ratio = config::compaction_promotion_ratio,
+ int64_t promotion_min_size = config::compaction_promotion_min_size_mbytes * 1024 * 1024,
+ int64_t compaction_min_size = config::compaction_min_size_mbytes * 1024 * 1024);
+
+ ~CloudSizeBasedCumulativeCompactionPolicy() {}
Review Comment:
warning: use '= default' to define a trivial destructor [modernize-use-equals-default]
```suggestion
~CloudSizeBasedCumulativeCompactionPolicy() = default;
```
##########
be/src/cloud/cloud_storage_engine.cpp:
##########
@@ -190,4 +234,418 @@ void CloudStorageEngine::_sync_tablets_thread_callback() {
}
}
+void CloudStorageEngine::get_cumu_compaction(
Review Comment:
warning: method 'get_cumu_compaction' can be made static [readability-convert-member-functions-to-static]
```suggestion
static void CloudStorageEngine::get_cumu_compaction(
```
##########
be/src/cloud/cloud_storage_engine.cpp:
##########
@@ -190,4 +234,418 @@
}
}
+void CloudStorageEngine::get_cumu_compaction(
+ int64_t tablet_id, std::vector<std::shared_ptr<CloudCumulativeCompaction>>& res) {
+ std::lock_guard lock(_compaction_mtx);
+ if (auto it = _submitted_cumu_compactions.find(tablet_id);
+ it != _submitted_cumu_compactions.end()) {
+ res = it->second;
+ }
+}
+
+void CloudStorageEngine::_adjust_compaction_thread_num() {
+ int base_thread_num = get_base_thread_num();
+ if (_base_compaction_thread_pool->max_threads() != base_thread_num) {
+ int old_max_threads = _base_compaction_thread_pool->max_threads();
+ Status status = _base_compaction_thread_pool->set_max_threads(base_thread_num);
+ if (status.ok()) {
+ VLOG_NOTICE << "update base compaction thread pool max_threads from " << old_max_threads
+ << " to " << base_thread_num;
+ }
+ }
+ if (_base_compaction_thread_pool->min_threads() != base_thread_num) {
+ int old_min_threads = _base_compaction_thread_pool->min_threads();
+ Status status = _base_compaction_thread_pool->set_min_threads(base_thread_num);
+ if (status.ok()) {
+ VLOG_NOTICE << "update base compaction thread pool min_threads from " << old_min_threads
+ << " to " << base_thread_num;
+ }
+ }
+
+ int cumu_thread_num = get_cumu_thread_num();
+ if (_cumu_compaction_thread_pool->max_threads() != cumu_thread_num) {
+ int old_max_threads = _cumu_compaction_thread_pool->max_threads();
+ Status status = _cumu_compaction_thread_pool->set_max_threads(cumu_thread_num);
+ if (status.ok()) {
+ VLOG_NOTICE << "update cumu compaction thread pool max_threads from " << old_max_threads
+ << " to " << cumu_thread_num;
+ }
+ }
+ if (_cumu_compaction_thread_pool->min_threads() != cumu_thread_num) {
+ int old_min_threads = _cumu_compaction_thread_pool->min_threads();
+ Status status = _cumu_compaction_thread_pool->set_min_threads(cumu_thread_num);
+ if (status.ok()) {
+ VLOG_NOTICE << "update cumu compaction thread pool min_threads from " << old_min_threads
+ << " to " << cumu_thread_num;
+ }
+ }
+}
+
+void CloudStorageEngine::_compaction_tasks_producer_callback() {
+ LOG(INFO) << "try to start compaction producer process!";
+
+ int round = 0;
+ CompactionType compaction_type;
+
+ // Used to record the time when the score metric was last updated.
+ // The update of the score metric is accompanied by the logic of selecting the tablet.
+ // If there is no slot available, the logic of selecting the tablet will be terminated,
+ // which causes the score metric update to be terminated.
+ // In order to avoid this situation, we need to update the score regularly.
+ int64_t last_cumulative_score_update_time = 0;
+ int64_t last_base_score_update_time = 0;
+ static const int64_t check_score_interval_ms = 5000; // 5 secs
+
+ int64_t interval = config::generate_compaction_tasks_interval_ms;
+ do {
+ if (!config::disable_auto_compaction) {
+ _adjust_compaction_thread_num();
+
+ bool check_score = false;
+ int64_t cur_time = UnixMillis();
+ if (round < config::cumulative_compaction_rounds_for_each_base_compaction_round) {
+ compaction_type = CompactionType::CUMULATIVE_COMPACTION;
+ round++;
+ if (cur_time - last_cumulative_score_update_time >= check_score_interval_ms) {
+ check_score = true;
+ last_cumulative_score_update_time = cur_time;
+ }
+ } else {
+ compaction_type = CompactionType::BASE_COMPACTION;
+ round = 0;
+ if (cur_time - last_base_score_update_time >= check_score_interval_ms) {
+ check_score = true;
+ last_base_score_update_time = cur_time;
+ }
+ }
+ std::unique_ptr<ThreadPool>& thread_pool =
+ (compaction_type == CompactionType::CUMULATIVE_COMPACTION)
+ ? _cumu_compaction_thread_pool
+ : _base_compaction_thread_pool;
+ VLOG_CRITICAL << "compaction thread pool. type: "
+ << (compaction_type == CompactionType::CUMULATIVE_COMPACTION ? "CUMU"
+ : "BASE")
+ << ", num_threads: " << thread_pool->num_threads()
+ << ", num_threads_pending_start: "
+ << thread_pool->num_threads_pending_start()
+ << ", num_active_threads: " << thread_pool->num_active_threads()
+ << ", max_threads: " << thread_pool->max_threads()
+ << ", min_threads: " << thread_pool->min_threads()
+ << ", num_total_queued_tasks: " << thread_pool->get_queue_size();
+ std::vector<CloudTabletSPtr> tablets_compaction =
+ _generate_cloud_compaction_tasks(compaction_type, check_score);
+
+ /// Regardless of whether the tablet is submitted for compaction or not,
+ /// we need to call 'reset_compaction' to clean up the base_compaction or cumulative_compaction objects
+ /// in the tablet, because these two objects store the tablet's own shared_ptr.
+ /// If it is not cleaned up, the reference count of the tablet will always be greater than 1,
+ /// thus cannot be collected by the garbage collector. (TabletManager::start_trash_sweep)
+ for (const auto& tablet : tablets_compaction) {
+ Status st = submit_compaction_task(tablet, compaction_type);
+ if (st.ok()) continue;
+ if ((!st.is<ErrorCode::BE_NO_SUITABLE_VERSION>() &&
+ !st.is<ErrorCode::CUMULATIVE_NO_SUITABLE_VERSION>()) ||
+ VLOG_DEBUG_IS_ON) {
+ LOG(WARNING) << "failed to submit compaction task for tablet: "
+ << tablet->tablet_id() << ", err: " << st;
+ }
+ }
+ interval = config::generate_compaction_tasks_interval_ms;
+ } else {
+ interval = config::check_auto_compaction_interval_seconds * 1000;
+ }
+ } while (!_stop_background_threads_latch.wait_for(std::chrono::milliseconds(interval)));
+}
+
+std::vector<CloudTabletSPtr> CloudStorageEngine::_generate_cloud_compaction_tasks(
+ CompactionType compaction_type, bool check_score) {
+ std::vector<std::shared_ptr<CloudTablet>> tablets_compaction;
+
+ int64_t max_compaction_score = 0;
+ std::unordered_set<int64_t> tablet_preparing_cumu_compaction;
+ std::unordered_map<int64_t, std::vector<std::shared_ptr<CloudCumulativeCompaction>>>
+ submitted_cumu_compactions;
+ std::unordered_map<int64_t, std::shared_ptr<CloudBaseCompaction>> submitted_base_compactions;
+ std::unordered_map<int64_t, std::shared_ptr<CloudFullCompaction>> submitted_full_compactions;
+ {
+ std::lock_guard lock(_compaction_mtx);
+ tablet_preparing_cumu_compaction = _tablet_preparing_cumu_compaction;
+ submitted_cumu_compactions = _submitted_cumu_compactions;
+ submitted_base_compactions = _submitted_base_compactions;
+ submitted_full_compactions = _submitted_full_compactions;
+ }
+
+ bool need_pick_tablet = true;
+ int thread_per_disk =
+ config::compaction_task_num_per_fast_disk; // all disks are fast in cloud mode
+ int num_cumu =
+ std::accumulate(submitted_cumu_compactions.begin(), submitted_cumu_compactions.end(), 0,
+ [](int a, auto& b) { return a + b.second.size(); });
+ int num_base = submitted_base_compactions.size() + submitted_full_compactions.size();
+ int n = thread_per_disk - num_cumu - num_base;
+ if (compaction_type == CompactionType::BASE_COMPACTION) {
+ // We need to reserve at least one thread for cumulative compaction,
+ // because base compactions may take too long to complete, which may
+ // leads to "too many rowsets" error.
+ int base_n = std::min(config::max_base_compaction_task_num_per_disk, thread_per_disk - 1) -
+ num_base;
+ n = std::min(base_n, n);
+ }
+ if (n <= 0) { // No threads available
+ if (!check_score) return tablets_compaction;
+ need_pick_tablet = false;
+ n = 0;
+ }
+
+ // Return true for skipping compaction
+ std::function<bool(CloudTablet*)> filter_out;
+ if (compaction_type == CompactionType::BASE_COMPACTION) {
+ filter_out = [&submitted_base_compactions, &submitted_full_compactions](CloudTablet* t) {
+ return !!submitted_base_compactions.count(t->tablet_id()) ||
+ !!submitted_full_compactions.count(t->tablet_id()) ||
+ t->tablet_state() != TABLET_RUNNING;
+ };
+ } else if (config::enable_parallel_cumu_compaction) {
+ filter_out = [&tablet_preparing_cumu_compaction](CloudTablet* t) {
+ return !!tablet_preparing_cumu_compaction.count(t->tablet_id()) ||
+ t->tablet_state() != TABLET_RUNNING;
+ };
+ } else {
+ filter_out = [&tablet_preparing_cumu_compaction, &submitted_cumu_compactions](CloudTablet* t) {
+ return !!tablet_preparing_cumu_compaction.count(t->tablet_id()) ||
+ !!submitted_cumu_compactions.count(t->tablet_id()) ||
+ t->tablet_state() != TABLET_RUNNING;
+ };
+ }
+
+ // Even if need_pick_tablet is false, we still need to call find_best_tablet_to_compaction(),
+ // So that we can update the max_compaction_score metric.
+ do {
+ std::vector<CloudTabletSPtr> tablets;
+ auto st = tablet_mgr().get_topn_tablets_to_compact(n, compaction_type, filter_out,
+ &tablets, &max_compaction_score);
+ if (!st.ok()) {
+ LOG(WARNING) << "failed to get tablets to compact, err=" << st;
+ break;
+ }
+ if (!need_pick_tablet) break;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (!need_pick_tablet) { break;
}
```
##########
be/src/cloud/cloud_cumulative_compaction_policy.h:
##########
@@ -0,0 +1,73 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <stddef.h>
+#include <stdint.h>
+
+#include <memory>
+#include <string>
+#include <vector>
+
+#include "common/config.h"
+#include "cloud/cloud_tablet.h"
+#include "olap/rowset/rowset.h"
+#include "olap/rowset/rowset_meta.h"
+
+namespace doris {
+
+class Tablet;
+struct Version;
+
+class CloudSizeBasedCumulativeCompactionPolicy {
+public:
+ CloudSizeBasedCumulativeCompactionPolicy(
+ int64_t promotion_size = config::compaction_promotion_size_mbytes * 1024 * 1024,
+ double promotion_ratio = config::compaction_promotion_ratio,
+ int64_t promotion_min_size = config::compaction_promotion_min_size_mbytes * 1024 * 1024,
+ int64_t compaction_min_size = config::compaction_min_size_mbytes * 1024 * 1024);
+
+ ~CloudSizeBasedCumulativeCompactionPolicy() {}
+
+ int64_t new_cumulative_point(CloudTablet* tablet, const RowsetSharedPtr& output_rowset,
+ Version& last_delete_version,
+ int64_t last_cumulative_point);
+
+ int pick_input_rowsets(CloudTablet* tablet, const std::vector<RowsetSharedPtr>& candidate_rowsets,
+ const int64_t max_compaction_score, const int64_t min_compaction_score,
+ std::vector<RowsetSharedPtr>* input_rowsets,
+ Version* last_delete_version, size_t* compaction_score,
+ bool allow_delete = false);
+
+private:
+ int64_t _level_size(const int64_t size);
Review Comment:
warning: parameter 'size' is const-qualified in the function declaration; const-qualification of parameters only has an effect in function definitions [readability-avoid-const-params-in-decls]
```suggestion
int64_t _level_size(int64_t size);
```
##########
be/src/cloud/cloud_storage_engine.cpp:
##########
@@ -190,4 +234,418 @@
}
}
+void CloudStorageEngine::get_cumu_compaction(
+ int64_t tablet_id, std::vector<std::shared_ptr<CloudCumulativeCompaction>>& res) {
+ std::lock_guard lock(_compaction_mtx);
+ if (auto it = _submitted_cumu_compactions.find(tablet_id);
+ it != _submitted_cumu_compactions.end()) {
+ res = it->second;
+ }
+}
+
+void CloudStorageEngine::_adjust_compaction_thread_num() {
+ int base_thread_num = get_base_thread_num();
+ if (_base_compaction_thread_pool->max_threads() != base_thread_num) {
+ int old_max_threads = _base_compaction_thread_pool->max_threads();
+ Status status = _base_compaction_thread_pool->set_max_threads(base_thread_num);
+ if (status.ok()) {
+ VLOG_NOTICE << "update base compaction thread pool max_threads from " << old_max_threads
+ << " to " << base_thread_num;
+ }
+ }
+ if (_base_compaction_thread_pool->min_threads() != base_thread_num) {
+ int old_min_threads = _base_compaction_thread_pool->min_threads();
+ Status status = _base_compaction_thread_pool->set_min_threads(base_thread_num);
+ if (status.ok()) {
+ VLOG_NOTICE << "update base compaction thread pool min_threads from " << old_min_threads
+ << " to " << base_thread_num;
+ }
+ }
+
+ int cumu_thread_num = get_cumu_thread_num();
+ if (_cumu_compaction_thread_pool->max_threads() != cumu_thread_num) {
+ int old_max_threads = _cumu_compaction_thread_pool->max_threads();
+ Status status = _cumu_compaction_thread_pool->set_max_threads(cumu_thread_num);
+ if (status.ok()) {
+ VLOG_NOTICE << "update cumu compaction thread pool max_threads from " << old_max_threads
+ << " to " << cumu_thread_num;
+ }
+ }
+ if (_cumu_compaction_thread_pool->min_threads() != cumu_thread_num) {
+ int old_min_threads = _cumu_compaction_thread_pool->min_threads();
+ Status status = _cumu_compaction_thread_pool->set_min_threads(cumu_thread_num);
+ if (status.ok()) {
+ VLOG_NOTICE << "update cumu compaction thread pool min_threads from " << old_min_threads
+ << " to " << cumu_thread_num;
+ }
+ }
+}
+
+void CloudStorageEngine::_compaction_tasks_producer_callback() {
+ LOG(INFO) << "try to start compaction producer process!";
+
+ int round = 0;
+ CompactionType compaction_type;
+
+ // Used to record the time when the score metric was last updated.
+ // The update of the score metric is accompanied by the logic of selecting the tablet.
+ // If there is no slot available, the logic of selecting the tablet will be terminated,
+ // which causes the score metric update to be terminated.
+ // In order to avoid this situation, we need to update the score regularly.
+ int64_t last_cumulative_score_update_time = 0;
+ int64_t last_base_score_update_time = 0;
+ static const int64_t check_score_interval_ms = 5000; // 5 secs
+
+ int64_t interval = config::generate_compaction_tasks_interval_ms;
+ do {
+ if (!config::disable_auto_compaction) {
+ _adjust_compaction_thread_num();
+
+ bool check_score = false;
+ int64_t cur_time = UnixMillis();
+ if (round < config::cumulative_compaction_rounds_for_each_base_compaction_round) {
+ compaction_type = CompactionType::CUMULATIVE_COMPACTION;
+ round++;
+ if (cur_time - last_cumulative_score_update_time >= check_score_interval_ms) {
+ check_score = true;
+ last_cumulative_score_update_time = cur_time;
+ }
+ } else {
+ compaction_type = CompactionType::BASE_COMPACTION;
+ round = 0;
+ if (cur_time - last_base_score_update_time >= check_score_interval_ms) {
+ check_score = true;
+ last_base_score_update_time = cur_time;
+ }
+ }
+ std::unique_ptr<ThreadPool>& thread_pool =
+ (compaction_type == CompactionType::CUMULATIVE_COMPACTION)
+ ? _cumu_compaction_thread_pool
+ : _base_compaction_thread_pool;
+ VLOG_CRITICAL << "compaction thread pool. type: "
+ << (compaction_type == CompactionType::CUMULATIVE_COMPACTION ? "CUMU"
+ : "BASE")
+ << ", num_threads: " << thread_pool->num_threads()
+ << ", num_threads_pending_start: "
+ << thread_pool->num_threads_pending_start()
+ << ", num_active_threads: " << thread_pool->num_active_threads()
+ << ", max_threads: " << thread_pool->max_threads()
+ << ", min_threads: " << thread_pool->min_threads()
+ << ", num_total_queued_tasks: " << thread_pool->get_queue_size();
+ std::vector<CloudTabletSPtr> tablets_compaction =
+ _generate_cloud_compaction_tasks(compaction_type, check_score);
+
+ /// Regardless of whether the tablet is submitted for compaction or not,
+ /// we need to call 'reset_compaction' to clean up the base_compaction or cumulative_compaction objects
+ /// in the tablet, because these two objects store the tablet's own shared_ptr.
+ /// If it is not cleaned up, the reference count of the tablet will always be greater than 1,
+ /// thus cannot be collected by the garbage collector. (TabletManager::start_trash_sweep)
+ for (const auto& tablet : tablets_compaction) {
+ Status st = submit_compaction_task(tablet, compaction_type);
+ if (st.ok()) continue;
+ if ((!st.is<ErrorCode::BE_NO_SUITABLE_VERSION>() &&
+ !st.is<ErrorCode::CUMULATIVE_NO_SUITABLE_VERSION>()) ||
+ VLOG_DEBUG_IS_ON) {
+ LOG(WARNING) << "failed to submit compaction task for tablet: "
+ << tablet->tablet_id() << ", err: " << st;
+ }
+ }
+ interval = config::generate_compaction_tasks_interval_ms;
+ } else {
+ interval = config::check_auto_compaction_interval_seconds * 1000;
+ }
+ } while (!_stop_background_threads_latch.wait_for(std::chrono::milliseconds(interval)));
+}
+
+std::vector<CloudTabletSPtr> CloudStorageEngine::_generate_cloud_compaction_tasks(
Review Comment:
warning: function '_generate_cloud_compaction_tasks' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
std::vector<CloudTabletSPtr> CloudStorageEngine::_generate_cloud_compaction_tasks(
^
```
<details>
<summary>Additional context</summary>
**be/src/cloud/cloud_storage_engine.cpp:359:** 85 lines including whitespace and comments (threshold 80)
```cpp
std::vector<CloudTabletSPtr> CloudStorageEngine::_generate_cloud_compaction_tasks(
^
```
</details>
##########
be/src/olap/compaction.cpp:
##########
@@ -844,4 +847,80 @@ void Compaction::_load_segment_to_cache() {
}
}
+void CloudCompactionMixin::build_basic_info() {
Review Comment:
warning: method 'build_basic_info' can be made static [readability-convert-member-functions-to-static]
be/src/olap/compaction.h:184:
```diff
- void build_basic_info();
+ static void build_basic_info();
```
##########
be/src/olap/compaction.cpp:
##########
@@ -844,4 +847,80 @@
}
}
+void CloudCompactionMixin::build_basic_info() {
+ _output_version =
+ Version(_input_rowsets.front()->start_version(), _input_rowsets.back()->end_version());
+
+ _newest_write_timestamp = _input_rowsets.back()->newest_write_timestamp();
+
+ std::vector<RowsetMetaSharedPtr> rowset_metas(_input_rowsets.size());
+ std::transform(_input_rowsets.begin(), _input_rowsets.end(), rowset_metas.begin(),
+ [](const RowsetSharedPtr& rowset) { return rowset->rowset_meta(); });
+ _cur_tablet_schema = _tablet->tablet_schema_with_merged_max_schema_version(rowset_metas);
+}
+
+int64_t CloudCompactionMixin::get_compaction_permits() {
Review Comment:
warning: method 'get_compaction_permits' can be made static [readability-convert-member-functions-to-static]
be/src/olap/compaction.h:188:
```diff
- int64_t get_compaction_permits();
+ static int64_t get_compaction_permits();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Implement cloud compaction [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31215:
URL: https://github.com/apache/doris/pull/31215#issuecomment-1956347644
TeamCity be ut coverage result:
Function Coverage: 35.58% (8556/24048)
Line Coverage: 27.43% (69439/253125)
Region Coverage: 26.59% (36027/135473)
Branch Coverage: 23.43% (18425/78648)
Coverage Report: http://coverage.selectdb-in.cc/coverage/dab4c3c2e9e6d569ef2db46369a2bbe26e41ae0b_dab4c3c2e9e6d569ef2db46369a2bbe26e41ae0b/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Implement cloud compaction [doris]
Posted by "luwei16 (via GitHub)" <gi...@apache.org>.
luwei16 commented on PR #31215:
URL: https://github.com/apache/doris/pull/31215#issuecomment-1970279244
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Implement cloud compaction [doris]
Posted by "luwei16 (via GitHub)" <gi...@apache.org>.
luwei16 commented on PR #31215:
URL: https://github.com/apache/doris/pull/31215#issuecomment-1966641358
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Implement cloud compaction [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #31215:
URL: https://github.com/apache/doris/pull/31215#discussion_r1507434763
##########
be/src/cloud/cloud_storage_engine.cpp:
##########
@@ -196,4 +246,454 @@ void CloudStorageEngine::_sync_tablets_thread_callback() {
}
}
+void CloudStorageEngine::get_cumu_compaction(
+ int64_t tablet_id, std::vector<std::shared_ptr<CloudCumulativeCompaction>>& res) {
+ std::lock_guard lock(_compaction_mtx);
+ if (auto it = _submitted_cumu_compactions.find(tablet_id);
+ it != _submitted_cumu_compactions.end()) {
+ res = it->second;
+ }
+}
+
+void CloudStorageEngine::_adjust_compaction_thread_num() {
+ int base_thread_num = get_base_thread_num();
+ if (_base_compaction_thread_pool->max_threads() != base_thread_num) {
+ int old_max_threads = _base_compaction_thread_pool->max_threads();
+ Status status = _base_compaction_thread_pool->set_max_threads(base_thread_num);
+ if (status.ok()) {
+ VLOG_NOTICE << "update base compaction thread pool max_threads from " << old_max_threads
+ << " to " << base_thread_num;
+ }
+ }
+ if (_base_compaction_thread_pool->min_threads() != base_thread_num) {
+ int old_min_threads = _base_compaction_thread_pool->min_threads();
+ Status status = _base_compaction_thread_pool->set_min_threads(base_thread_num);
+ if (status.ok()) {
+ VLOG_NOTICE << "update base compaction thread pool min_threads from " << old_min_threads
+ << " to " << base_thread_num;
+ }
+ }
+
+ int cumu_thread_num = get_cumu_thread_num();
+ if (_cumu_compaction_thread_pool->max_threads() != cumu_thread_num) {
+ int old_max_threads = _cumu_compaction_thread_pool->max_threads();
+ Status status = _cumu_compaction_thread_pool->set_max_threads(cumu_thread_num);
+ if (status.ok()) {
+ VLOG_NOTICE << "update cumu compaction thread pool max_threads from " << old_max_threads
+ << " to " << cumu_thread_num;
+ }
+ }
+ if (_cumu_compaction_thread_pool->min_threads() != cumu_thread_num) {
+ int old_min_threads = _cumu_compaction_thread_pool->min_threads();
+ Status status = _cumu_compaction_thread_pool->set_min_threads(cumu_thread_num);
+ if (status.ok()) {
+ VLOG_NOTICE << "update cumu compaction thread pool min_threads from " << old_min_threads
+ << " to " << cumu_thread_num;
+ }
+ }
+}
+
+void CloudStorageEngine::_compaction_tasks_producer_callback() {
+ LOG(INFO) << "try to start compaction producer process!";
+
+ int round = 0;
+ CompactionType compaction_type;
+
+ // Used to record the time when the score metric was last updated.
+ // The update of the score metric is accompanied by the logic of selecting the tablet.
+ // If there is no slot available, the logic of selecting the tablet will be terminated,
+ // which causes the score metric update to be terminated.
+ // In order to avoid this situation, we need to update the score regularly.
+ int64_t last_cumulative_score_update_time = 0;
+ int64_t last_base_score_update_time = 0;
+ static const int64_t check_score_interval_ms = 5000; // 5 secs
+
+ int64_t interval = config::generate_compaction_tasks_interval_ms;
+ do {
+ if (!config::disable_auto_compaction) {
+ _adjust_compaction_thread_num();
+
+ bool check_score = false;
+ int64_t cur_time = UnixMillis();
+ if (round < config::cumulative_compaction_rounds_for_each_base_compaction_round) {
+ compaction_type = CompactionType::CUMULATIVE_COMPACTION;
+ round++;
+ if (cur_time - last_cumulative_score_update_time >= check_score_interval_ms) {
+ check_score = true;
+ last_cumulative_score_update_time = cur_time;
+ }
+ } else {
+ compaction_type = CompactionType::BASE_COMPACTION;
+ round = 0;
+ if (cur_time - last_base_score_update_time >= check_score_interval_ms) {
+ check_score = true;
+ last_base_score_update_time = cur_time;
+ }
+ }
+ std::unique_ptr<ThreadPool>& thread_pool =
+ (compaction_type == CompactionType::CUMULATIVE_COMPACTION)
+ ? _cumu_compaction_thread_pool
+ : _base_compaction_thread_pool;
+ VLOG_CRITICAL << "compaction thread pool. type: "
+ << (compaction_type == CompactionType::CUMULATIVE_COMPACTION ? "CUMU"
+ : "BASE")
+ << ", num_threads: " << thread_pool->num_threads()
+ << ", num_threads_pending_start: "
+ << thread_pool->num_threads_pending_start()
+ << ", num_active_threads: " << thread_pool->num_active_threads()
+ << ", max_threads: " << thread_pool->max_threads()
+ << ", min_threads: " << thread_pool->min_threads()
+ << ", num_total_queued_tasks: " << thread_pool->get_queue_size();
+ std::vector<CloudTabletSPtr> tablets_compaction =
+ _generate_cloud_compaction_tasks(compaction_type, check_score);
+
+ /// Regardless of whether the tablet is submitted for compaction or not,
+ /// we need to call 'reset_compaction' to clean up the base_compaction or cumulative_compaction objects
+ /// in the tablet, because these two objects store the tablet's own shared_ptr.
+ /// If it is not cleaned up, the reference count of the tablet will always be greater than 1,
+ /// thus cannot be collected by the garbage collector. (TabletManager::start_trash_sweep)
+ for (const auto& tablet : tablets_compaction) {
+ Status st = submit_compaction_task(tablet, compaction_type);
+ if (st.ok()) continue;
+ if ((!st.is<ErrorCode::BE_NO_SUITABLE_VERSION>() &&
+ !st.is<ErrorCode::CUMULATIVE_NO_SUITABLE_VERSION>()) ||
+ VLOG_DEBUG_IS_ON) {
+ LOG(WARNING) << "failed to submit compaction task for tablet: "
+ << tablet->tablet_id() << ", err: " << st;
+ }
+ }
+ interval = config::generate_compaction_tasks_interval_ms;
+ } else {
+ interval = config::check_auto_compaction_interval_seconds * 1000;
+ }
+ } while (!_stop_background_threads_latch.wait_for(std::chrono::milliseconds(interval)));
+}
+
+std::vector<CloudTabletSPtr> CloudStorageEngine::_generate_cloud_compaction_tasks(
Review Comment:
warning: function '_generate_cloud_compaction_tasks' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
std::vector<CloudTabletSPtr> CloudStorageEngine::_generate_cloud_compaction_tasks(
^
```
<details>
<summary>Additional context</summary>
**be/src/cloud/cloud_storage_engine.cpp:371:** 86 lines including whitespace and comments (threshold 80)
```cpp
std::vector<CloudTabletSPtr> CloudStorageEngine::_generate_cloud_compaction_tasks(
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Implement cloud compaction [doris]
Posted by "luwei16 (via GitHub)" <gi...@apache.org>.
luwei16 commented on PR #31215:
URL: https://github.com/apache/doris/pull/31215#issuecomment-1970953419
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Implement cloud compaction [doris]
Posted by "luwei16 (via GitHub)" <gi...@apache.org>.
luwei16 commented on PR #31215:
URL: https://github.com/apache/doris/pull/31215#issuecomment-1970278696
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Implement cloud compaction [doris]
Posted by "luwei16 (via GitHub)" <gi...@apache.org>.
luwei16 commented on PR #31215:
URL: https://github.com/apache/doris/pull/31215#issuecomment-1969270723
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Implement cloud compaction [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #31215:
URL: https://github.com/apache/doris/pull/31215#discussion_r1504316798
##########
be/src/cloud/cloud_compaction_action.cpp:
##########
@@ -0,0 +1,338 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_compaction_action.h"
+
+// IWYU pragma: no_include <bits/chrono.h>
+#include <chrono> // IWYU pragma: keep
+#include <exception>
+#include <future>
+#include <memory>
+#include <mutex>
+#include <sstream>
+#include <string>
+#include <thread>
+#include <utility>
+
+#include "cloud/cloud_base_compaction.h"
+#include "cloud/cloud_compaction_action.h"
Review Comment:
warning: duplicate include [readability-duplicate-include]
be/src/cloud/cloud_compaction_action.cpp:30:
```diff
- #include "cloud/cloud_base_compaction.h"
- #include "cloud/cloud_compaction_action.h"
+ #include "cloud/cloud_base_compaction.h"
```
##########
be/src/cloud/cloud_vertical_rowset_writer.cpp:
##########
@@ -0,0 +1,71 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_vertical_rowset_writer.h"
+
+#include <fmt/format.h>
+#include <gen_cpp/olap_file.pb.h>
+
+#include <algorithm>
+#include <atomic>
+#include <memory>
+#include <mutex>
+#include <ostream>
+#include <string>
+#include <utility>
+
+#include "common/compiler_util.h" // IWYU pragma: keep
+#include "common/logging.h"
+#include "io/fs/file_system.h"
+#include "io/fs/file_writer.h"
+#include "olap/rowset/beta_rowset.h"
+#include "olap/rowset/rowset_meta.h"
+#include "olap/rowset/rowset_writer_context.h"
+#include "olap/rowset/vertical_beta_rowset_writer_helper.h"
+#include "util/slice.h"
+#include "util/spinlock.h"
+#include "vec/core/block.h"
+
+namespace doris {
+using namespace ErrorCode;
+
+CloudVerticalRowsetWriter::CloudVerticalRowsetWriter() : CloudRowsetWriter() {
Review Comment:
warning: initializer for base class 'doris::CloudRowsetWriter' is redundant [readability-redundant-member-init]
```suggestion
CloudVerticalRowsetWriter::CloudVerticalRowsetWriter() : {
```
##########
be/src/cloud/cloud_vertical_rowset_writer.cpp:
##########
@@ -0,0 +1,71 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_vertical_rowset_writer.h"
+
+#include <fmt/format.h>
+#include <gen_cpp/olap_file.pb.h>
+
+#include <algorithm>
+#include <atomic>
+#include <memory>
+#include <mutex>
+#include <ostream>
+#include <string>
+#include <utility>
+
+#include "common/compiler_util.h" // IWYU pragma: keep
+#include "common/logging.h"
+#include "io/fs/file_system.h"
+#include "io/fs/file_writer.h"
+#include "olap/rowset/beta_rowset.h"
+#include "olap/rowset/rowset_meta.h"
+#include "olap/rowset/rowset_writer_context.h"
+#include "olap/rowset/vertical_beta_rowset_writer_helper.h"
+#include "util/slice.h"
+#include "util/spinlock.h"
+#include "vec/core/block.h"
+
+namespace doris {
+using namespace ErrorCode;
+
+CloudVerticalRowsetWriter::CloudVerticalRowsetWriter() : CloudRowsetWriter() {
+ _helper = std::make_shared<VerticalBetaRowsetWriterHelper>(
+ &_segment_writers, _already_built, _rowset_meta, &_num_segment, _context,
+ &_num_rows_written, &_segments_encoded_key_bounds, &_segment_num_rows,
+ &_total_index_size, &_file_writers, &_total_data_size, &_lock);
+}
+
+CloudVerticalRowsetWriter::~CloudVerticalRowsetWriter() {
Review Comment:
warning: use '= default' to define a trivial destructor [modernize-use-equals-default]
```cpp
CloudVerticalRowsetWriter::~CloudVerticalRowsetWriter() {
^
```
##########
be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:
##########
@@ -0,0 +1,244 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/rowset/vertical_beta_rowset_writer_helper.h"
+
+#include <fmt/format.h>
+#include <gen_cpp/olap_file.pb.h>
+
+#include <algorithm>
+#include <atomic>
+#include <mutex>
+#include <ostream>
+#include <string>
+#include <utility>
+
+#include "common/compiler_util.h" // IWYU pragma: keep
+#include "common/logging.h"
+#include "io/fs/file_system.h"
+#include "io/fs/file_writer.h"
+#include "olap/rowset/beta_rowset.h"
+#include "olap/rowset/rowset_meta.h"
+#include "olap/rowset/rowset_writer_context.h"
+#include "util/slice.h"
+#include "util/spinlock.h"
+#include "vec/core/block.h"
+
+namespace doris {
+using namespace ErrorCode;
+
+VerticalBetaRowsetWriterHelper::VerticalBetaRowsetWriterHelper(
+ std::vector<std::unique_ptr<segment_v2::SegmentWriter>>* segment_writers,
+ bool already_built, RowsetMetaSharedPtr& rowset_meta, std::atomic<int32_t>* num_segment,
+ RowsetWriterContext& context, std::atomic<int64_t>* num_rows_written,
+ std::vector<KeyBoundsPB>* segments_encoded_key_bounds,
+ std::vector<uint32_t>* segment_num_rows, std::atomic<int64_t>* total_index_size,
+ std::vector<io::FileWriterPtr>* file_writers, std::atomic<int64_t>* total_data_size,
+ SpinLock* lock)
+ : _segment_writers(segment_writers),
+ _already_built(already_built),
+ _rowset_meta(rowset_meta),
+ _num_segment(num_segment),
+ _context(context),
+ _num_rows_written(num_rows_written),
+ _segments_encoded_key_bounds(segments_encoded_key_bounds),
+ _segment_num_rows(segment_num_rows),
+ _total_index_size(total_index_size),
+ _file_writers(file_writers),
+ _total_data_size(total_data_size),
+ _lock(lock) {}
+
+void VerticalBetaRowsetWriterHelper::destruct_writer() {
+ if (!_already_built) {
+ const auto& fs = _rowset_meta->fs();
+ if (!fs || !_rowset_meta->is_local()) { // Remote fs will delete them asynchronously
+ return;
+ }
+ for (auto& segment_writer : *_segment_writers) {
+ segment_writer.reset();
+ }
+ for (int i = 0; i < *_num_segment; ++i) {
+ auto path = BetaRowset::segment_file_path(_context.rowset_dir, _context.rowset_id, i);
+ // Even if an error is encountered, these files that have not been cleaned up
+ // will be cleaned up by the GC background. So here we only print the error
+ // message when we encounter an error.
+ WARN_IF_ERROR(fs->delete_file(path), fmt::format("Failed to delete file={}", path));
+ }
+ }
+}
+
+Status VerticalBetaRowsetWriterHelper::add_columns(const vectorized::Block* block,
+ const std::vector<uint32_t>& col_ids,
+ bool is_key, uint32_t max_rows_per_segment) {
+ VLOG_NOTICE << "VerticalBetaRowsetWriter::add_columns, columns: " << block->columns();
+ size_t num_rows = block->rows();
+ if (num_rows == 0) {
+ return Status::OK();
+ }
+ if (UNLIKELY(max_rows_per_segment > _context.max_rows_per_segment)) {
+ max_rows_per_segment = _context.max_rows_per_segment;
+ }
+
+ if (_segment_writers->empty()) {
+ // it must be key columns
+ DCHECK(is_key);
+ std::unique_ptr<segment_v2::SegmentWriter> writer;
+ RETURN_IF_ERROR(_create_segment_writer(col_ids, is_key, &writer));
+ _segment_writers->emplace_back(std::move(writer));
+ _cur_writer_idx = 0;
+ RETURN_IF_ERROR((*_segment_writers)[_cur_writer_idx]->append_block(block, 0, num_rows));
+ } else if (is_key) {
+ if ((*_segment_writers)[_cur_writer_idx]->num_rows_written() > max_rows_per_segment) {
+ // segment is full, need flush columns and create new segment writer
+ RETURN_IF_ERROR(_flush_columns(&(*_segment_writers)[_cur_writer_idx], true));
+
+ std::unique_ptr<segment_v2::SegmentWriter> writer;
+ RETURN_IF_ERROR(_create_segment_writer(col_ids, is_key, &writer));
+ _segment_writers->emplace_back(std::move(writer));
+ ++_cur_writer_idx;
+ }
+ RETURN_IF_ERROR((*_segment_writers)[_cur_writer_idx]->append_block(block, 0, num_rows));
+ } else {
+ // value columns
+ uint32_t num_rows_written = (*_segment_writers)[_cur_writer_idx]->num_rows_written();
+ VLOG_NOTICE << "num_rows_written: " << num_rows_written
+ << ", _cur_writer_idx: " << _cur_writer_idx;
+ uint32_t num_rows_key_group = (*_segment_writers)[_cur_writer_idx]->row_count();
+ // init if it's first value column write in current segment
+ if (_cur_writer_idx == 0 && num_rows_written == 0) {
+ VLOG_NOTICE << "init first value column segment writer";
+ RETURN_IF_ERROR((*_segment_writers)[_cur_writer_idx]->init(col_ids, is_key));
+ }
+ // when splitting segment, need to make rows align between key columns and value columns
+ size_t start_offset = 0;
+ size_t limit = num_rows;
+ if (num_rows_written + num_rows >= num_rows_key_group &&
+ _cur_writer_idx < _segment_writers->size() - 1) {
+ RETURN_IF_ERROR((*_segment_writers)[_cur_writer_idx]->append_block(
+ block, 0, num_rows_key_group - num_rows_written));
+ RETURN_IF_ERROR(_flush_columns(&(*_segment_writers)[_cur_writer_idx]));
+ start_offset = num_rows_key_group - num_rows_written;
+ limit = num_rows - start_offset;
+ ++_cur_writer_idx;
+ // switch to next writer
+ RETURN_IF_ERROR((*_segment_writers)[_cur_writer_idx]->init(col_ids, is_key));
+ num_rows_written = 0;
+ num_rows_key_group = (*_segment_writers)[_cur_writer_idx]->row_count();
+ }
+ if (limit > 0) {
+ RETURN_IF_ERROR(
+ (*_segment_writers)[_cur_writer_idx]->append_block(block, start_offset, limit));
+ DCHECK((*_segment_writers)[_cur_writer_idx]->num_rows_written() <=
+ (*_segment_writers)[_cur_writer_idx]->row_count());
+ }
+ }
+ if (is_key) {
+ *_num_rows_written += num_rows;
+ }
+ return Status::OK();
+}
+
+Status VerticalBetaRowsetWriterHelper::_flush_columns(
+ std::unique_ptr<segment_v2::SegmentWriter>* segment_writer, bool is_key) {
+ uint64_t index_size = 0;
+ VLOG_NOTICE << "flush columns index: " << _cur_writer_idx;
+ RETURN_IF_ERROR((*segment_writer)->finalize_columns_data());
+ RETURN_IF_ERROR((*segment_writer)->finalize_columns_index(&index_size));
+ if (is_key) {
+ _total_key_group_rows += (*segment_writer)->row_count();
+ // record segment key bound
+ KeyBoundsPB key_bounds;
+ Slice min_key = (*segment_writer)->min_encoded_key();
+ Slice max_key = (*segment_writer)->max_encoded_key();
+ DCHECK_LE(min_key.compare(max_key), 0);
+ key_bounds.set_min_key(min_key.to_string());
+ key_bounds.set_max_key(max_key.to_string());
+ _segments_encoded_key_bounds->emplace_back(key_bounds);
+ _segment_num_rows->resize(_cur_writer_idx + 1);
+ (*_segment_num_rows)[_cur_writer_idx] = (*_segment_writers)[_cur_writer_idx]->row_count();
+ }
+ *_total_index_size +=
+ static_cast<int64_t>(index_size) + (*segment_writer)->get_inverted_index_file_size();
+ return Status::OK();
+}
+
+Status VerticalBetaRowsetWriterHelper::flush_columns(bool is_key) {
+ if (_segment_writers->empty()) {
+ return Status::OK();
+ }
+
+ DCHECK(_cur_writer_idx < _segment_writers->size() && (*_segment_writers)[_cur_writer_idx]);
+ RETURN_IF_ERROR(_flush_columns(&(*_segment_writers)[_cur_writer_idx], is_key));
+ _cur_writer_idx = 0;
+ return Status::OK();
+}
+
+Status VerticalBetaRowsetWriterHelper::_create_segment_writer(
+ const std::vector<uint32_t>& column_ids, bool is_key,
+ std::unique_ptr<segment_v2::SegmentWriter>* writer) {
+ auto path = BetaRowset::segment_file_path(_context.rowset_dir, _context.rowset_id,
+ (*_num_segment)++);
+ auto fs = _rowset_meta->fs();
+ if (!fs) {
+ return Status::Error<INIT_FAILED>("get fs failed");
+ }
+ io::FileWriterPtr file_writer;
+ io::FileWriterOptions opts;
+ opts.create_empty_file = false;
+ Status st = fs->create_file(path, &file_writer, &opts);
+ if (!st.ok()) {
+ LOG(WARNING) << "failed to create writable file. path=" << path << ", err: " << st;
+ return st;
+ }
+
+ DCHECK(file_writer != nullptr);
+ segment_v2::SegmentWriterOptions writer_options;
+ writer_options.enable_unique_key_merge_on_write = _context.enable_unique_key_merge_on_write;
+ writer_options.rowset_ctx = &_context;
+ writer->reset(new segment_v2::SegmentWriter(
+ file_writer.get(), *_num_segment, _context.tablet_schema, _context.tablet,
+ _context.data_dir, _context.max_rows_per_segment, writer_options, nullptr));
+ {
+ std::lock_guard<SpinLock> l(*_lock);
+ _file_writers->push_back(std::move(file_writer));
+ }
+
+ auto s = (*writer)->init(column_ids, is_key);
+ if (!s.ok()) {
+ LOG(WARNING) << "failed to init segment writer: " << s.to_string();
+ writer->reset(nullptr);
+ return s;
+ }
+ return Status::OK();
+}
+
+Status VerticalBetaRowsetWriterHelper::final_flush() {
Review Comment:
warning: method 'final_flush' can be made static [readability-convert-member-functions-to-static]
be/src/olap/rowset/vertical_beta_rowset_writer_helper.h:51:
```diff
- Status final_flush();
+ static Status final_flush();
```
##########
be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:
##########
@@ -0,0 +1,244 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/rowset/vertical_beta_rowset_writer_helper.h"
+
+#include <fmt/format.h>
+#include <gen_cpp/olap_file.pb.h>
+
+#include <algorithm>
+#include <atomic>
+#include <mutex>
+#include <ostream>
+#include <string>
+#include <utility>
+
+#include "common/compiler_util.h" // IWYU pragma: keep
+#include "common/logging.h"
+#include "io/fs/file_system.h"
+#include "io/fs/file_writer.h"
+#include "olap/rowset/beta_rowset.h"
+#include "olap/rowset/rowset_meta.h"
+#include "olap/rowset/rowset_writer_context.h"
+#include "util/slice.h"
+#include "util/spinlock.h"
+#include "vec/core/block.h"
+
+namespace doris {
+using namespace ErrorCode;
+
+VerticalBetaRowsetWriterHelper::VerticalBetaRowsetWriterHelper(
+ std::vector<std::unique_ptr<segment_v2::SegmentWriter>>* segment_writers,
+ bool already_built, RowsetMetaSharedPtr& rowset_meta, std::atomic<int32_t>* num_segment,
+ RowsetWriterContext& context, std::atomic<int64_t>* num_rows_written,
+ std::vector<KeyBoundsPB>* segments_encoded_key_bounds,
+ std::vector<uint32_t>* segment_num_rows, std::atomic<int64_t>* total_index_size,
+ std::vector<io::FileWriterPtr>* file_writers, std::atomic<int64_t>* total_data_size,
+ SpinLock* lock)
+ : _segment_writers(segment_writers),
+ _already_built(already_built),
+ _rowset_meta(rowset_meta),
+ _num_segment(num_segment),
+ _context(context),
+ _num_rows_written(num_rows_written),
+ _segments_encoded_key_bounds(segments_encoded_key_bounds),
+ _segment_num_rows(segment_num_rows),
+ _total_index_size(total_index_size),
+ _file_writers(file_writers),
+ _total_data_size(total_data_size),
+ _lock(lock) {}
+
+void VerticalBetaRowsetWriterHelper::destruct_writer() {
+ if (!_already_built) {
+ const auto& fs = _rowset_meta->fs();
+ if (!fs || !_rowset_meta->is_local()) { // Remote fs will delete them asynchronously
+ return;
+ }
+ for (auto& segment_writer : *_segment_writers) {
+ segment_writer.reset();
+ }
+ for (int i = 0; i < *_num_segment; ++i) {
+ auto path = BetaRowset::segment_file_path(_context.rowset_dir, _context.rowset_id, i);
+ // Even if an error is encountered, these files that have not been cleaned up
+ // will be cleaned up by the GC background. So here we only print the error
+ // message when we encounter an error.
+ WARN_IF_ERROR(fs->delete_file(path), fmt::format("Failed to delete file={}", path));
+ }
+ }
+}
+
+Status VerticalBetaRowsetWriterHelper::add_columns(const vectorized::Block* block,
Review Comment:
warning: function 'add_columns' has cognitive complexity of 79 (threshold 50) [readability-function-cognitive-complexity]
```cpp
Status VerticalBetaRowsetWriterHelper::add_columns(const vectorized::Block* block,
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:88:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (num_rows == 0) {
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:91:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (UNLIKELY(max_rows_per_segment > _context.max_rows_per_segment)) {
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:95:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (_segment_writers->empty()) {
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:99:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_create_segment_writer(col_ids, is_key, &writer));
^
```
**be/src/common/status.h:541:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:99:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_create_segment_writer(col_ids, is_key, &writer));
^
```
**be/src/common/status.h:543:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:102:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR((*_segment_writers)[_cur_writer_idx]->append_block(block, 0, num_rows));
^
```
**be/src/common/status.h:541:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:102:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR((*_segment_writers)[_cur_writer_idx]->append_block(block, 0, num_rows));
^
```
**be/src/common/status.h:543:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:103:** +1, nesting level increased to 1
```cpp
} else if (is_key) {
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:104:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if ((*_segment_writers)[_cur_writer_idx]->num_rows_written() > max_rows_per_segment) {
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:106:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_flush_columns(&(*_segment_writers)[_cur_writer_idx], true));
^
```
**be/src/common/status.h:541:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:106:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(_flush_columns(&(*_segment_writers)[_cur_writer_idx], true));
^
```
**be/src/common/status.h:543:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:109:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_create_segment_writer(col_ids, is_key, &writer));
^
```
**be/src/common/status.h:541:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:109:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(_create_segment_writer(col_ids, is_key, &writer));
^
```
**be/src/common/status.h:543:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:113:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR((*_segment_writers)[_cur_writer_idx]->append_block(block, 0, num_rows));
^
```
**be/src/common/status.h:541:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:113:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR((*_segment_writers)[_cur_writer_idx]->append_block(block, 0, num_rows));
^
```
**be/src/common/status.h:543:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:114:** +1, nesting level increased to 1
```cpp
} else {
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:121:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (_cur_writer_idx == 0 && num_rows_written == 0) {
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:121:** +1
```cpp
if (_cur_writer_idx == 0 && num_rows_written == 0) {
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:123:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR((*_segment_writers)[_cur_writer_idx]->init(col_ids, is_key));
^
```
**be/src/common/status.h:541:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:123:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR((*_segment_writers)[_cur_writer_idx]->init(col_ids, is_key));
^
```
**be/src/common/status.h:543:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:128:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (num_rows_written + num_rows >= num_rows_key_group &&
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:130:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR((*_segment_writers)[_cur_writer_idx]->append_block(
^
```
**be/src/common/status.h:541:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:130:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR((*_segment_writers)[_cur_writer_idx]->append_block(
^
```
**be/src/common/status.h:543:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:132:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_flush_columns(&(*_segment_writers)[_cur_writer_idx]));
^
```
**be/src/common/status.h:541:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:132:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(_flush_columns(&(*_segment_writers)[_cur_writer_idx]));
^
```
**be/src/common/status.h:543:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:137:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR((*_segment_writers)[_cur_writer_idx]->init(col_ids, is_key));
^
```
**be/src/common/status.h:541:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:137:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR((*_segment_writers)[_cur_writer_idx]->init(col_ids, is_key));
^
```
**be/src/common/status.h:543:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:141:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (limit > 0) {
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:142:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(
^
```
**be/src/common/status.h:541:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:142:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(
^
```
**be/src/common/status.h:543:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:148:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (is_key) {
^
```
</details>
##########
be/src/olap/rowset/vertical_beta_rowset_writer_helper.h:
##########
@@ -0,0 +1,82 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <stddef.h>
+#include <stdint.h>
+
+#include <memory>
+#include <vector>
+
+#include "common/status.h"
+#include "olap/rowset/beta_rowset_writer.h"
+#include "olap/rowset/segment_v2/segment_writer.h"
+
+namespace doris {
+namespace vectorized {
+class Block;
+} // namespace vectorized
+
+class VerticalBetaRowsetWriterHelper {
+public:
+ VerticalBetaRowsetWriterHelper(
+ std::vector<std::unique_ptr<segment_v2::SegmentWriter>>* segment_writers,
+ bool already_built, RowsetMetaSharedPtr& rowset_meta, std::atomic<int32_t>* num_segment,
+ RowsetWriterContext& context, std::atomic<int64_t>* _num_rows_written,
+ std::vector<KeyBoundsPB>* _segments_encoded_key_bounds,
+ std::vector<uint32_t>* _segment_num_rows, std::atomic<int64_t>* _total_index_size,
+ std::vector<io::FileWriterPtr>* _file_writers, std::atomic<int64_t>* _total_data_size,
+ SpinLock* _lock);
+ ~VerticalBetaRowsetWriterHelper() = default;
+
+ Status add_columns(const vectorized::Block* block, const std::vector<uint32_t>& col_ids,
+ bool is_key, uint32_t max_rows_per_segment);
+
+ Status flush_columns(bool is_key);
+
+ Status final_flush();
+
+ int64_t num_rows() const { return _total_key_group_rows; }
+
+ void destruct_writer();
+
+private:
+ Status _flush_columns(std::unique_ptr<segment_v2::SegmentWriter>* segment_writer,
+ bool is_key = false);
+ Status _create_segment_writer(const std::vector<uint32_t>& column_ids, bool is_key,
+ std::unique_ptr<segment_v2::SegmentWriter>* writer);
+
+private:
Review Comment:
warning: redundant access specifier has the same accessibility as the previous access specifier [readability-redundant-access-specifiers]
```suggestion
```
<details>
<summary>Additional context</summary>
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.h:57:** previously declared here
```cpp
private:
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Implement cloud compaction [doris]
Posted by "luwei16 (via GitHub)" <gi...@apache.org>.
luwei16 commented on PR #31215:
URL: https://github.com/apache/doris/pull/31215#issuecomment-1970760128
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Implement cloud compaction [doris]
Posted by "luwei16 (via GitHub)" <gi...@apache.org>.
luwei16 commented on PR #31215:
URL: https://github.com/apache/doris/pull/31215#issuecomment-1970554486
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Implement cloud compaction [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #31215:
URL: https://github.com/apache/doris/pull/31215#discussion_r1506920166
##########
be/src/olap/rowset/vertical_beta_rowset_writer_helper.h:
##########
@@ -0,0 +1,83 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <stddef.h>
+#include <stdint.h>
+
+#include <memory>
+#include <vector>
+
+#include "common/status.h"
+#include "olap/rowset/beta_rowset_writer.h"
+#include "olap/rowset/segment_v2/segment_writer.h"
+
+namespace doris {
+namespace vectorized {
+class Block;
+} // namespace vectorized
+
+class VerticalBetaRowsetWriterHelper {
+public:
+ VerticalBetaRowsetWriterHelper(
+ std::vector<std::unique_ptr<segment_v2::SegmentWriter>>* segment_writers,
+ bool& already_built, RowsetMetaSharedPtr& rowset_meta,
+ std::atomic<int32_t>* num_segment, RowsetWriterContext& context,
+ std::atomic<int64_t>* _num_rows_written,
+ std::vector<KeyBoundsPB>* _segments_encoded_key_bounds,
+ std::vector<uint32_t>* _segment_num_rows, std::atomic<int64_t>* _total_index_size,
+ std::vector<io::FileWriterPtr>* _file_writers, std::atomic<int64_t>* _total_data_size,
+ SpinLock* _lock);
+ ~VerticalBetaRowsetWriterHelper() = default;
+
+ Status add_columns(const vectorized::Block* block, const std::vector<uint32_t>& col_ids,
+ bool is_key, uint32_t max_rows_per_segment);
+
+ Status flush_columns(bool is_key);
+
+ Status final_flush();
+
+ int64_t num_rows() const { return _total_key_group_rows; }
+
+ void destruct_writer();
+
+private:
+ Status _flush_columns(std::unique_ptr<segment_v2::SegmentWriter>* segment_writer,
+ bool is_key = false);
+ Status _create_segment_writer(const std::vector<uint32_t>& column_ids, bool is_key,
+ std::unique_ptr<segment_v2::SegmentWriter>* writer);
+
+private:
Review Comment:
warning: redundant access specifier has the same accessibility as the previous access specifier [readability-redundant-access-specifiers]
```suggestion
```
<details>
<summary>Additional context</summary>
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.h:58:** previously declared here
```cpp
private:
^
```
</details>
##########
be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:
##########
@@ -0,0 +1,244 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/rowset/vertical_beta_rowset_writer_helper.h"
+
+#include <fmt/format.h>
+#include <gen_cpp/olap_file.pb.h>
+
+#include <algorithm>
+#include <atomic>
+#include <mutex>
+#include <ostream>
+#include <string>
+#include <utility>
+
+#include "common/compiler_util.h" // IWYU pragma: keep
+#include "common/logging.h"
+#include "io/fs/file_system.h"
+#include "io/fs/file_writer.h"
+#include "olap/rowset/beta_rowset.h"
+#include "olap/rowset/rowset_meta.h"
+#include "olap/rowset/rowset_writer_context.h"
+#include "util/slice.h"
+#include "util/spinlock.h"
+#include "vec/core/block.h"
+
+namespace doris {
+using namespace ErrorCode;
+
+VerticalBetaRowsetWriterHelper::VerticalBetaRowsetWriterHelper(
+ std::vector<std::unique_ptr<segment_v2::SegmentWriter>>* segment_writers,
+ bool& already_built, RowsetMetaSharedPtr& rowset_meta, std::atomic<int32_t>* num_segment,
+ RowsetWriterContext& context, std::atomic<int64_t>* num_rows_written,
+ std::vector<KeyBoundsPB>* segments_encoded_key_bounds,
+ std::vector<uint32_t>* segment_num_rows, std::atomic<int64_t>* total_index_size,
+ std::vector<io::FileWriterPtr>* file_writers, std::atomic<int64_t>* total_data_size,
+ SpinLock* lock)
+ : _segment_writers(segment_writers),
+ _already_built(already_built),
+ _rowset_meta(rowset_meta),
+ _num_segment(num_segment),
+ _context(context),
+ _num_rows_written(num_rows_written),
+ _segments_encoded_key_bounds(segments_encoded_key_bounds),
+ _segment_num_rows(segment_num_rows),
+ _total_index_size(total_index_size),
+ _file_writers(file_writers),
+ _total_data_size(total_data_size),
+ _lock(lock) {}
+
+void VerticalBetaRowsetWriterHelper::destruct_writer() {
+ if (!_already_built) {
+ const auto& fs = _rowset_meta->fs();
+ if (!fs || !_rowset_meta->is_local()) { // Remote fs will delete them asynchronously
+ return;
+ }
+ for (auto& segment_writer : *_segment_writers) {
+ segment_writer.reset();
+ }
+ for (int i = 0; i < *_num_segment; ++i) {
+ auto path = BetaRowset::segment_file_path(_context.rowset_dir, _context.rowset_id, i);
+ // Even if an error is encountered, these files that have not been cleaned up
+ // will be cleaned up by the GC background. So here we only print the error
+ // message when we encounter an error.
+ WARN_IF_ERROR(fs->delete_file(path), fmt::format("Failed to delete file={}", path));
+ }
+ }
+}
+
+Status VerticalBetaRowsetWriterHelper::add_columns(const vectorized::Block* block,
+ const std::vector<uint32_t>& col_ids,
+ bool is_key, uint32_t max_rows_per_segment) {
+ VLOG_NOTICE << "VerticalBetaRowsetWriter::add_columns, columns: " << block->columns();
+ size_t num_rows = block->rows();
+ if (num_rows == 0) {
+ return Status::OK();
+ }
+ if (UNLIKELY(max_rows_per_segment > _context.max_rows_per_segment)) {
+ max_rows_per_segment = _context.max_rows_per_segment;
+ }
+
+ if (_segment_writers->empty()) {
+ // it must be key columns
+ DCHECK(is_key);
+ std::unique_ptr<segment_v2::SegmentWriter> writer;
+ RETURN_IF_ERROR(_create_segment_writer(col_ids, is_key, &writer));
+ _segment_writers->emplace_back(std::move(writer));
+ _cur_writer_idx = 0;
+ RETURN_IF_ERROR((*_segment_writers)[_cur_writer_idx]->append_block(block, 0, num_rows));
+ } else if (is_key) {
+ if ((*_segment_writers)[_cur_writer_idx]->num_rows_written() > max_rows_per_segment) {
+ // segment is full, need flush columns and create new segment writer
+ RETURN_IF_ERROR(_flush_columns(&(*_segment_writers)[_cur_writer_idx], true));
+
+ std::unique_ptr<segment_v2::SegmentWriter> writer;
+ RETURN_IF_ERROR(_create_segment_writer(col_ids, is_key, &writer));
+ _segment_writers->emplace_back(std::move(writer));
+ ++_cur_writer_idx;
+ }
+ RETURN_IF_ERROR((*_segment_writers)[_cur_writer_idx]->append_block(block, 0, num_rows));
+ } else {
+ // value columns
+ uint32_t num_rows_written = (*_segment_writers)[_cur_writer_idx]->num_rows_written();
+ VLOG_NOTICE << "num_rows_written: " << num_rows_written
+ << ", _cur_writer_idx: " << _cur_writer_idx;
+ uint32_t num_rows_key_group = (*_segment_writers)[_cur_writer_idx]->row_count();
+ // init if it's first value column write in current segment
+ if (_cur_writer_idx == 0 && num_rows_written == 0) {
+ VLOG_NOTICE << "init first value column segment writer";
+ RETURN_IF_ERROR((*_segment_writers)[_cur_writer_idx]->init(col_ids, is_key));
+ }
+ // when splitting segment, need to make rows align between key columns and value columns
+ size_t start_offset = 0;
+ size_t limit = num_rows;
+ if (num_rows_written + num_rows >= num_rows_key_group &&
+ _cur_writer_idx < _segment_writers->size() - 1) {
+ RETURN_IF_ERROR((*_segment_writers)[_cur_writer_idx]->append_block(
+ block, 0, num_rows_key_group - num_rows_written));
+ RETURN_IF_ERROR(_flush_columns(&(*_segment_writers)[_cur_writer_idx]));
+ start_offset = num_rows_key_group - num_rows_written;
+ limit = num_rows - start_offset;
+ ++_cur_writer_idx;
+ // switch to next writer
+ RETURN_IF_ERROR((*_segment_writers)[_cur_writer_idx]->init(col_ids, is_key));
+ num_rows_written = 0;
+ num_rows_key_group = (*_segment_writers)[_cur_writer_idx]->row_count();
+ }
+ if (limit > 0) {
+ RETURN_IF_ERROR(
+ (*_segment_writers)[_cur_writer_idx]->append_block(block, start_offset, limit));
+ DCHECK((*_segment_writers)[_cur_writer_idx]->num_rows_written() <=
+ (*_segment_writers)[_cur_writer_idx]->row_count());
+ }
+ }
+ if (is_key) {
+ *_num_rows_written += num_rows;
+ }
+ return Status::OK();
+}
+
+Status VerticalBetaRowsetWriterHelper::_flush_columns(
+ std::unique_ptr<segment_v2::SegmentWriter>* segment_writer, bool is_key) {
+ uint64_t index_size = 0;
+ VLOG_NOTICE << "flush columns index: " << _cur_writer_idx;
+ RETURN_IF_ERROR((*segment_writer)->finalize_columns_data());
+ RETURN_IF_ERROR((*segment_writer)->finalize_columns_index(&index_size));
+ if (is_key) {
+ _total_key_group_rows += (*segment_writer)->row_count();
+ // record segment key bound
+ KeyBoundsPB key_bounds;
+ Slice min_key = (*segment_writer)->min_encoded_key();
+ Slice max_key = (*segment_writer)->max_encoded_key();
+ DCHECK_LE(min_key.compare(max_key), 0);
+ key_bounds.set_min_key(min_key.to_string());
+ key_bounds.set_max_key(max_key.to_string());
+ _segments_encoded_key_bounds->emplace_back(key_bounds);
+ _segment_num_rows->resize(_cur_writer_idx + 1);
+ (*_segment_num_rows)[_cur_writer_idx] = (*_segment_writers)[_cur_writer_idx]->row_count();
+ }
+ *_total_index_size +=
+ static_cast<int64_t>(index_size) + (*segment_writer)->get_inverted_index_file_size();
+ return Status::OK();
+}
+
+Status VerticalBetaRowsetWriterHelper::flush_columns(bool is_key) {
+ if (_segment_writers->empty()) {
+ return Status::OK();
+ }
+
+ DCHECK(_cur_writer_idx < _segment_writers->size() && (*_segment_writers)[_cur_writer_idx]);
+ RETURN_IF_ERROR(_flush_columns(&(*_segment_writers)[_cur_writer_idx], is_key));
+ _cur_writer_idx = 0;
+ return Status::OK();
+}
+
+Status VerticalBetaRowsetWriterHelper::_create_segment_writer(
+ const std::vector<uint32_t>& column_ids, bool is_key,
+ std::unique_ptr<segment_v2::SegmentWriter>* writer) {
+ auto path = BetaRowset::segment_file_path(_context.rowset_dir, _context.rowset_id,
+ (*_num_segment)++);
+ auto fs = _rowset_meta->fs();
+ if (!fs) {
+ return Status::Error<INIT_FAILED>("get fs failed");
+ }
+ io::FileWriterPtr file_writer;
+ io::FileWriterOptions opts;
+ opts.create_empty_file = false;
+ Status st = fs->create_file(path, &file_writer, &opts);
+ if (!st.ok()) {
+ LOG(WARNING) << "failed to create writable file. path=" << path << ", err: " << st;
+ return st;
+ }
+
+ DCHECK(file_writer != nullptr);
+ segment_v2::SegmentWriterOptions writer_options;
+ writer_options.enable_unique_key_merge_on_write = _context.enable_unique_key_merge_on_write;
+ writer_options.rowset_ctx = &_context;
+ writer->reset(new segment_v2::SegmentWriter(
+ file_writer.get(), *_num_segment, _context.tablet_schema, _context.tablet,
+ _context.data_dir, _context.max_rows_per_segment, writer_options, nullptr));
+ {
+ std::lock_guard<SpinLock> l(*_lock);
+ _file_writers->push_back(std::move(file_writer));
+ }
+
+ auto s = (*writer)->init(column_ids, is_key);
+ if (!s.ok()) {
+ LOG(WARNING) << "failed to init segment writer: " << s.to_string();
+ writer->reset(nullptr);
+ return s;
+ }
+ return Status::OK();
+}
+
+Status VerticalBetaRowsetWriterHelper::final_flush() {
Review Comment:
warning: method 'final_flush' can be made static [readability-convert-member-functions-to-static]
be/src/olap/rowset/vertical_beta_rowset_writer_helper.h:52:
```diff
- Status final_flush();
+ static Status final_flush();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Implement cloud compaction [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #31215:
URL: https://github.com/apache/doris/pull/31215#discussion_r1497198972
##########
be/src/cloud/cloud_base_compaction.cpp:
##########
@@ -0,0 +1,387 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_base_compaction.h"
+
+#include <boost/container_hash/hash.hpp>
+
+#include "cloud/cloud_meta_mgr.h"
+#include "cloud/config.h"
+#include "common/config.h"
+#include "common/sync_point.h"
+#include "gen_cpp/cloud.pb.h"
+#include "olap/compaction.h"
+#include "olap/task/engine_checksum_task.h"
+#include "service/backend_options.h"
+#include "util/thread.h"
+#include "util/uuid_generator.h"
+
+namespace doris {
+using namespace ErrorCode;
+
+bvar::Adder<uint64_t> base_output_size("base_compaction", "output_size");
+
+CloudBaseCompaction::CloudBaseCompaction(CloudStorageEngine& engine, CloudTabletSPtr tablet)
+ : CloudCompactionMixin(engine, std::move(tablet),
+ "BaseCompaction:" + std::to_string(tablet->tablet_id())) {
+ auto uuid = UUIDGenerator::instance()->next_uuid();
+ std::stringstream ss;
+ ss << uuid;
+ _uuid = ss.str();
+}
+
+CloudBaseCompaction::~CloudBaseCompaction() = default;
+
+Status CloudBaseCompaction::prepare_compact() {
+ if (_tablet->tablet_state() != TABLET_RUNNING) {
+ return Status::InternalError("invalid tablet state. tablet_id={}", _tablet->tablet_id());
+ }
+
+ bool need_sync_tablet = true;
+ {
+ std::shared_lock rlock(_tablet->get_header_lock());
+ // If number of rowsets is equal to approximate_num_rowsets, it is very likely that this tablet has been
+ // synchronized with meta-service.
+ if (_tablet->tablet_meta()->all_rs_metas().size() >=
+ cloud_tablet()->fetch_add_approximate_num_rowsets(0) &&
+ cloud_tablet()->last_sync_time_s > 0) {
+ need_sync_tablet = false;
+ }
+ }
+ if (need_sync_tablet) {
+ RETURN_IF_ERROR(cloud_tablet()->sync_rowsets());
+ }
+
+ RETURN_IF_ERROR(pick_rowsets_to_compact());
+
+ // prepare compaction job
+ cloud::TabletJobInfoPB job;
+ auto idx = job.mutable_idx();
+ idx->set_tablet_id(_tablet->tablet_id());
+ idx->set_table_id(_tablet->table_id());
+ idx->set_index_id(_tablet->index_id());
+ idx->set_partition_id(_tablet->partition_id());
+ auto compaction_job = job.add_compaction();
+ compaction_job->set_id(_uuid);
+ compaction_job->set_initiator(BackendOptions::get_localhost() + ':' +
+ std::to_string(config::heartbeat_service_port));
+ compaction_job->set_type(cloud::TabletCompactionJobPB::BASE);
+ compaction_job->set_base_compaction_cnt(_base_compaction_cnt);
+ compaction_job->set_cumulative_compaction_cnt(_cumulative_compaction_cnt);
+ using namespace std::chrono;
+ int64_t now = duration_cast<seconds>(system_clock::now().time_since_epoch()).count();
+ _expiration = now + config::compaction_timeout_seconds;
+ compaction_job->set_expiration(_expiration);
+ compaction_job->set_lease(now + config::lease_compaction_interval_seconds * 4);
+ cloud::StartTabletJobResponse resp;
+ //auto st = cloud::meta_mgr()->prepare_tablet_job(job, &resp);
+ auto st = _engine.meta_mgr().prepare_tablet_job(job, &resp);
+ if (!st.ok()) {
+ if (resp.status().code() == cloud::STALE_TABLET_CACHE) {
+ // set last_sync_time to 0 to force sync tablet next time
+ cloud_tablet()->last_sync_time_s = 0;
+ } else if (resp.status().code() == cloud::TABLET_NOT_FOUND) {
+ // tablet not found
+ cloud_tablet()->recycle_cached_data();
+ }
+ return st;
+ }
+
+ for (auto& rs : _input_rowsets) {
+ _input_row_num += rs->num_rows();
+ _input_segments += rs->num_segments();
+ _input_rowsets_size += rs->data_disk_size();
+ }
+ LOG_INFO("start CloudBaseCompaction, tablet_id={}, range=[{}-{}]", _tablet->tablet_id(),
+ _input_rowsets.front()->start_version(), _input_rowsets.back()->end_version())
+ .tag("job_id", _uuid)
+ .tag("input_rowsets", _input_rowsets.size())
+ .tag("input_rows", _input_row_num)
+ .tag("input_segments", _input_segments)
+ .tag("input_data_size", _input_rowsets_size);
+ return st;
+}
+
+void CloudBaseCompaction::_filter_input_rowset() {
Review Comment:
warning: method '_filter_input_rowset' can be made static [readability-convert-member-functions-to-static]
be/src/cloud/cloud_base_compaction.h:46:
```diff
- void _filter_input_rowset();
+ static void _filter_input_rowset();
```
##########
be/src/cloud/cloud_cumulative_compaction.cpp:
##########
@@ -0,0 +1,477 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_cumulative_compaction.h"
+
+#include "cloud/cloud_meta_mgr.h"
+#include "cloud/config.h"
+#include "common/config.h"
+#include "common/logging.h"
+#include "common/status.h"
+#include "common/sync_point.h"
+#include "gen_cpp/cloud.pb.h"
+#include "olap/compaction.h"
+#include "olap/cumulative_compaction_policy.h"
+#include "service/backend_options.h"
+#include "util/trace.h"
+#include "util/uuid_generator.h"
+
+namespace doris {
+using namespace ErrorCode;
+
+bvar::Adder<uint64_t> cumu_output_size("cumu_compaction", "output_size");
+
+CloudCumulativeCompaction::CloudCumulativeCompaction(CloudStorageEngine& engine,
+ CloudTabletSPtr tablet)
+ : CloudCompactionMixin(engine, std::move(tablet),
+ "BaseCompaction:" + std::to_string(tablet->tablet_id())) {
+ auto uuid = UUIDGenerator::instance()->next_uuid();
+ std::stringstream ss;
+ ss << uuid;
+ _uuid = ss.str();
+}
+
+CloudCumulativeCompaction::~CloudCumulativeCompaction() = default;
+
+Status CloudCumulativeCompaction::prepare_compact() {
Review Comment:
warning: function 'prepare_compact' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status CloudCumulativeCompaction::prepare_compact() {
^
```
<details>
<summary>Additional context</summary>
**be/src/cloud/cloud_cumulative_compaction.cpp:49:** 115 lines including whitespace and comments (threshold 80)
```cpp
Status CloudCumulativeCompaction::prepare_compact() {
^
```
</details>
##########
be/src/cloud/cloud_cumulative_compaction.cpp:
##########
@@ -0,0 +1,477 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_cumulative_compaction.h"
+
+#include "cloud/cloud_meta_mgr.h"
+#include "cloud/config.h"
+#include "common/config.h"
+#include "common/logging.h"
+#include "common/status.h"
+#include "common/sync_point.h"
+#include "gen_cpp/cloud.pb.h"
+#include "olap/compaction.h"
+#include "olap/cumulative_compaction_policy.h"
+#include "service/backend_options.h"
+#include "util/trace.h"
+#include "util/uuid_generator.h"
+
+namespace doris {
+using namespace ErrorCode;
+
+bvar::Adder<uint64_t> cumu_output_size("cumu_compaction", "output_size");
+
+CloudCumulativeCompaction::CloudCumulativeCompaction(CloudStorageEngine& engine,
+ CloudTabletSPtr tablet)
+ : CloudCompactionMixin(engine, std::move(tablet),
+ "BaseCompaction:" + std::to_string(tablet->tablet_id())) {
+ auto uuid = UUIDGenerator::instance()->next_uuid();
+ std::stringstream ss;
+ ss << uuid;
+ _uuid = ss.str();
+}
+
+CloudCumulativeCompaction::~CloudCumulativeCompaction() = default;
+
+Status CloudCumulativeCompaction::prepare_compact() {
+ if (_tablet->tablet_state() != TABLET_RUNNING) {
+ return Status::InternalError("invalid tablet state. tablet_id={}", _tablet->tablet_id());
+ }
+
+ std::vector<std::shared_ptr<CloudCumulativeCompaction>> cumu_compactions;
+ _engine.get_cumu_compaction(_tablet->tablet_id(), cumu_compactions);
+ if (!cumu_compactions.empty()) {
+ for (auto& cumu : cumu_compactions) {
+ _max_conflict_version =
+ std::max(_max_conflict_version, cumu->_input_rowsets.back()->end_version());
+ }
+ }
+
+ int tried = 0;
+PREPARE_TRY_AGAIN:
+
+ bool need_sync_tablet = true;
+ {
+ std::shared_lock rlock(_tablet->get_header_lock());
+ // If number of rowsets is equal to approximate_num_rowsets, it is very likely that this tablet has been
+ // synchronized with meta-service.
+ if (_tablet->tablet_meta()->all_rs_metas().size() >=
+ cloud_tablet()->fetch_add_approximate_num_rowsets(0) &&
+ cloud_tablet()->last_sync_time_s > 0) {
+ need_sync_tablet = false;
+ }
+ }
+ if (need_sync_tablet) {
+ RETURN_IF_ERROR(cloud_tablet()->sync_rowsets());
+ }
+
+ // pick rowsets to compact
+ auto st = pick_rowsets_to_compact();
+ if (!st.ok()) {
+ if (tried == 0 && _last_delete_version.first != -1) {
+ // we meet a delete version, should increase the cumulative point to let base compaction handle the delete version.
+ // plus 1 to skip the delete version.
+ // NOTICE: after that, the cumulative point may be larger than max version of this tablet, but it doesn't matter.
+ update_cumulative_point();
+ }
+ return st;
+ }
+
+ // prepare compaction job
+ cloud::TabletJobInfoPB job;
+ auto idx = job.mutable_idx();
+ idx->set_tablet_id(_tablet->tablet_id());
+ idx->set_table_id(_tablet->table_id());
+ idx->set_index_id(_tablet->index_id());
+ idx->set_partition_id(_tablet->partition_id());
+ auto compaction_job = job.add_compaction();
+ compaction_job->set_id(_uuid);
+ compaction_job->set_initiator(BackendOptions::get_localhost() + ':' +
+ std::to_string(config::heartbeat_service_port));
+ compaction_job->set_type(cloud::TabletCompactionJobPB::CUMULATIVE);
+ compaction_job->set_base_compaction_cnt(_base_compaction_cnt);
+ compaction_job->set_cumulative_compaction_cnt(_cumulative_compaction_cnt);
+ using namespace std::chrono;
+ int64_t now = duration_cast<seconds>(system_clock::now().time_since_epoch()).count();
+ _expiration = now + config::compaction_timeout_seconds;
+ compaction_job->set_expiration(_expiration);
+ compaction_job->set_lease(now + config::lease_compaction_interval_seconds * 4);
+ if (config::enable_parallel_cumu_compaction) {
+ // Set input version range to let meta-service judge version range conflict
+ compaction_job->add_input_versions(_input_rowsets.front()->start_version());
+ compaction_job->add_input_versions(_input_rowsets.back()->end_version());
+ }
+ cloud::StartTabletJobResponse resp;
+ st = _engine.meta_mgr().prepare_tablet_job(job, &resp);
+ if (!st.ok()) {
+ if (resp.status().code() == cloud::STALE_TABLET_CACHE) {
+ // set last_sync_time to 0 to force sync tablet next time
+ cloud_tablet()->last_sync_time_s = 0;
+ } else if (resp.status().code() == cloud::TABLET_NOT_FOUND) {
+ // tablet not found
+ cloud_tablet()->recycle_cached_data();
+ } else if (resp.status().code() == cloud::JOB_TABLET_BUSY) {
+ if (config::enable_parallel_cumu_compaction && resp.version_in_compaction_size() > 0 &&
+ ++tried <= 2) {
+ _max_conflict_version = *std::max_element(resp.version_in_compaction().begin(),
+ resp.version_in_compaction().end());
+ LOG_INFO("retry pick input rowsets")
+ .tag("job_id", _uuid)
+ .tag("max_conflict_version", _max_conflict_version)
+ .tag("tried", tried)
+ .tag("msg", resp.status().msg());
+ goto PREPARE_TRY_AGAIN;
+ } else {
+ LOG_WARNING("failed to prepare cumu compaction")
+ .tag("job_id", _uuid)
+ .tag("msg", resp.status().msg());
+ return Status::Error<CUMULATIVE_NO_SUITABLE_VERSION>("no suitable versions");
+ }
+ }
+ return st;
+ }
+
+ for (auto& rs : _input_rowsets) {
+ _input_row_num += rs->num_rows();
+ _input_segments += rs->num_segments();
+ _input_rowsets_size += rs->data_disk_size();
+ }
+ LOG_INFO("start CloudCumulativeCompaction, tablet_id={}, range=[{}-{}]", _tablet->tablet_id(),
+ _input_rowsets.front()->start_version(), _input_rowsets.back()->end_version())
+ .tag("job_id", _uuid)
+ .tag("input_rowsets", _input_rowsets.size())
+ .tag("input_rows", _input_row_num)
+ .tag("input_segments", _input_segments)
+ .tag("input_data_size", _input_rowsets_size)
+ .tag("tablet_max_version", cloud_tablet()->max_version_unlocked())
+ .tag("cumulative_point", cloud_tablet()->cumulative_layer_point())
+ .tag("num_rowsets", cloud_tablet()->fetch_add_approximate_num_rowsets(0))
+ .tag("cumu_num_rowsets", cloud_tablet()->fetch_add_approximate_cumu_num_rowsets(0));
+ return st;
+}
+
+Status CloudCumulativeCompaction::execute_compact() {
+ TEST_SYNC_POINT_RETURN_WITH_VALUE("CloudCumulativeCompaction::execute_compact_impl",
+ Status::OK(), this);
+ using namespace std::chrono;
+ auto start = steady_clock::now();
+ RETURN_IF_ERROR(CloudCompactionMixin::execute_compact());
+ LOG_INFO("finish CloudCumulativeCompaction, tablet_id={}, cost={}ms", _tablet->tablet_id(),
+ duration_cast<milliseconds>(steady_clock::now() - start).count())
+ .tag("job_id", _uuid)
+ .tag("input_rowsets", _input_rowsets.size())
+ .tag("input_rows", _input_row_num)
+ .tag("input_segments", _input_segments)
+ .tag("input_data_size", _input_rowsets_size)
+ .tag("output_rows", _output_rowset->num_rows())
+ .tag("output_segments", _output_rowset->num_segments())
+ .tag("output_data_size", _output_rowset->data_disk_size())
+ .tag("tablet_max_version", _tablet->max_version_unlocked())
+ .tag("cumulative_point", cloud_tablet()->cumulative_layer_point())
+ .tag("num_rowsets", cloud_tablet()->fetch_add_approximate_num_rowsets(0))
+ .tag("cumu_num_rowsets", cloud_tablet()->fetch_add_approximate_cumu_num_rowsets(0));
+
+ _state = CompactionState::SUCCESS;
+
+ DorisMetrics::instance()->cumulative_compaction_deltas_total->increment(_input_rowsets.size());
+ DorisMetrics::instance()->cumulative_compaction_bytes_total->increment(_input_rowsets_size);
+ cumu_output_size << _output_rowset->data_disk_size();
+
+ return Status::OK();
+}
+
+Status CloudCumulativeCompaction::modify_rowsets() {
Review Comment:
warning: function 'modify_rowsets' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status CloudCumulativeCompaction::modify_rowsets() {
^
```
<details>
<summary>Additional context</summary>
**be/src/cloud/cloud_cumulative_compaction.cpp:196:** 102 lines including whitespace and comments (threshold 80)
```cpp
Status CloudCumulativeCompaction::modify_rowsets() {
^
```
</details>
##########
be/src/cloud/cloud_base_compaction.cpp:
##########
@@ -0,0 +1,387 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_base_compaction.h"
+
+#include <boost/container_hash/hash.hpp>
+
+#include "cloud/cloud_meta_mgr.h"
+#include "cloud/config.h"
+#include "common/config.h"
+#include "common/sync_point.h"
+#include "gen_cpp/cloud.pb.h"
+#include "olap/compaction.h"
+#include "olap/task/engine_checksum_task.h"
+#include "service/backend_options.h"
+#include "util/thread.h"
+#include "util/uuid_generator.h"
+
+namespace doris {
+using namespace ErrorCode;
+
+bvar::Adder<uint64_t> base_output_size("base_compaction", "output_size");
+
+CloudBaseCompaction::CloudBaseCompaction(CloudStorageEngine& engine, CloudTabletSPtr tablet)
+ : CloudCompactionMixin(engine, std::move(tablet),
+ "BaseCompaction:" + std::to_string(tablet->tablet_id())) {
+ auto uuid = UUIDGenerator::instance()->next_uuid();
+ std::stringstream ss;
+ ss << uuid;
+ _uuid = ss.str();
+}
+
+CloudBaseCompaction::~CloudBaseCompaction() = default;
+
+Status CloudBaseCompaction::prepare_compact() {
+ if (_tablet->tablet_state() != TABLET_RUNNING) {
+ return Status::InternalError("invalid tablet state. tablet_id={}", _tablet->tablet_id());
+ }
+
+ bool need_sync_tablet = true;
+ {
+ std::shared_lock rlock(_tablet->get_header_lock());
+ // If number of rowsets is equal to approximate_num_rowsets, it is very likely that this tablet has been
+ // synchronized with meta-service.
+ if (_tablet->tablet_meta()->all_rs_metas().size() >=
+ cloud_tablet()->fetch_add_approximate_num_rowsets(0) &&
+ cloud_tablet()->last_sync_time_s > 0) {
+ need_sync_tablet = false;
+ }
+ }
+ if (need_sync_tablet) {
+ RETURN_IF_ERROR(cloud_tablet()->sync_rowsets());
+ }
+
+ RETURN_IF_ERROR(pick_rowsets_to_compact());
+
+ // prepare compaction job
+ cloud::TabletJobInfoPB job;
+ auto idx = job.mutable_idx();
+ idx->set_tablet_id(_tablet->tablet_id());
+ idx->set_table_id(_tablet->table_id());
+ idx->set_index_id(_tablet->index_id());
+ idx->set_partition_id(_tablet->partition_id());
+ auto compaction_job = job.add_compaction();
+ compaction_job->set_id(_uuid);
+ compaction_job->set_initiator(BackendOptions::get_localhost() + ':' +
+ std::to_string(config::heartbeat_service_port));
+ compaction_job->set_type(cloud::TabletCompactionJobPB::BASE);
+ compaction_job->set_base_compaction_cnt(_base_compaction_cnt);
+ compaction_job->set_cumulative_compaction_cnt(_cumulative_compaction_cnt);
+ using namespace std::chrono;
+ int64_t now = duration_cast<seconds>(system_clock::now().time_since_epoch()).count();
+ _expiration = now + config::compaction_timeout_seconds;
+ compaction_job->set_expiration(_expiration);
+ compaction_job->set_lease(now + config::lease_compaction_interval_seconds * 4);
+ cloud::StartTabletJobResponse resp;
+ //auto st = cloud::meta_mgr()->prepare_tablet_job(job, &resp);
+ auto st = _engine.meta_mgr().prepare_tablet_job(job, &resp);
+ if (!st.ok()) {
+ if (resp.status().code() == cloud::STALE_TABLET_CACHE) {
+ // set last_sync_time to 0 to force sync tablet next time
+ cloud_tablet()->last_sync_time_s = 0;
+ } else if (resp.status().code() == cloud::TABLET_NOT_FOUND) {
+ // tablet not found
+ cloud_tablet()->recycle_cached_data();
+ }
+ return st;
+ }
+
+ for (auto& rs : _input_rowsets) {
+ _input_row_num += rs->num_rows();
+ _input_segments += rs->num_segments();
+ _input_rowsets_size += rs->data_disk_size();
+ }
+ LOG_INFO("start CloudBaseCompaction, tablet_id={}, range=[{}-{}]", _tablet->tablet_id(),
+ _input_rowsets.front()->start_version(), _input_rowsets.back()->end_version())
+ .tag("job_id", _uuid)
+ .tag("input_rowsets", _input_rowsets.size())
+ .tag("input_rows", _input_row_num)
+ .tag("input_segments", _input_segments)
+ .tag("input_data_size", _input_rowsets_size);
+ return st;
+}
+
+void CloudBaseCompaction::_filter_input_rowset() {
+ // if dup_key and no delete predicate
+ // we skip big files to save resources
+ if (_tablet->keys_type() != KeysType::DUP_KEYS) {
+ return;
+ }
+ for (auto& rs : _input_rowsets) {
+ if (rs->rowset_meta()->has_delete_predicate()) {
+ return;
+ }
+ }
+ int64_t max_size = config::base_compaction_dup_key_max_file_size_mbytes * 1024 * 1024;
+ // first find a proper rowset for start
+ auto rs_iter = _input_rowsets.begin();
+ while (rs_iter != _input_rowsets.end()) {
+ if ((*rs_iter)->rowset_meta()->total_disk_size() >= max_size) {
+ rs_iter = _input_rowsets.erase(rs_iter);
+ } else {
+ break;
+ }
+ }
+}
+
+Status CloudBaseCompaction::pick_rowsets_to_compact() {
+ _input_rowsets.clear();
+ {
+ std::shared_lock rlock(_tablet->get_header_lock());
+ _base_compaction_cnt = cloud_tablet()->base_compaction_cnt();
+ _cumulative_compaction_cnt = cloud_tablet()->cumulative_compaction_cnt();
+ _input_rowsets = cloud_tablet()->pick_candidate_rowsets_to_base_compaction();
+ }
+ if (auto st = check_version_continuity(_input_rowsets); !st.ok()) {
+ DCHECK(false) << st;
+ return st;
+ }
+ _filter_input_rowset();
+ if (_input_rowsets.size() <= 1) {
+ return Status::Error<BE_NO_SUITABLE_VERSION>(
+ "insuffient compation input rowset, #rowsets={}", _input_rowsets.size());
+ }
+
+ if (_input_rowsets.size() == 2 && _input_rowsets[0]->end_version() == 1) {
+ // the tablet is with rowset: [0-1], [2-y]
+ // and [0-1] has no data. in this situation, no need to do base compaction.
+ return Status::Error<BE_NO_SUITABLE_VERSION>("no suitable versions for compaction");
+ }
+
+ // 1. cumulative rowset must reach base_compaction_min_rowset_num threshold
+ if (_input_rowsets.size() > config::base_compaction_min_rowset_num) {
+ VLOG_NOTICE << "satisfy the base compaction policy. tablet=" << _tablet->tablet_id()
+ << ", num_cumulative_rowsets=" << _input_rowsets.size() - 1
+ << ", base_compaction_num_cumulative_rowsets="
+ << config::base_compaction_min_rowset_num;
+ return Status::OK();
+ }
+
+ // 2. the ratio between base rowset and all input cumulative rowsets reaches the threshold
+ // `_input_rowsets` has been sorted by end version, so we consider `_input_rowsets[0]` is the base rowset.
+ int64_t base_size = _input_rowsets.front()->data_disk_size();
+ int64_t cumulative_total_size = 0;
+ for (auto it = _input_rowsets.begin() + 1; it != _input_rowsets.end(); ++it) {
+ cumulative_total_size += (*it)->data_disk_size();
+ }
+
+ double base_cumulative_delta_ratio = config::base_compaction_min_data_ratio;
+ if (base_size == 0) {
+ // base_size == 0 means this may be a base version [0-1], which has no data.
+ // set to 1 to void divide by zero
+ base_size = 1;
+ }
+ double cumulative_base_ratio = static_cast<double>(cumulative_total_size) / base_size;
+
+ if (cumulative_base_ratio > base_cumulative_delta_ratio) {
+ VLOG_NOTICE << "satisfy the base compaction policy. tablet=" << _tablet->tablet_id()
+ << ", cumulative_total_size=" << cumulative_total_size
+ << ", base_size=" << base_size
+ << ", cumulative_base_ratio=" << cumulative_base_ratio
+ << ", policy_ratio=" << base_cumulative_delta_ratio;
+ return Status::OK();
+ }
+
+ // 3. the interval since last base compaction reaches the threshold
+ int64_t base_creation_time = _input_rowsets[0]->creation_time();
+ int64_t interval_threshold = config::base_compaction_interval_seconds_since_last_operation;
+ int64_t interval_since_last_base_compaction = time(nullptr) - base_creation_time;
+ if (interval_since_last_base_compaction > interval_threshold) {
+ VLOG_NOTICE << "satisfy the base compaction policy. tablet=" << _tablet->tablet_id()
+ << ", interval_since_last_base_compaction="
+ << interval_since_last_base_compaction
+ << ", interval_threshold=" << interval_threshold;
+ return Status::OK();
+ }
+
+ VLOG_NOTICE << "don't satisfy the base compaction policy. tablet=" << _tablet->tablet_id()
+ << ", num_cumulative_rowsets=" << _input_rowsets.size() - 1
+ << ", cumulative_base_ratio=" << cumulative_base_ratio
+ << ", interval_since_last_base_compaction=" << interval_since_last_base_compaction;
+ return Status::Error<BE_NO_SUITABLE_VERSION>("no suitable versions for compaction");
+}
+
+Status CloudBaseCompaction::execute_compact() {
+ if (config::enable_base_compaction_idle_sched) {
+ Thread::set_idle_sched();
+ }
+
+ SCOPED_ATTACH_TASK(_mem_tracker);
+
+ using namespace std::chrono;
+ auto start = steady_clock::now();
+ RETURN_IF_ERROR(CloudCompactionMixin::execute_compact());
+ LOG_INFO("finish CloudBaseCompaction, tablet_id={}, cost={}ms", _tablet->tablet_id(),
+ duration_cast<milliseconds>(steady_clock::now() - start).count())
+ .tag("job_id", _uuid)
+ .tag("input_rowsets", _input_rowsets.size())
+ .tag("input_rows", _input_row_num)
+ .tag("input_segments", _input_segments)
+ .tag("input_data_size", _input_rowsets_size)
+ .tag("output_rows", _output_rowset->num_rows())
+ .tag("output_segments", _output_rowset->num_segments())
+ .tag("output_data_size", _output_rowset->data_disk_size());
+
+ //_compaction_succeed = true;
+ _state = CompactionState::SUCCESS;
+
+ DorisMetrics::instance()->base_compaction_deltas_total->increment(_input_rowsets.size());
+ DorisMetrics::instance()->base_compaction_bytes_total->increment(_input_rowsets_size);
+ base_output_size << _output_rowset->data_disk_size();
+
+ return Status::OK();
+}
+
+Status CloudBaseCompaction::modify_rowsets() {
Review Comment:
warning: function 'modify_rowsets' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status CloudBaseCompaction::modify_rowsets() {
^
```
<details>
<summary>Additional context</summary>
**be/src/cloud/cloud_base_compaction.cpp:249:** 82 lines including whitespace and comments (threshold 80)
```cpp
Status CloudBaseCompaction::modify_rowsets() {
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Implement cloud compaction [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31215:
URL: https://github.com/apache/doris/pull/31215#issuecomment-1972948544
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Implement cloud compaction [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31215:
URL: https://github.com/apache/doris/pull/31215#issuecomment-1970687908
TeamCity be ut coverage result:
Function Coverage: 35.53% (8555/24076)
Line Coverage: 27.29% (69414/254327)
Region Coverage: 26.48% (36000/135948)
Branch Coverage: 23.31% (18394/78922)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a7bf44581a441089b9c5ac6ab06d6ae548e04e78_a7bf44581a441089b9c5ac6ab06d6ae548e04e78/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Implement cloud compaction [doris]
Posted by "luwei16 (via GitHub)" <gi...@apache.org>.
luwei16 commented on PR #31215:
URL: https://github.com/apache/doris/pull/31215#issuecomment-1956242705
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Implement cloud compaction [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #31215:
URL: https://github.com/apache/doris/pull/31215#discussion_r1504307847
##########
be/src/cloud/cloud_compaction_action.cpp:
##########
@@ -0,0 +1,338 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_compaction_action.h"
+
+// IWYU pragma: no_include <bits/chrono.h>
+#include <chrono> // IWYU pragma: keep
+#include <exception>
+#include <future>
+#include <memory>
+#include <mutex>
+#include <sstream>
+#include <string>
+#include <thread>
+#include <utility>
+
+#include "cloud/cloud_base_compaction.h"
+#include "cloud/cloud_compaction_action.h"
+#include "cloud/cloud_cumulative_compaction.h"
+#include "cloud/cloud_full_compaction.h"
+#include "cloud/cloud_tablet.h"
+#include "cloud/cloud_tablet_mgr.h"
+#include "common/logging.h"
+#include "common/status.h"
+#include "gutil/strings/substitute.h"
+#include "http/http_channel.h"
+#include "http/http_headers.h"
+#include "http/http_request.h"
+#include "http/http_status.h"
+#include "olap/base_compaction.h"
+#include "olap/cumulative_compaction.h"
+#include "olap/cumulative_compaction_policy.h"
+#include "olap/cumulative_compaction_time_series_policy.h"
+#include "olap/full_compaction.h"
+#include "olap/olap_define.h"
+#include "olap/storage_engine.h"
+#include "olap/tablet_manager.h"
+#include "util/doris_metrics.h"
+#include "util/stopwatch.hpp"
+
+namespace doris {
+using namespace ErrorCode;
+
+namespace {}
+
+const static std::string HEADER_JSON = "application/json";
+
+CloudCompactionAction::CloudCompactionAction(CompactionActionType ctype, ExecEnv* exec_env,
+ CloudStorageEngine& engine, TPrivilegeHier::type hier,
+ TPrivilegeType::type ptype)
+ : HttpHandlerWithAuth(exec_env, hier, ptype), _engine(engine), _type(ctype) {}
+
+/// check param and fetch tablet_id from req
+static Status _check_param(HttpRequest* req, uint64_t* tablet_id, uint64_t* table_id) {
+ // req tablet id and table id, we have to set only one of them.
+ std::string req_tablet_id = req->param(TABLET_ID_KEY);
+ std::string req_table_id = req->param(TABLE_ID_KEY);
+ if (req_tablet_id == "") {
+ if (req_table_id == "") {
+ // both tablet id and table id are empty, return error.
+ return Status::InternalError(
+ "tablet id and table id can not be empty at the same time!");
+ } else {
+ try {
+ *table_id = std::stoull(req_table_id);
+ } catch (const std::exception& e) {
+ return Status::InternalError("convert tablet_id or table_id failed, {}", e.what());
+ }
+ return Status::OK();
+ }
+ } else {
+ if (req_table_id == "") {
+ try {
+ *tablet_id = std::stoull(req_tablet_id);
+ } catch (const std::exception& e) {
+ return Status::InternalError("convert tablet_id or table_id failed, {}", e.what());
+ }
+ return Status::OK();
+ } else {
+ // both tablet id and table id are not empty, return err.
+ return Status::InternalError("tablet id and table id can not be set at the same time!");
+ }
+ }
+}
+
+// for viewing the compaction status
+Status CloudCompactionAction::_handle_show_compaction(HttpRequest* req, std::string* json_result) {
Review Comment:
warning: method '_handle_show_compaction' can be made static [readability-convert-member-functions-to-static]
be/src/cloud/cloud_compaction_action.h:61:
```diff
- Status _handle_show_compaction(HttpRequest* req, std::string* json_result);
+ static Status _handle_show_compaction(HttpRequest* req, std::string* json_result);
```
##########
be/src/cloud/cloud_compaction_action.cpp:
##########
@@ -0,0 +1,338 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_compaction_action.h"
+
+// IWYU pragma: no_include <bits/chrono.h>
+#include <chrono> // IWYU pragma: keep
+#include <exception>
+#include <future>
+#include <memory>
+#include <mutex>
+#include <sstream>
+#include <string>
+#include <thread>
+#include <utility>
+
+#include "cloud/cloud_base_compaction.h"
+#include "cloud/cloud_compaction_action.h"
+#include "cloud/cloud_cumulative_compaction.h"
+#include "cloud/cloud_full_compaction.h"
Review Comment:
warning: duplicate include [readability-duplicate-include]
be/src/cloud/cloud_compaction_action.cpp:32:
```diff
- #include "cloud/cloud_base_compaction.h"
- #include "cloud/cloud_compaction_action.h"
+ #include "cloud/cloud_base_compaction.h"
```
##########
be/src/cloud/cloud_compaction_action.cpp:
##########
@@ -0,0 +1,338 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_compaction_action.h"
+
+// IWYU pragma: no_include <bits/chrono.h>
+#include <chrono> // IWYU pragma: keep
+#include <exception>
+#include <future>
+#include <memory>
+#include <mutex>
+#include <sstream>
+#include <string>
+#include <thread>
+#include <utility>
+
+#include "cloud/cloud_base_compaction.h"
+#include "cloud/cloud_compaction_action.h"
+#include "cloud/cloud_cumulative_compaction.h"
+#include "cloud/cloud_full_compaction.h"
+#include "cloud/cloud_tablet.h"
+#include "cloud/cloud_tablet_mgr.h"
+#include "common/logging.h"
+#include "common/status.h"
+#include "gutil/strings/substitute.h"
+#include "http/http_channel.h"
+#include "http/http_headers.h"
+#include "http/http_request.h"
+#include "http/http_status.h"
+#include "olap/base_compaction.h"
+#include "olap/cumulative_compaction.h"
+#include "olap/cumulative_compaction_policy.h"
+#include "olap/cumulative_compaction_time_series_policy.h"
+#include "olap/full_compaction.h"
+#include "olap/olap_define.h"
+#include "olap/storage_engine.h"
+#include "olap/tablet_manager.h"
+#include "util/doris_metrics.h"
+#include "util/stopwatch.hpp"
+
+namespace doris {
+using namespace ErrorCode;
+
+namespace {}
+
+const static std::string HEADER_JSON = "application/json";
+
+CloudCompactionAction::CloudCompactionAction(CompactionActionType ctype, ExecEnv* exec_env,
+ CloudStorageEngine& engine, TPrivilegeHier::type hier,
+ TPrivilegeType::type ptype)
+ : HttpHandlerWithAuth(exec_env, hier, ptype), _engine(engine), _type(ctype) {}
+
+/// check param and fetch tablet_id from req
+static Status _check_param(HttpRequest* req, uint64_t* tablet_id, uint64_t* table_id) {
+ // req tablet id and table id, we have to set only one of them.
+ std::string req_tablet_id = req->param(TABLET_ID_KEY);
+ std::string req_table_id = req->param(TABLE_ID_KEY);
+ if (req_tablet_id == "") {
+ if (req_table_id == "") {
+ // both tablet id and table id are empty, return error.
+ return Status::InternalError(
+ "tablet id and table id can not be empty at the same time!");
+ } else {
+ try {
+ *table_id = std::stoull(req_table_id);
+ } catch (const std::exception& e) {
+ return Status::InternalError("convert tablet_id or table_id failed, {}", e.what());
+ }
+ return Status::OK();
+ }
+ } else {
+ if (req_table_id == "") {
Review Comment:
warning: the 'empty' method should be used to check for emptiness instead of comparing to an empty object [readability-container-size-empty]
```suggestion
if (req_table_id.empty()) {
```
<details>
<summary>Additional context</summary>
**/usr/include/c++/11/bits/basic_string.h:1022:** method 'basic_string'::empty() defined here
```cpp
empty() const _GLIBCXX_NOEXCEPT
^
```
</details>
##########
be/src/cloud/cloud_vertical_rowset_writer.cpp:
##########
@@ -0,0 +1,71 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_vertical_rowset_writer.h"
+
+#include <fmt/format.h>
+#include <gen_cpp/olap_file.pb.h>
+
+#include <algorithm>
+#include <atomic>
+#include <memory>
+#include <mutex>
+#include <ostream>
+#include <string>
+#include <utility>
+
+#include "common/compiler_util.h" // IWYU pragma: keep
+#include "common/logging.h"
+#include "io/fs/file_system.h"
+#include "io/fs/file_writer.h"
+#include "olap/rowset/beta_rowset.h"
+#include "olap/rowset/rowset_meta.h"
+#include "olap/rowset/rowset_writer_context.h"
+#include "olap/rowset/vertical_beta_rowset_writer_helper.h"
+#include "util/slice.h"
+#include "util/spinlock.h"
+#include "vec/core/block.h"
+
+namespace doris {
+using namespace ErrorCode;
+
+CloudVerticalRowsetWriter::CloudVerticalRowsetWriter() : CloudRowsetWriter() {
+ _helper = std::make_shared<VerticalBetaRowsetWriterHelper>(
Review Comment:
warning: initializer for base class 'doris::CloudRowsetWriter' is redundant [readability-redundant-member-init]
```suggestion
: {
```
##########
be/src/olap/rowset/vertical_beta_rowset_writer_helper.h:
##########
@@ -0,0 +1,82 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <stddef.h>
Review Comment:
warning: inclusion of deprecated C++ header 'stddef.h'; consider using 'cstddef' instead [modernize-deprecated-headers]
```suggestion
#include <cstddef>
```
##########
be/src/olap/rowset/vertical_beta_rowset_writer_helper.h:
##########
@@ -0,0 +1,82 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <stddef.h>
+#include <stdint.h>
Review Comment:
warning: inclusion of deprecated C++ header 'stdint.h'; consider using 'cstdint' instead [modernize-deprecated-headers]
```suggestion
#include <cstdint>
```
##########
be/src/cloud/cloud_compaction_action.h:
##########
@@ -0,0 +1,79 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <stdint.h>
+
+#include <string>
+
+#include "cloud/cloud_storage_engine.h"
+#include "common/status.h"
+#include "http/action/compaction_action.h"
+#include "http/http_handler_with_auth.h"
+#include "olap/tablet.h"
+
+namespace doris {
+class HttpRequest;
+
+class ExecEnv;
+
+/*
+enum class CompactionActionType {
+ SHOW_INFO = 1,
+ RUN_COMPACTION = 2,
+ RUN_COMPACTION_STATUS = 3,
+};
+*/
+
+/*
+const std::string PARAM_COMPACTION_TYPE = "compact_type";
+const std::string PARAM_COMPACTION_BASE = "base";
+const std::string PARAM_COMPACTION_CUMULATIVE = "cumulative";
+const std::string PARAM_COMPACTION_FULL = "full";
+*/
+
+/// This action is used for viewing the compaction status.
+/// See compaction-action.md for details.
+class CloudCompactionAction : public HttpHandlerWithAuth {
+public:
+ CloudCompactionAction(CompactionActionType ctype, ExecEnv* exec_env, CloudStorageEngine& engine,
+ TPrivilegeHier::type hier, TPrivilegeType::type ptype);
+
+ ~CloudCompactionAction() override = default;
+
+ void handle(HttpRequest* req) override;
+
+private:
+ Status _handle_show_compaction(HttpRequest* req, std::string* json_result);
+
+ /// execute compaction request to run compaction task
+ /// param compact_type in req to distinguish the task type, base or cumulative
+ Status _handle_run_compaction(HttpRequest* req, std::string* json_result);
+
+ /// thread callback function for the tablet to do compaction
+ Status _execute_compaction_callback(CloudTabletSPtr tablet, const std::string& compaction_type);
+
+ /// fetch compaction running status
+ Status _handle_run_status_compaction(HttpRequest* req, std::string* json_result);
+
+private:
Review Comment:
warning: redundant access specifier has the same accessibility as the previous access specifier [readability-redundant-access-specifiers]
```suggestion
```
<details>
<summary>Additional context</summary>
**be/src/cloud/cloud_compaction_action.h:60:** previously declared here
```cpp
private:
^
```
</details>
##########
be/src/cloud/cloud_compaction_action.cpp:
##########
@@ -0,0 +1,338 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_compaction_action.h"
+
+// IWYU pragma: no_include <bits/chrono.h>
+#include <chrono> // IWYU pragma: keep
+#include <exception>
+#include <future>
+#include <memory>
+#include <mutex>
+#include <sstream>
+#include <string>
+#include <thread>
+#include <utility>
+
+#include "cloud/cloud_base_compaction.h"
+#include "cloud/cloud_compaction_action.h"
+#include "cloud/cloud_cumulative_compaction.h"
+#include "cloud/cloud_full_compaction.h"
+#include "cloud/cloud_tablet.h"
+#include "cloud/cloud_tablet_mgr.h"
+#include "common/logging.h"
+#include "common/status.h"
+#include "gutil/strings/substitute.h"
+#include "http/http_channel.h"
+#include "http/http_headers.h"
+#include "http/http_request.h"
+#include "http/http_status.h"
+#include "olap/base_compaction.h"
+#include "olap/cumulative_compaction.h"
+#include "olap/cumulative_compaction_policy.h"
+#include "olap/cumulative_compaction_time_series_policy.h"
+#include "olap/full_compaction.h"
+#include "olap/olap_define.h"
+#include "olap/storage_engine.h"
+#include "olap/tablet_manager.h"
+#include "util/doris_metrics.h"
+#include "util/stopwatch.hpp"
+
+namespace doris {
+using namespace ErrorCode;
+
+namespace {}
+
+const static std::string HEADER_JSON = "application/json";
+
+CloudCompactionAction::CloudCompactionAction(CompactionActionType ctype, ExecEnv* exec_env,
+ CloudStorageEngine& engine, TPrivilegeHier::type hier,
+ TPrivilegeType::type ptype)
+ : HttpHandlerWithAuth(exec_env, hier, ptype), _engine(engine), _type(ctype) {}
+
+/// check param and fetch tablet_id from req
+static Status _check_param(HttpRequest* req, uint64_t* tablet_id, uint64_t* table_id) {
+ // req tablet id and table id, we have to set only one of them.
+ std::string req_tablet_id = req->param(TABLET_ID_KEY);
+ std::string req_table_id = req->param(TABLE_ID_KEY);
+ if (req_tablet_id == "") {
Review Comment:
warning: the 'empty' method should be used to check for emptiness instead of comparing to an empty object [readability-container-size-empty]
```suggestion
if (req_tablet_id.empty()) {
```
<details>
<summary>Additional context</summary>
**/usr/include/c++/11/bits/basic_string.h:1022:** method 'basic_string'::empty() defined here
```cpp
empty() const _GLIBCXX_NOEXCEPT
^
```
</details>
##########
be/src/cloud/cloud_compaction_action.h:
##########
@@ -0,0 +1,79 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <stdint.h>
Review Comment:
warning: inclusion of deprecated C++ header 'stdint.h'; consider using 'cstdint' instead [modernize-deprecated-headers]
```suggestion
#include <cstdint>
```
##########
be/src/olap/rowset/vertical_beta_rowset_writer_helper.h:
##########
@@ -0,0 +1,82 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <stddef.h>
+#include <stdint.h>
+
+#include <memory>
+#include <vector>
+
+#include "common/status.h"
+#include "olap/rowset/beta_rowset_writer.h"
+#include "olap/rowset/segment_v2/segment_writer.h"
+
+namespace doris {
+namespace vectorized {
+class Block;
+} // namespace vectorized
+
+class VerticalBetaRowsetWriterHelper {
+public:
+ VerticalBetaRowsetWriterHelper(
+ std::vector<std::unique_ptr<segment_v2::SegmentWriter>>* segment_writers,
+ bool already_built, RowsetMetaSharedPtr& rowset_meta, std::atomic<int32_t>* num_segment,
+ RowsetWriterContext& context, std::atomic<int64_t>* _num_rows_written,
+ std::vector<KeyBoundsPB>* _segments_encoded_key_bounds,
+ std::vector<uint32_t>* _segment_num_rows, std::atomic<int64_t>* _total_index_size,
+ std::vector<io::FileWriterPtr>* _file_writers, std::atomic<int64_t>* _total_data_size,
+ SpinLock* _lock);
+ ~VerticalBetaRowsetWriterHelper() = default;
+
+ Status add_columns(const vectorized::Block* block, const std::vector<uint32_t>& col_ids,
+ bool is_key, uint32_t max_rows_per_segment);
+
+ Status flush_columns(bool is_key);
+
+ Status final_flush();
+
+ int64_t num_rows() const { return _total_key_group_rows; }
+
+ void destruct_writer();
+
+private:
+ Status _flush_columns(std::unique_ptr<segment_v2::SegmentWriter>* segment_writer,
+ bool is_key = false);
+ Status _create_segment_writer(const std::vector<uint32_t>& column_ids, bool is_key,
+ std::unique_ptr<segment_v2::SegmentWriter>* writer);
Review Comment:
warning: redundant access specifier has the same accessibility as the previous access specifier [readability-redundant-access-specifiers]
```suggestion
```
<details>
<summary>Additional context</summary>
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.h:55:** previously declared here
```cpp
private:
^
```
</details>
##########
be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:
##########
@@ -0,0 +1,244 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/rowset/vertical_beta_rowset_writer_helper.h"
+
+#include <fmt/format.h>
+#include <gen_cpp/olap_file.pb.h>
+
+#include <algorithm>
+#include <atomic>
+#include <mutex>
+#include <ostream>
+#include <string>
+#include <utility>
+
+#include "common/compiler_util.h" // IWYU pragma: keep
+#include "common/logging.h"
+#include "io/fs/file_system.h"
+#include "io/fs/file_writer.h"
+#include "olap/rowset/beta_rowset.h"
+#include "olap/rowset/rowset_meta.h"
+#include "olap/rowset/rowset_writer_context.h"
+#include "util/slice.h"
+#include "util/spinlock.h"
+#include "vec/core/block.h"
+
+namespace doris {
+using namespace ErrorCode;
+
+VerticalBetaRowsetWriterHelper::VerticalBetaRowsetWriterHelper(
+ std::vector<std::unique_ptr<segment_v2::SegmentWriter>>* segment_writers,
+ bool already_built, RowsetMetaSharedPtr& rowset_meta, std::atomic<int32_t>* num_segment,
+ RowsetWriterContext& context, std::atomic<int64_t>* num_rows_written,
+ std::vector<KeyBoundsPB>* segments_encoded_key_bounds,
+ std::vector<uint32_t>* segment_num_rows, std::atomic<int64_t>* total_index_size,
+ std::vector<io::FileWriterPtr>* file_writers, std::atomic<int64_t>* total_data_size,
+ SpinLock* lock)
+ : _segment_writers(segment_writers),
+ _already_built(already_built),
+ _rowset_meta(rowset_meta),
+ _num_segment(num_segment),
+ _context(context),
+ _num_rows_written(num_rows_written),
+ _segments_encoded_key_bounds(segments_encoded_key_bounds),
+ _segment_num_rows(segment_num_rows),
+ _total_index_size(total_index_size),
+ _file_writers(file_writers),
+ _total_data_size(total_data_size),
+ _lock(lock) {}
+
+void VerticalBetaRowsetWriterHelper::destruct_writer() {
+ if (!_already_built) {
+ const auto& fs = _rowset_meta->fs();
+ if (!fs || !_rowset_meta->is_local()) { // Remote fs will delete them asynchronously
+ return;
+ }
+ for (auto& segment_writer : *_segment_writers) {
+ segment_writer.reset();
+ }
+ for (int i = 0; i < *_num_segment; ++i) {
+ auto path = BetaRowset::segment_file_path(_context.rowset_dir, _context.rowset_id, i);
+ // Even if an error is encountered, these files that have not been cleaned up
+ // will be cleaned up by the GC background. So here we only print the error
+ // message when we encounter an error.
+ WARN_IF_ERROR(fs->delete_file(path), fmt::format("Failed to delete file={}", path));
+ }
+ }
+}
+
+Status VerticalBetaRowsetWriterHelper::add_columns(const vectorized::Block* block,
+ const std::vector<uint32_t>& col_ids,
+ bool is_key, uint32_t max_rows_per_segment) {
+ VLOG_NOTICE << "VerticalBetaRowsetWriter::add_columns, columns: " << block->columns();
+ size_t num_rows = block->rows();
+ if (num_rows == 0) {
+ return Status::OK();
+ }
+ if (UNLIKELY(max_rows_per_segment > _context.max_rows_per_segment)) {
+ max_rows_per_segment = _context.max_rows_per_segment;
+ }
+
+ if (_segment_writers->empty()) {
+ // it must be key columns
+ DCHECK(is_key);
+ std::unique_ptr<segment_v2::SegmentWriter> writer;
+ RETURN_IF_ERROR(_create_segment_writer(col_ids, is_key, &writer));
+ _segment_writers->emplace_back(std::move(writer));
+ _cur_writer_idx = 0;
+ RETURN_IF_ERROR((*_segment_writers)[_cur_writer_idx]->append_block(block, 0, num_rows));
+ } else if (is_key) {
+ if ((*_segment_writers)[_cur_writer_idx]->num_rows_written() > max_rows_per_segment) {
+ // segment is full, need flush columns and create new segment writer
+ RETURN_IF_ERROR(_flush_columns(&(*_segment_writers)[_cur_writer_idx], true));
+
+ std::unique_ptr<segment_v2::SegmentWriter> writer;
+ RETURN_IF_ERROR(_create_segment_writer(col_ids, is_key, &writer));
+ _segment_writers->emplace_back(std::move(writer));
+ ++_cur_writer_idx;
+ }
+ RETURN_IF_ERROR((*_segment_writers)[_cur_writer_idx]->append_block(block, 0, num_rows));
+ } else {
+ // value columns
+ uint32_t num_rows_written = (*_segment_writers)[_cur_writer_idx]->num_rows_written();
+ VLOG_NOTICE << "num_rows_written: " << num_rows_written
+ << ", _cur_writer_idx: " << _cur_writer_idx;
+ uint32_t num_rows_key_group = (*_segment_writers)[_cur_writer_idx]->row_count();
+ // init if it's first value column write in current segment
+ if (_cur_writer_idx == 0 && num_rows_written == 0) {
+ VLOG_NOTICE << "init first value column segment writer";
+ RETURN_IF_ERROR((*_segment_writers)[_cur_writer_idx]->init(col_ids, is_key));
+ }
+ // when splitting segment, need to make rows align between key columns and value columns
+ size_t start_offset = 0;
+ size_t limit = num_rows;
+ if (num_rows_written + num_rows >= num_rows_key_group &&
+ _cur_writer_idx < _segment_writers->size() - 1) {
+ RETURN_IF_ERROR((*_segment_writers)[_cur_writer_idx]->append_block(
+ block, 0, num_rows_key_group - num_rows_written));
+ RETURN_IF_ERROR(_flush_columns(&(*_segment_writers)[_cur_writer_idx]));
+ start_offset = num_rows_key_group - num_rows_written;
+ limit = num_rows - start_offset;
+ ++_cur_writer_idx;
+ // switch to next writer
+ RETURN_IF_ERROR((*_segment_writers)[_cur_writer_idx]->init(col_ids, is_key));
+ num_rows_written = 0;
+ num_rows_key_group = (*_segment_writers)[_cur_writer_idx]->row_count();
+ }
+ if (limit > 0) {
+ RETURN_IF_ERROR(
+ (*_segment_writers)[_cur_writer_idx]->append_block(block, start_offset, limit));
+ DCHECK((*_segment_writers)[_cur_writer_idx]->num_rows_written() <=
+ (*_segment_writers)[_cur_writer_idx]->row_count());
+ }
+ }
+ if (is_key) {
+ *_num_rows_written += num_rows;
+ }
+ return Status::OK();
+}
+
+Status VerticalBetaRowsetWriterHelper::_flush_columns(
+ std::unique_ptr<segment_v2::SegmentWriter>* segment_writer, bool is_key) {
+ uint64_t index_size = 0;
+ VLOG_NOTICE << "flush columns index: " << _cur_writer_idx;
+ RETURN_IF_ERROR((*segment_writer)->finalize_columns_data());
+ RETURN_IF_ERROR((*segment_writer)->finalize_columns_index(&index_size));
+ if (is_key) {
+ _total_key_group_rows += (*segment_writer)->row_count();
+ // record segment key bound
+ KeyBoundsPB key_bounds;
+ Slice min_key = (*segment_writer)->min_encoded_key();
+ Slice max_key = (*segment_writer)->max_encoded_key();
+ DCHECK_LE(min_key.compare(max_key), 0);
+ key_bounds.set_min_key(min_key.to_string());
+ key_bounds.set_max_key(max_key.to_string());
+ _segments_encoded_key_bounds->emplace_back(key_bounds);
+ _segment_num_rows->resize(_cur_writer_idx + 1);
+ (*_segment_num_rows)[_cur_writer_idx] = (*_segment_writers)[_cur_writer_idx]->row_count();
+ }
+ *_total_index_size +=
+ static_cast<int64_t>(index_size) + (*segment_writer)->get_inverted_index_file_size();
+ return Status::OK();
+}
+
+Status VerticalBetaRowsetWriterHelper::flush_columns(bool is_key) {
+ if (_segment_writers->empty()) {
+ return Status::OK();
+ }
+
+ DCHECK(_cur_writer_idx < _segment_writers->size() && (*_segment_writers)[_cur_writer_idx]);
+ RETURN_IF_ERROR(_flush_columns(&(*_segment_writers)[_cur_writer_idx], is_key));
+ _cur_writer_idx = 0;
+ return Status::OK();
+}
+
+Status VerticalBetaRowsetWriterHelper::_create_segment_writer(
+ const std::vector<uint32_t>& column_ids, bool is_key,
+ std::unique_ptr<segment_v2::SegmentWriter>* writer) {
+ auto path = BetaRowset::segment_file_path(_context.rowset_dir, _context.rowset_id,
+ (*_num_segment)++);
+ auto fs = _rowset_meta->fs();
+ if (!fs) {
+ return Status::Error<INIT_FAILED>("get fs failed");
+ }
+ io::FileWriterPtr file_writer;
+ io::FileWriterOptions opts;
+ opts.create_empty_file = false;
+ Status st = fs->create_file(path, &file_writer, &opts);
+ if (!st.ok()) {
+ LOG(WARNING) << "failed to create writable file. path=" << path << ", err: " << st;
+ return st;
+ }
+
+ DCHECK(file_writer != nullptr);
+ segment_v2::SegmentWriterOptions writer_options;
+ writer_options.enable_unique_key_merge_on_write = _context.enable_unique_key_merge_on_write;
+ writer_options.rowset_ctx = &_context;
+ writer->reset(new segment_v2::SegmentWriter(
+ file_writer.get(), *_num_segment, _context.tablet_schema, _context.tablet,
+ _context.data_dir, _context.max_rows_per_segment, writer_options, nullptr));
+ {
+ std::lock_guard<SpinLock> l(*_lock);
+ _file_writers->push_back(std::move(file_writer));
+ }
+
+ auto s = (*writer)->init(column_ids, is_key);
+ if (!s.ok()) {
+ LOG(WARNING) << "failed to init segment writer: " << s.to_string();
+ writer->reset(nullptr);
+ return s;
+ }
+ return Status::OK();
+}
Review Comment:
warning: method 'final_flush' can be made static [readability-convert-member-functions-to-static]
be/src/olap/rowset/vertical_beta_rowset_writer_helper.h:49:
```diff
- Status final_flush();
+ static Status final_flush();
```
##########
be/src/cloud/cloud_compaction_action.cpp:
##########
@@ -0,0 +1,338 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_compaction_action.h"
+
+// IWYU pragma: no_include <bits/chrono.h>
+#include <chrono> // IWYU pragma: keep
+#include <exception>
+#include <future>
+#include <memory>
+#include <mutex>
+#include <sstream>
+#include <string>
+#include <thread>
+#include <utility>
+
+#include "cloud/cloud_base_compaction.h"
+#include "cloud/cloud_compaction_action.h"
+#include "cloud/cloud_cumulative_compaction.h"
+#include "cloud/cloud_full_compaction.h"
+#include "cloud/cloud_tablet.h"
+#include "cloud/cloud_tablet_mgr.h"
+#include "common/logging.h"
+#include "common/status.h"
+#include "gutil/strings/substitute.h"
+#include "http/http_channel.h"
+#include "http/http_headers.h"
+#include "http/http_request.h"
+#include "http/http_status.h"
+#include "olap/base_compaction.h"
+#include "olap/cumulative_compaction.h"
+#include "olap/cumulative_compaction_policy.h"
+#include "olap/cumulative_compaction_time_series_policy.h"
+#include "olap/full_compaction.h"
+#include "olap/olap_define.h"
+#include "olap/storage_engine.h"
+#include "olap/tablet_manager.h"
+#include "util/doris_metrics.h"
+#include "util/stopwatch.hpp"
+
+namespace doris {
+using namespace ErrorCode;
+
+namespace {}
+
+const static std::string HEADER_JSON = "application/json";
+
+CloudCompactionAction::CloudCompactionAction(CompactionActionType ctype, ExecEnv* exec_env,
+ CloudStorageEngine& engine, TPrivilegeHier::type hier,
+ TPrivilegeType::type ptype)
+ : HttpHandlerWithAuth(exec_env, hier, ptype), _engine(engine), _type(ctype) {}
+
+/// check param and fetch tablet_id from req
+static Status _check_param(HttpRequest* req, uint64_t* tablet_id, uint64_t* table_id) {
+ // req tablet id and table id, we have to set only one of them.
+ std::string req_tablet_id = req->param(TABLET_ID_KEY);
+ std::string req_table_id = req->param(TABLE_ID_KEY);
+ if (req_tablet_id == "") {
+ if (req_table_id == "") {
Review Comment:
warning: the 'empty' method should be used to check for emptiness instead of comparing to an empty object [readability-container-size-empty]
```suggestion
if (req_table_id.empty()) {
```
<details>
<summary>Additional context</summary>
**/usr/include/c++/11/bits/basic_string.h:1022:** method 'basic_string'::empty() defined here
```cpp
empty() const _GLIBCXX_NOEXCEPT
^
```
</details>
##########
be/src/cloud/cloud_vertical_rowset_writer.h:
##########
@@ -0,0 +1,60 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <stddef.h>
Review Comment:
warning: inclusion of deprecated C++ header 'stddef.h'; consider using 'cstddef' instead [modernize-deprecated-headers]
```suggestion
#include <cstddef>
```
##########
be/src/cloud/cloud_storage_engine.cpp:
##########
@@ -190,4 +235,454 @@ void CloudStorageEngine::_sync_tablets_thread_callback() {
}
}
+void CloudStorageEngine::get_cumu_compaction(
+ int64_t tablet_id, std::vector<std::shared_ptr<CloudCumulativeCompaction>>& res) {
+ std::lock_guard lock(_compaction_mtx);
+ if (auto it = _submitted_cumu_compactions.find(tablet_id);
+ it != _submitted_cumu_compactions.end()) {
+ res = it->second;
+ }
+}
+
+void CloudStorageEngine::_adjust_compaction_thread_num() {
Review Comment:
warning: method '_adjust_compaction_thread_num' can be made static [readability-convert-member-functions-to-static]
be/src/cloud/cloud_storage_engine.h:89:
```diff
- void _adjust_compaction_thread_num();
+ static void _adjust_compaction_thread_num();
```
##########
be/src/cloud/cloud_vertical_rowset_writer.h:
##########
@@ -0,0 +1,60 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <stddef.h>
+#include <stdint.h>
Review Comment:
warning: inclusion of deprecated C++ header 'stdint.h'; consider using 'cstdint' instead [modernize-deprecated-headers]
```suggestion
#include <cstdint>
```
##########
be/src/cloud/cloud_vertical_rowset_writer.cpp:
##########
@@ -0,0 +1,71 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_vertical_rowset_writer.h"
+
+#include <fmt/format.h>
+#include <gen_cpp/olap_file.pb.h>
+
+#include <algorithm>
+#include <atomic>
+#include <memory>
+#include <mutex>
+#include <ostream>
+#include <string>
+#include <utility>
+
+#include "common/compiler_util.h" // IWYU pragma: keep
+#include "common/logging.h"
+#include "io/fs/file_system.h"
+#include "io/fs/file_writer.h"
+#include "olap/rowset/beta_rowset.h"
+#include "olap/rowset/rowset_meta.h"
+#include "olap/rowset/rowset_writer_context.h"
+#include "olap/rowset/vertical_beta_rowset_writer_helper.h"
+#include "util/slice.h"
+#include "util/spinlock.h"
+#include "vec/core/block.h"
+
+namespace doris {
+using namespace ErrorCode;
+
+CloudVerticalRowsetWriter::CloudVerticalRowsetWriter() : CloudRowsetWriter() {
+ _helper = std::make_shared<VerticalBetaRowsetWriterHelper>(
+ &_segment_writers, _already_built, _rowset_meta, &_num_segment, _context,
+ &_num_rows_written, &_segments_encoded_key_bounds, &_segment_num_rows,
+ &_total_index_size, &_file_writers, &_total_data_size, &_lock);
+}
+
+CloudVerticalRowsetWriter::~CloudVerticalRowsetWriter() {
+ _helper->destruct_writer();
Review Comment:
warning: use '= default' to define a trivial destructor [modernize-use-equals-default]
```cpp
CloudVerticalRowsetWriter::~CloudVerticalRowsetWriter() {
^
```
##########
be/src/cloud/cloud_storage_engine.cpp:
##########
@@ -190,4 +235,454 @@
}
}
+void CloudStorageEngine::get_cumu_compaction(
+ int64_t tablet_id, std::vector<std::shared_ptr<CloudCumulativeCompaction>>& res) {
+ std::lock_guard lock(_compaction_mtx);
+ if (auto it = _submitted_cumu_compactions.find(tablet_id);
+ it != _submitted_cumu_compactions.end()) {
+ res = it->second;
+ }
+}
+
+void CloudStorageEngine::_adjust_compaction_thread_num() {
+ int base_thread_num = get_base_thread_num();
+ if (_base_compaction_thread_pool->max_threads() != base_thread_num) {
+ int old_max_threads = _base_compaction_thread_pool->max_threads();
+ Status status = _base_compaction_thread_pool->set_max_threads(base_thread_num);
+ if (status.ok()) {
+ VLOG_NOTICE << "update base compaction thread pool max_threads from " << old_max_threads
+ << " to " << base_thread_num;
+ }
+ }
+ if (_base_compaction_thread_pool->min_threads() != base_thread_num) {
+ int old_min_threads = _base_compaction_thread_pool->min_threads();
+ Status status = _base_compaction_thread_pool->set_min_threads(base_thread_num);
+ if (status.ok()) {
+ VLOG_NOTICE << "update base compaction thread pool min_threads from " << old_min_threads
+ << " to " << base_thread_num;
+ }
+ }
+
+ int cumu_thread_num = get_cumu_thread_num();
+ if (_cumu_compaction_thread_pool->max_threads() != cumu_thread_num) {
+ int old_max_threads = _cumu_compaction_thread_pool->max_threads();
+ Status status = _cumu_compaction_thread_pool->set_max_threads(cumu_thread_num);
+ if (status.ok()) {
+ VLOG_NOTICE << "update cumu compaction thread pool max_threads from " << old_max_threads
+ << " to " << cumu_thread_num;
+ }
+ }
+ if (_cumu_compaction_thread_pool->min_threads() != cumu_thread_num) {
+ int old_min_threads = _cumu_compaction_thread_pool->min_threads();
+ Status status = _cumu_compaction_thread_pool->set_min_threads(cumu_thread_num);
+ if (status.ok()) {
+ VLOG_NOTICE << "update cumu compaction thread pool min_threads from " << old_min_threads
+ << " to " << cumu_thread_num;
+ }
+ }
+}
+
+void CloudStorageEngine::_compaction_tasks_producer_callback() {
+ LOG(INFO) << "try to start compaction producer process!";
+
+ int round = 0;
+ CompactionType compaction_type;
+
+ // Used to record the time when the score metric was last updated.
+ // The update of the score metric is accompanied by the logic of selecting the tablet.
+ // If there is no slot available, the logic of selecting the tablet will be terminated,
+ // which causes the score metric update to be terminated.
+ // In order to avoid this situation, we need to update the score regularly.
+ int64_t last_cumulative_score_update_time = 0;
+ int64_t last_base_score_update_time = 0;
+ static const int64_t check_score_interval_ms = 5000; // 5 secs
+
+ int64_t interval = config::generate_compaction_tasks_interval_ms;
+ do {
+ if (!config::disable_auto_compaction) {
+ _adjust_compaction_thread_num();
+
+ bool check_score = false;
+ int64_t cur_time = UnixMillis();
+ if (round < config::cumulative_compaction_rounds_for_each_base_compaction_round) {
+ compaction_type = CompactionType::CUMULATIVE_COMPACTION;
+ round++;
+ if (cur_time - last_cumulative_score_update_time >= check_score_interval_ms) {
+ check_score = true;
+ last_cumulative_score_update_time = cur_time;
+ }
+ } else {
+ compaction_type = CompactionType::BASE_COMPACTION;
+ round = 0;
+ if (cur_time - last_base_score_update_time >= check_score_interval_ms) {
+ check_score = true;
+ last_base_score_update_time = cur_time;
+ }
+ }
+ std::unique_ptr<ThreadPool>& thread_pool =
+ (compaction_type == CompactionType::CUMULATIVE_COMPACTION)
+ ? _cumu_compaction_thread_pool
+ : _base_compaction_thread_pool;
+ VLOG_CRITICAL << "compaction thread pool. type: "
+ << (compaction_type == CompactionType::CUMULATIVE_COMPACTION ? "CUMU"
+ : "BASE")
+ << ", num_threads: " << thread_pool->num_threads()
+ << ", num_threads_pending_start: "
+ << thread_pool->num_threads_pending_start()
+ << ", num_active_threads: " << thread_pool->num_active_threads()
+ << ", max_threads: " << thread_pool->max_threads()
+ << ", min_threads: " << thread_pool->min_threads()
+ << ", num_total_queued_tasks: " << thread_pool->get_queue_size();
+ std::vector<CloudTabletSPtr> tablets_compaction =
+ _generate_cloud_compaction_tasks(compaction_type, check_score);
+
+ /// Regardless of whether the tablet is submitted for compaction or not,
+ /// we need to call 'reset_compaction' to clean up the base_compaction or cumulative_compaction objects
+ /// in the tablet, because these two objects store the tablet's own shared_ptr.
+ /// If it is not cleaned up, the reference count of the tablet will always be greater than 1,
+ /// thus cannot be collected by the garbage collector. (TabletManager::start_trash_sweep)
+ for (const auto& tablet : tablets_compaction) {
+ Status st = submit_compaction_task(tablet, compaction_type);
+ if (st.ok()) continue;
+ if ((!st.is<ErrorCode::BE_NO_SUITABLE_VERSION>() &&
+ !st.is<ErrorCode::CUMULATIVE_NO_SUITABLE_VERSION>()) ||
+ VLOG_DEBUG_IS_ON) {
+ LOG(WARNING) << "failed to submit compaction task for tablet: "
+ << tablet->tablet_id() << ", err: " << st;
+ }
+ }
+ interval = config::generate_compaction_tasks_interval_ms;
+ } else {
+ interval = config::check_auto_compaction_interval_seconds * 1000;
+ }
+ } while (!_stop_background_threads_latch.wait_for(std::chrono::milliseconds(interval)));
+}
+
+std::vector<CloudTabletSPtr> CloudStorageEngine::_generate_cloud_compaction_tasks(
+ CompactionType compaction_type, bool check_score) {
+ std::vector<std::shared_ptr<CloudTablet>> tablets_compaction;
+
+ int64_t max_compaction_score = 0;
+ std::unordered_set<int64_t> tablet_preparing_cumu_compaction;
+ std::unordered_map<int64_t, std::vector<std::shared_ptr<CloudCumulativeCompaction>>>
+ submitted_cumu_compactions;
+ std::unordered_map<int64_t, std::shared_ptr<CloudBaseCompaction>> submitted_base_compactions;
+ std::unordered_map<int64_t, std::shared_ptr<CloudFullCompaction>> submitted_full_compactions;
+ {
+ std::lock_guard lock(_compaction_mtx);
+ tablet_preparing_cumu_compaction = _tablet_preparing_cumu_compaction;
+ submitted_cumu_compactions = _submitted_cumu_compactions;
+ submitted_base_compactions = _submitted_base_compactions;
+ submitted_full_compactions = _submitted_full_compactions;
+ }
+
+ bool need_pick_tablet = true;
+ int thread_per_disk =
+ config::compaction_task_num_per_fast_disk; // all disks are fast in cloud mode
+ int num_cumu =
+ std::accumulate(submitted_cumu_compactions.begin(), submitted_cumu_compactions.end(), 0,
+ [](int a, auto& b) { return a + b.second.size(); });
+ int num_base = submitted_base_compactions.size() + submitted_full_compactions.size();
+ int n = thread_per_disk - num_cumu - num_base;
+ if (compaction_type == CompactionType::BASE_COMPACTION) {
+ // We need to reserve at least one thread for cumulative compaction,
+ // because base compactions may take too long to complete, which may
+ // leads to "too many rowsets" error.
+ int base_n = std::min(config::max_base_compaction_task_num_per_disk, thread_per_disk - 1) -
+ num_base;
+ n = std::min(base_n, n);
+ }
+ if (n <= 0) { // No threads available
+ if (!check_score) return tablets_compaction;
+ need_pick_tablet = false;
+ n = 0;
+ }
+
+ // Return true for skipping compaction
+ std::function<bool(CloudTablet*)> filter_out;
+ if (compaction_type == CompactionType::BASE_COMPACTION) {
+ filter_out = [&submitted_base_compactions, &submitted_full_compactions](CloudTablet* t) {
+ return !!submitted_base_compactions.count(t->tablet_id()) ||
+ !!submitted_full_compactions.count(t->tablet_id()) ||
+ t->tablet_state() != TABLET_RUNNING;
+ };
+ } else if (config::enable_parallel_cumu_compaction) {
+ filter_out = [&tablet_preparing_cumu_compaction](CloudTablet* t) {
+ return !!tablet_preparing_cumu_compaction.count(t->tablet_id()) ||
+ t->tablet_state() != TABLET_RUNNING;
+ };
+ } else {
+ filter_out = [&tablet_preparing_cumu_compaction,
+ &submitted_cumu_compactions](CloudTablet* t) {
+ return !!tablet_preparing_cumu_compaction.count(t->tablet_id()) ||
+ !!submitted_cumu_compactions.count(t->tablet_id()) ||
+ t->tablet_state() != TABLET_RUNNING;
+ };
+ }
+
+ // Even if need_pick_tablet is false, we still need to call find_best_tablet_to_compaction(),
+ // So that we can update the max_compaction_score metric.
+ do {
+ std::vector<CloudTabletSPtr> tablets;
+ auto st = tablet_mgr().get_topn_tablets_to_compact(n, compaction_type, filter_out, &tablets,
+ &max_compaction_score);
+ if (!st.ok()) {
+ LOG(WARNING) << "failed to get tablets to compact, err=" << st;
+ break;
+ }
+ if (!need_pick_tablet) break;
+ tablets_compaction = std::move(tablets);
+ } while (false);
+
+ if (max_compaction_score > 0) {
+ if (compaction_type == CompactionType::BASE_COMPACTION) {
+ DorisMetrics::instance()->tablet_base_max_compaction_score->set_value(
+ max_compaction_score);
+ } else {
+ DorisMetrics::instance()->tablet_cumulative_max_compaction_score->set_value(
+ max_compaction_score);
+ }
+ }
+
+ return tablets_compaction;
+}
+
+Status CloudStorageEngine::_submit_base_compaction_task(const CloudTabletSPtr& tablet) {
+ using namespace std::chrono;
+ {
+ std::lock_guard lock(_compaction_mtx);
+ // Take a placeholder for base compaction
+ auto [_, success] = _submitted_base_compactions.emplace(tablet->tablet_id(), nullptr);
+ if (!success) {
+ return Status::AlreadyExist(
+ "other base compaction or full compaction is submitted, tablet_id={}",
+ tablet->tablet_id());
+ }
+ }
+ auto compaction = std::make_shared<CloudBaseCompaction>(*this, tablet);
+ auto st = compaction->prepare_compact();
+ if (!st.ok()) {
+ long now = duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
+ tablet->set_last_base_compaction_failure_time(now);
+ std::lock_guard lock(_compaction_mtx);
+ _submitted_base_compactions.erase(tablet->tablet_id());
+ return st;
+ }
+ {
+ std::lock_guard lock(_compaction_mtx);
+ _submitted_base_compactions[tablet->tablet_id()] = compaction;
+ }
+ st = _base_compaction_thread_pool->submit_func([=, this, compaction = std::move(compaction)]() {
+ auto st = compaction->execute_compact();
+ if (!st.ok()) {
+ // Error log has been output in `execute_compact`
+ long now = duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
+ tablet->set_last_base_compaction_failure_time(now);
+ }
+ std::lock_guard lock(_compaction_mtx);
+ _submitted_base_compactions.erase(tablet->tablet_id());
+ });
+ if (!st.ok()) {
+ std::lock_guard lock(_compaction_mtx);
+ _submitted_base_compactions.erase(tablet->tablet_id());
+ return Status::InternalError("failed to submit base compaction, tablet_id={}",
+ tablet->tablet_id());
+ }
+ return st;
+}
+
+Status CloudStorageEngine::_submit_cumulative_compaction_task(const CloudTabletSPtr& tablet) {
+ using namespace std::chrono;
+ {
+ std::lock_guard lock(_compaction_mtx);
+ if (!config::enable_parallel_cumu_compaction &&
+ _submitted_cumu_compactions.count(tablet->tablet_id())) {
+ return Status::AlreadyExist("other cumu compaction is submitted, tablet_id={}",
+ tablet->tablet_id());
+ }
+ auto [_, success] = _tablet_preparing_cumu_compaction.insert(tablet->tablet_id());
+ if (!success) {
+ return Status::AlreadyExist("other cumu compaction is preparing, tablet_id={}",
+ tablet->tablet_id());
+ }
+ }
+ auto compaction = std::make_shared<CloudCumulativeCompaction>(*this, tablet);
+ auto st = compaction->prepare_compact();
+ if (!st.ok()) {
+ long now = duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
+ if (st.is<ErrorCode::CUMULATIVE_NO_SUITABLE_VERSION>()) {
+ // Backoff strategy if no suitable version
+ tablet->last_cumu_no_suitable_version_ms = now;
+ }
+ tablet->set_last_cumu_compaction_failure_time(now);
+ std::lock_guard lock(_compaction_mtx);
+ _tablet_preparing_cumu_compaction.erase(tablet->tablet_id());
+ return st;
+ }
+ {
+ std::lock_guard lock(_compaction_mtx);
+ _tablet_preparing_cumu_compaction.erase(tablet->tablet_id());
+ _submitted_cumu_compactions[tablet->tablet_id()].push_back(compaction);
+ }
+ auto erase_submitted_cumu_compaction = [=, this]() {
+ std::lock_guard lock(_compaction_mtx);
+ auto it = _submitted_cumu_compactions.find(tablet->tablet_id());
+ DCHECK(it != _submitted_cumu_compactions.end());
+ auto& compactions = it->second;
+ auto it1 = std::find(compactions.begin(), compactions.end(), compaction);
+ DCHECK(it1 != compactions.end());
+ compactions.erase(it1);
+ if (compactions.empty()) { // No compactions on this tablet, erase key
+ _submitted_cumu_compactions.erase(it);
+ // No cumu compaction on this tablet, reset `last_cumu_no_suitable_version_ms` to enable this tablet to
+ // enter the compaction scheduling candidate set. The purpose of doing this is to have at least one BE perform
+ // cumu compaction on tablet which has suitable versions for cumu compaction.
+ tablet->last_cumu_no_suitable_version_ms = 0;
+ }
+ };
+ st = _cumu_compaction_thread_pool->submit_func([=, compaction = std::move(compaction)]() {
+ auto st = compaction->execute_compact();
+ if (!st.ok()) {
+ // Error log has been output in `execute_compact`
+ long now = duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
+ tablet->set_last_cumu_compaction_failure_time(now);
+ }
+ erase_submitted_cumu_compaction();
+ });
+ if (!st.ok()) {
+ erase_submitted_cumu_compaction();
+ return Status::InternalError("failed to submit cumu compaction, tablet_id={}",
+ tablet->tablet_id());
+ }
+ return st;
+}
+
+Status CloudStorageEngine::_submit_full_compaction_task(const CloudTabletSPtr& tablet) {
+ using namespace std::chrono;
+ {
+ std::lock_guard lock(_compaction_mtx);
+ // Take a placeholder for full compaction
+ auto [_, success] = _submitted_full_compactions.emplace(tablet->tablet_id(), nullptr);
+ if (!success) {
+ return Status::AlreadyExist(
+ "other full compaction or base compaction is submitted, tablet_id={}",
+ tablet->tablet_id());
+ }
+ }
+ //auto compaction = std::make_shared<CloudFullCompaction>(tablet);
+ auto compaction = std::make_shared<CloudFullCompaction>(*this, tablet);
+ auto st = compaction->prepare_compact();
+ if (!st.ok()) {
+ long now = duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
+ tablet->set_last_full_compaction_failure_time(now);
+ std::lock_guard lock(_compaction_mtx);
+ _submitted_full_compactions.erase(tablet->tablet_id());
+ return st;
+ }
+ {
+ std::lock_guard lock(_compaction_mtx);
+ _submitted_full_compactions[tablet->tablet_id()] = compaction;
+ }
+ st = _base_compaction_thread_pool->submit_func([=, this, compaction = std::move(compaction)]() {
+ auto st = compaction->execute_compact();
+ if (!st.ok()) {
+ // Error log has been output in `execute_compact`
+ long now = duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
+ tablet->set_last_full_compaction_failure_time(now);
+ }
+ std::lock_guard lock(_compaction_mtx);
+ _submitted_full_compactions.erase(tablet->tablet_id());
+ });
+ if (!st.ok()) {
+ std::lock_guard lock(_compaction_mtx);
+ _submitted_full_compactions.erase(tablet->tablet_id());
+ return Status::InternalError("failed to submit full compaction, tablet_id={}",
+ tablet->tablet_id());
+ }
+ return st;
+}
+
+Status CloudStorageEngine::submit_compaction_task(const CloudTabletSPtr& tablet,
+ CompactionType compaction_type) {
+ DCHECK(compaction_type == CompactionType::CUMULATIVE_COMPACTION ||
+ compaction_type == CompactionType::BASE_COMPACTION ||
+ compaction_type == CompactionType::FULL_COMPACTION);
+ switch (compaction_type) {
+ case CompactionType::BASE_COMPACTION:
+ RETURN_IF_ERROR(_submit_base_compaction_task(tablet));
+ return Status::OK();
+ case CompactionType::CUMULATIVE_COMPACTION:
+ RETURN_IF_ERROR(_submit_cumulative_compaction_task(tablet));
+ return Status::OK();
+ case CompactionType::FULL_COMPACTION:
+ RETURN_IF_ERROR(_submit_full_compaction_task(tablet));
+ return Status::OK();
+ default:
+ return Status::InternalError("unknown compaction type!");
+ }
+}
+
+void CloudStorageEngine::_lease_compaction_thread_callback() {
+ while (!_stop_background_threads_latch.wait_for(
+ std::chrono::seconds(config::lease_compaction_interval_seconds))) {
+ std::vector<std::shared_ptr<CloudBaseCompaction>> base_compactions;
+ std::vector<std::shared_ptr<CloudCumulativeCompaction>> cumu_compactions;
+ {
+ std::lock_guard lock(_compaction_mtx);
+ for (auto& [_, base] : _submitted_base_compactions) {
+ if (base) { // `base` might be a nullptr placeholder
+ base_compactions.push_back(base);
+ }
+ }
+ for (auto& [_, cumus] : _submitted_cumu_compactions) {
+ for (auto& cumu : cumus) {
+ cumu_compactions.push_back(cumu);
+ }
+ }
+ }
+ // TODO(plat1ko): Support batch lease rpc
+ for (auto& comp : cumu_compactions) {
+ comp->do_lease();
+ }
+ for (auto& comp : base_compactions) {
+ comp->do_lease();
+ }
+ }
+}
+
+Status CloudStorageEngine::get_compaction_status_json(std::string* result) {
Review Comment:
warning: method 'get_compaction_status_json' can be made static [readability-convert-member-functions-to-static]
be/src/cloud/cloud_storage_engine.h:80:
```diff
- Status get_compaction_status_json(std::string* result);
+ static Status get_compaction_status_json(std::string* result);
```
##########
be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:
##########
@@ -0,0 +1,244 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/rowset/vertical_beta_rowset_writer_helper.h"
+
+#include <fmt/format.h>
+#include <gen_cpp/olap_file.pb.h>
+
+#include <algorithm>
+#include <atomic>
+#include <mutex>
+#include <ostream>
+#include <string>
+#include <utility>
+
+#include "common/compiler_util.h" // IWYU pragma: keep
+#include "common/logging.h"
+#include "io/fs/file_system.h"
+#include "io/fs/file_writer.h"
+#include "olap/rowset/beta_rowset.h"
+#include "olap/rowset/rowset_meta.h"
+#include "olap/rowset/rowset_writer_context.h"
+#include "util/slice.h"
+#include "util/spinlock.h"
+#include "vec/core/block.h"
+
+namespace doris {
+using namespace ErrorCode;
+
+VerticalBetaRowsetWriterHelper::VerticalBetaRowsetWriterHelper(
+ std::vector<std::unique_ptr<segment_v2::SegmentWriter>>* segment_writers,
+ bool already_built, RowsetMetaSharedPtr& rowset_meta, std::atomic<int32_t>* num_segment,
+ RowsetWriterContext& context, std::atomic<int64_t>* num_rows_written,
+ std::vector<KeyBoundsPB>* segments_encoded_key_bounds,
+ std::vector<uint32_t>* segment_num_rows, std::atomic<int64_t>* total_index_size,
+ std::vector<io::FileWriterPtr>* file_writers, std::atomic<int64_t>* total_data_size,
+ SpinLock* lock)
+ : _segment_writers(segment_writers),
+ _already_built(already_built),
+ _rowset_meta(rowset_meta),
+ _num_segment(num_segment),
+ _context(context),
+ _num_rows_written(num_rows_written),
+ _segments_encoded_key_bounds(segments_encoded_key_bounds),
+ _segment_num_rows(segment_num_rows),
+ _total_index_size(total_index_size),
+ _file_writers(file_writers),
+ _total_data_size(total_data_size),
+ _lock(lock) {}
+
+void VerticalBetaRowsetWriterHelper::destruct_writer() {
+ if (!_already_built) {
+ const auto& fs = _rowset_meta->fs();
+ if (!fs || !_rowset_meta->is_local()) { // Remote fs will delete them asynchronously
+ return;
+ }
+ for (auto& segment_writer : *_segment_writers) {
+ segment_writer.reset();
+ }
+ for (int i = 0; i < *_num_segment; ++i) {
+ auto path = BetaRowset::segment_file_path(_context.rowset_dir, _context.rowset_id, i);
+ // Even if an error is encountered, these files that have not been cleaned up
+ // will be cleaned up by the GC background. So here we only print the error
+ // message when we encounter an error.
+ WARN_IF_ERROR(fs->delete_file(path), fmt::format("Failed to delete file={}", path));
+ }
+ }
+}
Review Comment:
warning: function 'add_columns' has cognitive complexity of 79 (threshold 50) [readability-function-cognitive-complexity]
```cpp
Status VerticalBetaRowsetWriterHelper::add_columns(const vectorized::Block* block,
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:86:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (num_rows == 0) {
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:89:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (UNLIKELY(max_rows_per_segment > _context.max_rows_per_segment)) {
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:93:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (_segment_writers->empty()) {
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:97:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_create_segment_writer(col_ids, is_key, &writer));
^
```
**be/src/common/status.h:541:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:97:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_create_segment_writer(col_ids, is_key, &writer));
^
```
**be/src/common/status.h:543:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:100:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR((*_segment_writers)[_cur_writer_idx]->append_block(block, 0, num_rows));
^
```
**be/src/common/status.h:541:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:100:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR((*_segment_writers)[_cur_writer_idx]->append_block(block, 0, num_rows));
^
```
**be/src/common/status.h:543:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:101:** +1, nesting level increased to 1
```cpp
} else if (is_key) {
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:102:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if ((*_segment_writers)[_cur_writer_idx]->num_rows_written() > max_rows_per_segment) {
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:104:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_flush_columns(&(*_segment_writers)[_cur_writer_idx], true));
^
```
**be/src/common/status.h:541:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:104:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(_flush_columns(&(*_segment_writers)[_cur_writer_idx], true));
^
```
**be/src/common/status.h:543:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:107:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_create_segment_writer(col_ids, is_key, &writer));
^
```
**be/src/common/status.h:541:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:107:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(_create_segment_writer(col_ids, is_key, &writer));
^
```
**be/src/common/status.h:543:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:111:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR((*_segment_writers)[_cur_writer_idx]->append_block(block, 0, num_rows));
^
```
**be/src/common/status.h:541:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:111:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR((*_segment_writers)[_cur_writer_idx]->append_block(block, 0, num_rows));
^
```
**be/src/common/status.h:543:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:112:** +1, nesting level increased to 1
```cpp
} else {
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:119:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (_cur_writer_idx == 0 && num_rows_written == 0) {
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:119:** +1
```cpp
if (_cur_writer_idx == 0 && num_rows_written == 0) {
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:121:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR((*_segment_writers)[_cur_writer_idx]->init(col_ids, is_key));
^
```
**be/src/common/status.h:541:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:121:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR((*_segment_writers)[_cur_writer_idx]->init(col_ids, is_key));
^
```
**be/src/common/status.h:543:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:126:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (num_rows_written + num_rows >= num_rows_key_group &&
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:128:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR((*_segment_writers)[_cur_writer_idx]->append_block(
^
```
**be/src/common/status.h:541:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:128:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR((*_segment_writers)[_cur_writer_idx]->append_block(
^
```
**be/src/common/status.h:543:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:130:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_flush_columns(&(*_segment_writers)[_cur_writer_idx]));
^
```
**be/src/common/status.h:541:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:130:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(_flush_columns(&(*_segment_writers)[_cur_writer_idx]));
^
```
**be/src/common/status.h:543:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:135:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR((*_segment_writers)[_cur_writer_idx]->init(col_ids, is_key));
^
```
**be/src/common/status.h:541:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:135:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR((*_segment_writers)[_cur_writer_idx]->init(col_ids, is_key));
^
```
**be/src/common/status.h:543:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:139:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (limit > 0) {
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:140:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(
^
```
**be/src/common/status.h:541:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:140:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(
^
```
**be/src/common/status.h:543:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/vertical_beta_rowset_writer_helper.cpp:146:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (is_key) {
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Implement cloud compaction [doris]
Posted by "luwei16 (via GitHub)" <gi...@apache.org>.
luwei16 commented on PR #31215:
URL: https://github.com/apache/doris/pull/31215#issuecomment-1970481755
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Implement cloud compaction [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31215:
URL: https://github.com/apache/doris/pull/31215#issuecomment-1970308262
TeamCity be ut coverage result:
Function Coverage: 35.54% (8554/24068)
Line Coverage: 27.30% (69414/254277)
Region Coverage: 26.48% (35992/135924)
Branch Coverage: 23.31% (18393/78908)
Coverage Report: http://coverage.selectdb-in.cc/coverage/837504f95132bd625757e4c18634f6bca058145f_837504f95132bd625757e4c18634f6bca058145f/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Implement cloud compaction [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31215:
URL: https://github.com/apache/doris/pull/31215#issuecomment-1970903260
TeamCity be ut coverage result:
Function Coverage: 35.53% (8553/24075)
Line Coverage: 27.28% (69390/254332)
Region Coverage: 26.47% (35986/135952)
Branch Coverage: 23.30% (18388/78924)
Coverage Report: http://coverage.selectdb-in.cc/coverage/1170d4ac4a73bacfa568dadebf2e6372e31b6b81_1170d4ac4a73bacfa568dadebf2e6372e31b6b81/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Implement cloud compaction [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31215:
URL: https://github.com/apache/doris/pull/31215#issuecomment-1956152080
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Implement cloud compaction [doris]
Posted by "platoneko (via GitHub)" <gi...@apache.org>.
platoneko commented on code in PR #31215:
URL: https://github.com/apache/doris/pull/31215#discussion_r1508802051
##########
be/src/cloud/cloud_tablet.h:
##########
@@ -109,6 +109,69 @@ class CloudTablet final : public BaseTablet {
void set_cumulative_compaction_cnt(int64_t cnt) { _cumulative_compaction_cnt = cnt; }
void set_cumulative_layer_point(int64_t new_point);
+ int64_t last_cumu_compaction_failure_time() { return _last_cumu_compaction_failure_millis; }
+ void set_last_cumu_compaction_failure_time(int64_t millis) {
+ _last_cumu_compaction_failure_millis = millis;
+ }
+
+ int64_t last_base_compaction_failure_time() { return _last_base_compaction_failure_millis; }
+ void set_last_base_compaction_failure_time(int64_t millis) {
+ _last_base_compaction_failure_millis = millis;
+ }
+
+ int64_t last_full_compaction_failure_time() { return _last_full_compaction_failure_millis; }
+ void set_last_full_compaction_failure_time(int64_t millis) {
+ _last_full_compaction_failure_millis = millis;
+ }
+
+ int64_t last_cumu_compaction_success_time() { return _last_cumu_compaction_success_millis; }
+ void set_last_cumu_compaction_success_time(int64_t millis) {
+ _last_cumu_compaction_success_millis = millis;
+ }
+
+ int64_t last_base_compaction_success_time() { return _last_base_compaction_success_millis; }
+ void set_last_base_compaction_success_time(int64_t millis) {
+ _last_base_compaction_success_millis = millis;
+ }
+
+ int64_t last_full_compaction_success_time() { return _last_full_compaction_success_millis; }
+ void set_last_full_compaction_success_time(int64_t millis) {
+ _last_full_compaction_success_millis = millis;
+ }
+
+ int64_t last_base_compaction_schedule_time() { return _last_base_compaction_schedule_millis; }
+ void set_last_base_compaction_schedule_time(int64_t millis) {
+ _last_base_compaction_schedule_millis = millis;
+ }
+
+ std::vector<RowsetSharedPtr> pick_candidate_rowsets_to_base_compaction();
Review Comment:
这个函数应该可以不用了，直接用 `traverse_rowsets`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Implement cloud compaction [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31215:
URL: https://github.com/apache/doris/pull/31215#issuecomment-1970542286
TeamCity be ut coverage result:
Function Coverage: 35.54% (8553/24068)
Line Coverage: 27.29% (69406/254284)
Region Coverage: 26.47% (35985/135932)
Branch Coverage: 23.30% (18386/78914)
Coverage Report: http://coverage.selectdb-in.cc/coverage/837504f95132bd625757e4c18634f6bca058145f_837504f95132bd625757e4c18634f6bca058145f/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Implement cloud compaction [doris]
Posted by "luwei16 (via GitHub)" <gi...@apache.org>.
luwei16 commented on PR #31215:
URL: https://github.com/apache/doris/pull/31215#issuecomment-1970645065
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Implement cloud compaction [doris]
Posted by "luwei16 (via GitHub)" <gi...@apache.org>.
luwei16 commented on PR #31215:
URL: https://github.com/apache/doris/pull/31215#issuecomment-1968983956
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) Implement cloud compaction [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31215:
URL: https://github.com/apache/doris/pull/31215#issuecomment-1971043329
TeamCity be ut coverage result:
Function Coverage: 35.52% (8552/24076)
Line Coverage: 27.28% (69383/254364)
Region Coverage: 26.46% (35978/135963)
Branch Coverage: 23.29% (18382/78930)
Coverage Report: http://coverage.selectdb-in.cc/coverage/741d380bfc98f1e2d944c0d4bb2a37c595d07bd9_741d380bfc98f1e2d944c0d4bb2a37c595d07bd9/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org