You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/06/01 10:00:12 UTC
[GitHub] [incubator-doris] yixiutt opened a new pull request, #9915: [feature](compaction) add some optimize for compaction
yixiutt opened a new pull request, #9915:
URL: https://github.com/apache/incubator-doris/pull/9915
1. seperate base and cu compaction thread pool
origin thread pool may all be held by base compaction, a case is
be has 8 dirs, each dir can have 4 task at the same time, even
every dir will reserve one slot for cumu compaction, (4-1)*8=24 is
much more bigger than max compaction thread num 10.
2.set base compaction idle schedule to run in low priority
3.remove base compaction for dup key if no delete_predict applied,
base compaction waste a lot of cpu&io for dup model.
4.add a max_rowset_num limit for base compaction in case base compaction
run too long
# Proposed changes
Issue Number: close #xxx
## Problem Summary:
Describe the overview of changes.
## Checklist(Required)
1. Does it affect the original behavior: (Yes/No/I Don't know)
2. Has unit tests been added: (Yes/No/No Need)
3. Has document been added or modified: (Yes/No/No Need)
4. Does it need to update dependencies: (Yes/No)
5. Are there any changes that cannot be rolled back: (Yes/No)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yixiutt commented on a diff in pull request #9915: [feature](compaction) add some optimize for compaction
Posted by GitBox <gi...@apache.org>.
yixiutt commented on code in PR #9915:
URL: https://github.com/apache/incubator-doris/pull/9915#discussion_r887444794
##########
be/src/olap/compaction.cpp:
##########
@@ -158,6 +158,8 @@ Status Compaction::do_compaction_impl(int64_t permits) {
<< ". tablet=" << _tablet->full_name() << ", output_version=" << _output_version
<< ", current_max_version=" << current_max_version
<< ", disk=" << _tablet->data_dir()->path() << ", segments=" << segments_num
+ << ", input_row_num=" << _input_row_num
+ << ", output_row_num=" << _output_rowset->num_rows()
Review Comment:
we can do it later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a diff in pull request #9915: [feature](compaction) add some optimize for compaction
Posted by GitBox <gi...@apache.org>.
morningman commented on code in PR #9915:
URL: https://github.com/apache/incubator-doris/pull/9915#discussion_r887004196
##########
be/src/olap/tablet_manager.cpp:
##########
@@ -636,6 +636,13 @@ TabletSharedPtr TabletManager::find_best_tablet_to_compaction(
<< ", tablet_id=" << tablet_ptr->tablet_id();
continue;
}
+ if (compaction_type == CompactionType::BASE_COMPACTION &&
Review Comment:
We should make sure that the rowset is large enough to not perform base comapction. Instead of simply prohibiting base compaction of duplicate tables. Otherwise we might end up with a lot of small files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yixiutt closed pull request #9915: [feature](compaction) add some optimize for compaction
Posted by GitBox <gi...@apache.org>.
yixiutt closed pull request #9915: [feature](compaction) add some optimize for compaction
URL: https://github.com/apache/incubator-doris/pull/9915
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yixiutt commented on a diff in pull request #9915: [feature](compaction) add some optimize for compaction
Posted by GitBox <gi...@apache.org>.
yixiutt commented on code in PR #9915:
URL: https://github.com/apache/incubator-doris/pull/9915#discussion_r887438098
##########
be/src/olap/olap_server.cpp:
##########
@@ -304,6 +309,46 @@ void StorageEngine::_tablet_checkpoint_callback(const std::vector<DataDir*>& dat
} while (!_stop_background_threads_latch.wait_for(std::chrono::seconds(interval)));
}
+void StorageEngine::_adjust_compaction_thread_num() {
+ if (_base_compaction_thread_pool->max_threads() != config::max_base_compaction_threads) {
+ int old_max_threads = _base_compaction_thread_pool->max_threads();
+ Status status =
+ _base_compaction_thread_pool->set_max_threads(config::max_base_compaction_threads);
+ if (status.ok()) {
+ LOG(INFO) << "update base compaction thread pool max_threads from " << old_max_threads
+ << " to " << config::max_base_compaction_threads;
+ }
+ }
+ if (_base_compaction_thread_pool->min_threads() != config::max_base_compaction_threads) {
+ int old_min_threads = _base_compaction_thread_pool->min_threads();
+ Status status =
+ _base_compaction_thread_pool->set_min_threads(config::max_base_compaction_threads);
+ if (status.ok()) {
+ LOG(INFO) << "update base compaction thread pool min_threads from " << old_min_threads
+ << " to " << config::max_base_compaction_threads;
+ }
+ }
+
+ if (_cumu_compaction_thread_pool->max_threads() != config::max_cumu_compaction_threads) {
+ int old_max_threads = _cumu_compaction_thread_pool->max_threads();
+ Status status =
+ _cumu_compaction_thread_pool->set_max_threads(config::max_cumu_compaction_threads);
+ if (status.ok()) {
+ LOG(INFO) << "update cumu compaction thread pool max_threads from " << old_max_threads
Review Comment:
ok
##########
be/src/olap/olap_server.cpp:
##########
@@ -304,6 +309,46 @@ void StorageEngine::_tablet_checkpoint_callback(const std::vector<DataDir*>& dat
} while (!_stop_background_threads_latch.wait_for(std::chrono::seconds(interval)));
}
+void StorageEngine::_adjust_compaction_thread_num() {
+ if (_base_compaction_thread_pool->max_threads() != config::max_base_compaction_threads) {
+ int old_max_threads = _base_compaction_thread_pool->max_threads();
+ Status status =
+ _base_compaction_thread_pool->set_max_threads(config::max_base_compaction_threads);
+ if (status.ok()) {
+ LOG(INFO) << "update base compaction thread pool max_threads from " << old_max_threads
+ << " to " << config::max_base_compaction_threads;
+ }
+ }
+ if (_base_compaction_thread_pool->min_threads() != config::max_base_compaction_threads) {
+ int old_min_threads = _base_compaction_thread_pool->min_threads();
+ Status status =
+ _base_compaction_thread_pool->set_min_threads(config::max_base_compaction_threads);
+ if (status.ok()) {
+ LOG(INFO) << "update base compaction thread pool min_threads from " << old_min_threads
+ << " to " << config::max_base_compaction_threads;
+ }
+ }
+
+ if (_cumu_compaction_thread_pool->max_threads() != config::max_cumu_compaction_threads) {
+ int old_max_threads = _cumu_compaction_thread_pool->max_threads();
+ Status status =
+ _cumu_compaction_thread_pool->set_max_threads(config::max_cumu_compaction_threads);
+ if (status.ok()) {
+ LOG(INFO) << "update cumu compaction thread pool max_threads from " << old_max_threads
+ << " to " << config::max_cumu_compaction_threads;
+ }
+ }
+ if (_cumu_compaction_thread_pool->min_threads() != config::max_cumu_compaction_threads) {
+ int old_min_threads = _cumu_compaction_thread_pool->min_threads();
+ Status status =
+ _cumu_compaction_thread_pool->set_min_threads(config::max_cumu_compaction_threads);
+ if (status.ok()) {
+ LOG(INFO) << "update cumu compaction thread pool min_threads from " << old_min_threads
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yixiutt commented on a diff in pull request #9915: [feature](compaction) add some optimize for compaction
Posted by GitBox <gi...@apache.org>.
yixiutt commented on code in PR #9915:
URL: https://github.com/apache/incubator-doris/pull/9915#discussion_r887442783
##########
be/src/olap/tablet_manager.cpp:
##########
@@ -636,6 +636,13 @@ TabletSharedPtr TabletManager::find_best_tablet_to_compaction(
<< ", tablet_id=" << tablet_ptr->tablet_id();
continue;
}
+ if (compaction_type == CompactionType::BASE_COMPACTION &&
Review Comment:
i'll fix it, hanlde it in pick rowset, filter big rowset if no delete exist
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] Gabriel39 commented on a diff in pull request #9915: [feature](compaction) add some optimize for compaction
Posted by GitBox <gi...@apache.org>.
Gabriel39 commented on code in PR #9915:
URL: https://github.com/apache/incubator-doris/pull/9915#discussion_r887425174
##########
be/src/olap/olap_server.cpp:
##########
@@ -304,6 +309,46 @@ void StorageEngine::_tablet_checkpoint_callback(const std::vector<DataDir*>& dat
} while (!_stop_background_threads_latch.wait_for(std::chrono::seconds(interval)));
}
+void StorageEngine::_adjust_compaction_thread_num() {
+ if (_base_compaction_thread_pool->max_threads() != config::max_base_compaction_threads) {
+ int old_max_threads = _base_compaction_thread_pool->max_threads();
+ Status status =
+ _base_compaction_thread_pool->set_max_threads(config::max_base_compaction_threads);
+ if (status.ok()) {
+ LOG(INFO) << "update base compaction thread pool max_threads from " << old_max_threads
+ << " to " << config::max_base_compaction_threads;
+ }
+ }
+ if (_base_compaction_thread_pool->min_threads() != config::max_base_compaction_threads) {
+ int old_min_threads = _base_compaction_thread_pool->min_threads();
+ Status status =
+ _base_compaction_thread_pool->set_min_threads(config::max_base_compaction_threads);
+ if (status.ok()) {
+ LOG(INFO) << "update base compaction thread pool min_threads from " << old_min_threads
+ << " to " << config::max_base_compaction_threads;
+ }
+ }
+
+ if (_cumu_compaction_thread_pool->max_threads() != config::max_cumu_compaction_threads) {
+ int old_max_threads = _cumu_compaction_thread_pool->max_threads();
+ Status status =
+ _cumu_compaction_thread_pool->set_max_threads(config::max_cumu_compaction_threads);
+ if (status.ok()) {
+ LOG(INFO) << "update cumu compaction thread pool max_threads from " << old_max_threads
Review Comment:
Change to debug level log?
##########
be/src/olap/compaction.cpp:
##########
@@ -158,6 +158,8 @@ Status Compaction::do_compaction_impl(int64_t permits) {
<< ". tablet=" << _tablet->full_name() << ", output_version=" << _output_version
<< ", current_max_version=" << current_max_version
<< ", disk=" << _tablet->data_dir()->path() << ", segments=" << segments_num
+ << ", input_row_num=" << _input_row_num
+ << ", output_row_num=" << _output_rowset->num_rows()
Review Comment:
Should we also print filtered rows and merged rows?
##########
be/src/olap/olap_server.cpp:
##########
@@ -304,6 +309,46 @@ void StorageEngine::_tablet_checkpoint_callback(const std::vector<DataDir*>& dat
} while (!_stop_background_threads_latch.wait_for(std::chrono::seconds(interval)));
}
+void StorageEngine::_adjust_compaction_thread_num() {
+ if (_base_compaction_thread_pool->max_threads() != config::max_base_compaction_threads) {
+ int old_max_threads = _base_compaction_thread_pool->max_threads();
+ Status status =
+ _base_compaction_thread_pool->set_max_threads(config::max_base_compaction_threads);
+ if (status.ok()) {
+ LOG(INFO) << "update base compaction thread pool max_threads from " << old_max_threads
+ << " to " << config::max_base_compaction_threads;
+ }
+ }
+ if (_base_compaction_thread_pool->min_threads() != config::max_base_compaction_threads) {
+ int old_min_threads = _base_compaction_thread_pool->min_threads();
+ Status status =
+ _base_compaction_thread_pool->set_min_threads(config::max_base_compaction_threads);
+ if (status.ok()) {
+ LOG(INFO) << "update base compaction thread pool min_threads from " << old_min_threads
+ << " to " << config::max_base_compaction_threads;
+ }
+ }
+
+ if (_cumu_compaction_thread_pool->max_threads() != config::max_cumu_compaction_threads) {
+ int old_max_threads = _cumu_compaction_thread_pool->max_threads();
+ Status status =
+ _cumu_compaction_thread_pool->set_max_threads(config::max_cumu_compaction_threads);
+ if (status.ok()) {
+ LOG(INFO) << "update cumu compaction thread pool max_threads from " << old_max_threads
+ << " to " << config::max_cumu_compaction_threads;
+ }
+ }
+ if (_cumu_compaction_thread_pool->min_threads() != config::max_cumu_compaction_threads) {
+ int old_min_threads = _cumu_compaction_thread_pool->min_threads();
+ Status status =
+ _cumu_compaction_thread_pool->set_min_threads(config::max_cumu_compaction_threads);
+ if (status.ok()) {
+ LOG(INFO) << "update cumu compaction thread pool min_threads from " << old_min_threads
Review Comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org