You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/06/27 03:01:25 UTC
[GitHub] [doris] yixiutt opened a new pull request, #10434: [Optimize](Compaction) compaction task producer add quiet period
yixiutt opened a new pull request, #10434:
URL: https://github.com/apache/doris/pull/10434
In some case, if a dir has no more compaction tasks, it still
scan tablet meta to find best tablet to try compaction, which
makes compaction task producer thread always run full。
Add a quiet period if a dir continuely fails to do compaction to
optimize this case.
case:
# Proposed changes
Issue Number: close #xxx
## Problem Summary:
Describe the overview of changes.
## Checklist(Required)
1. Does it affect the original behavior: (Yes/No/I Don't know)
2. Has unit tests been added: (Yes/No/No Need)
3. Has document been added or modified: (Yes/No/No Need)
4. Does it need to update dependencies: (Yes/No)
5. Are there any changes that cannot be rolled back: (Yes/No)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] compiletheworld commented on a diff in pull request #10434: [Optimize](Compaction) compaction task producer add quiet period
Posted by GitBox <gi...@apache.org>.
compiletheworld commented on code in PR #10434:
URL: https://github.com/apache/doris/pull/10434#discussion_r907993041
##########
be/src/olap/data_dir.h:
##########
@@ -139,6 +139,41 @@ class DataDir {
// storage_root/trash/20150619154308.delete_counter/tablet_path/segment_path/tablet_uid
Status move_to_trash(const FilePathDesc& segment_path_desc);
+ // add data_dir consecutive prepared failed times to make it
+ // goes to quiet period to save cpu cost when no tablet needs to compact
+ void set_tablet_prepare_compact_failed(int64_t now, bool failed) {
+ std::unique_lock<std::mutex> lck(_compaction_quiet_mutex);
+ if (!failed) {
+ _compaction_last_prepared_failed_ts = 0;
+ _compaction_quiet_period_s = 0;
+ _compaction_sleep_cv.notify_one();
+ } else {
+ if (_compaction_last_prepared_failed_ts == 0) {
+ _compaction_last_prepared_failed_ts = now;
+ return;
+ }
+ if (_compaction_last_prepared_failed_ts != 0 &&
+ now - _compaction_last_prepared_failed_ts >=
+ config::max_prepare_failure_interval_before_quiet) {
+ _compaction_quiet_period_s = now + config::data_dir_quiet_period_s;
+ _compaction_last_prepared_failed_ts = 0;
+ }
+ }
+ }
+ void wait_in_quiet_period(int64_t wait_ms) {
+ std::unique_lock<std::mutex> lock(_compaction_quiet_mutex);
+ // It is necessary to wake up the thread on timeout to prevent deadlock
+ // in case of no running compaction task.
+ _compaction_sleep_cv.wait_for(lock, std::chrono::milliseconds(wait_ms),
+ [=] { return _compaction_quiet_period_s == 0; });
Review Comment:
Explicitly capturing `this` is straightforward and better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] yixiutt commented on a diff in pull request #10434: [Optimize](Compaction) compaction task producer add quiet period
Posted by GitBox <gi...@apache.org>.
yixiutt commented on code in PR #10434:
URL: https://github.com/apache/doris/pull/10434#discussion_r908005218
##########
be/src/olap/data_dir.h:
##########
@@ -139,6 +139,41 @@ class DataDir {
// storage_root/trash/20150619154308.delete_counter/tablet_path/segment_path/tablet_uid
Status move_to_trash(const FilePathDesc& segment_path_desc);
+ // add data_dir consecutive prepared failed times to make it
+ // goes to quiet period to save cpu cost when no tablet needs to compact
+ void set_tablet_prepare_compact_failed(int64_t now, bool failed) {
+ std::unique_lock<std::mutex> lck(_compaction_quiet_mutex);
+ if (!failed) {
+ _compaction_last_prepared_failed_ts = 0;
+ _compaction_quiet_period_s = 0;
+ _compaction_sleep_cv.notify_one();
+ } else {
+ if (_compaction_last_prepared_failed_ts == 0) {
+ _compaction_last_prepared_failed_ts = now;
+ return;
+ }
+ if (_compaction_last_prepared_failed_ts != 0 &&
+ now - _compaction_last_prepared_failed_ts >=
+ config::max_prepare_failure_interval_before_quiet) {
+ _compaction_quiet_period_s = now + config::data_dir_quiet_period_s;
+ _compaction_last_prepared_failed_ts = 0;
+ }
+ }
+ }
+ void wait_in_quiet_period(int64_t wait_ms) {
+ std::unique_lock<std::mutex> lock(_compaction_quiet_mutex);
+ // It is necessary to wake up the thread on timeout to prevent deadlock
+ // in case of no running compaction task.
+ _compaction_sleep_cv.wait_for(lock, std::chrono::milliseconds(wait_ms),
+ [=] { return _compaction_quiet_period_s == 0; });
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] yixiutt closed pull request #10434: [Optimize](Compaction) compaction task producer add quiet period
Posted by GitBox <gi...@apache.org>.
yixiutt closed pull request #10434: [Optimize](Compaction) compaction task producer add quiet period
URL: https://github.com/apache/doris/pull/10434
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org