You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/10/19 14:15:51 UTC
[GitHub] [doris] yixiutt opened a new pull request, #13495: [enhancement](compaction) opt compaction task producer and quick compaction
yixiutt opened a new pull request, #13495:
URL: https://github.com/apache/doris/pull/13495
1.remove quick_compaction's rowset pick policy, call cu compaction when trigger quick compaction
2. skip tablet's compaction task when compaction score is too small
# Proposed changes
Issue Number: close #xxx
## Problem summary
Describe your changes.
## Checklist(Required)
1. Does it affect the original behavior:
- [ ] Yes
- [ ] No
- [ ] I don't know
2. Has unit tests been added:
- [ ] Yes
- [ ] No
- [ ] No Need
3. Has document been added or modified:
- [ ] Yes
- [ ] No
- [ ] No Need
4. Does it need to update dependencies:
- [ ] Yes
- [ ] No
5. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [ ] No
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #13495: [enhancement](compaction) opt compaction task producer and quick compaction
Posted by GitBox <gi...@apache.org>.
dataroaring merged PR #13495:
URL: https://github.com/apache/doris/pull/13495
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13495: [enhancement](compaction) opt compaction task producer and quick compaction
Posted by GitBox <gi...@apache.org>.
hello-stephen commented on PR #13495:
URL: https://github.com/apache/doris/pull/13495#issuecomment-1284807380
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 39.94 seconds
load time: 591 seconds
storage size: 17154768597 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221020100542_clickbench_pr_31632.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zhannngchen commented on a diff in pull request #13495: [enhancement](compaction) opt compaction task producer and quick compaction
Posted by GitBox <gi...@apache.org>.
zhannngchen commented on code in PR #13495:
URL: https://github.com/apache/doris/pull/13495#discussion_r1006681881
##########
be/src/common/config.h:
##########
@@ -253,59 +253,41 @@ CONF_Bool(enable_vectorized_compaction, "true");
// whether enable vectorized schema change/material-view/rollup task.
CONF_Bool(enable_vectorized_alter_table, "true");
-// check the configuration of auto compaction in seconds when auto compaction disabled
-CONF_mInt32(check_auto_compaction_interval_seconds, "5");
+// This config can be set to limit thread number in compaction thread pool.
+CONF_mInt32(max_base_compaction_threads, "4");
+CONF_mInt32(max_cumu_compaction_threads, "10");
-CONF_mInt64(base_compaction_num_cumulative_deltas, "5");
-CONF_mDouble(base_cumulative_delta_ratio, "0.3");
-CONF_mInt64(base_compaction_interval_seconds_since_last_operation, "86400");
-CONF_mInt32(base_compaction_write_mbytes_per_sec, "5");
CONF_Bool(enable_base_compaction_idle_sched, "true");
-
-// dup key not compaction big files
-CONF_Bool(enable_dup_key_base_compaction_skip_big_file, "true");
+CONF_mInt64(base_compaction_min_rowset_num, "5");
+CONF_mDouble(base_compaction_min_data_ratio, "0.3");
CONF_mInt64(base_compaction_dup_key_max_file_size_mbytes, "1024");
-// In size_based policy, output rowset of cumulative compaction total disk size exceed this config size,
+// output rowset of cumulative compaction total disk size exceed this config size,
// this rowset will be given to base compaction, unit is m byte.
-CONF_mInt64(cumulative_size_based_promotion_size_mbytes, "1024");
Review Comment:
We can clean some useless configs, but we'd better not to change the config name for compatibility consideration? @morningman pls help to review this kind of change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] freemandealer commented on pull request #13495: [enhancement](compaction) opt compaction task producer and quick compaction
Posted by GitBox <gi...@apache.org>.
freemandealer commented on PR #13495:
URL: https://github.com/apache/doris/pull/13495#issuecomment-1298417747
This pr will break somethings:
1. disable_auto_compaction setting to true won't stop CumuThread working as expected
2. page cache is used when doing compaction. see the print below:
```
(gdb) bt
#0 doris::StoragePageCache::insert (this=this@entry=0x55eb5ef87630 <doris::StoragePageCache::create_global_cache(unsigned long, int, unsigned int)::instance>, key=..., data=...,
handle=handle@entry=0x7ff503b30440, page_type=doris::segment_v2::INDEX_PAGE, in_memory=false) at /mnt/disk1/zhangzhengyu/project/doris-build/be/src/olap/page_cache.cpp:66
#1 0x000055eb586e7457 in doris::segment_v2::PageIO::read_and_decompress_page (opts=..., handle=0x55eb6ddb3308, body=body@entry=0x7ff503b305c0, footer=footer@entry=0x7ff503b30640)
at /mnt/disk1/zhangzhengyu/project/doris-build/be/src/olap/rowset/segment_v2/page_io.cpp:204
#2 0x000055eb586f85f1 in operator() (__closure=__closure@entry=0x7ff503b308e8) at /mnt/disk1/zhangzhengyu/project/doris-build/be/src/olap/rowset/segment_v2/segment.cpp:208
#3 0x000055eb586f8c3a in operator() (__closure=0x7ff503b308e0) at /mnt/disk1/zhangzhengyu/project/doris-build/be/src/util/once.h:57
#4 std::__invoke_impl<void, doris::DorisCallOnce<doris::Status>::call<doris::segment_v2::Segment::load_index()::<lambda()> >(doris::segment_v2::Segment::load_index()::<lambda()>)::<lambda()> > (__f=...)
at /mnt/disk1/zhangzhengyu/build-dep/ldb_toolchain/include/c++/11/bits/invoke.h:61
#5 std::__invoke<doris::DorisCallOnce<doris::Status>::call<doris::segment_v2::Segment::load_index()::<lambda()> >(doris::segment_v2::Segment::load_index()::<lambda()>)::<lambda()> > (__fn=...)
at /mnt/disk1/zhangzhengyu/build-dep/ldb_toolchain/include/c++/11/bits/invoke.h:96
#6 operator() (__closure=<optimized out>) at /mnt/disk1/zhangzhengyu/build-dep/ldb_toolchain/include/c++/11/mutex:776
#7 operator() (__closure=<optimized out>) at /mnt/disk1/zhangzhengyu/build-dep/ldb_toolchain/include/c++/11/mutex:712
#8 _FUN () at /mnt/disk1/zhangzhengyu/build-dep/ldb_toolchain/include/c++/11/mutex:712
#9 0x00007ff5e3bcd4df in __pthread_once_slow (once_control=0x55eb6ddb32ac, init_routine=0x55eb5e675600 <__once_proxy>) at pthread_once.c:116
#10 0x000055eb586f6aeb in __gthread_once (__func=<optimized out>, __once=0x55eb6ddb32ac) at /mnt/disk1/zhangzhengyu/build-dep/ldb_toolchain/include/x86_64-linux-gnu/c++/11/bits/gthr-default.h:700
#11 std::call_once<doris::DorisCallOnce<doris::Status>::call<doris::segment_v2::Segment::load_index()::<lambda()> >(doris::segment_v2::Segment::load_index()::<lambda()>)::<lambda()> > (__f=..., __once=...)
at /mnt/disk1/zhangzhengyu/build-dep/ldb_toolchain/include/c++/11/mutex:783
#12 doris::DorisCallOnce<doris::Status>::call<doris::segment_v2::Segment::load_index()::<lambda()> > (fn=..., this=0x55eb6ddb32a8) at /mnt/disk1/zhangzhengyu/project/doris-build/be/src/util/once.h:57
#13 doris::segment_v2::Segment::load_index (this=0x55eb6ddb3180) at /mnt/disk1/zhangzhengyu/project/doris-build/be/src/olap/rowset/segment_v2/segment.cpp:218
#14 doris::segment_v2::Segment::new_iterator (this=0x55eb6ddb3180, schema=..., read_options=..., iter=iter@entry=0x7ff503b30c00)
at /mnt/disk1/zhangzhengyu/project/doris-build/be/src/olap/rowset/segment_v2/segment.cpp:112
#15 0x000055eb589ca230 in doris::BetaRowsetReader::init (this=0x55ec2a761a40, read_context=0x7ff503b313f8) at /mnt/disk1/zhangzhengyu/build-dep/ldb_toolchain/include/c++/11/bits/shared_ptr_base.h:1290
#16 0x000055eb5b70526b in doris::vectorized::BlockReader::_init_collect_iter (this=this@entry=0x7ff503b313a0, read_params=..., valid_rs_readers=valid_rs_readers@entry=0x7ff503b30f90)
at /mnt/disk1/zhangzhengyu/build-dep/ldb_toolchain/include/c++/11/bits/shared_ptr_base.h:1290
#17 0x000055eb5b70924b in doris::vectorized::BlockReader::init (this=this@entry=0x7ff503b313a0, read_params=...) at /mnt/disk1/zhangzhengyu/project/doris-build/be/src/vec/olap/block_reader.cpp:136
#18 0x000055eb583ca14a in doris::Merger::vmerge_rowsets (tablet=..., reader_type=<optimized out>, cur_tablet_schema=..., src_rowset_readers=..., dst_rowset_writer=dst_rowset_writer@entry=0x55ebcb7aa000,
stats_output=0x7ff503b31a90) at /mnt/disk1/zhangzhengyu/project/doris-build/be/src/olap/merger.cpp:138
#19 0x000055eb583c2ffc in doris::Compaction::do_compaction_impl (this=this@entry=0x55edd5833e00, permits=permits@entry=24) at /mnt/disk1/zhangzhengyu/build-dep/ldb_toolchain/include/c++/11/ext/atomicity.h:109
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13495: [enhancement](compaction) opt compaction task producer and quick compaction
Posted by GitBox <gi...@apache.org>.
hello-stephen commented on PR #13495:
URL: https://github.com/apache/doris/pull/13495#issuecomment-1285557846
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 41.14 seconds
load time: 594 seconds
storage size: 17154768704 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221020213919_clickbench_pr_32056.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zhannngchen commented on pull request #13495: [enhancement](compaction) opt compaction task producer and quick compaction
Posted by GitBox <gi...@apache.org>.
zhannngchen commented on PR #13495:
URL: https://github.com/apache/doris/pull/13495#issuecomment-1293308573
Can you provide some test result for this change?
e.g. under the high frequency load scenario, like flink connector, is there regression comparing with quick compaction?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] dataroaring commented on a diff in pull request #13495: [enhancement](compaction) opt compaction task producer and quick compaction
Posted by GitBox <gi...@apache.org>.
dataroaring commented on code in PR #13495:
URL: https://github.com/apache/doris/pull/13495#discussion_r1000184979
##########
be/src/olap/delta_writer.cpp:
##########
@@ -114,9 +114,8 @@ Status DeltaWriter::init() {
// check tablet version number
if (_tablet->version_count() > config::max_tablet_version_num) {
Review Comment:
Should we trigger a compaction before encountering an error. e.g.
if (_tablet->version_count() > config::max_tablet_version_num - 10) {
// submit a compaction task?
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] yixiutt commented on a diff in pull request #13495: [enhancement](compaction) opt compaction task producer and quick compaction
Posted by GitBox <gi...@apache.org>.
yixiutt commented on code in PR #13495:
URL: https://github.com/apache/doris/pull/13495#discussion_r1000322963
##########
be/src/olap/delta_writer.cpp:
##########
@@ -114,9 +114,8 @@ Status DeltaWriter::init() {
// check tablet version number
if (_tablet->version_count() > config::max_tablet_version_num) {
Review Comment:
add a logic:
if (_tablet->version_count() > config::max_tablet_version_num - 100) trigger compaction.
And I‘ve test this patch with flink checkpoint 1s and 20 concurrency stream load, compaction is under 40
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] dataroaring commented on a diff in pull request #13495: [enhancement](compaction) opt compaction task producer and quick compaction
Posted by GitBox <gi...@apache.org>.
dataroaring commented on code in PR #13495:
URL: https://github.com/apache/doris/pull/13495#discussion_r1000659159
##########
docs/en/docs/admin-manual/config/be-config.md:
##########
@@ -117,13 +117,7 @@ Default: true
Whether to enable vectorized compaction
-### `base_compaction_interval_seconds_since_last_operation`
-
-Default: 86400
-
-One of the triggering conditions of BaseCompaction: the interval since the last BaseCompaction
-
-### `base_compaction_num_cumulative_deltas`
+### `base_compaction_min_rowset_num`
Default: 5
Review Comment:
The limit of the number of Cumulative files to be reached. After reaching this limit, BaseCompaction will be triggered. Clumulative files == rowsets?
##########
be/src/common/config.h:
##########
@@ -253,59 +253,41 @@ CONF_Bool(enable_vectorized_compaction, "true");
// whether enable vectorized schema change/material-view/rollup task.
CONF_Bool(enable_vectorized_alter_table, "true");
-// check the configuration of auto compaction in seconds when auto compaction disabled
-CONF_mInt32(check_auto_compaction_interval_seconds, "5");
+// This config can be set to limit thread number in compaction thread pool.
+CONF_mInt32(max_base_compaction_threads, "4");
+CONF_mInt32(max_cumu_compaction_threads, "10");
-CONF_mInt64(base_compaction_num_cumulative_deltas, "5");
-CONF_mDouble(base_cumulative_delta_ratio, "0.3");
-CONF_mInt64(base_compaction_interval_seconds_since_last_operation, "86400");
-CONF_mInt32(base_compaction_write_mbytes_per_sec, "5");
CONF_Bool(enable_base_compaction_idle_sched, "true");
-
-// dup key not compaction big files
-CONF_Bool(enable_dup_key_base_compaction_skip_big_file, "true");
+CONF_mInt64(base_compaction_min_rowset_num, "5");
+CONF_mDouble(base_compaction_min_data_ratio, "0.3");
CONF_mInt64(base_compaction_dup_key_max_file_size_mbytes, "1024");
-// In size_based policy, output rowset of cumulative compaction total disk size exceed this config size,
+// output rowset of cumulative compaction total disk size exceed this config size,
// this rowset will be given to base compaction, unit is m byte.
-CONF_mInt64(cumulative_size_based_promotion_size_mbytes, "1024");
+CONF_mInt64(compaction_promotion_size_mbytes, "1024");
-// In size_based policy, output rowset of cumulative compaction total disk size exceed this config ratio of
+// output rowset of cumulative compaction total disk size exceed this config ratio of
// base rowset's total disk size, this rowset will be given to base compaction. The value must be between
// 0 and 1.
-CONF_mDouble(cumulative_size_based_promotion_ratio, "0.05");
+CONF_mDouble(compaction_promotion_ratio, "0.05");
-// In size_based policy, the smallest size of rowset promotion. When the rowset is less than this config, this
+// the smallest size of rowset promotion. When the rowset is less than this config, this
// rowset will be not given to base compaction. The unit is m byte.
-CONF_mInt64(cumulative_size_based_promotion_min_size_mbytes, "64");
+CONF_mInt64(compaction_promotion_min_size_mbytes, "64");
// The lower bound size to do cumulative compaction. When total disk size of candidate rowsets is less than
// this size, size_based policy may not do to cumulative compaction. The unit is m byte.
-CONF_mInt64(cumulative_size_based_compaction_lower_size_mbytes, "64");
+CONF_mInt64(compaction_min_size_mbytes, "64");
// cumulative compaction policy: min and max delta file's number
-CONF_mInt64(min_cumulative_compaction_num_singleton_deltas, "5");
-CONF_mInt64(max_cumulative_compaction_num_singleton_deltas, "1000");
-
-// if compaction of a tablet failed, this tablet should not be chosen to
-// compaction until this interval passes.
-CONF_mInt64(min_compaction_failure_interval_sec, "5"); // 5 seconds
-
-// This config can be set to limit thread number in compaction thread pool.
-CONF_mInt32(max_base_compaction_threads, "4");
-CONF_mInt32(max_cumu_compaction_threads, "10");
-
-// This config can be set to limit thread number in smallcompaction thread pool.
-CONF_mInt32(quick_compaction_max_threads, "10");
-
-// Thread count to do tablet meta checkpoint, -1 means use the data directories count.
-CONF_Int32(max_meta_checkpoint_threads, "-1");
+CONF_mInt64(cumulative_compaction_min_deltas, "5");
+CONF_mInt64(cumulative_compaction_max_deltas, "1000");
Review Comment:
Can we rename deltas to segments?
##########
be/src/common/config.h:
##########
@@ -253,59 +253,41 @@ CONF_Bool(enable_vectorized_compaction, "true");
// whether enable vectorized schema change/material-view/rollup task.
CONF_Bool(enable_vectorized_alter_table, "true");
-// check the configuration of auto compaction in seconds when auto compaction disabled
-CONF_mInt32(check_auto_compaction_interval_seconds, "5");
+// This config can be set to limit thread number in compaction thread pool.
+CONF_mInt32(max_base_compaction_threads, "4");
+CONF_mInt32(max_cumu_compaction_threads, "10");
-CONF_mInt64(base_compaction_num_cumulative_deltas, "5");
-CONF_mDouble(base_cumulative_delta_ratio, "0.3");
-CONF_mInt64(base_compaction_interval_seconds_since_last_operation, "86400");
-CONF_mInt32(base_compaction_write_mbytes_per_sec, "5");
CONF_Bool(enable_base_compaction_idle_sched, "true");
-
-// dup key not compaction big files
-CONF_Bool(enable_dup_key_base_compaction_skip_big_file, "true");
+CONF_mInt64(base_compaction_min_rowset_num, "5");
+CONF_mDouble(base_compaction_min_data_ratio, "0.3");
CONF_mInt64(base_compaction_dup_key_max_file_size_mbytes, "1024");
-// In size_based policy, output rowset of cumulative compaction total disk size exceed this config size,
+// output rowset of cumulative compaction total disk size exceed this config size,
// this rowset will be given to base compaction, unit is m byte.
-CONF_mInt64(cumulative_size_based_promotion_size_mbytes, "1024");
+CONF_mInt64(compaction_promotion_size_mbytes, "1024");
-// In size_based policy, output rowset of cumulative compaction total disk size exceed this config ratio of
+// output rowset of cumulative compaction total disk size exceed this config ratio of
// base rowset's total disk size, this rowset will be given to base compaction. The value must be between
// 0 and 1.
-CONF_mDouble(cumulative_size_based_promotion_ratio, "0.05");
+CONF_mDouble(compaction_promotion_ratio, "0.05");
-// In size_based policy, the smallest size of rowset promotion. When the rowset is less than this config, this
+// the smallest size of rowset promotion. When the rowset is less than this config, this
// rowset will be not given to base compaction. The unit is m byte.
-CONF_mInt64(cumulative_size_based_promotion_min_size_mbytes, "64");
+CONF_mInt64(compaction_promotion_min_size_mbytes, "64");
// The lower bound size to do cumulative compaction. When total disk size of candidate rowsets is less than
// this size, size_based policy may not do to cumulative compaction. The unit is m byte.
-CONF_mInt64(cumulative_size_based_compaction_lower_size_mbytes, "64");
+CONF_mInt64(compaction_min_size_mbytes, "64");
Review Comment:
And the config is strange, if we insert small data each, e.g. 1row, then all rowset would not be compacted until total size reached 64MB, however the version num is limited by 2000?
##########
be/src/olap/cumulative_compaction.cpp:
##########
@@ -143,8 +142,7 @@ Status CumulativeCompaction::pick_rowsets_to_compact() {
int64_t last_cumu = _tablet->last_cumu_compaction_success_time();
int64_t last_base = _tablet->last_base_compaction_success_time();
if (last_cumu != 0 || last_base != 0) {
- int64_t interval_threshold =
- config::base_compaction_interval_seconds_since_last_operation * 1000;
Review Comment:
we should define a const instead of using 86400 in different places.
##########
be/src/common/config.h:
##########
@@ -253,59 +253,41 @@ CONF_Bool(enable_vectorized_compaction, "true");
// whether enable vectorized schema change/material-view/rollup task.
CONF_Bool(enable_vectorized_alter_table, "true");
-// check the configuration of auto compaction in seconds when auto compaction disabled
-CONF_mInt32(check_auto_compaction_interval_seconds, "5");
+// This config can be set to limit thread number in compaction thread pool.
+CONF_mInt32(max_base_compaction_threads, "4");
+CONF_mInt32(max_cumu_compaction_threads, "10");
-CONF_mInt64(base_compaction_num_cumulative_deltas, "5");
-CONF_mDouble(base_cumulative_delta_ratio, "0.3");
-CONF_mInt64(base_compaction_interval_seconds_since_last_operation, "86400");
-CONF_mInt32(base_compaction_write_mbytes_per_sec, "5");
CONF_Bool(enable_base_compaction_idle_sched, "true");
-
-// dup key not compaction big files
-CONF_Bool(enable_dup_key_base_compaction_skip_big_file, "true");
+CONF_mInt64(base_compaction_min_rowset_num, "5");
+CONF_mDouble(base_compaction_min_data_ratio, "0.3");
CONF_mInt64(base_compaction_dup_key_max_file_size_mbytes, "1024");
-// In size_based policy, output rowset of cumulative compaction total disk size exceed this config size,
+// output rowset of cumulative compaction total disk size exceed this config size,
// this rowset will be given to base compaction, unit is m byte.
-CONF_mInt64(cumulative_size_based_promotion_size_mbytes, "1024");
+CONF_mInt64(compaction_promotion_size_mbytes, "1024");
-// In size_based policy, output rowset of cumulative compaction total disk size exceed this config ratio of
+// output rowset of cumulative compaction total disk size exceed this config ratio of
// base rowset's total disk size, this rowset will be given to base compaction. The value must be between
// 0 and 1.
-CONF_mDouble(cumulative_size_based_promotion_ratio, "0.05");
+CONF_mDouble(compaction_promotion_ratio, "0.05");
-// In size_based policy, the smallest size of rowset promotion. When the rowset is less than this config, this
+// the smallest size of rowset promotion. When the rowset is less than this config, this
// rowset will be not given to base compaction. The unit is m byte.
-CONF_mInt64(cumulative_size_based_promotion_min_size_mbytes, "64");
+CONF_mInt64(compaction_promotion_min_size_mbytes, "64");
// The lower bound size to do cumulative compaction. When total disk size of candidate rowsets is less than
// this size, size_based policy may not do to cumulative compaction. The unit is m byte.
-CONF_mInt64(cumulative_size_based_compaction_lower_size_mbytes, "64");
+CONF_mInt64(compaction_min_size_mbytes, "64");
Review Comment:
If it is used for cu, we'd better use cumulative_compaction_min_size_mbytes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] yixiutt commented on pull request #13495: [enhancement](compaction) opt compaction task producer and quick compaction
Posted by GitBox <gi...@apache.org>.
yixiutt commented on PR #13495:
URL: https://github.com/apache/doris/pull/13495#issuecomment-1293312142
> Can you provide some test result for this change? e.g. under the high frequency load scenario, like flink connector, is there regression comparing with quick compaction?
I've test in flink high frequency load such as 20 thread load, 1s checkpoint and that compaction score is under 40
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org