You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by da...@apache.org on 2023/01/13 15:12:48 UTC
[doris] branch master updated: [conf](compaction) enable vertical_compaction ordered_data_compaction (#14945)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new d8990522fb [conf](compaction) enable vertical_compaction ordered_data_compaction (#14945)
d8990522fb is described below
commit d8990522fbb5be4941a791b6136f91e81af771f2
Author: yixiutt <10...@users.noreply.github.com>
AuthorDate: Fri Jan 13 23:12:42 2023 +0800
[conf](compaction) enable vertical_compaction ordered_data_compaction (#14945)
---
be/src/common/config.h | 6 ++--
be/src/olap/compaction.cpp | 4 +--
docs/en/docs/admin-manual/config/be-config.md | 42 ++++++++++++++++++++----
docs/zh-CN/docs/admin-manual/config/be-config.md | 42 ++++++++++++++++++++----
4 files changed, 77 insertions(+), 17 deletions(-)
diff --git a/be/src/common/config.h b/be/src/common/config.h
index e3abebd903..e0eb895175 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -265,15 +265,15 @@ CONF_Bool(enable_vectorized_compaction, "true");
// whether enable vectorized schema change/material-view/rollup task.
CONF_Bool(enable_vectorized_alter_table, "true");
// whether enable vertical compaction
-CONF_mBool(enable_vertical_compaction, "false");
+CONF_mBool(enable_vertical_compaction, "true");
// whether enable ordered data compaction
-CONF_mBool(enable_ordered_data_compaction, "false");
+CONF_mBool(enable_ordered_data_compaction, "true");
// In vertical compaction, column number for every group
CONF_mInt32(vertical_compaction_num_columns_per_group, "5");
// In vertical compaction, max memory usage for row_source_buffer
CONF_Int32(vertical_compaction_max_row_source_memory_mb, "200");
// In vertical compaction, max dest segment file size
-CONF_mInt64(max_segment_size_in_vertical_compaction, "268435456");
+CONF_mInt64(vertical_compaction_max_segment_size, "268435456");
// In ordered data compaction, min segment size for input rowset
CONF_mInt32(ordered_data_compaction_min_segment_size, "10485760");
diff --git a/be/src/olap/compaction.cpp b/be/src/olap/compaction.cpp
index e2c7473f5f..76c7fc3374 100644
--- a/be/src/olap/compaction.cpp
+++ b/be/src/olap/compaction.cpp
@@ -100,8 +100,8 @@ int64_t Compaction::get_avg_segment_rows() {
// take care of empty rowset
// input_rowsets_size is total disk_size of input_rowset, this size is the
// final size after codec and compress, so expect dest segment file size
- // in disk is config::max_segment_size_in_vertical_compaction
- return config::max_segment_size_in_vertical_compaction /
+ // in disk is config::vertical_compaction_max_segment_size
+ return config::vertical_compaction_max_segment_size /
(_input_rowsets_size / (_input_row_num + 1) + 1);
}
diff --git a/docs/en/docs/admin-manual/config/be-config.md b/docs/en/docs/admin-manual/config/be-config.md
index 8339bd6c1b..d9d32f5edc 100644
--- a/docs/en/docs/admin-manual/config/be-config.md
+++ b/docs/en/docs/admin-manual/config/be-config.md
@@ -445,6 +445,42 @@ There are two ways to configure BE configuration items:
* Description: Whether to enable vectorized compaction
* Default value: true
+#### `enable_vertical_compaction`
+
+* Type: bool
+* Description: Whether to enable vertical compaction
+* Default value: true
+
+#### `vertical_compaction_num_columns_per_group`
+
+* Type: bool
+* Description: In vertical compaction, column number for every group
+* Default value: true
+
+#### `vertical_compaction_max_row_source_memory_mb`
+
+* Type: bool
+* Description: In vertical compaction, max memory usage for row_source_buffer
+* Default value: true
+
+#### `vertical_compaction_max_segment_size`
+
+* Type: bool
+* Description: In vertical compaction, max dest segment file size
+* Default value: true
+
+#### `enable_ordered_data_compaction`
+
+* Type: bool
+* Description: Whether to enable ordered data compaction
+* Default value: true
+
+#### `ordered_data_compaction_min_segment_size`
+
+* Type: bool
+* Description: In ordered data compaction, min segment size for input rowset
+* Default value: true
+
#### `max_base_compaction_threads`
* Type: int32
@@ -473,12 +509,6 @@ There are two ways to configure BE configuration items:
* Default value: 10000
* Dynamically modifiable: Yes
-#### `compaction_tablet_compaction_score_factor`
-
-* Type: int32
-* Description: Coefficient for compaction score when calculating tablet score to find a tablet for compaction.
-* Default value: 1
-
#### `compaction_promotion_size_mbytes`
* Type: int64
diff --git a/docs/zh-CN/docs/admin-manual/config/be-config.md b/docs/zh-CN/docs/admin-manual/config/be-config.md
index b05dc0bb76..17be7394af 100644
--- a/docs/zh-CN/docs/admin-manual/config/be-config.md
+++ b/docs/zh-CN/docs/admin-manual/config/be-config.md
@@ -459,6 +459,42 @@ BE 重启后该配置将失效。如果想持久化修改结果，使用如下
* 描述：是否开启向量化compaction
* 默认值：true
+#### `enable_vertical_compaction`
+
+* 类型: bool
+* 描述: 是否开启列式compaction
+* 默认值: true
+
+#### `vertical_compaction_num_columns_per_group`
+
+* 类型: bool
+* 描述: 在列式compaction中, 组成一个合并组的列个数
+* 默认值: true
+
+#### `vertical_compaction_max_row_source_memory_mb`
+
+* 类型: bool
+* 描述: 在列式compaction中, row_source_buffer能使用的最大内存
+* 默认值: true
+
+#### `vertical_compaction_max_segment_size`
+
+* 类型: bool
+* 描述: 在列式compaction中, 输出的segment文件最大值
+* 默认值: true
+
+#### `enable_ordered_data_compaction`
+
+* 类型: bool
+* 描述: 是否开启有序数据的compaction
+* 默认值: true
+
+#### `ordered_data_compaction_min_segment_size`
+
+* 类型: bool
+* 描述: 在有序数据compaction中, 满足要求的最小segment大小
+* 默认值: true
+
#### `max_base_compaction_threads`
* 类型：int32
@@ -487,12 +523,6 @@ BE 重启后该配置将失效。如果想持久化修改结果，使用如下
* 默认值：10000
* 可动态修改：是
-#### `compaction_tablet_compaction_score_factor`
-
-* 类型：int32
-* 描述：选择tablet进行compaction时，计算 tablet score 的公式中 compaction score的权重。
-* 默认值：1
-
#### `compaction_promotion_size_mbytes`
* 类型：int64
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org