You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by ji...@apache.org on 2022/07/27 07:42:58 UTC
[doris-website] branch master updated: config
This is an automated email from the ASF dual-hosted git repository.
jiafengzheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new d8c8faa8661 config
d8c8faa8661 is described below
commit d8c8faa86619e776675137b326f45cdd9fe8b688
Author: jiafeng.zhang <zh...@gmail.com>
AuthorDate: Wed Jul 27 15:42:48 2022 +0800
config
---
docs/admin-manual/config/be-config.md | 49 +++++++++++++---------
docs/admin-manual/config/fe-config.md | 10 ++++-
.../current/admin-manual/config/be-config.md | 49 +++++++++++++---------
.../current/admin-manual/config/fe-config.md | 10 ++++-
4 files changed, 74 insertions(+), 44 deletions(-)
diff --git a/docs/admin-manual/config/be-config.md b/docs/admin-manual/config/be-config.md
index f16194f7b3b..0e1aec9be4b 100644
--- a/docs/admin-manual/config/be-config.md
+++ b/docs/admin-manual/config/be-config.md
@@ -672,11 +672,17 @@ Default: 10737418240
BloomFilter/Min/Max and other statistical information cache capacity
+### `kafka_api_version_request`
+
+Default: true
+
+If the dependent Kafka version is lower than 0.10.0.0, this value should be set to false.
+
### `kafka_broker_version_fallback`
Default: 0.10.0
-If the dependent Kafka version is lower than the Kafka client version that routine load depends on, the value set by the fallback version kafka_broker_version_fallback will be used, and the valid values are: 0.9.0, 0.8.2, 0.8.1, 0.8.0.
+If the dependent Kafka version is lower than 0.10.0.0, the value set by the fallback version kafka_broker_version_fallback will be used if the value of kafka_api_version_request is set to false, and the valid values are: 0.9.0.x, 0.8.x.y.
### `load_data_reserve_hours`
@@ -1437,36 +1443,18 @@ The size of the buffer before flashing
* Default: 3
-### `track_new_delete`
+### `enable_tcmalloc_hook`
* Type: bool
* Description: Whether Hook TCmalloc new/delete, currently consume/release tls mem tracker in Hook.
* Default: true
-### `mem_tracker_level`
-
-* Type: int16
-* Description: The level at which MemTracker is displayed on the Web page equal or lower than this level will be displayed on the Web page
- ```
- OVERVIEW = 0
- TASK = 1
- INSTANCE = 2
- VERBOSE = 3
- ```
-* Default: 0
-
### `mem_tracker_consume_min_size_bytes`
* Type: int32
* Description: The minimum length of TCMalloc Hook when consume/release MemTracker. Consume size smaller than this value will continue to accumulate to avoid frequent calls to consume/release of MemTracker. Decreasing this value will increase the frequency of consume/release. Increasing this value will cause MemTracker statistics to be inaccurate. Theoretically, the statistical value of a MemTracker differs from the true value = ( mem_tracker_consume_min_size_bytes * the number of BE thr [...]
* Default: 1048576
-### `memory_leak_detection`
-
-* Type: bool
-* Description: Whether to start memory leak detection, when MemTracker is a negative value, it is considered that a memory leak has occurred, but the actual MemTracker records inaccurately will also cause a negative value, so this feature is in the experimental stage.
-* Default: false
-
### `max_segment_num_per_rowset`
* Type: int32
@@ -1531,3 +1519,24 @@ Translated with www.DeepL.com/Translator (free version)
* Type: int32
* Description: The maximum amount of data read by each OlapScanner.
* Default: 1024
+
+### `enable_quick_compaction`
+* Type: bool
+* Description: enable quick compaction,It is mainly used in the scenario of frequent import of small amount of data. The problem of -235 can be effectively avoided by merging the imported versions in time through the mechanism of rapid compaction. The definition of small amount of data is currently defined according to the number of rows
+* Default: false
+
+### `quick_compaction_max_rows`
+* Type: int32
+* Description: When the number of imported rows is less than this value, it is considered that this import is an import of small amount of data, which will be selected during quick compaction
+* Default: 1000
+
+### `quick_compaction_batch_size`
+* Type: int32
+* Description: trigger time, when import times reach quick_compaction_batch_size will trigger immediately
+* Default: 10
+
+### `quick_compaction_min_rowsets`
+* Type: int32
+* Description: at least the number of versions to be compaction, and the number of rowsets with a small amount of data in the selection. If it is greater than this value, the real compaction will be carried out
+* Default: 10
+
diff --git a/docs/admin-manual/config/fe-config.md b/docs/admin-manual/config/fe-config.md
index f37e64564ee..13c6e8374f5 100644
--- a/docs/admin-manual/config/fe-config.md
+++ b/docs/admin-manual/config/fe-config.md
@@ -1458,6 +1458,12 @@ MasterOnly：true
Default broker load timeout
+### broker_timeout_ms
+
+Default：10000 （10s）
+
+Default broker RPC timeout
+
### load_running_job_num_limit
Default：0
@@ -2078,11 +2084,11 @@ Whether to allow multiple replicas of the same tablet to be distributed on the s
### min_version_count_indicate_replica_compaction_too_slow
-Default: 300
+Default: 200
Dynamically configured: true
-Only for Master FE: true
+Only for Master FE: false
The version count threshold used to judge whether replica compaction is too slow
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/config/be-config.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/config/be-config.md
index 8e41779a83f..fc7cc62b94c 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/config/be-config.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/config/be-config.md
@@ -673,11 +673,17 @@ load tablets from header failed, failed tablets size: xxx, path=xxx
BloomFilter/Min/Max等统计信息缓存的容量
+### `kafka_api_version_request`
+
+默认值：true
+
+如果依赖的 kafka 版本低于0.10.0.0, 该值应该被设置为 false。
+
### `kafka_broker_version_fallback`
默认值：0.10.0
-如果依赖的 kafka 版本低于routine load依赖的 kafka 客户端版本, 将使用回退版本 kafka_broker_version_fallback 设置的值，有效值为：0.9.0、0.8.2、0.8.1、0.8.0。
+如果依赖的 kafka 版本低于0.10.0.0, 当 kafka_api_version_request 值为 false 的时候，将使用回退版本 kafka_broker_version_fallback 设置的值，有效值为：0.9.0.x、0.8.x.y。
### `load_data_reserve_hours`
@@ -1456,36 +1462,18 @@ webserver默认工作线程数
```
* 默认值: 3
-### `track_new_delete`
+### `enable_tcmalloc_hook`
* 类型：bool
* 描述：是否Hook TCmalloc new/delete，目前在Hook中统计thread local MemTracker。
* 默认值：true
-### `mem_tracker_level`
-
-* 类型: int16
-* 描述: MemTracker在Web页面上展示的级别，等于或低于这个级别的MemTracker会在Web页面上展示
- ```
- OVERVIEW = 0
- TASK = 1
- INSTANCE = 2
- VERBOSE = 3
- ```
-* 默认值: 0
-
### `mem_tracker_consume_min_size_bytes`
* 类型: int32
* 描述: TCMalloc Hook consume/release MemTracker时的最小长度，小于该值的consume size会持续累加，避免频繁调用MemTracker的consume/release，减小该值会增加consume/release的频率，增大该值会导致MemTracker统计不准，理论上一个MemTracker的统计值与真实值相差 = (mem_tracker_consume_min_size_bytes * 这个MemTracker所在的BE线程数)。
* 默认值: 1048576
-### `memory_leak_detection`
-
-* 类型: bool
-* 描述: 是否启动内存泄漏检测，当 MemTracker 为负值时认为发生了内存泄漏，但实际 MemTracker 记录不准确时也会导致负值，所以这个功能处于实验阶段。
-* 默认值: false
-
### `max_segment_num_per_rowset`
* 类型: int32
@@ -1553,3 +1541,24 @@ webserver默认工作线程数
* 类型: int32
* 描述: String 类型最大长度的软限，单位是字节
* 默认值: 1048576
+
+### `enable_quick_compaction`
+* 类型: bool
+* 描述: 是否开启quick_compaction,主要用在小数据量频繁导入的场景,通过快速compaction的机制及时合并导入版本可以有效避免-235的问题，小数据量的定义目前是根据行数来定义
+* 默认值: false
+
+### `quick_compaction_max_rows`
+* 类型: int32
+* 描述: 当导入的行数小于这个值认为这次导入是小数据量的导入，在快速合并时会被选中
+* 默认值: 1000
+
+### `quick_compaction_batch_size`
+* 类型: int32
+* 描述: 快速合并的触发时机，导入次数达到quick_compaction_batch_size时触发一次
+* 默认值: 10
+
+### `quick_compaction_min_rowsets`
+* 类型: int32
+* 描述: 最少进行合并的版本数，当选中的小数据量的rowset个数，大于这个值是才会进行真正的合并
+* 默认值: 10
+
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/config/fe-config.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/config/fe-config.md
index b99d1569a19..bd09d69e539 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/config/fe-config.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/config/fe-config.md
@@ -1467,6 +1467,12 @@ load 最大超时时间，适用于除 stream load 之外的所有类型的加
Broker load 的默认超时时间
+### `broker_timeout_ms`
+
+默认值：10000 （10秒）
+
+Broker rpc 的默认超时时间
+
### `load_running_job_num_limit`
默认值：0
@@ -2123,11 +2129,11 @@ load 标签清理器将每隔 `label_clean_interval_second` 运行一次以清
### `min_version_count_indicate_replica_compaction_too_slow`
-默认值：300
+默认值：200
是否可以动态配置：true
-是否为 Master FE 节点独有的配置项：true
+是否为 Master FE 节点独有的配置项：false
版本计数阈值，用来判断副本做 compaction 的速度是否太慢
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org