You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by da...@apache.org on 2023/01/31 13:36:44 UTC
[doris] branch master updated: [enhancement](config) remove config load_process_max_memory_limit_bytes (#15686)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 6470ae58ea [enhancement](config) remove config load_process_max_memory_limit_bytes (#15686)
6470ae58ea is described below
commit 6470ae58eacd078874f95923e0f37bd6040c90c7
Author: zhannngchen <48...@users.noreply.github.com>
AuthorDate: Tue Jan 31 21:36:34 2023 +0800
[enhancement](config) remove config load_process_max_memory_limit_bytes (#15686)
---
be/src/common/config.h | 8 +-------
be/src/runtime/load_channel_mgr.cpp | 3 +--
docs/en/docs/admin-manual/config/be-config.md | 6 ------
.../maint-monitor/memory-management/memory-tracker.md | 2 +-
docs/zh-CN/docs/admin-manual/config/be-config.md | 6 ------
.../maint-monitor/memory-management/memory-tracker.md | 2 +-
6 files changed, 4 insertions(+), 23 deletions(-)
diff --git a/be/src/common/config.h b/be/src/common/config.h
index 7e796e6e62..a8ec1620ac 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -494,13 +494,7 @@ CONF_mInt64(write_buffer_size, "209715200");
// max buffer size used in memtable for the aggregated table, default 400MB
CONF_mInt64(write_buffer_size_for_agg, "419430400");
-// following 2 configs limit the memory consumption of load process on a Backend.
-// eg: memory limit to 80% of mem limit config but up to 100GB(default)
-// NOTICE(cmy): set these default values very large because we don't want to
-// impact the load performance when user upgrading Doris.
-// user should set these configs properly if necessary.
-CONF_Int64(load_process_max_memory_limit_bytes, "107374182400"); // 100GB
-CONF_Int32(load_process_max_memory_limit_percent, "50"); // 50%
+CONF_Int32(load_process_max_memory_limit_percent, "50"); // 50%
// If the memory consumption of load jobs exceed load_process_max_memory_limit,
// all load jobs will hang there to wait for memtable flush. We should have a
diff --git a/be/src/runtime/load_channel_mgr.cpp b/be/src/runtime/load_channel_mgr.cpp
index bd00718472..fca3e0c0b0 100644
--- a/be/src/runtime/load_channel_mgr.cpp
+++ b/be/src/runtime/load_channel_mgr.cpp
@@ -43,8 +43,7 @@ static int64_t calc_process_max_load_memory(int64_t process_mem_limit) {
return -1;
}
int32_t max_load_memory_percent = config::load_process_max_memory_limit_percent;
- int64_t max_load_memory_bytes = process_mem_limit * max_load_memory_percent / 100;
- return std::min<int64_t>(max_load_memory_bytes, config::load_process_max_memory_limit_bytes);
+ return process_mem_limit * max_load_memory_percent / 100;
}
static int64_t calc_channel_timeout_s(int64_t timeout_in_req_s) {
diff --git a/docs/en/docs/admin-manual/config/be-config.md b/docs/en/docs/admin-manual/config/be-config.md
index 7823885289..8eb9019061 100644
--- a/docs/en/docs/admin-manual/config/be-config.md
+++ b/docs/en/docs/admin-manual/config/be-config.md
@@ -672,12 +672,6 @@ Metrics: {"filtered_rows":0,"input_row_num":3346807,"input_rowsets_count":42,"in
* Description: The load error log will be deleted after this time
* Default value: 48 (h)
-#### `load_process_max_memory_limit_bytes`
-
-* Description: The upper limit of memory occupied by all imported threads on a single node.
- - Set these default values very large, because we don't want to affect load performance when users upgrade Doris. If necessary, the user should set these configurations correctly.
-* Default value: 107374182400 (100G)
-
#### `load_process_max_memory_limit_percent`
* Description: The percentage of the upper memory limit occupied by all imported threads on a single node, the default is 50%
diff --git a/docs/en/docs/admin-manual/maint-monitor/memory-management/memory-tracker.md b/docs/en/docs/admin-manual/maint-monitor/memory-management/memory-tracker.md
index 8fa8fa8d16..c037b350b3 100644
--- a/docs/en/docs/admin-manual/maint-monitor/memory-management/memory-tracker.md
+++ b/docs/en/docs/admin-manual/maint-monitor/memory-management/memory-tracker.md
@@ -91,7 +91,7 @@ For the memory statistics results of historical queries, you can view the `peakM
### Load Type `/mem_tracker?type=load`
-1. Limit: The import is divided into two stages: Fragment Scan and Load Channel to write Segment to disk. The upper memory limit of the Scan phase can be viewed and modified through `show session variables`; the segment write disk phase does not have a separate memory upper limit for each import, but the total upper limit of all imports, corresponding to `load_process_max_memory_limit_percent` and ` in be.conf load_process_max_memory_limit_bytes`.
+1. Limit: The import is divided into two stages: Fragment Scan and Load Channel to write Segment to disk. The upper memory limit of the Scan phase can be viewed and modified through `show session variables`; the segment write disk phase does not have a separate memory upper limit for each import, but the total upper limit of all imports, corresponding to `load_process_max_memory_limit_percent`.
2. Label: The label naming rule of a single import Scan stage Tracker is `Load#Id=xxx`; the Label naming rule of a single import Segment write disk stage Tracker is `LoadChannel#senderIp=xxx#loadID=xxx`.
3. Parent Label: Parent is the Tracker of `Load#Id=xxx`, which records the memory used by different operators during the import Scan stage; Parent is the Tracker of `LoadChannelMgrTrackerSet`, which records the Insert and The memory used by the Flush disk process is associated with the last `loadID` of the Label to write to the disk stage Tracker of the Segment.
diff --git a/docs/zh-CN/docs/admin-manual/config/be-config.md b/docs/zh-CN/docs/admin-manual/config/be-config.md
index 754c11294e..f3f337afab 100644
--- a/docs/zh-CN/docs/admin-manual/config/be-config.md
+++ b/docs/zh-CN/docs/admin-manual/config/be-config.md
@@ -686,12 +686,6 @@ Metrics: {"filtered_rows":0,"input_row_num":3346807,"input_rowsets_count":42,"in
* 描述: load错误日志将在此时间后删除
* 默认值: 48（h）
-#### `load_process_max_memory_limit_bytes`
-
-* 描述: 单节点上所有的导入线程占据的内存上限。
- - 将这些默认值设置得很大，因为我们不想在用户升级 Doris 时影响负载性能。 如有必要，用户应正确设置这些配置。
-* 默认值: 107374182400 （100G）
-
#### `load_process_max_memory_limit_percent`
* 描述: 单节点上所有的导入线程占据的内存上限比例
diff --git a/docs/zh-CN/docs/admin-manual/maint-monitor/memory-management/memory-tracker.md b/docs/zh-CN/docs/admin-manual/maint-monitor/memory-management/memory-tracker.md
index befc9d743d..b50b79f587 100644
--- a/docs/zh-CN/docs/admin-manual/maint-monitor/memory-management/memory-tracker.md
+++ b/docs/zh-CN/docs/admin-manual/maint-monitor/memory-management/memory-tracker.md
@@ -91,7 +91,7 @@ https://shimo.im/docs/DT6JXDRkdTvdyV3G
### Load Type `/mem_tracker?type=load`
-1. Limit: 导入分为 Fragment Scan 和 Load Channel 写 Segment 到磁盘两个阶段。Scan 阶段的内存上限通过`show session variables`查看和修改`load_mem_limit`；Segment 写磁盘阶段每个导入没有单独的内存上限，而是所有导入的总上限，对应 be.conf 中的 `load_process_max_memory_limit_percent`和`load_process_max_memory_limit_bytes`。
+1. Limit: 导入分为 Fragment Scan 和 Load Channel 写 Segment 到磁盘两个阶段。Scan 阶段的内存上限通过`show session variables`查看和修改`load_mem_limit`；Segment 写磁盘阶段每个导入没有单独的内存上限，而是所有导入的总上限，对应 be.conf 中的 `load_process_max_memory_limit_percent`。
2. Label: 单个导入 Scan 阶段 Tracker 的 Label 命名规则为`Load#Id=xxx`；单个导入 Segment 写磁盘阶段 Tracker 的 Label 命名规则为`LoadChannel#senderIp=xxx#loadID=xxx`。
3. Parent Label: Parent是 `Load#Id=xxx` 的 Tracker 记录导入 Scan 阶段不同算子执行过程使用的内存；Parent是 `LoadChannelMgrTrackerSet` 的 Tracker 记录 Segment 写磁盘阶段每个中间数据结构 MemTable 的 Insert 和 Flush 磁盘过程使用的内存，用 Label 最后的 `loadID` 关联 Segment 写磁盘阶段 Tracker。
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org