You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2021/07/21 09:48:51 UTC
[GitHub] [incubator-doris] huangmengbin opened a new pull request #6290: [Config] Add default configuration of load_parallelism
huangmengbin opened a new pull request #6290:
URL: https://github.com/apache/incubator-doris/pull/6290
## Proposed changes
- Make load_parallelism configurable.
- Different clusters should be configured with different load_parallelism values.
- Some user don't know how to set load_parallelism, or don't know the best load_parallelism value.
## Types of changes
configuration
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [ ] Documentation Update (if none of the other choices apply)
- [ ] Code refactor (Modify the code structure, format the code, etc...)
- [ ] Optimization. Including functional usability improvements and performance improvements.
- [ ] Dependency. Such as changes related to third-party components.
- [ ] Other.
## Checklist
_Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._
- [ ] I have created an issue on (Fix #ISSUE) and described the bug/feature there in detail
- [ ] Compiling and unit tests pass locally with my changes
- [ ] I have added tests that prove my fix is effective or that my feature works
- [x] If these changes need document changes, I have updated the document
- [x] Any dependent changes have been merged
## Further comments
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #6290: [Config] Add default configuration of load_parallelism
Posted by GitBox <gi...@apache.org>.
morningman merged pull request #6290:
URL: https://github.com/apache/incubator-doris/pull/6290
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] huangmengbin commented on a change in pull request #6290: [Config] Add default configuration of load_parallelism
Posted by GitBox <gi...@apache.org>.
huangmengbin commented on a change in pull request #6290:
URL: https://github.com/apache/incubator-doris/pull/6290#discussion_r679604253
##########
File path: docs/zh-CN/administrator-guide/config/fe_config.md
##########
@@ -1049,6 +1049,16 @@ colocote join PlanFragment instance 的 memory_limit = exec_mem_limit / min (que
导出检查器的运行间隔
+### load_parallelism
+
+默认值：1
+
+是否可以动态配置：true
+
+是否为 Master FE 节点独有的配置项：true
+
+单个节点导入的默认并发度
Review comment:
@EmmyMiao87 是的，我们团队也留意到了那段代码，不过之前没有写入文档之中。感谢你的提醒！不知道现在这种写法是否ok？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] huangmengbin commented on a change in pull request #6290: [Config] Add default configuration of load_parallelism
Posted by GitBox <gi...@apache.org>.
huangmengbin commented on a change in pull request #6290:
URL: https://github.com/apache/incubator-doris/pull/6290#discussion_r679603497
##########
File path: fe/fe-core/src/main/java/org/apache/doris/common/Config.java
##########
@@ -128,6 +128,13 @@
@ConfField(mutable = true, masterOnly = true)
public static int label_keep_max_second = 3 * 24 * 3600; // 3 days
+ /**
+ * The default parallelism of the load execution plan
+ * on a single node when the broker load is submitted
+ */
+ @ConfField(mutable = true, masterOnly = true)
+ public static int load_parallelism = 1;
Review comment:
done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #6290: [Config] Add default configuration of load_parallelism
Posted by GitBox <gi...@apache.org>.
morningman commented on pull request #6290:
URL: https://github.com/apache/incubator-doris/pull/6290#issuecomment-894763681
@EmmyMiao87 FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] huangmengbin commented on a change in pull request #6290: [Config] Add default configuration of load_parallelism
Posted by GitBox <gi...@apache.org>.
huangmengbin commented on a change in pull request #6290:
URL: https://github.com/apache/incubator-doris/pull/6290#discussion_r679604253
##########
File path: docs/zh-CN/administrator-guide/config/fe_config.md
##########
@@ -1049,6 +1049,16 @@ colocote join PlanFragment instance 的 memory_limit = exec_mem_limit / min (que
导出检查器的运行间隔
+### load_parallelism
+
+默认值：1
+
+是否可以动态配置：true
+
+是否为 Master FE 节点独有的配置项：true
+
+单个节点导入的默认并发度
Review comment:
是的，我们团队也留意到了那段代码，不过之前没有写入文档之中。感谢你的提醒！不知道现在这种写法是否ok？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] huangmengbin commented on a change in pull request #6290: [Config] Add default configuration of load_parallelism
Posted by GitBox <gi...@apache.org>.
huangmengbin commented on a change in pull request #6290:
URL: https://github.com/apache/incubator-doris/pull/6290#discussion_r679604253
##########
File path: docs/zh-CN/administrator-guide/config/fe_config.md
##########
@@ -1049,6 +1049,16 @@ colocote join PlanFragment instance 的 memory_limit = exec_mem_limit / min (que
导出检查器的运行间隔
+### load_parallelism
+
+默认值：1
+
+是否可以动态配置：true
+
+是否为 Master FE 节点独有的配置项：true
+
+单个节点导入的默认并发度
Review comment:
@EmmyMiao87 是的，我们团队也留意到了那段代码，不过之前没有写入文档之中。感谢你的提醒！不知道现在这种写法是否ok？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] huangmengbin commented on a change in pull request #6290: [Config] Add default configuration of load_parallelism
Posted by GitBox <gi...@apache.org>.
huangmengbin commented on a change in pull request #6290:
URL: https://github.com/apache/incubator-doris/pull/6290#discussion_r679604253
##########
File path: docs/zh-CN/administrator-guide/config/fe_config.md
##########
@@ -1049,6 +1049,16 @@ colocote join PlanFragment instance 的 memory_limit = exec_mem_limit / min (que
导出检查器的运行间隔
+### load_parallelism
+
+默认值：1
+
+是否可以动态配置：true
+
+是否为 Master FE 节点独有的配置项：true
+
+单个节点导入的默认并发度
Review comment:
是的，我们团队也留意到了那段代码，不过之前没有写入文档之中。感谢你的指正！不知道新的写法是否ok？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #6290: [Config] Add default configuration of load_parallelism
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #6290:
URL: https://github.com/apache/incubator-doris/pull/6290#issuecomment-894763676
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] EmmyMiao87 commented on a change in pull request #6290: [Config] Add default configuration of load_parallelism
Posted by GitBox <gi...@apache.org>.
EmmyMiao87 commented on a change in pull request #6290:
URL: https://github.com/apache/incubator-doris/pull/6290#discussion_r677086844
##########
File path: fe/fe-core/src/main/java/org/apache/doris/common/Config.java
##########
@@ -128,6 +128,13 @@
@ConfField(mutable = true, masterOnly = true)
public static int label_keep_max_second = 3 * 24 * 3600; // 3 days
+ /**
+ * The default parallelism of the load execution plan
+ * on a single node when the broker load is submitted
+ */
+ @ConfField(mutable = true, masterOnly = true)
+ public static int load_parallelism = 1;
Review comment:
```suggestion
public static int default_load_parallelism = 1;
```
##########
File path: docs/zh-CN/administrator-guide/config/fe_config.md
##########
@@ -1049,6 +1049,16 @@ colocote join PlanFragment instance 的 memory_limit = exec_mem_limit / min (que
导出检查器的运行间隔
+### load_parallelism
+
+默认值：1
+
+是否可以动态配置：true
+
+是否为 Master FE 节点独有的配置项：true
+
+单个节点导入的默认并发度
Review comment:
首先这里要强调，这是一个默认的导入并发度。如果用户指定自定义的并发度，则使用自定义并发度。
其次导入任务的并发度取决于多个参数，这只是其中之一，参见 BrokerScanNode 的 line 355 ~ 362
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org