You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2021/04/25 11:39:28 UTC
[GitHub] [incubator-doris] morningman opened a new pull request #5703: [Feature] Support create history dynamic partition
morningman opened a new pull request #5703:
URL: https://github.com/apache/incubator-doris/pull/5703
## Proposed changes
1. Add a new dynamic partition property `create_history_partition`.
If set to true, Doris will create all partitions from `start` to `end`.
2. Add a new FE config `max_dynamic_partition_num`
To limit the number of partitions created when creating one table.
## Types of changes
What types of changes does your code introduce to Doris?
_Put an `x` in the boxes that apply_
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [ ] Documentation Update (if none of the other choices apply)
- [ ] Code refactor (Modify the code structure, format the code, etc...)
## Checklist
_Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._
- [ ] I have created an issue on (Fix #ISSUE) and described the bug/feature there in detail
- [ ] Compiling and unit tests pass locally with my changes
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] If these changes need document changes, I have updated the document
- [ ] Any dependent changes have been merged
## Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] EmmyMiao87 commented on a change in pull request #5703: [Feature] Support create history dynamic partition
Posted by GitBox <gi...@apache.org>.
EmmyMiao87 commented on a change in pull request #5703:
URL: https://github.com/apache/incubator-doris/pull/5703#discussion_r628110103
##########
File path: fe/fe-core/src/main/java/org/apache/doris/common/Config.java
##########
@@ -1367,4 +1367,12 @@
*/
@ConfField
public static int grpc_max_message_size_bytes = 1 * 1024 * 1024 * 1024; // 1GB
+
+ /**
+ * Used to limit the maximum number of partitions that can be created when creating a dynamic partition table,
+ * to avoid creating too many partitions at one time.
+ * The number is determined by "start" and "end" in the dynamic partition parameters.
+ */
+ @ConfField(mutable = true, masterOnly = true)
+ public static int max_dynamic_partition_num = 500;
Review comment:
新配置加个文档说明吧
##########
File path: docs/zh-CN/administrator-guide/dynamic-partition.md
##########
@@ -122,6 +122,12 @@ under the License.
* `dynamic_partition.start_day_of_month`
当 `time_unit` 为 `MONTH` 时，该参数用于指定每月的起始日期。取值为 1 到 28。其中 1 表示每月1号，28 表示每月28号。默认为 1，即表示每月以1号位起始点。暂不支持以29、30、31号为起始日，以避免因闰年或闰月带来的歧义。
+
+* `dynamic_partition.create_history_partition`
+
+ 默认为 false。当置为 true 时，Doris 会自动创建由 start 到 end 的所有分区。同时，FE 的参数 `max_dynamic_partition_num` 会限制总分区数量，以避免一次性创建过多分区。
Review comment:
如果 max_dynamic_partition_num 大于 end - start 则无法创建动态分区，并且报错xxx。描述一下
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #5703: [Feature] Support create history dynamic partition
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #5703:
URL: https://github.com/apache/incubator-doris/pull/5703#discussion_r628687111
##########
File path: docs/zh-CN/administrator-guide/dynamic-partition.md
##########
@@ -122,6 +122,12 @@ under the License.
* `dynamic_partition.start_day_of_month`
当 `time_unit` 为 `MONTH` 时，该参数用于指定每月的起始日期。取值为 1 到 28。其中 1 表示每月1号，28 表示每月28号。默认为 1，即表示每月以1号位起始点。暂不支持以29、30、31号为起始日，以避免因闰年或闰月带来的歧义。
+
+* `dynamic_partition.create_history_partition`
+
+ 默认为 false。当置为 true 时，Doris 会自动创建由 start 到 end 的所有分区。同时，FE 的参数 `max_dynamic_partition_num` 会限制总分区数量，以避免一次性创建过多分区。
Review comment:
OK
##########
File path: fe/fe-core/src/main/java/org/apache/doris/common/Config.java
##########
@@ -1367,4 +1367,12 @@
*/
@ConfField
public static int grpc_max_message_size_bytes = 1 * 1024 * 1024 * 1024; // 1GB
+
+ /**
+ * Used to limit the maximum number of partitions that can be created when creating a dynamic partition table,
+ * to avoid creating too many partitions at one time.
+ * The number is determined by "start" and "end" in the dynamic partition parameters.
+ */
+ @ConfField(mutable = true, masterOnly = true)
+ public static int max_dynamic_partition_num = 500;
Review comment:
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #5703: [Feature] Support create history dynamic partition
Posted by GitBox <gi...@apache.org>.
morningman merged pull request #5703:
URL: https://github.com/apache/incubator-doris/pull/5703
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] EmmyMiao87 commented on a change in pull request #5703: [Feature] Support create history dynamic partition
Posted by GitBox <gi...@apache.org>.
EmmyMiao87 commented on a change in pull request #5703:
URL: https://github.com/apache/incubator-doris/pull/5703#discussion_r628688739
##########
File path: docs/zh-CN/administrator-guide/dynamic-partition.md
##########
@@ -122,6 +122,12 @@ under the License.
* `dynamic_partition.start_day_of_month`
当 `time_unit` 为 `MONTH` 时，该参数用于指定每月的起始日期。取值为 1 到 28。其中 1 表示每月1号，28 表示每月28号。默认为 1，即表示每月以1号位起始点。暂不支持以29、30、31号为起始日，以避免因闰年或闰月带来的歧义。
+
+* `dynamic_partition.create_history_partition`
+
+ 默认为 false。当置为 true 时，Doris 会自动创建由 start 到 end 的所有分区。同时，FE 的参数 `max_dynamic_partition_num` 会限制总分区数量，以避免一次性创建过多分区。当 end - start 的值大于 `max_dynamic_partition_num` 值时，操作将被禁止。
+
+ 当不指定 `start` 属性时，该参数不生效。
Review comment:
缩进问题
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org