You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by mo...@apache.org on 2021/07/14 06:02:45 UTC
[incubator-doris] branch master updated: [Doc] Optimize dynamic
partition docs (#6217)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new f4c5c6c [Doc] Optimize dynamic partition docs (#6217)
f4c5c6c is described below
commit f4c5c6ccc650012a0db7ddda8a38f4c65cc5c9be
Author: qiye <ji...@gmail.com>
AuthorDate: Wed Jul 14 14:02:33 2021 +0800
[Doc] Optimize dynamic partition docs (#6217)
optimize dynamic partition docs.
add feature introduced in #6129
---
docs/en/administrator-guide/dynamic-partition.md | 58 ++++++++++++++++++++-
.../zh-CN/administrator-guide/dynamic-partition.md | 60 ++++++++++++++++++++--
2 files changed, 113 insertions(+), 5 deletions(-)
diff --git a/docs/en/administrator-guide/dynamic-partition.md b/docs/en/administrator-guide/dynamic-partition.md
index cc5a12e..9a947dd 100644
--- a/docs/en/administrator-guide/dynamic-partition.md
+++ b/docs/en/administrator-guide/dynamic-partition.md
@@ -127,13 +127,13 @@ The rules of dynamic partition are prefixed with `dynamic_partition.`:
* `dynamic_partition.create_history_partition`
- The default is false. When set to true, Doris will automatically create all partitions from start to end. At the same time, the parameter `max_dynamic_partition_num` of FE will limit the total number of partitions to avoid creating too many partitions at once. If (end - start) is larger than `max_dynamic_partition_num`, the operation will fail.
+ The default is false. When set to true, Doris will automatically create all partitions, as described in the creation rules below. At the same time, the parameter `max_dynamic_partition_num` of FE will limit the total number of partitions to avoid creating too many partitions at once. When the number of partitions expected to be created is greater than `max_dynamic_partition_num`, the operation will fail.
When the `start` attribute is not specified, this parameter has no effect.
* `dynamic_partition.history_partition_num`
- When `create_history_partition` is `true`, this parameter is used to specify the number of history partitions.
+ When `create_history_partition` is `true`, this parameter is used to specify the number of history partitions. The default value is -1, which means it is not set.
* `dynamic_partition.hot_partition_num`
@@ -151,6 +151,60 @@ The rules of dynamic partition are prefixed with `dynamic_partition.`:
p20210523: ["2021-05-23", "2021-05-24") storage_medium=SSD storage_cooldown_time=2021-05-25 00:00:00
```
+#### Create History Partition Rules
+
+When `create_history_partition` is `true`, i.e. history partition creation is enabled, Doris determines the number of history partitions to be created based on `dynamic_partition.start` and `dynamic_partition.history_partition_num`.
+
+Assuming the number of history partitions to be created is `expect_create_partition_num`, the number is as follows according to different settings.
+
+1. `create_history_partition` = `true`
+ - `dynamic_partition.history_partition_num` is not set, i.e. -1.
+ `expect_create_partition_num` = `end` - `start`;
+
+ - `dynamic_partition.history_partition_num` is set
+ `expect_create_partition_num` = `end` - max(`start`, `-histoty_partition_num`);
+
+2. `create_history_partition` = `false`
+ No history partition will be created, `expect_create_partition_num` = `end` - 0;
+
+When `expect_create_partition_num` is greater than `max_dynamic_partition_num` (default 500), creating too many partitions is prohibited.
+
+**Examples:**
+
+1. Suppose today is 2021-05-20, partition by day, and the attributes of dynamic partition are set to `create_history_partition=true, end=3, start=-3, history_partition_num=1`, then the system will automatically create the following partitions.
+
+ ```
+ p20210519
+ p20210520
+ p20210521
+ p20210522
+ p20210523
+ ```
+
+2. `history_partition_num=5` and keep the rest attributes as in 1, then the system will automatically create the following partitions.
+
+ ```
+ p20210517
+ p20210518
+ p20210519
+ p20210520
+ p20210521
+ p20210522
+ p20210523
+ ```
+
+3. `history_partition_num=-1` i.e., if you do not set the number of history partitions and keep the rest of the attributes as in 1, the system will automatically create the following partitions.
+
+ ```
+ p20210517
+ p20210518
+ p20210519
+ p20210520
+ p20210521
+ p20210522
+ p20210523
+ ```
+
### Notice
If some partitions between `dynamic_partition.start` and `dynamic_partition.end` are lost due to some unexpected circumstances when using dynamic partition, the lost partitions between the current time and `dynamic_partition.end` will be recreated, but the lost partitions between `dynamic_partition.start` and the current time will not be recreated.
diff --git a/docs/zh-CN/administrator-guide/dynamic-partition.md b/docs/zh-CN/administrator-guide/dynamic-partition.md
index 2c49d3d..76589c6 100644
--- a/docs/zh-CN/administrator-guide/dynamic-partition.md
+++ b/docs/zh-CN/administrator-guide/dynamic-partition.md
@@ -125,13 +125,13 @@ under the License.
* `dynamic_partition.create_history_partition`
- 默认为 false。当置为 true 时，Doris 会自动创建由 start 到 end 的所有分区。同时，FE 的参数 `max_dynamic_partition_num` 会限制总分区数量，以避免一次性创建过多分区。当 end - start 的值大于 `max_dynamic_partition_num` 值时，操作将被禁止。
+ 默认为 false。当置为 true 时，Doris 会自动创建所有分区，具体创建规则见下文。同时，FE 的参数 `max_dynamic_partition_num` 会限制总分区数量，以避免一次性创建过多分区。当期望创建的分区个数大于 `max_dynamic_partition_num` 值时，操作将被禁止。
当不指定 `start` 属性时，该参数不生效。
* `dynamic_partition.history_partition_num`
- 当 `create_history_partition` 为 `true` 时，该参数用于指定创建历史分区数量。
+ 当 `create_history_partition` 为 `true` 时，该参数用于指定创建历史分区数量。默认值为 -1， 即未设置。
* `dynamic_partition.hot_partition_num`
@@ -148,7 +148,61 @@ under the License.
p20210522：["2021-05-22", "2021-05-23") storage_medium=SSD storage_cooldown_time=2021-05-24 00:00:00
p20210523：["2021-05-23", "2021-05-24") storage_medium=SSD storage_cooldown_time=2021-05-25 00:00:00
```
-
+
+#### 创建历史分区规则
+
+当 `create_history_partition` 为 `true`，即开启创建历史分区功能时，Doris 会根据 `dynamic_partition.start` 和 `dynamic_partition.history_partition_num` 来决定创建历史分区的个数。
+
+假设需要创建的历史分区数量为 `expect_create_partition_num`，根据不同的设置具体数量如下：
+
+1. `create_history_partition` = `true`
+ - `dynamic_partition.history_partition_num` 未设置，即 -1.
+ `expect_create_partition_num` = `end` - `start`;
+
+ - `dynamic_partition.history_partition_num` 已设置
+ `expect_create_partition_num` = `end` - max(`start`, `-histoty_partition_num`);
+
+2. `create_history_partition` = `false`
+ 不会创建历史分区，`expect_create_partition_num` = `end` - 0;
+
+当 `expect_create_partition_num` 大于 `max_dynamic_partition_num`（默认500）时，禁止创建过多分区。
+
+**举例说明：**
+
+1. 假设今天是 2021-05-20，按天分区，动态分区的属性设置为：`create_history_partition=true, end=3, start=-3, history_partition_num=1`，则系统会自动创建以下分区：
+
+ ```
+ p20210519
+ p20210520
+ p20210521
+ p20210522
+ p20210523
+ ```
+
+2. `history_partition_num=5`，其余属性与 1 中保持一直，则系统会自动创建以下分区：
+
+ ```
+ p20210517
+ p20210518
+ p20210519
+ p20210520
+ p20210521
+ p20210522
+ p20210523
+ ```
+
+3. `history_partition_num=-1` 即不设置历史分区数量，其余属性与 1 中保持一直，则系统会自动创建以下分区：
+
+ ```
+ p20210517
+ p20210518
+ p20210519
+ p20210520
+ p20210521
+ p20210522
+ p20210523
+ ```
+
### 注意事项
动态分区使用过程中，如果因为一些意外情况导致 `dynamic_partition.start` 和 `dynamic_partition.end` 之间的某些分区丢失，那么当前时间与 `dynamic_partition.end` 之间的丢失分区会被重新创建，`dynamic_partition.start`与当前时间之间的丢失分区不会重新创建。
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org