[GitHub] [incubator-doris] WingsGo opened a new issue #2262: [Proposal] Support Dynamic Partition in Doris

[GitBox <gi...@apache.org>]

WingsGo opened a new issue #2262: [Proposal] Support Dynamic Partition in Doris
URL: https://github.com/apache/incubator-doris/issues/2262
 
 
   ## Background:
     In some scenarios, when a user creates an Olap table that is range partition by time, the user needs to periodically add and remove partitions to ensure that the data is valid. As a result, adding and removing partitions dynamically can be very useful for users.
   
   ## How to Use
   When create a Olap Table in Doris, users can specify following properties:
   | Properties | Instructions | Value |
   | :-----| :----: | :----: |
   | dynamic_partition.window_timeunit | Dynamic partition units for olap tables | day/week/month |
   | dynamic_partition.window_end | Dynamic partition end location for olap tables | Integer |
   | dynamic_partition.template | The prefix partition name template | String |
   | dynamic_partition.buckets | The number of buckets in a partition | Integer |
   
   For example
   ```
   CREATE TABLE `dynamic_partition` (
     `olap_date` date NULL COMMENT "",
     `event_name` varchar(256) NULL COMMENT "",
     `distinct_id` varchar(36) NULL COMMENT "",
     `content` varchar(256) NULL COMMENT ""
   ) ENGINE=OLAP
   DUPLICATE KEY(`olap_date`, `event_name`, `distinct_id`)
   PARTITION BY RANGE(`olap_date`)
   (PARTITION p20191120 VALUES LESS THAN ("2019-11-21"),
   PARTITION p20191121 VALUES LESS THAN ("2019-11-22"))
   DISTRIBUTED BY HASH(`distinct_id`) BUCKETS 100
   PROPERTIES (
     "storage_type" = "COLUMN",
     "dynamic_partition.timeunit" = "day",
     "dynamic_partition.end" = 3,
     "dynamic_partition.format" = "p",
     "dynamic_partition.buckets" = "100"
   );
   ```
   In the example, the user specifies the automatically add partition units of time, and the number of partitions to be created in advance.dynamic_partition.format specifies the prefix of partition name, in this case, take 2019-11-21 as an example, a three-day partition named p20191122, p20191123 and p20191124 will be created in advance, and the number of buckets for each partition will be 100.
   
   ## How to realize
   1. When create table, FE parse these properties, check if these properties are valid and persistents them to metadata.
   2. Once FE starts, create a daemon called DynamicPartitionTask to check all tables properties if they have dynamic_partition property, if the dynamic_partition property is set, FE execute DDL stmt to add partitions.
   3. User can set how often the daemon thread execute in fe.conf with key `dynamic_partition_check_interval_seconds`, the default value is 600.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org