[GitHub] [incubator-doris] morningman edited a comment on issue #2663: [Proposal] Support partition overwrite operation

[GitBox <gi...@apache.org>]

morningman edited a comment on issue #2663: [Proposal] Support partition overwrite operation
URL: https://github.com/apache/incubator-doris/issues/2663#issuecomment-583703429
 
 
   Redesign according to new requirements
   
   We hope that the Temporary Partition feature can be more flexible. While supporting overwrite, it can also support the modification of existing partitions, such as supporting partition splitting or merging, or modifying the number of buckets.
   
   ## Add Temp Partition
   
   ```
   ALTER TABLE tbl
   ADD TEMPORARY PARTITION tp1 values ​​[(lower), (high)) BUCKETS 10,
   ADD TEMPORARY PARTITION tp2 values ​​((lower), (high)) BUCKETS 20,
   ADD TEMPORARY PARTITION tp3 values ​​[(lower), (high));
   ```
   
   Temporary partitions can also specify partition ranges and number of buckets. If no bucket number is specified, the default bucket number is used. There is no limit on the range of the temporary partition. However, all ranges of the temporary partitions of a table must not conflict.
   
   ## Delete Temp Partition
   
   ```
   ALTER TABLE tbl DROP TEMPORARY PARTITION (tp1, tp2);
   ```
   
   ## Replace Temp Partition
   
   ```
   ALTER TABLE tbl REPLACE PARTITION (p1, p2) WITH TEMPORARY PARTITION (tp1, tp2, tp3);
   ```
   
   The above statement replaces the existing partitions `p1, p2` with `tp1, tp2, tp3`. In the replace partition operation, as long as after deleting original partitions and replaced with the temp partitions, the range of the entire table does not conflict, then the replace operation will be allowed.
   
   Replacement is an atomic operation.
   
   ## Load/Read/ to Temp Partitions
   
   1. BROKER LOAD
   
       ```
       LOAD LABEL db1.label1
       DATA INFILE ("hdfs: // files")
       INTO TABLE `my_table`
       TEMPORARY PARTITION (p1, p2)
       ...
       ```
   
   2. STREAM LOAD
   
       ```
       curl -H "temporary partition: p1, p2" ...
       ```
   
   3. INSERT INTO
   
       ```
       INSERT INTO tbl TEMPORARY PARTITION (p1, p2) ...
       ```
   
   4. SELECT
   
       ```
       SELECT * FROM tbl TEMPORARY PARTITION (p1, p2) ...
       ```
   
   ## Usage
   
   * Overwrite
   
       First create a temporary partition with the same range as the partition to be overwritten. Then load the data to the temporary partition, and finally replace the partition with the temporary partition by the replace command to complete the overwrite.
   
   * Partition split and merge
   
       First create two temporary partitions whose range is the splitted range. Use the insert into statement to load data from the original partition into two temporary partitions. Finally, use the replace command to replace the partition with temporary partitions to complete the partition split.
   
       Merging partitions is to create a temporary partition whose range covers multiple partitions' ranges, and then perform the same operation as split.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org