You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2021/02/20 13:13:38 UTC

[GitHub] [incubator-doris] qidaye opened a new issue #5402: [Proposal] Support of LIST partition

qidaye opened a new issue #5402:
URL: https://github.com/apache/incubator-doris/issues/5402


   # List parttion
   
   Doris currently only supports Range partitioning, where data is usually partitioned by time columns. 
   
   However, in some scenarios, users want to partition by some enumerated values of columns, such as by city, etc.
   
   # Design 
   
   To add support for List partitioning, the following functional points need to be considered.
   
   1. Support for List partition syntax in create table statements.
   2. Support for adding and deleting List partition syntax. 
   3. Support for List partitioning in various load operations.
   4. Support for List partition pruning during query.
   
   List partitioned tables do not need to consider dynamic partitioning.
   
   # Detailed design
   
   ## Syntax
   
   The main changes involved here include.
   
   1. Implementation of the subclass ListPartitionDesc of the parsing class PartitionDesc
   2. Implementation of metadata class PartitionInfo subclass ListPartitionInfo
   3. Support for parsing and checking ListPartitionDesc in CreateTableStmt
   4. Support for the creation of List Partition tables in Catalog class. 
   5. Metadata persistence-related changes.
   
   The syntax is referenced from MySQL and Oracle
   
   ### Single partition column 
   
     ```sql
     CREATE TABLE tb1 (
         k1 int, k2 varchar(128), k3 int, v1 int, v2 int
     )
     PARTITION BY LIST(k1)
     (
         PARTITION p1 VALUES IN ("1", "3", "5"),
         PARTITION p2 VALUES IN ("2", "4", "6"),
         ...
     )
     ...
     ;
     ```
   
   ### Multi-partition columns
   
   ```sql
   CREATE TABLE tb2 (
       k1 int, k2 varchar(128), k3 int, v1 int, v2 int
   )
   PARTITION BY LIST(k1, k2)
   (
       PARTITION p1 VALUES IN (("1", "beijing"), ("1", "shanghai")),
       PARTITION p2 VALUES IN (("2", "beijing"), ("2", "shanghai"), ("2", "tianjin")),
       PARTITION p3 VALUES IN (("3", "beijing")),
       ...
   )
   ...
   ;
   ``` 
   
   **NOTE**: Each partition needs to ensure that the partition values are unique.
   
   ### Add partition 
   
   ```sql
   ALTER TABLE tb1 ADD PARTITION p4 VALUES IN ("7", "8", "9");
   
   ALTER TABLE tb2 ADD PARTITION p4 VALUES IN (("4", "tianjin"));
   ```
   
   ## Load
   
   The current load methods of Doris include Stream Load, INSERT, Routine Load, Broker Load, Hadoop Load, Spark Load.
   
   Among them, Stream Load, INSERT, Routine Load, and Broker Load all use TabletSink class for data distribution. Our first phase supports List partition support for these load operations.
   
   The main changes involved include: 
   
   1. Changes related to the Descriptors.TOlapTablePartitionParam structure in the Thrift structure TOlapTableSink
   2. Changes related to the OlapTablePartition object in the OlapTableSink class on the BE side.
   
   ## Query
   
   The query mainly needs to implement the List Partition pruning function.
   
   The main changes involved include: 
   
   1. Implementing the subclass ListPartitionPruner of PartitionPruner
   
   ## Partition related
   
   Support operations related to partitioned tables, such as recover, truncate, temporary partition, restore, replace, etc.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] HappenLee commented on issue #5402: [Proposal] Support of LIST partition

Posted by GitBox <gi...@apache.org>.
HappenLee commented on issue #5402:
URL: https://github.com/apache/incubator-doris/issues/5402#issuecomment-782718290


   Nice Job~~


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] moonming commented on issue #5402: [Proposal] Support of LIST partition

Posted by GitBox <gi...@apache.org>.
moonming commented on issue #5402:
URL: https://github.com/apache/incubator-doris/issues/5402#issuecomment-782776844


   Please send the proposal to the mailing list too


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] morningman closed issue #5402: [Proposal] Support of LIST partition

Posted by GitBox <gi...@apache.org>.
morningman closed issue #5402:
URL: https://github.com/apache/incubator-doris/issues/5402


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] qidaye commented on issue #5402: [Proposal] Support of LIST partition

Posted by GitBox <gi...@apache.org>.
qidaye commented on issue #5402:
URL: https://github.com/apache/incubator-doris/issues/5402#issuecomment-782779462


   > Please send the proposal to the mailing list too
   
   Already sent to dev@doris.apache.org
   
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] morningman closed issue #5402: [Proposal] Support of LIST partition

Posted by GitBox <gi...@apache.org>.
morningman closed issue #5402:
URL: https://github.com/apache/incubator-doris/issues/5402


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org