[GitHub] [incubator-doris] morningman opened a new issue #2823: [Load] Invalid file group aggregation when handling broker load job

[GitBox <gi...@apache.org>]

morningman opened a new issue #2823: [Load] Invalid file group aggregation when handling broker load job
URL: https://github.com/apache/incubator-doris/issues/2823
 
 
   **Describe the bug**
   
   First, In the broker load, we allow users to add multiple data descriptions. Each data description
    represents a description of a file (or set of files). Including file path, delimiter, table and 
   partitions to be loaded, and other information.
   
   When the user specifies multiple data descriptions, Doris currently aggregates the data 
   descriptions belonging to the same table and generates a unified load task.
   
   The problem here is that although different data descriptions point to the same table, 
   they may specify different partitions. Therefore, the aggregation of data description
    should not only consider the table level, but also the partition level.
   
   Examples are as follows:
   
   data description 1 is: 
   ```
   DATA INFILE("hdfs://hdfs_host:hdfs_port/input/file1")
   INTO TABLE `tbl1`
   PARTITION (p1, p2)
   ```
   
   data description 2 is:
   ```
   DATA INFILE("hdfs://hdfs_host:hdfs_port/input/file2")
   INTO TABLE `tbl1`
   PARTITION (p3, p4)
   ```
   What user expects is to load file1 into partition p1 and p2 of tbl1, and load file2 into paritition
   p3 and p4 of same table. But currently, it will be aggregated together, which result in loading
   file1 and file2 into all partitions p1, p2, p3 and p4.
   
   Second, there is a problem with the code implementation. In the constructor of 
   `OlapTableSink.java`, we pass in a string of partition names separated by commas. 
   But at the `OlapTableSink` level, we should be able to pass in a list of partition ids directly,
    instead of names.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [incubator-doris] morningman closed issue #2823: [Load] Invalid file group aggregation when handling broker load job

[GitBox <gi...@apache.org>]

morningman closed issue #2823: [Load] Invalid file group aggregation when handling broker load job
URL: https://github.com/apache/incubator-doris/issues/2823
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org