You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@carbondata.apache.org by "Indhumathi Muthumurugesh (Jira)" <ji...@apache.org> on 2021/05/12 06:41:00 UTC

[jira] [Updated] (CARBONDATA-4183) Local sort Partition Load and Compaction improvement

     [ https://issues.apache.org/jira/browse/CARBONDATA-4183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Indhumathi Muthumurugesh updated CARBONDATA-4183:
-------------------------------------------------
    Description: Currently, number of tasks for partition table local sort load, is decided based on input file size. In this case, the data will not be properly sorted, as tasks launched is more. For compaction, number of tasks is equal to number of partitions. If data is huge for a partition, then there can be chances, that compaction will fail with OOM with less memory configurations.

> Local sort Partition Load and Compaction improvement
> ----------------------------------------------------
>
>                 Key: CARBONDATA-4183
>                 URL: https://issues.apache.org/jira/browse/CARBONDATA-4183
>             Project: CarbonData
>          Issue Type: Improvement
>            Reporter: Indhumathi Muthumurugesh
>            Priority: Major
>
> Currently, number of tasks for partition table local sort load, is decided based on input file size. In this case, the data will not be properly sorted, as tasks launched is more. For compaction, number of tasks is equal to number of partitions. If data is huge for a partition, then there can be chances, that compaction will fail with OOM with less memory configurations.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)