You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@carbondata.apache.org by "xuchuanyin (JIRA)" <ji...@apache.org> on 2018/04/03 02:02:00 UTC

[jira] [Created] (CARBONDATA-2309) Add strategy to generate bigger carbondata files in case of small amount of data

xuchuanyin created CARBONDATA-2309:
--------------------------------------

             Summary: Add strategy to generate bigger carbondata files in case of small amount of data
                 Key: CARBONDATA-2309
                 URL: https://issues.apache.org/jira/browse/CARBONDATA-2309
             Project: CarbonData
          Issue Type: Improvement
          Components: data-load
            Reporter: xuchuanyin
            Assignee: xuchuanyin


In some scenario, the input amount of loading data is small, but carbondata still distribute them to each executors (nodes) to do local-sort, thus resulting to small carbondata files generated by each executor. 

In  some extreme conditions, if the cluster is big enough or if the amount of data is small enough, the carbondata file contains only one blocklet or page.

I  think a new strategy should be introduced to solve the above problem.

The new strategy should:
 # be able to control the minimum amount of input data for each node
 # ignore data locality otherwise it may always choose a small portion of particular nodes



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)