[jira] [Updated] (KYLIN-702) When Kylin create the flat hive table, it generates large number of small files in HDFS

["Shaofeng SHI (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KYLIN-702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Shaofeng SHI updated KYLIN-702:
-------------------------------
    Affects Version/s: v1.1
                       v1.0
                       v1.1.1
        Fix Version/s: v1.2

I think this change need be introduced into 1.x, as this will reduce hadoop job scheduling effort, especially for big cubes; 

> When Kylin create the flat hive table, it generates large number of small files in HDFS 
> ----------------------------------------------------------------------------------------
>
>                 Key: KYLIN-702
>                 URL: https://issues.apache.org/jira/browse/KYLIN-702
>             Project: Kylin
>          Issue Type: Improvement
>          Components: General
>    Affects Versions: v1.1, v1.0, v0.7.1, v1.1.1
>            Reporter: Shaofeng SHI
>            Assignee: Shaofeng SHI
>             Fix For: v2.0, v1.2
>
>
> When I build a cube, I noticed that when build the dictionary and calculate the cube, there are a large number of mappers be started (more than 10,000); With the log I noticed many mappers has 0 or much less records to process, this confused me; 
> Then I checked the storage location of the flat table, found there are many files; I did a count and found it is the same number as the mappers; 
> Too many mappers will cause much overhead, and downgrade the cluster's performance; Kylin should ask Hive to merge those small files during creating the flat table step. 
> In my hadoop cluster, the hive.merge.mapredfiles was set to false (default value); After changing it to true for Kylin's job, the intermediate table's file number was reduced to 4, each be up to 256M, looks good; Check hive configuration at: https://cwiki.apache.org/confluence/display/Hive/AdminManual+Configuration



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)