[jira] [Created] (KYLIN-2404) Add "hive.merge.mapfiles" and "hive.merge.mapredfiles" to kylin_hive_conf.xml

["Shaofeng SHI (JIRA)" <ji...@apache.org>]

Shaofeng SHI created KYLIN-2404:
-----------------------------------

             Summary: Add "hive.merge.mapfiles" and "hive.merge.mapredfiles" to kylin_hive_conf.xml
                 Key: KYLIN-2404
                 URL: https://issues.apache.org/jira/browse/KYLIN-2404
             Project: Kylin
          Issue Type: Improvement
          Components: Job Engine
            Reporter: Shaofeng SHI
            Assignee: Shaofeng SHI
            Priority: Minor
             Fix For: v2.0.0


Since 1.5.3, Kylin uses a "redistribute" step to merge the small files to proper size after creating the intermediate hive table. While in some users' environment, hive merge small files is enabled by default, that will cause additional CPU and will impact on the cube building performance (in extreme case the files will be merged to 256MB, then only very small number of mappers be started in building).

So Kylin should explicitly tell Hive to disable the merge small files feature when  creating and redistributing the intermediate flat table. Will add "hive.merge.mapfiles" and "hive.merge.mapredfiles" to conf/kylin_hive_conf.xml with value "false".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)