[jira] [Created] (KYLIN-4048) Too long spark cube building time for too many eviction and loading for dict slices

["Liu Shaohui (JIRA)" <ji...@apache.org>]

Liu Shaohui created KYLIN-4048:
----------------------------------

             Summary: Too long spark cube building time for too many eviction and loading for dict slices
                 Key: KYLIN-4048
                 URL: https://issues.apache.org/jira/browse/KYLIN-4048
             Project: Kylin
          Issue Type: Improvement
    Affects Versions: v2.5.2
            Reporter: Liu Shaohui


In our cluster, a cube building costs too long. In the log for spark, we found there are too many eviction and loading for dict slices in AppendTrieDictionary.
{code:java}
$ grep "read slice from" spark.log | wc -l
119721
$ grep "Evict slice with key" spark.log| wc -l
119634
{code}
The reason is that the memory of spark executor(4G) is not enough to hold all the slices of dict(3.3G in hdfs) in memory, which cause the bad performance of cube building.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)