[jira] [Commented] (KYLIN-4048) Too long spark cube building time for too many eviction and loading for dict slices

["Shao Feng Shi (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KYLIN-4048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047664#comment-17047664 ] 

Shao Feng Shi commented on KYLIN-4048:
--------------------------------------

This should be related with the huge global dictionary. And the "shrunk dictionary" has solved this problem I think. It was enabled by default in Kylin 2.6.

> Too long spark cube building time for too many eviction and loading for dict slices
> -----------------------------------------------------------------------------------
>
>                 Key: KYLIN-4048
>                 URL: https://issues.apache.org/jira/browse/KYLIN-4048
>             Project: Kylin
>          Issue Type: Improvement
>    Affects Versions: v2.5.2
>            Reporter: Shaohui Liu
>            Priority: Major
>
> In our cluster, a cube building costs too long. In the log for spark, we found there are too many eviction and loading for dict slices in AppendTrieDictionary.
> {code:java}
> $ grep "read slice from" spark.log | wc -l
> 119721
> $ grep "Evict slice with key" spark.log| wc -l
> 119634
> {code}
> The reason is that the memory of spark executor(4G) is not enough to hold all the slices of dict(3.3G in hdfs) in memory, which cause the bad performance of cube building.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)