You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by maoxiaomao <la...@163.com> on 2019/08/29 07:02:01 UTC
Merge Job Java Heap Error
Hi,
I building a MERGE job, and it got the following error when Step1. Merge Dictionary. I merge 3 segment ,and my dict file size is 185.7M*3 (557M)，it's really larger than normal.
I wonder:
Is there any size limit for dictionary？
How can I make it's size smaller?
env:
Kylin 2.6.1
Error: java.lang.OutOfMemoryError: Java heap space
at org.apache.kylin.dict.TrieDictionary.readFields(TrieDictionary.java:341)
at org.apache.kylin.dict.TrieDictionaryForest.readFields(TrieDictionaryForest.java:234)
at org.apache.kylin.dict.DictionaryInfoSerializer.deserialize(DictionaryInfoSerializer.java:80)
at org.apache.kylin.dict.DictionaryInfoSerializer.deserialize(DictionaryInfoSerializer.java:35)
at org.apache.kylin.common.persistence.ContentReader.readContent(ContentReader.java:40)
at org.apache.kylin.common.persistence.ResourceStore.getResource(ResourceStore.java:269)
at org.apache.kylin.common.persistence.ResourceStore.getResource(ResourceStore.java:256)
at org.apache.kylin.dict.DictionaryManager.load(DictionaryManager.java:397)
at org.apache.kylin.dict.DictionaryManager$1.load(DictionaryManager.java:80)
at org.apache.kylin.dict.DictionaryManager$1.load(DictionaryManager.java:77)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2379)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2257)
at com.google.common.cache.LocalCache.get(LocalCache.java:4000)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4004)
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4874)
at org.apache.kylin.dict.DictionaryManager.getDictionaryInfo(DictionaryManager.java:101)
at org.apache.kylin.engine.mr.steps.MergeDictionaryMapper.doMap(MergeDictionaryMapper.java:103)
at org.apache.kylin.engine.mr.steps.MergeDictionaryMapper.doMap(MergeDictionaryMapper.java:66)
at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:77)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
--
Re: Merge Job Java Heap Error
Posted by ShaoFeng Shi <sh...@apache.org>.
Dictionary will be loaded into Kylin's memory, so based on your JVM heap
configuration, the dictionary couldn't be very big.
Usually, the dictionary can only support dimension whose cardinality is
less than 5 million (smaller is better). For larger cardinality, please use
"fixed_length" encoding or "integer" encoding (if the column is in int
format).
Best regards,
Shaofeng Shi 史少锋
Apache Kylin PMC
Email: shaofengshi@apache.org
Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscribe@kylin.apache.org
Join Kylin dev mail group: dev-subscribe@kylin.apache.org
maoxiaomao <la...@163.com> 于2019年8月29日周四 下午3:02写道：
> Hi,
> I building a MERGE job, and it got the following error when Step1.
> Merge Dictionary. I merge 3 segment ,and my dict file size is 185.7M*3
> (557M)，it's really larger than normal.
> I wonder:
> Is there any size limit for dictionary？
> How can I make it's size smaller?
> env:
> Kylin 2.6.1
>
> Error: java.lang.OutOfMemoryError: Java heap space
> at org.apache.kylin.dict.TrieDictionary.readFields(TrieDictionary.java:341)
> at
> org.apache.kylin.dict.TrieDictionaryForest.readFields(TrieDictionaryForest.java:234)
> at
> org.apache.kylin.dict.DictionaryInfoSerializer.deserialize(DictionaryInfoSerializer.java:80)
> at
> org.apache.kylin.dict.DictionaryInfoSerializer.deserialize(DictionaryInfoSerializer.java:35)
> at
> org.apache.kylin.common.persistence.ContentReader.readContent(ContentReader.java:40)
> at
> org.apache.kylin.common.persistence.ResourceStore.getResource(ResourceStore.java:269)
> at
> org.apache.kylin.common.persistence.ResourceStore.getResource(ResourceStore.java:256)
> at
> org.apache.kylin.dict.DictionaryManager.load(DictionaryManager.java:397)
> at
> org.apache.kylin.dict.DictionaryManager$1.load(DictionaryManager.java:80)
> at
> org.apache.kylin.dict.DictionaryManager$1.load(DictionaryManager.java:77)
> at
> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599)
> at
> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2379)
> at
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342)
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2257)
> at com.google.common.cache.LocalCache.get(LocalCache.java:4000)
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4004)
> at
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4874)
> at
> org.apache.kylin.dict.DictionaryManager.getDictionaryInfo(DictionaryManager.java:101)
> at
> org.apache.kylin.engine.mr.steps.MergeDictionaryMapper.doMap(MergeDictionaryMapper.java:103)
> at
> org.apache.kylin.engine.mr.steps.MergeDictionaryMapper.doMap(MergeDictionaryMapper.java:66)
> at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:77)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
>
>
>
>
>
> --
> [image: 点击参加抽奖]
> <ht...@163.com>
>
>
>
>