You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by 仇同心 <qi...@jd.com> on 2017/02/14 09:14:19 UTC

java.lang.RuntimeException: Too big dictionary, dictionary cannot be bigger than 2GB

Hi ,all
  The first step in cube merge, an error :

   java.lang.RuntimeException: Too big dictionary, dictionary cannot be bigger than 2GB
       at org.apache.kylin.dict.TrieDictionaryBuilder.buildTrieBytes(TrieDictionaryBuilder.java:421)
       at org.apache.kylin.dict.TrieDictionaryBuilder.build(TrieDictionaryBuilder.java:408)
       at org.apache.kylin.dict.DictionaryGenerator$StringDictBuilder.build(DictionaryGenerator.java:165)
       at org.apache.kylin.dict.DictionaryGenerator.buildDictionary(DictionaryGenerator.java:81)
       at org.apache.kylin.dict.DictionaryGenerator.buildDictionary(DictionaryGenerator.java:73)
       at org.apache.kylin.dict.DictionaryGenerator.mergeDictionaries(DictionaryGenerator.java:102)
       at org.apache.kylin.dict.DictionaryManager.mergeDictionary(DictionaryManager.java:268)
       at org.apache.kylin.engine.mr.steps.MergeDictionaryStep.mergeDictionaries(MergeDictionaryStep.java:145)
       at org.apache.kylin.engine.mr.steps.MergeDictionaryStep.makeDictForNewSegment(MergeDictionaryStep.java:135)
       at org.apache.kylin.engine.mr.steps.MergeDictionaryStep.doWork(MergeDictionaryStep.java:67)
       at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
       at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:57)
       at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
       at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:136)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
       at java.lang.Thread.run(Thread.java:745)


     “SALE_ORD_ID”  Cardinality ：157644463
     SALE    COUNT_DISTINCT      Value:SALE_ORD_ID, Type:column   bitmap

I'm wondering that the high base fields can't do count_distinct accurate statistical metrics ??

Re: java.lang.RuntimeException: Too big dictionary, dictionary cannot be bigger than 2GB

Posted by Alberto Ramón <a....@gmail.com>.
Max cardinality of defeault Dic is 2 millons
Why encode  Sale_ord_id as Dic? if this is an int, you can use integer Dic

Please check:
http://apache-kylin.74782.x6.nabble.com/create-dictionary-error-td7155.html
http://mail-archives.apache.org/mod_mbox/kylin-user/201702.mbox/%3CCAEcyM17BTkhVpFcZLP6%2Boawx%3D1eap%3DZS_ER1HJbhevJPBE71-g%40mail.gmail.com%3E



2017-02-14 10:14 GMT+01:00 仇同心 <qi...@jd.com>:

> Hi ,all
>
>   The first step in cube merge, an error :
>
>
>
>    java.lang.RuntimeException: Too big dictionary, dictionary cannot be
> bigger than 2GB
>
>        at org.apache.kylin.dict.TrieDictionaryBuilder.buildTrieBytes(
> TrieDictionaryBuilder.java:421)
>
>        at org.apache.kylin.dict.TrieDictionaryBuilder.build(
> TrieDictionaryBuilder.java:408)
>
>        at org.apache.kylin.dict.DictionaryGenerator$
> StringDictBuilder.build(DictionaryGenerator.java:165)
>
>        at org.apache.kylin.dict.DictionaryGenerator.buildDictionary(
> DictionaryGenerator.java:81)
>
>        at org.apache.kylin.dict.DictionaryGenerator.buildDictionary(
> DictionaryGenerator.java:73)
>
>        at org.apache.kylin.dict.DictionaryGenerator.mergeDictionaries(
> DictionaryGenerator.java:102)
>
>        at org.apache.kylin.dict.DictionaryManager.mergeDictionary(
> DictionaryManager.java:268)
>
>        at org.apache.kylin.engine.mr.steps.MergeDictionaryStep.
> mergeDictionaries(MergeDictionaryStep.java:145)
>
>        at org.apache.kylin.engine.mr.steps.MergeDictionaryStep.
> makeDictForNewSegment(MergeDictionaryStep.java:135)
>
>        at org.apache.kylin.engine.mr.steps.MergeDictionaryStep.
> doWork(MergeDictionaryStep.java:67)
>
>        at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:113)
>
>        at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(
> DefaultChainedExecutable.java:57)
>
>        at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:113)
>
>        at org.apache.kylin.job.impl.threadpool.DefaultScheduler$
> JobRunner.run(DefaultScheduler.java:136)
>
>        at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
>
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
>
>        at java.lang.Thread.run(Thread.java:745)
>
>
>
>
>
>      “SALE_ORD_ID”  Cardinality ：157644463
>
>      SALE    COUNT_DISTINCT      Value:SALE_ORD_ID, Type:column   bitmap
>
>
>
> I'm wondering that the high base fields can't do count_distinct accurate
> statistical metrics ??
>
>
>
>
>
>
>