You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by Lu Zhe <tc...@hotmail.com> on 2018/04/10 07:29:50 UTC

Dict value is too long

 Kylin 2.3.1

Build Dimension Dictionary时报错：

java.lang.IllegalStateException: maxValueLength is negative (-26578). Dict value is too long, whose length is larger than 32767
        at org.apache.kylin.dict.TrieDictionary.init(TrieDictionary.java:104)
        at org.apache.kylin.dict.TrieDictionary.readFields(TrieDictionary.java:339)
        at org.apache.kylin.dict.lookup.SnapshotTable.readData(SnapshotTable.java:250)
        at org.apache.kylin.dict.lookup.SnapshotTableSerializer.deserialize(SnapshotTableSerializer.java:74)
        at org.apache.kylin.dict.lookup.SnapshotTableSerializer.deserialize(SnapshotTableSerializer.java:48)
        at org.apache.kylin.common.persistence.ResourceStore.getResource(ResourceStore.java:171)
        at org.apache.kylin.dict.lookup.SnapshotManager.load(SnapshotManager.java:196)
        at org.apache.kylin.dict.lookup.SnapshotManager.checkDupByContent(SnapshotManager.java:178)
        at org.apache.kylin.dict.lookup.SnapshotManager.trySaveNewSnapshot(SnapshotManager.java:140)
        at org.apache.kylin.dict.lookup.SnapshotManager.buildSnapshot(SnapshotManager.java:120)
        at org.apache.kylin.cube.CubeManager$DictionaryAssist.buildSnapshotTable(CubeManager.java:1055)
        at org.apache.kylin.cube.CubeManager.buildSnapshotTable(CubeManager.java:971)
        at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:87)
        at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:49)
        at org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:71)
        at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:97)
        at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
        at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:162)
        at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:67)
        at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:162)
        at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:300)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

result code:2

原本怀疑是字段值过长，因为上次遇到的情况就是这样，后来我限制了字段值的长度就好了。
这次又遇到这个问题，不是很理解。

RE: Dict value is too long

Posted by ro...@stratebi.com.
Hi,

 

Looks like there is a dimension with high cardinality (HC) that does not fit
in dictionary. Please check if you have a HC dimension and try to change its
Rowkey Encoding from dictionary (dic) to other encoding, such as fixed
length. Have you tried this already?

 

Regards,

 

 

 

From: Lu Zhe [mailto:tcluzhe@hotmail.com] 
Sent: martes, 10 de abril de 2018 9:37
To: user@kylin.apache.org
Subject: 答复: Dict value is too long

 

抱歉，我修改下格式。

 

java.lang.IllegalStateException: maxValueLength is negative (-26578). Dict
value is too long, whose length is larger than 32767
 at org.apache.kylin.dict.TrieDictionary.init(TrieDictionary.java:104)
 at org.apache.kylin.dict.TrieDictionary.readFields(TrieDictionary.java:339)
 at
org.apache.kylin.dict.lookup.SnapshotTable.readData(SnapshotTable.java:250)
 at
org.apache.kylin.dict.lookup.SnapshotTableSerializer.deserialize(SnapshotTab
leSerializer.java:74)
 at
org.apache.kylin.dict.lookup.SnapshotTableSerializer.deserialize(SnapshotTab
leSerializer.java:48)
 at
org.apache.kylin.common.persistence.ResourceStore.getResource(ResourceStore.
java:171)
 at
org.apache.kylin.dict.lookup.SnapshotManager.load(SnapshotManager.java:196)
 at
org.apache.kylin.dict.lookup.SnapshotManager.checkDupByContent(SnapshotManag
er.java:178)
 at
org.apache.kylin.dict.lookup.SnapshotManager.trySaveNewSnapshot(SnapshotMana
ger.java:140)
 at
org.apache.kylin.dict.lookup.SnapshotManager.buildSnapshot(SnapshotManager.j
ava:120)
 at
org.apache.kylin.cube.CubeManager$DictionaryAssist.buildSnapshotTable(CubeMa
nager.java:1055)
 at
org.apache.kylin.cube.CubeManager.buildSnapshotTable(CubeManager.java:971)
 at
org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGe
neratorCLI.java:87)
 at
org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGe
neratorCLI.java:49)
 at
org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob
.java:71)
 at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:97)
 at
org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellEx
ecutable.java:63)
 at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable
.java:162)
 at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChaine
dExecutable.java:67)
 at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable
.java:162)
 at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultS
cheduler.java:300)
 at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
42)
 at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
17)
 at java.lang.Thread.run(Thread.java:745)

result code:2

 

原本怀疑是字段值过长，因为上次遇到的情况就是这样，后来我限制了字段值的长度就
好了。
这次又遇到这个问题，不是很理解。

 

  _____  

发件人: Lu Zhe < <ma...@hotmail.com> tcluzhe@hotmail.com>
发送时间: 2018年4月10日 15:29
收件人:  <ma...@kylin.apache.org> user@kylin.apache.org
主题: Dict value is too long 

 

 Kylin 2.3.1





Build Dimension Dictionary时报错：

java.lang.IllegalStateException: maxValueLength is negative (-26578). Dict
value is too long, whose length is larger than 32767
                    at
org.apache.kylin.dict.TrieDictionary.init(TrieDictionary.java:104)
                    at
org.apache.kylin.dict.TrieDictionary.readFields(TrieDictionary.java:339)
                    at
org.apache.kylin.dict.lookup.SnapshotTable.readData(SnapshotTable.java:250)
                    at org.apache.kylin.dict.lookup.SnapshotTableSerializer.
deserialize(SnapshotTableSerializer.java:74)
                    at org.apache.kylin.dict.lookup.SnapshotTableSerializer.
deserialize(SnapshotTableSerializer.java:48)
                    at
org.apache.kylin.common.persistence.ResourceStore.getResource(ResourceStore.
java:171)
                    at
org.apache.kylin.dict.lookup.SnapshotManager.load(SnapshotManager.java:196)
                    at
org.apache.kylin.dict.lookup.SnapshotManager.checkDupByContent(SnapshotManag
er.java:178)
                    at
org.apache.kylin.dict.lookup.SnapshotManager.trySaveNewSnapshot(SnapshotMana
ger.java:140)
                    at
org.apache.kylin.dict.lookup.SnapshotManager.buildSnapshot(SnapshotManager.j
ava:120)
                    at
org.apache.kylin.cube.CubeManager$DictionaryAssist.buildSnapshotTable(CubeMa
nager.java:1055)
                    at
org.apache.kylin.cube.CubeManager.buildSnapshotTable(CubeManager.java:971)
                    at
org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGe
neratorCLI.java:87)
                    at
org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGe
neratorCLI.java:49)
                    at org.apache.kylin.engine.mr.steps.CreateDictionaryJob.
run(CreateDictionaryJob.java:71)
                    at
org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:97)
                    at
org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellEx
ecutable.java:63)
                    at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable
.java:162)
                    at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChaine
dExecutable.java:67)
                    at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable
.java:162)
                    at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultS
cheduler.java:300)
                    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
42)
                    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
17)
                    at java.lang.Thread.run(Thread.java:745)
 
result code:2

 

原本怀疑是字段值过长，因为上次遇到的情况就是这样，后来我限制了字段值的长度就
好了。

这次又遇到这个问题，不是很理解。

答复: Dict value is too long

Posted by Lu Zhe <tc...@hotmail.com>.
抱歉，我修改下格式。

java.lang.IllegalStateException: maxValueLength is negative (-26578). Dict value is too long, whose length is larger than 32767
 at org.apache.kylin.dict.TrieDictionary.init(TrieDictionary.java:104)
 at org.apache.kylin.dict.TrieDictionary.readFields(TrieDictionary.java:339)
 at org.apache.kylin.dict.lookup.SnapshotTable.readData(SnapshotTable.java:250)
 at org.apache.kylin.dict.lookup.SnapshotTableSerializer.deserialize(SnapshotTableSerializer.java:74)
 at org.apache.kylin.dict.lookup.SnapshotTableSerializer.deserialize(SnapshotTableSerializer.java:48)
 at org.apache.kylin.common.persistence.ResourceStore.getResource(ResourceStore.java:171)
 at org.apache.kylin.dict.lookup.SnapshotManager.load(SnapshotManager.java:196)
 at org.apache.kylin.dict.lookup.SnapshotManager.checkDupByContent(SnapshotManager.java:178)
 at org.apache.kylin.dict.lookup.SnapshotManager.trySaveNewSnapshot(SnapshotManager.java:140)
 at org.apache.kylin.dict.lookup.SnapshotManager.buildSnapshot(SnapshotManager.java:120)
 at org.apache.kylin.cube.CubeManager$DictionaryAssist.buildSnapshotTable(CubeManager.java:1055)
 at org.apache.kylin.cube.CubeManager.buildSnapshotTable(CubeManager.java:971)
 at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:87)
 at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:49)
 at org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:71)
 at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:97)
 at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
 at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:162)
 at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:67)
 at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:162)
 at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:300)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)
result code:2

原本怀疑是字段值过长，因为上次遇到的情况就是这样，后来我限制了字段值的长度就好了。
这次又遇到这个问题，不是很理解。

________________________________
发件人: Lu Zhe <tc...@hotmail.com>
发送时间: 2018年4月10日 15:29
收件人: user@kylin.apache.org
主题: Dict value is too long

 Kylin 2.3.1

Build Dimension Dictionary时报错：

java.lang.IllegalStateException: maxValueLength is negative (-26578). Dict value is too long, whose length is larger than 32767
        at org.apache.kylin.dict.TrieDictionary.init(TrieDictionary.java:104)
        at org.apache.kylin.dict.TrieDictionary.readFields(TrieDictionary.java:339)
        at org.apache.kylin.dict.lookup.SnapshotTable.readData(SnapshotTable.java:250)
        at org.apache.kylin.dict.lookup.SnapshotTableSerializer.deserialize(SnapshotTableSerializer.java:74)
        at org.apache.kylin.dict.lookup.SnapshotTableSerializer.deserialize(SnapshotTableSerializer.java:48)
        at org.apache.kylin.common.persistence.ResourceStore.getResource(ResourceStore.java:171)
        at org.apache.kylin.dict.lookup.SnapshotManager.load(SnapshotManager.java:196)
        at org.apache.kylin.dict.lookup.SnapshotManager.checkDupByContent(SnapshotManager.java:178)
        at org.apache.kylin.dict.lookup.SnapshotManager.trySaveNewSnapshot(SnapshotManager.java:140)
        at org.apache.kylin.dict.lookup.SnapshotManager.buildSnapshot(SnapshotManager.java:120)
        at org.apache.kylin.cube.CubeManager$DictionaryAssist.buildSnapshotTable(CubeManager.java:1055)
        at org.apache.kylin.cube.CubeManager.buildSnapshotTable(CubeManager.java:971)
        at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:87)
        at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:49)
        at org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:71)
        at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:97)
        at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
        at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:162)
        at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:67)
        at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:162)
        at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:300)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

result code:2

原本怀疑是字段值过长，因为上次遇到的情况就是这样，后来我限制了字段值的长度就好了。
这次又遇到这个问题，不是很理解。