You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by ShaoFeng Shi <sh...@apache.org> on 2016/05/03 11:39:46 UTC
Re: java.lang.IllegalArgumentException:KeyValue size too large at
Building Dimension Dictionary
What's the "hbase.client.keyvalue.maxsize" in your hbase configuration?
Guess it is bigger than 10485760. Please get that value and then set
"kylin.hbase.client.keyvalue.maxsize" conf/kylin.properteis to the same or
less than hbase's. Kylin uses the later one to deciside whether put a
resource to hbase or not;
2016-05-03 17:09 GMT+08:00 彭丽 <pe...@outlook.com>:
> Hi,
>
> I meet a problem when the step of building dimension dictionary. Fact
> table has 400000 records. 3 lookup tables have 200000 , 10000, 200 records
> separately.
>
>
>
> java.lang.IllegalArgumentException: KeyValue size too large
>
> at
> org.apache.hadoop.hbase.client.HTable.validatePut(HTable.java:1413)
>
> at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:1035)
>
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:999)
>
> at
>
> org.apache.kylin.storage.hbase.HBaseResourceStore.putResourceImpl(HBaseResou
> rceStore.java:244)
>
> at
>
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.
> java:210)
>
> at
> org.apache.kylin.dict.DictionaryManager.save(DictionaryManager.java:378)
>
> at
>
> org.apache.kylin.dict.DictionaryManager.saveNewDict(DictionaryManager.java:1
> 99)
>
> at
>
> org.apache.kylin.dict.DictionaryManager.trySaveNewDict(DictionaryManager.jav
> a:166)
>
> at
>
> org.apache.kylin.dict.DictionaryManager.buildDictionary(DictionaryManager.ja
> va:293)
>
> at
> org.apache.kylin.cube.CubeManager.buildDictionary(CubeManager.java:167)
>
> at
>
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGe
> neratorCLI.java:51)
>
> at
>
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGe
> neratorCLI.java:42)
>
> at
>
> org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob
> .java:56)
>
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>
> at
>
> org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellEx
> ecutable.java:60)
>
> at
>
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable
> .java:114)
>
> at
>
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChaine
> dExecutable.java:50)
>
> at
>
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable
> .java:114)
>
> at
>
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultS
> cheduler.java:124)
>
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
> 45)
>
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
> 15)
>
> at java.lang.Thread.run(Thread.java:745)
>
>
>
> result code:2
>
>
>
> Thanks,
>
>
>
> Laney.
>
>
>
>
--
Best regards,
Shaofeng Shi
Re: java.lang.IllegalArgumentException:KeyValue size too large at
Building Dimension Dictionary
Posted by hongbin ma <ma...@apache.org>.
is it possible that one of the dimensions specified as "dict encoding"
turns out to be having huge cardinality/dimension length? if that's the
case, the generated dictionary might be too large to fix. You should think
about using other dimension encodings like "fixed length" or "extended
column (https://issues.apache.org/jira/browse/KYLIN-1313)"
On Tue, May 3, 2016 at 5:39 PM, ShaoFeng Shi <sh...@apache.org> wrote:
> What's the "hbase.client.keyvalue.maxsize" in your hbase configuration?
> Guess it is bigger than 10485760. Please get that value and then set
> "kylin.hbase.client.keyvalue.maxsize" conf/kylin.properteis to the same or
> less than hbase's. Kylin uses the later one to deciside whether put a
> resource to hbase or not;
>
>
>
> 2016-05-03 17:09 GMT+08:00 彭丽 <pe...@outlook.com>:
>
> > Hi,
> >
> > I meet a problem when the step of building dimension dictionary. Fact
> > table has 400000 records. 3 lookup tables have 200000 , 10000, 200
> records
> > separately.
> >
> >
> >
> > java.lang.IllegalArgumentException: KeyValue size too large
> >
> > at
> > org.apache.hadoop.hbase.client.HTable.validatePut(HTable.java:1413)
> >
> > at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:1035)
> >
> > at org.apache.hadoop.hbase.client.HTable.put(HTable.java:999)
> >
> > at
> >
> >
> org.apache.kylin.storage.hbase.HBaseResourceStore.putResourceImpl(HBaseResou
> > rceStore.java:244)
> >
> > at
> >
> >
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.
> > java:210)
> >
> > at
> > org.apache.kylin.dict.DictionaryManager.save(DictionaryManager.java:378)
> >
> > at
> >
> >
> org.apache.kylin.dict.DictionaryManager.saveNewDict(DictionaryManager.java:1
> > 99)
> >
> > at
> >
> >
> org.apache.kylin.dict.DictionaryManager.trySaveNewDict(DictionaryManager.jav
> > a:166)
> >
> > at
> >
> >
> org.apache.kylin.dict.DictionaryManager.buildDictionary(DictionaryManager.ja
> > va:293)
> >
> > at
> > org.apache.kylin.cube.CubeManager.buildDictionary(CubeManager.java:167)
> >
> > at
> >
> >
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGe
> > neratorCLI.java:51)
> >
> > at
> >
> >
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGe
> > neratorCLI.java:42)
> >
> > at
> >
> >
> org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob
> > .java:56)
> >
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> >
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> >
> > at
> >
> >
> org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellEx
> > ecutable.java:60)
> >
> > at
> >
> >
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable
> > .java:114)
> >
> > at
> >
> >
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChaine
> > dExecutable.java:50)
> >
> > at
> >
> >
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable
> > .java:114)
> >
> > at
> >
> >
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultS
> > cheduler.java:124)
> >
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
> > 45)
> >
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
> > 15)
> >
> > at java.lang.Thread.run(Thread.java:745)
> >
> >
> >
> > result code:2
> >
> >
> >
> > Thanks,
> >
> >
> >
> > Laney.
> >
> >
> >
> >
>
>
> --
> Best regards,
>
> Shaofeng Shi
>
--
Regards,
*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone