Re: #3 Step Name: Build Dimension Dictionary

[Li Yang <li...@apache.org>]

In 0.8 branch, Kylin start to read Hive table via the HCat APIs. The one
HDFS file limitation is gone.  However the small assumption still exists.

On Wed, May 27, 2015 at 4:19 PM, hongbin ma <ma...@apache.org> wrote:

> for now , kylin assumes dimension tables are relatively small, and there
> exists only one hdfs file for that table.
>
> On Wed, May 27, 2015 at 4:17 PM, Luke Han <lu...@gmail.com> wrote:
>
> > Forward to mailing list for further support.
> >
> > Thanks.
> >
> > 在 2015年5月27日星期三 UTC+8下午4:11:09，donald fossouo写道：
> >>
> >> Hi i just start with kylin after build test cube and my own on HDP
> >> environment :
> >>
> >> HDP 2.2
> >>
> >> Hive 0.14 , hadoop 2.6
> >>
> >> I got the following error in the 3rd step :
> >>
> >>
> >>
> >> java.lang.IllegalStateException: Expect 1 and only 1 non-zero file under
> >> hdfs://my-name-node:8020/apps/hive/warehouse/dim_organisations, but
> find 0
> >>         at
> >> org.apache.kylin.dict.lookup.HiveTable.findOnlyFile(HiveTable.java:123)
> >>         at
> >>
> org.apache.kylin.dict.lookup.HiveTable.computeHDFSLocation(HiveTable.java:107)
> >>
> >>         at
> >>
> org.apache.kylin.dict.lookup.HiveTable.getHDFSLocation(HiveTable.java:83)
> >>         at
> >> org.apache.kylin.dict.lookup.HiveTable.getFileTable(HiveTable.java:76)
> >>         at
> >> org.apache.kylin.dict.lookup.HiveTable.getSignature(HiveTable.java:71)
> >>         at
> >>
> org.apache.kylin.dict.DictionaryManager.buildDictionary(DictionaryManager.java:164)
> >>
> >>         at
> >> org.apache.kylin.cube.CubeManager.buildDictionary(CubeManager.java:154)
> >>         at
> >>
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:53)
> >>
> >>         at
> >>
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:42)
> >>
> >>         at
> >>
> org.apache.kylin.job.hadoop.dict.CreateDictionaryJob.run(CreateDictionaryJob.java:53)
> >>
> >>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> >>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> >>         at
> >>
> org.apache.kylin.job.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
> >>
> >>         at
> >>
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> >>
> >>         at
> >>
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
> >>
> >>         at
> >>
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> >>
> >>         at
> >>
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:132)
> >>
> >>         at
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >>
> >>         at
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >>
> >>         at java.lang.Thread.run(Thread.java:745)
> >>
> >> result code:2
> >>
> >> What is the cause of the problem?
> >>
> >>
>
>
> --
> Regards,
>
> *Bin Mahone | 马洪宾*
> Apache Kylin: http://kylin.io
> Github: https://github.com/binmahone
>