Fwd: Re: org.apache.kylin.dict.TrieDictionary: Not a valid value

[ShaoFeng Shi <sh...@apache.org>]

Forward to dev mailing list for wider audience;

---------- Forwarded message ----------
From: ShaoFeng Shi <sh...@apache.org>
Date: 2016-11-26 15:59 GMT+08:00
Subject: Re: Re: org.apache.kylin.dict.TrieDictionary: Not a valid value
To: user <us...@kylin.apache.org>


Hi pengwen,

I carefully checked that and I think it was caused by some dirtly data on
fact table (fk but doesn't exist in lookup table); The invalid value "532"
should be a FK column value. Since the lookup table is defined as "inner
join" with the fact table, Kylin delegates building the dictionary from
lookup table's PK; so if there is unmatched record, this error may happen.
The discussion about this issue is in JIRA KYLIN-1855.

Regarding why in-mem building got failed but layer cubing got success, that
is because the layer cubing tolerates some insane record (< 100 in one
mapper), but in-mem building didn't. I opened JIRA KYLIN-2231 for tracking
this.

Thanks for the reporting!


2016-11-25 23:33 GMT+08:00 ypw89724@126.com <yp...@126.com>:

> This attachment is diagnosis.
> Tonight I tried to use the layer cubing (kylin.cube.algorithm:layer),I
> found its ok.
>
> ------------------------------
> 袁鹏文  Pengwen Yuan
>
>
> *From:* ShaoFeng Shi <sh...@apache.org>
> *Date:* 2016-11-25 17:31
> *To:* user <us...@kylin.apache.org>
> *Subject:* Re: org.apache.kylin.dict.TrieDictionary: Not a valid value
> Hi Pengwen,
>
> This question also be asked by other people before, but we wasn't able to
> reproduce. Could you share your model and cube definition JSON? Thanks!
>
> 2016-11-25 15:41 GMT+08:00 ypw89724@126.com <yp...@126.com>:
>
>> hi all:
>>
>> When I build cube at "#15 Step Name: Build Cube", I got an error :
>>
>>
>> 2016-11-25 14:17:23,724 INFO [main] org.apache.kylin.engine.mr.steps.InMemCuboidMapper: Reserve 1079 MB = 200 (MR reserve) + 879 (SYS reserve)
>> 2016-11-25 14:17:23,732 INFO [pool-5-thread-1] org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder: Dogged Cube Build start
>> 2016-11-25 14:17:23,758 INFO [pool-5-thread-1] org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder: Split #1 kickoff
>> 2016-11-25 14:17:23,771 INFO [Thread-12] org.apache.kylin.cube.inmemcubing.InMemCubeBuilder: In Mem Cube Build start, CUBE_OLP_SGM_HOTELMONTHREPORT
>> 2016-11-25 14:17:25,984 WARN [Thread-12] org.apache.kylin.common.util.MemoryBudgetController: Higher system avail 8886 MB in markLow()
>> 2016-11-25 14:17:25,987 INFO [Thread-12] org.apache.kylin.cube.inmemcubing.InMemCubeBuilder: Calculating base cuboid 1272016-11-25 14:17:26,134 ERROR [Thread-12] org.apache.kylin.dict.TrieDictionary: Not a valid value: 532
>> 2016-11-25 14:17:27,136 ERROR [pool-5-thread-1] org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder: Dogged Cube Build error
>> java.io.IOException: java.lang.IllegalArgumentException: Value not exists!
>> 	at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.abort(DoggedCubeBuilder.java:194)
>> 	at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.checkException(DoggedCubeBuilder.java:167)
>> 	at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.build(DoggedCubeBuilder.java:114)
>> 	at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder.build(DoggedCubeBuilder.java:73)
>> 	at org.apache.kylin.cube.inmemcubing.AbstractInMemCubeBuilder$1.run(AbstractInMemCubeBuilder.java:80)
>> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> Caused by: java.lang.IllegalArgumentException: Value not exists!
>> 	at org.apache.kylin.common.util.Dictionary.getIdFromValueBytes(Dictionary.java:162)
>> 	at org.apache.kylin.dict.TrieDictionary.getIdFromValueImpl(TrieDictionary.java:167)
>> 	at org.apache.kylin.common.util.Dictionary.getIdFromValue(Dictionary.java:98)
>>
>>
>> My Kylin version : 1.5.4.1
>>
>> hadoop : 2.6.0-cdh5.8.0
>>
>> HBase : 1.2.0-cdh5.8.0
>>
>>
>>
>>
>> ------------------------------
>> 袁鹏文  Pengwen Yuan
>>
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>


-- 
Best regards,

Shaofeng Shi 史少锋




-- 
Best regards,

Shaofeng Shi 史少锋