You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by "Shaofeng SHI (JIRA)" <ji...@apache.org> on 2016/11/26 08:01:10 UTC
[jira] [Updated] (KYLIN-2231) Layer cubing tolerates some insane
records, but in-mem cubing doesn't
[ https://issues.apache.org/jira/browse/KYLIN-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shaofeng SHI updated KYLIN-2231:
--------------------------------
Description:
User pengwen yuan in user@kylin.apache.org mailing list reports that, he got error when using "in-mem" cubing, but got success after switch to "layer" cubing algorithm.
Checked the code I think it is because the layer cubing tolerates on minor insane records (see BaseCuboidMapperBase.handleErrorRecord()), but in-mem doesn't.
There should be no difference in the behavior. Need make them consistent.
The error in in-mem cubing is:
{code}
2016-11-25 16:23:10,198 ERROR [Thread-12] org.apache.kylin.dict.TrieDictionary: Not a valid value: 532
2016-11-25 16:23:11,199 ERROR [pool-5-thread-1] org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder: Dogged Cube Build error
java.io.IOException: java.lang.IllegalArgumentException: Value not exists!
at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.abort(DoggedCubeBuilder.java:194)
at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.checkException(DoggedCubeBuilder.java:167)
at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.build(DoggedCubeBuilder.java:114)
at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder.build(DoggedCubeBuilder.java:73)
at org.apache.kylin.cube.inmemcubing.AbstractInMemCubeBuilder$1.run(AbstractInMemCubeBuilder.java:80)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Value not exists!
at org.apache.kylin.common.util.Dictionary.getIdFromValueBytes(Dictionary.java:162)
at org.apache.kylin.dict.TrieDictionary.getIdFromValueImpl(TrieDictionary.java:167)
at org.apache.kylin.common.util.Dictionary.getIdFromValue(Dictionary.java:98)
at org.apache.kylin.dimension.DictionaryDimEnc$DictionarySerializer.serialize(DictionaryDimEnc.java:121)
at org.apache.kylin.cube.gridtable.CubeCodeSystem.encodeColumnValue(CubeCodeSystem.java:121)
at org.apache.kylin.cube.gridtable.CubeCodeSystem.encodeColumnValue(CubeCodeSystem.java:110)
at org.apache.kylin.gridtable.GTRecord.setValues(GTRecord.java:93)
at org.apache.kylin.gridtable.GTRecord.setValues(GTRecord.java:81)
at org.apache.kylin.cube.inmemcubing.InMemCubeBuilderInputConverter.convert(InMemCubeBuilderInputConverter.java:74)
at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder$InputConverter$1.next(InMemCubeBuilder.java:542)
at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder$InputConverter$1.next(InMemCubeBuilder.java:523)
at org.apache.kylin.gridtable.GTAggregateScanner.iterator(GTAggregateScanner.java:139)
at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.createBaseCuboid(InMemCubeBuilder.java:339)
at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.build(InMemCubeBuilder.java:166)
at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.build(InMemCubeBuilder.java:135)
at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$SplitThread.run(DoggedCubeBuilder.java:282)
2016-11-25 16:23:11,202 INFO [pool-5-thread-1] org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder: Dogged Cube Build end, totally took 3410 ms
{code}
was:
User pengwen yuan in user@kylin.apache.org mailing list reports that, he got error when using "in-mem" cubing, but got success after switch to "layer" cubing algorithm.
Checked the code I think it is because the layer cubing tolerates on minor insane records (see BaseCuboidMapperBase.handleErrorRecord()), but in-mem doesn't.
There should be no difference in the behavior. Need make them consistent.
> Layer cubing tolerates some insane records, but in-mem cubing doesn't
> ---------------------------------------------------------------------
>
> Key: KYLIN-2231
> URL: https://issues.apache.org/jira/browse/KYLIN-2231
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
> Reporter: Shaofeng SHI
> Priority: Minor
>
> User pengwen yuan in user@kylin.apache.org mailing list reports that, he got error when using "in-mem" cubing, but got success after switch to "layer" cubing algorithm.
> Checked the code I think it is because the layer cubing tolerates on minor insane records (see BaseCuboidMapperBase.handleErrorRecord()), but in-mem doesn't.
> There should be no difference in the behavior. Need make them consistent.
> The error in in-mem cubing is:
> {code}
> 2016-11-25 16:23:10,198 ERROR [Thread-12] org.apache.kylin.dict.TrieDictionary: Not a valid value: 532
> 2016-11-25 16:23:11,199 ERROR [pool-5-thread-1] org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder: Dogged Cube Build error
> java.io.IOException: java.lang.IllegalArgumentException: Value not exists!
> at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.abort(DoggedCubeBuilder.java:194)
> at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.checkException(DoggedCubeBuilder.java:167)
> at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.build(DoggedCubeBuilder.java:114)
> at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder.build(DoggedCubeBuilder.java:73)
> at org.apache.kylin.cube.inmemcubing.AbstractInMemCubeBuilder$1.run(AbstractInMemCubeBuilder.java:80)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalArgumentException: Value not exists!
> at org.apache.kylin.common.util.Dictionary.getIdFromValueBytes(Dictionary.java:162)
> at org.apache.kylin.dict.TrieDictionary.getIdFromValueImpl(TrieDictionary.java:167)
> at org.apache.kylin.common.util.Dictionary.getIdFromValue(Dictionary.java:98)
> at org.apache.kylin.dimension.DictionaryDimEnc$DictionarySerializer.serialize(DictionaryDimEnc.java:121)
> at org.apache.kylin.cube.gridtable.CubeCodeSystem.encodeColumnValue(CubeCodeSystem.java:121)
> at org.apache.kylin.cube.gridtable.CubeCodeSystem.encodeColumnValue(CubeCodeSystem.java:110)
> at org.apache.kylin.gridtable.GTRecord.setValues(GTRecord.java:93)
> at org.apache.kylin.gridtable.GTRecord.setValues(GTRecord.java:81)
> at org.apache.kylin.cube.inmemcubing.InMemCubeBuilderInputConverter.convert(InMemCubeBuilderInputConverter.java:74)
> at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder$InputConverter$1.next(InMemCubeBuilder.java:542)
> at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder$InputConverter$1.next(InMemCubeBuilder.java:523)
> at org.apache.kylin.gridtable.GTAggregateScanner.iterator(GTAggregateScanner.java:139)
> at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.createBaseCuboid(InMemCubeBuilder.java:339)
> at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.build(InMemCubeBuilder.java:166)
> at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.build(InMemCubeBuilder.java:135)
> at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$SplitThread.run(DoggedCubeBuilder.java:282)
> 2016-11-25 16:23:11,202 INFO [pool-5-thread-1] org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder: Dogged Cube Build end, totally took 3410 ms
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)