You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by "JiangQin (JIRA)" <ji...@apache.org> on 2017/09/12 02:11:00 UTC
[jira] [Issue Comment Deleted] (KYLIN-2845) Merging cube get error
IllegalStateException
[ https://issues.apache.org/jira/browse/KYLIN-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
JiangQin updated KYLIN-2845:
----------------------------
Comment: was deleted
(was: The phenomenon takes place when the length of dimensions varies from one segment to another.
With the tool org.apache.kylin.engine.mr.common.CubeStatsReader, I found that the 'Length of dimension' of one column has changed from 1 to 2.
FYI, I did not change the length of this column in hive table , nor did I make any difference in Kylin Cube design ('Advanced Setting'->'Rowkeys')
And the source code I pasted, 'Preconditions.checkState', seems to block this change and throws an exception.)
> Merging cube get error IllegalStateException
> --------------------------------------------
>
> Key: KYLIN-2845
> URL: https://issues.apache.org/jira/browse/KYLIN-2845
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
> Affects Versions: v2.0.0
> Reporter: JiangQin
> Assignee: Dong Li
>
> When trying to merge cubes, I encountered following errors :
> Error: java.lang.IllegalStateException
> at com.google.common.base.Preconditions.checkState(Preconditions.java:129)
> at org.apache.kylin.cube.kv.RowKeyEncoder.encode(RowKeyEncoder.java:123)
> at org.apache.kylin.engine.mr.steps.MergeCuboidMapper.doMap(MergeCuboidMapper.java:246)
> I digged into the source code and found the following code:
> {code:java}
> public void encode(ByteArray bodyBytes, ByteArray outputBuf) {
> Preconditions.checkState(bodyBytes.length() == bodyLength);
> Preconditions.checkState(bodyBytes.length() + getHeaderLength() == outputBuf.length(), //
> "bodybytes length: " + bodyBytes.length() + " outputBuf length: " + outputBuf.length() + " header length: " + getHeaderLength());
> System.arraycopy(bodyBytes.array(), bodyBytes.offset(), outputBuf.array(), getHeaderLength(), bodyLength);
> //fill shard and cuboid
> fillHeader(outputBuf.array());
> }
> {code}
> And also I found an old similar bug [https://issues.apache.org/jira/browse/KYLIN-1004].
> The cube can build correctly, but the merge just does not work.
> Here is the detail for this error:
> 2017-09-05 09:35:57,801 INFO [main] org.apache.kylin.cube.CubeManager: Reloaded cube cube_fact_dui_assemble_log being CUBE[name=cube_fact_dui_assemble_log] having 7 segments
> 2017-09-05 09:35:57,801 INFO [main] org.apache.kylin.cube.CubeManager: Loaded 1 cubes, fail on 0 cubes
> 2017-09-05 09:35:57,805 INFO [main] org.apache.kylin.dict.DictionaryManager: DictionaryManager(1908999587) loading DictionaryInfo(loadDictObj:true) at /dict/DIM_DUI.DIM_CALENDAR/P_DAY/7dbd2f63-ffc6-4eb6-972a-b712954bcff0.dict
> 2017-09-05 09:35:57,821 INFO [main] org.apache.kylin.dict.DictionaryManager: DictionaryManager(1908999587) loading DictionaryInfo(loadDictObj:true) at /dict/DIM_DUI.DIM_DEVELOPER/DEVELOPER_ID/31e3578a-94fe-4d99-b00f-c523f59b8eb2.dict
> 2017-09-05 09:35:57,833 INFO [main] org.apache.kylin.dict.DictionaryManager: DictionaryManager(1908999587) loading DictionaryInfo(loadDictObj:true) at /dict/DIM_DUI.DIM_DEVELOPER/DEVELOPER_ID/9e365c29-52d8-4397-acb9-63cb9214a05e.dict
> 2017-09-05 09:35:57,834 INFO [main] org.apache.kylin.engine.mr.KylinMapper: Accepting Mapper Key with ordinal: 1
> 2017-09-05 09:35:57,837 ERROR [main] org.apache.kylin.engine.mr.KylinMapper:
> java.lang.IllegalStateException
> at com.google.common.base.Preconditions.checkState(Preconditions.java:129)
> at org.apache.kylin.cube.kv.RowKeyEncoder.encode(RowKeyEncoder.java:123)
> at org.apache.kylin.engine.mr.steps.MergeCuboidMapper.doMap(MergeCuboidMapper.java:246)
> at org.apache.kylin.engine.mr.steps.MergeCuboidMapper.doMap(MergeCuboidMapper.java:63)
> at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:48)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
> 2017-09-05 09:35:57,866 INFO [main] org.apache.hadoop.mapred.MapTask: Starting flush of map output
> 2017-09-05 09:35:57,958 INFO [main] org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
> 2017-09-05 09:35:57,964 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.IllegalStateException
> at com.google.common.base.Preconditions.checkState(Preconditions.java:129)
> at org.apache.kylin.cube.kv.RowKeyEncoder.encode(RowKeyEncoder.java:123)
> at org.apache.kylin.engine.mr.steps.MergeCuboidMapper.doMap(MergeCuboidMapper.java:246)
> at org.apache.kylin.engine.mr.steps.MergeCuboidMapper.doMap(MergeCuboidMapper.java:63)
> at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:48)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)