You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by Fan Xie <fa...@kyligence.io> on 2017/03/09 08:18:35 UTC
答复: Extract Fact Table Distinct Columns Error while don't have any distinct columns---kylin 2.0 beta
The FactDistinctColumnsMapper is used by all kinds of dictionary encoding dimensions. I guest that there may be some very long values in some of your dimension in your test data and it's size exceed 4096 bytes. And thus cause a buffer overflow exception.
________________________________
发件人: 赵天烁 <zh...@meizu.com>
发送时间: 2017年3月9日 15:56:47
收件人: user
主题: Extract Fact Table Distinct Columns Error while don't have any distinct columns---kylin 2.0 beta
run into java.nio.BufferOverflowException while building a cube:
Error: java.nio.BufferOverflowException at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:183) at java.nio.ByteBuffer.put(ByteBuffer.java:832) at org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper.doMap(FactDistinctColumnsMapper.java:157) at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:48) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
but I don't have any distinct column,the only meature I have are as follow:
[cid:_Foxmail.1@bf2308b4-3f6d-5a7f-a980-a55601976ad3]
a count and two raw measure.I'm curious about does raw measure use the same kind of mapper as count distinct ?
________________________________
赵天烁
Kevin Zhao
zhaotianshuo@meizu.com<ma...@meizu.com>
珠海市魅族科技有限公司
MEIZU Technology Co., Ltd.
广东省珠海市科技创新海岸魅族科技楼
MEIZU Tech Bldg., Technology & Innovation Coast
Zhuhai, 519085, Guangdong, China
meizu.com
Re: 答复: Extract Fact Table Distinct Columns Error while don't have any distinct columns---kylin 2.0 beta
Posted by Li Yang <li...@apache.org>.
Not at the moment. Please open a JIRA, it may catch the 2.0 release.
On Thu, Mar 9, 2017 at 4:53 PM, 赵天烁 <zh...@meizu.com> wrote:
> ok,is there any way to walk around that issue?
>
> ------------------------------
>
> 赵天烁
>
> Kevin Zhao
>
> *zhaotianshuo@meizu.com <zh...@meizu.com>*
>
>
>
> 珠海市魅族科技有限公司
>
> MEIZU Technology Co., Ltd.
>
> 广东省珠海市科技创新海岸魅族科技楼
>
> MEIZU Tech Bldg., Technology & Innovation Coast
>
> Zhuhai, 519085, Guangdong, China
>
> meizu.com
>
>
> *发件人：* Fan Xie <fa...@kyligence.io>
> *发送时间：* 2017-03-09 16:51
> *收件人：* user@kylin.apache.org
> *主题：* 答复: Extract Fact Table Distinct Columns Error while don't have any
> distinct columns---kylin 2.0 beta
>
> The FactDistinctColumnsMapper is used by all kinds of dictionary encoding
> dimensions. I guest that there may be some very long values in some of your
> dimension in your test data and it's size exceed 4096 bytes. And thus
> cause a buffer overflow exception.
> ------------------------------
> *发件人:* 赵天烁 <zh...@meizu.com>
> *发送时间:* 2017年3月9日 15:56:47
> *收件人:* user
> *主题:* Extract Fact Table Distinct Columns Error while don't have any
> distinct columns---kylin 2.0 beta
>
> run into java.nio.BufferOverflowException while building a cube:
>
> Error: java.nio.BufferOverflowException at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:183)
> at java.nio.ByteBuffer.put(ByteBuffer.java:832) at
> org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper.doMap(
> FactDistinctColumnsMapper.java:157) at org.apache.kylin.engine.mr.
> KylinMapper.map(KylinMapper.java:48) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:415) at
> org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1556) at org.apache.hadoop.mapred.
> YarnChild.main(YarnChild.java:162)
>
> but I don't have any distinct column,the only meature I have are as follow:
> a count and two raw measure.I'm curious about does raw measure use the
> same kind of mapper as count distinct ?
>
>
> ------------------------------
>
> 赵天烁
>
> Kevin Zhao
>
> *zhaotianshuo@meizu.com <zh...@meizu.com>*
>
>
>
> 珠海市魅族科技有限公司
>
> MEIZU Technology Co., Ltd.
>
> 广东省珠海市科技创新海岸魅族科技楼
>
> MEIZU Tech Bldg., Technology & Innovation Coast
>
> Zhuhai, 519085, Guangdong, China
>
> meizu.com
>
>
回复: 答复: Extract Fact Table Distinct Columns Error while don't have any distinct columns---kylin 2.0 beta
Posted by 赵天烁 <zh...@meizu.com>.
ok,is there any way to walk around that issue?
________________________________
赵天烁
Kevin Zhao
zhaotianshuo@meizu.com<ma...@meizu.com>
珠海市魅族科技有限公司
MEIZU Technology Co., Ltd.
广东省珠海市科技创新海岸魅族科技楼
MEIZU Tech Bldg., Technology & Innovation Coast
Zhuhai, 519085, Guangdong, China
meizu.com
发件人： Fan Xie<ma...@kyligence.io>
发送时间： 2017-03-09 16:51
收件人： user@kylin.apache.org<ma...@kylin.apache.org>
主题： 答复: Extract Fact Table Distinct Columns Error while don't have any distinct columns---kylin 2.0 beta
The FactDistinctColumnsMapper is used by all kinds of dictionary encoding dimensions. I guest that there may be some very long values in some of your dimension in your test data and it's size exceed 4096 bytes. And thus cause a buffer overflow exception.
________________________________
发件人: 赵天烁 <zh...@meizu.com>
发送时间: 2017年3月9日 15:56:47
收件人: user
主题: Extract Fact Table Distinct Columns Error while don't have any distinct columns---kylin 2.0 beta
run into java.nio.BufferOverflowException while building a cube:
Error: java.nio.BufferOverflowException at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:183) at java.nio.ByteBuffer.put(ByteBuffer.java:832) at org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper.doMap(FactDistinctColumnsMapper.java:157) at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:48) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
but I don't have any distinct column,the only meature I have are as follow:
[cid:_Foxmail.1@a035c257-7349-2008-c921-14b6ff365591]
a count and two raw measure.I'm curious about does raw measure use the same kind of mapper as count distinct ?
________________________________
赵天烁
Kevin Zhao
zhaotianshuo@meizu.com<ma...@meizu.com>
珠海市魅族科技有限公司
MEIZU Technology Co., Ltd.
广东省珠海市科技创新海岸魅族科技楼
MEIZU Tech Bldg., Technology & Innovation Coast
Zhuhai, 519085, Guangdong, China
meizu.com