You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by "zhaoguoxian (Jira)" <ji...@apache.org> on 2019/12/27 10:32:00 UTC
[jira] [Commented] (KYLIN-2654) Kylin failed to build cube at 3rd
step on Hadoop 2.6.0-cdh5.4.4 + jdk 1.8
[ https://issues.apache.org/jira/browse/KYLIN-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17004079#comment-17004079 ]
zhaoguoxian commented on KYLIN-2654:
------------------------------------
I encountered the same problem. Then your yarn environment is cdh and your 'mapreduce.job.map.output.collector.class' param is org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator. you can not use your Custom writable type or comparator，example，kylin`s org.apache.kylin.engine.mr.steps.SelfDefineSortableKey. my solution was by setting the param
mapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.MapTask$MapOutputBuffer. btw. my environment was cdh5.5.2
> Kylin failed to build cube at 3rd step on Hadoop 2.6.0-cdh5.4.4 + jdk 1.8
> -------------------------------------------------------------------------
>
> Key: KYLIN-2654
> URL: https://issues.apache.org/jira/browse/KYLIN-2654
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
> Affects Versions: v2.0.0
> Reporter: Shao Feng Shi
> Assignee: Dong Li
> Priority: Major
>
> From community user qiumingming:
> When I use kylin 2.0 to build cube, I got a error at Extract Fact Table
> Distinct Columns step, mr job failed. My Hadoop version is 2.6.0-cdh5.4.4.
> Below is my error info:
> {code}
> Error: java.io.IOException: Initialization of all the collectors failed.
> Error in last collector was :Cannot find serializer for
> org.apache.kylin.engine.mr.steps.SelfDefineSortableKey at
> org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:423) at
> org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:82) at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:718)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:790) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:350) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by:
> java.io.IOException: Cannot find serializer for
> org.apache.kylin.engine.mr.steps.SelfDefineSortableKey at
> org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator.init(NativeMapOutputCollectorDelegator.java:147)
> at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:411)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)