You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by 凡梦星尘 <el...@gmail.com> on 2017/06/06 02:00:09 UTC
Kylin2.0.0在Spark模式下报Class未注册的问题
Hi guys：
按照官方的参考：https://kylin.apache.org/docs20/tutorial/cube_spark.html
尝试最新的版本的KYLIN2.0.0版本下的Spark模式，配置启动都是OK，导入sample数据也可以正常构建CUBE，但是当构建自有数据时就失败了，异常的堆栈信息如下，这是个BUG吗？目前有没解决的办法呢？
Job aborted due to stage failure: Task 1 in stage 4.0 failed 4 times,
most recent failure: Lost task 1.3 in stage 4.0 (TID 29,
test-hdp-cluster01): com.esotericsoftware.kryo.KryoException:
java.lang.IllegalArgumentException: Class is not registered:
org.apache.kylin.job.shaded.org.roaringbitmap.buffer.MappeableBitmapContainer
Note: To register this class use:
kryo.register(org.apache.kylin.job.shaded.org.roaringbitmap.buffer.MappeableBitmapContainer.class);
Serialization trace:
values (org.apache.kylin.job.shaded.org.roaringbitmap.buffer.MutableRoaringArray)
highLowContainer
(org.apache.kylin.job.shaded.org.roaringbitmap.buffer.MutableRoaringBitmap)
bitmap (org.apache.kylin.measure.bitmap.RoaringBitmapCounter)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:585)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:318)
at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:293)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:37)
at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
at org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:194)
at org.apache.spark.serializer.SerializationStream.writeAll(Serializer.scala:153)
at org.apache.spark.storage.BlockManager.dataSerializeStream(BlockManager.scala:1252)
at org.apache.spark.storage.BlockManager.dataSerialize(BlockManager.scala:1258)
at org.apache.spark.storage.MemoryStore.putArray(MemoryStore.scala:136)
at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:811)
at org.apache.spark.storage.BlockManager.putArray(BlockManager.scala:687)
at org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:175)
at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:78)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:268)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Class is not
registered: org.apache.kylin.job.shaded.org.roaringbitmap.buffer.MappeableBitmapContainer
Note: To register this class use:
kryo.register(org.apache.kylin.job.shaded.org.roaringbitmap.buffer.MappeableBitmapContainer.class);
at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:442)
at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:79)
at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:472)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:565)
at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:318)
at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:293)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
... 33 more
Driver stacktrace:
Re: Kylin2.0.0在Spark模式下报Class未注册的问题
Posted by Li Yang <li...@apache.org>.
Billy really loves coding... just cannot help it. ;-)
On Tue, Jun 6, 2017 at 10:20 AM, Billy Liu <bi...@apache.org> wrote:
> This is known issue. Could you try the latest code on the master branch? It
> should be fixed already.
> https://issues.apache.org/jira/browse/KYLIN-2592
>
> 2017-06-06 10:00 GMT+08:00 凡梦星尘 <el...@gmail.com>:
>
> > Hi guys：
> >
> > 按照官方的参考：https://kylin.apache.org/docs20/tutorial/cube_spark.html
> > 尝试最新的版本的KYLIN2.0.0版本下的Spark模式，配置启动都是OK，导入sample数据也可以正常构建CUBE，
> > 但是当构建自有数据时就失败了，异常的堆栈信息如下，这是个BUG吗？目前有没解决的办法呢？
> >
> > Job aborted due to stage failure: Task 1 in stage 4.0 failed 4 times,
> > most recent failure: Lost task 1.3 in stage 4.0 (TID 29,
> > test-hdp-cluster01): com.esotericsoftware.kryo.KryoException:
> > java.lang.IllegalArgumentException: Class is not registered:
> > org.apache.kylin.job.shaded.org.roaringbitmap.buffer.
> > MappeableBitmapContainer
> > Note: To register this class use:
> > kryo.register(org.apache.kylin.job.shaded.org.roaringbitmap.buffer.
> > MappeableBitmapContainer.class);
> > Serialization trace:
> > values (org.apache.kylin.job.shaded.org.roaringbitmap.buffer.
> > MutableRoaringArray)
> > highLowContainer
> > (org.apache.kylin.job.shaded.org.roaringbitmap.buffer.
> > MutableRoaringBitmap)
> > bitmap (org.apache.kylin.measure.bitmap.RoaringBitmapCounter)
> > at com.esotericsoftware.kryo.serializers.FieldSerializer$
> > ObjectField.write(FieldSerializer.java:585)
> > at com.esotericsoftware.kryo.serializers.FieldSerializer.
> > write(FieldSerializer.java:213)
> > at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
> > at com.esotericsoftware.kryo.serializers.FieldSerializer$
> > ObjectField.write(FieldSerializer.java:564)
> > at com.esotericsoftware.kryo.serializers.FieldSerializer.
> > write(FieldSerializer.java:213)
> > at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
> > at com.esotericsoftware.kryo.serializers.FieldSerializer$
> > ObjectField.write(FieldSerializer.java:564)
> > at com.esotericsoftware.kryo.serializers.FieldSerializer.
> > write(FieldSerializer.java:213)
> > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.
> > java:568)
> > at com.esotericsoftware.kryo.serializers.
> DefaultArraySerializers$
> > ObjectArraySerializer.write(DefaultArraySerializers.java:318)
> > at com.esotericsoftware.kryo.serializers.
> DefaultArraySerializers$
> > ObjectArraySerializer.write(DefaultArraySerializers.java:293)
> > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.
> > java:568)
> > at com.twitter.chill.Tuple2Serializer.write(
> > TupleSerializers.scala:37)
> > at com.twitter.chill.Tuple2Serializer.write(
> > TupleSerializers.scala:33)
> > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.
> > java:568)
> > at org.apache.spark.serializer.KryoSerializationStream.
> > writeObject(KryoSerializer.scala:194)
> > at org.apache.spark.serializer.SerializationStream.writeAll(
> > Serializer.scala:153)
> > at org.apache.spark.storage.BlockManager.dataSerializeStream(
> > BlockManager.scala:1252)
> > at org.apache.spark.storage.BlockManager.dataSerialize(
> > BlockManager.scala:1258)
> > at org.apache.spark.storage.MemoryStore.putArray(
> > MemoryStore.scala:136)
> > at org.apache.spark.storage.BlockManager.doPut(
> > BlockManager.scala:811)
> > at org.apache.spark.storage.BlockManager.putArray(
> > BlockManager.scala:687)
> > at org.apache.spark.CacheManager.putInBlockManager(
> > CacheManager.scala:175)
> > at org.apache.spark.CacheManager.getOrCompute(CacheManager.
> > scala:78)
> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:268)
> > at org.apache.spark.rdd.MapPartitionsRDD.compute(
> > MapPartitionsRDD.scala:38)
> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.
> scala:306)
> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
> > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.
> > scala:66)
> > at org.apache.spark.scheduler.Task.run(Task.scala:89)
> > at org.apache.spark.executor.Executor$TaskRunner.run(
> > Executor.scala:227)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> > ThreadPoolExecutor.java:1142)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:748)
> > Caused by: java.lang.IllegalArgumentException: Class is not
> > registered: org.apache.kylin.job.shaded.org.roaringbitmap.buffer.
> > MappeableBitmapContainer
> > Note: To register this class use:
> > kryo.register(org.apache.kylin.job.shaded.org.roaringbitmap.buffer.
> > MappeableBitmapContainer.class);
> > at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:442)
> > at com.esotericsoftware.kryo.util.DefaultClassResolver.
> writeClass(
> > DefaultClassResolver.java:79)
> > at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:472)
> > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.
> > java:565)
> > at com.esotericsoftware.kryo.serializers.
> DefaultArraySerializers$
> > ObjectArraySerializer.write(DefaultArraySerializers.java:318)
> > at com.esotericsoftware.kryo.serializers.
> DefaultArraySerializers$
> > ObjectArraySerializer.write(DefaultArraySerializers.java:293)
> > at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
> > at com.esotericsoftware.kryo.serializers.FieldSerializer$
> > ObjectField.write(FieldSerializer.java:564)
> > ... 33 more
> >
> > Driver stacktrace:
> >
>
Re: Kylin2.0.0在Spark模式下报Class未注册的问题
Posted by Billy Liu <bi...@apache.org>.
This is known issue. Could you try the latest code on the master branch? It
should be fixed already.
https://issues.apache.org/jira/browse/KYLIN-2592
2017-06-06 10:00 GMT+08:00 凡梦星尘 <el...@gmail.com>:
> Hi guys：
>
> 按照官方的参考：https://kylin.apache.org/docs20/tutorial/cube_spark.html
> 尝试最新的版本的KYLIN2.0.0版本下的Spark模式，配置启动都是OK，导入sample数据也可以正常构建CUBE，
> 但是当构建自有数据时就失败了，异常的堆栈信息如下，这是个BUG吗？目前有没解决的办法呢？
>
> Job aborted due to stage failure: Task 1 in stage 4.0 failed 4 times,
> most recent failure: Lost task 1.3 in stage 4.0 (TID 29,
> test-hdp-cluster01): com.esotericsoftware.kryo.KryoException:
> java.lang.IllegalArgumentException: Class is not registered:
> org.apache.kylin.job.shaded.org.roaringbitmap.buffer.
> MappeableBitmapContainer
> Note: To register this class use:
> kryo.register(org.apache.kylin.job.shaded.org.roaringbitmap.buffer.
> MappeableBitmapContainer.class);
> Serialization trace:
> values (org.apache.kylin.job.shaded.org.roaringbitmap.buffer.
> MutableRoaringArray)
> highLowContainer
> (org.apache.kylin.job.shaded.org.roaringbitmap.buffer.
> MutableRoaringBitmap)
> bitmap (org.apache.kylin.measure.bitmap.RoaringBitmapCounter)
> at com.esotericsoftware.kryo.serializers.FieldSerializer$
> ObjectField.write(FieldSerializer.java:585)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.
> write(FieldSerializer.java:213)
> at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
> at com.esotericsoftware.kryo.serializers.FieldSerializer$
> ObjectField.write(FieldSerializer.java:564)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.
> write(FieldSerializer.java:213)
> at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
> at com.esotericsoftware.kryo.serializers.FieldSerializer$
> ObjectField.write(FieldSerializer.java:564)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.
> write(FieldSerializer.java:213)
> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.
> java:568)
> at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$
> ObjectArraySerializer.write(DefaultArraySerializers.java:318)
> at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$
> ObjectArraySerializer.write(DefaultArraySerializers.java:293)
> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.
> java:568)
> at com.twitter.chill.Tuple2Serializer.write(
> TupleSerializers.scala:37)
> at com.twitter.chill.Tuple2Serializer.write(
> TupleSerializers.scala:33)
> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.
> java:568)
> at org.apache.spark.serializer.KryoSerializationStream.
> writeObject(KryoSerializer.scala:194)
> at org.apache.spark.serializer.SerializationStream.writeAll(
> Serializer.scala:153)
> at org.apache.spark.storage.BlockManager.dataSerializeStream(
> BlockManager.scala:1252)
> at org.apache.spark.storage.BlockManager.dataSerialize(
> BlockManager.scala:1258)
> at org.apache.spark.storage.MemoryStore.putArray(
> MemoryStore.scala:136)
> at org.apache.spark.storage.BlockManager.doPut(
> BlockManager.scala:811)
> at org.apache.spark.storage.BlockManager.putArray(
> BlockManager.scala:687)
> at org.apache.spark.CacheManager.putInBlockManager(
> CacheManager.scala:175)
> at org.apache.spark.CacheManager.getOrCompute(CacheManager.
> scala:78)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:268)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(
> MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.
> scala:66)
> at org.apache.spark.scheduler.Task.run(Task.scala:89)
> at org.apache.spark.executor.Executor$TaskRunner.run(
> Executor.scala:227)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.IllegalArgumentException: Class is not
> registered: org.apache.kylin.job.shaded.org.roaringbitmap.buffer.
> MappeableBitmapContainer
> Note: To register this class use:
> kryo.register(org.apache.kylin.job.shaded.org.roaringbitmap.buffer.
> MappeableBitmapContainer.class);
> at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:442)
> at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(
> DefaultClassResolver.java:79)
> at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:472)
> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.
> java:565)
> at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$
> ObjectArraySerializer.write(DefaultArraySerializers.java:318)
> at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$
> ObjectArraySerializer.write(DefaultArraySerializers.java:293)
> at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
> at com.esotericsoftware.kryo.serializers.FieldSerializer$
> ObjectField.write(FieldSerializer.java:564)
> ... 33 more
>
> Driver stacktrace:
>