how to register CompactBuffer in Kryo

[donhoff_h <16...@qq.com>]

Hi, all


I wrote a spark program which uses the Kryo serialization. When I count a rdd which type is RDD[(String,String)], it reported an Exception like the following :
* Class is not registered: org.apache.spark.util.collection.CompactBuffer[]
* Note: To register this class use: kryo.register(org.apache.spark.util.collection.CompactBuffer[].class);



I found that CompactBuffer is a private class that I could not use the API sparkConf.registerKryoClasses to register it. Although I found a solution from Andras Barjak that is using the statement : kryo.register(ClassTag(Class.forName("org.apache.spark.util.collection.CompactBuffer")).wrap.runtimeClass), I tried it with sparkConfg.registerKryoClasses, and only got another Exception :
* Class is not registered: scala.reflect.ManifestFactory$$anon$1


Could anybody tell me :
1.Why should I register CompactBuffer[] ? My program never used it.
2.How to solve this problem?


Many Thanks!

Re: how to register CompactBuffer in Kryo

[Ted Yu <yu...@gmail.com>]

For the exception w.r.t. ManifestFactory , there is SPARK-6497 which is
Open.

FYI

On Fri, Aug 28, 2015 at 8:25 AM, donhoff_h <16...@qq.com> wrote:

> Hi, all
>
> I wrote a spark program which uses the Kryo serialization. When I count a
> rdd which type is RDD[(String,String)], it reported an Exception like the
> following :
> * Class is not registered: org.apache.spark.util.collection.CompactBuffer[]
> * Note: To register this class use:
> kryo.register(org.apache.spark.util.collection.CompactBuffer[].class);
>
> I found that CompactBuffer is a private class that I could not use the API
> sparkConf.registerKryoClasses to register it. Although I found a solution
> from Andras Barjak that is using the statement :
> kryo.register(ClassTag(Class.forName("org.apache.spark.util.collection.CompactBuffer")).wrap.runtimeClass),
> I tried it with sparkConfg.registerKryoClasses, and only got another
> Exception :
> * Class is not registered: scala.reflect.ManifestFactory$$anon$1
>
> Could anybody tell me :
> 1.Why should I register CompactBuffer[] ? My program never used it.
> 2.How to solve this problem?
>
> Many Thanks!
>
>