You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Arun Lists <li...@gmail.com> on 2015/02/02 03:08:42 UTC
ClassNotFoundException when registering classes with Kryo
Here is the relevant snippet of code in my main program:
===========================================
sparkConf.set("spark.serializer",
"org.apache.spark.serializer.KryoSerializer")
sparkConf.set("spark.kryo.registrationRequired", "true")
val summaryDataClass = classOf[SummaryData]
val summaryViewClass = classOf[SummaryView]
sparkConf.registerKryoClasses(Array(
summaryDataClass, summaryViewClass))
===========================================
I get the following error:
Exception in thread "main" java.lang.reflect.InvocationTargetException
...
Caused by: org.apache.spark.SparkException: Failed to load class to
register with Kryo
...
Caused by: java.lang.ClassNotFoundException:
com.dtex.analysis.transform.SummaryData
Note that the class in question SummaryData is in the same package as the
main program and hence in the same jar.
What do I need to do to make this work?
Thanks,
arun
Re: ClassNotFoundException when registering classes with Kryo
Posted by Arun Lists <li...@gmail.com>.
Thanks for the notification!
For now, I'll use the Kryo serializer without registering classes until the
bug fix has been merged into the next version of Spark (I guess that will
be 1.3, right?).
arun
On Sun, Feb 1, 2015 at 10:58 PM, Shixiong Zhu <zs...@gmail.com> wrote:
> It's a bug that has been fixed in
> https://github.com/apache/spark/pull/4258 but not yet been merged.
>
> Best Regards,
> Shixiong Zhu
>
> 2015-02-02 10:08 GMT+08:00 Arun Lists <li...@gmail.com>:
>
>> Here is the relevant snippet of code in my main program:
>>
>> ===========================================
>>
>> sparkConf.set("spark.serializer",
>> "org.apache.spark.serializer.KryoSerializer")
>> sparkConf.set("spark.kryo.registrationRequired", "true")
>> val summaryDataClass = classOf[SummaryData]
>> val summaryViewClass = classOf[SummaryView]
>> sparkConf.registerKryoClasses(Array(
>>
>> summaryDataClass, summaryViewClass))
>>
>> ===========================================
>>
>> I get the following error:
>>
>> Exception in thread "main" java.lang.reflect.InvocationTargetException
>> ...
>>
>> Caused by: org.apache.spark.SparkException: Failed to load class to
>> register with Kryo
>> ...
>>
>> Caused by: java.lang.ClassNotFoundException:
>> com.dtex.analysis.transform.SummaryData
>>
>>
>> Note that the class in question SummaryData is in the same package as the
>> main program and hence in the same jar.
>>
>> What do I need to do to make this work?
>>
>> Thanks,
>> arun
>>
>>
>>
>
Re: ClassNotFoundException when registering classes with Kryo
Posted by Shixiong Zhu <zs...@gmail.com>.
It's a bug that has been fixed in https://github.com/apache/spark/pull/4258
but not yet been merged.
Best Regards,
Shixiong Zhu
2015-02-02 10:08 GMT+08:00 Arun Lists <li...@gmail.com>:
> Here is the relevant snippet of code in my main program:
>
> ===========================================
>
> sparkConf.set("spark.serializer",
> "org.apache.spark.serializer.KryoSerializer")
> sparkConf.set("spark.kryo.registrationRequired", "true")
> val summaryDataClass = classOf[SummaryData]
> val summaryViewClass = classOf[SummaryView]
> sparkConf.registerKryoClasses(Array(
>
> summaryDataClass, summaryViewClass))
>
> ===========================================
>
> I get the following error:
>
> Exception in thread "main" java.lang.reflect.InvocationTargetException
> ...
>
> Caused by: org.apache.spark.SparkException: Failed to load class to
> register with Kryo
> ...
>
> Caused by: java.lang.ClassNotFoundException:
> com.dtex.analysis.transform.SummaryData
>
>
> Note that the class in question SummaryData is in the same package as the
> main program and hence in the same jar.
>
> What do I need to do to make this work?
>
> Thanks,
> arun
>
>
>