You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by dash <bs...@nd.edu> on 2014/07/02 19:03:40 UTC
Worker can not find custom KryoRegistrator
Hi, I'm using Spark 1.1.0 standalone with 5 workers and 1 driver, and Kryo
settings are
When I submit this job, the driver works fine but workers will throw
ClassNotFoundException saying they can not found HDTMKryoRegistrator.
Any idea about this problem? I googled this but there is only one post looks
similar to my problem and seems it is fixed in 0.6
https://groups.google.com/forum/#!topic/spark-users/YHHkg7ELwRs
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Worker-can-not-find-custom-KryoRegistrator-tp8659.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Worker can not find custom KryoRegistrator
Posted by "Baoxu Shi(Dash)" <bs...@nd.edu>.
Don’t know why the setting does not appear in the last mail:
.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.set("spark.kryo.registrator", new HDTMKryoRegistrator().getClass.getName)
On Jul 2, 2014, at 1:03 PM, dash <bs...@nd.edu> wrote:
> Hi, I'm using Spark 1.1.0 standalone with 5 workers and 1 driver, and Kryo
> settings are
>
>
>
> When I submit this job, the driver works fine but workers will throw
> ClassNotFoundException saying they can not found HDTMKryoRegistrator.
>
> Any idea about this problem? I googled this but there is only one post looks
> similar to my problem and seems it is fixed in 0.6
>
> https://groups.google.com/forum/#!topic/spark-users/YHHkg7ELwRs
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Worker-can-not-find-custom-KryoRegistrator-tp8659.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.