You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Mudassar Sarwar <mu...@northbaysolutions.net> on 2014/10/01 15:13:02 UTC
KryoSerializer exception in Spark Streaming JAVA
Hi,
I'm implementing KryoSerializer for my custom class. Here is class
public class ImpressionFactsValue implements KryoSerializable {
private int hits;
public ImpressionFactsValue() {
}
public int getHits() {
return hits;
}
public void setHits(int hits) {
this.hits = hits;
}
public void read(Kryo kryo, Input input) {
this.hits = input.readInt();
//kryo.readClass(input);
}
public void write(Kryo arg0, Output output) {
output.writeInt(this.hits);
//arg0.writeClass(output, ImpressionFactsValue.class);
}
}
Here is my KryoRegistrator
class MyRegistrator implements KryoRegistrator {
public void registerClasses(Kryo kryo) {
kryo.register(ImpressionFactsValue.class);
}
}
Here is spark conf
conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer");
conf.set("spark.kryo.registrator", "com.MyRegistrator");
Getting exception
14/10/01 17:15:38 ERROR serializer.KryoSerializer: Failed to run
spark.kryo.registrator
java.lang.IllegalAccessException: Class
org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$2 can not access
a member of class com.MyRegistrator with modifiers ""
at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:95)
at java.lang.Class.newInstance0(Class.java:366)
at java.lang.Class.newInstance(Class.java:325)
at
org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$2.apply(KryoSerializer.scala:78)
at
org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$2.apply(KryoSerializer.scala:76)
at scala.Option.foreach(Option.scala:236)
at
org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:76)
at
org.apache.spark.serializer.KryoSerializerInstance.<init>(KryoSerializer.scala:133)
at
org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:95)
at
org.apache.spark.util.collection.ExternalAppendOnlyMap.<init>(ExternalAppendOnlyMap.scala:109)
at org.apache.spark.Aggregator.combineValuesByKey(Aggregator.scala:57)
at
org.apache.spark.rdd.PairRDDFunctions$$anonfun$1.apply(PairRDDFunctions.scala:96)
at
org.apache.spark.rdd.PairRDDFunctions$$anonfun$1.apply(PairRDDFunctions.scala:95)
at org.apache.spark.rdd.RDD$$anonfun$15.apply(RDD.scala:594)
at org.apache.spark.rdd.RDD$$anonfun$15.apply(RDD.scala:594)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:158)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
at org.apache.spark.scheduler.Task.run(Task.scala:51)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:183)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Please help to resolve this
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/KryoSerializer-exception-in-Spark-Streaming-JAVA-tp15479.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: KryoSerializer exception in Spark Streaming JAVA
Posted by Tathagata Das <ta...@gmail.com>.
Also please make sure you are specifying the fully qualified name of
registrator class in the sparkconf configuration correctly.
On Dec 11, 2014 5:57 PM, "bonnahu" <bo...@gmail.com> wrote:
> class MyRegistrator implements KryoRegistrator {
>
> public void registerClasses(Kryo kryo) {
> kryo.register(ImpressionFactsValue.class);
> }
>
> }
>
> change this class to public and give a try
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/KryoSerializer-exception-in-Spark-Streaming-JAVA-tp15479p20647.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: KryoSerializer exception in Spark Streaming JAVA
Posted by bonnahu <bo...@gmail.com>.
class MyRegistrator implements KryoRegistrator {
public void registerClasses(Kryo kryo) {
kryo.register(ImpressionFactsValue.class);
}
}
change this class to public and give a try
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/KryoSerializer-exception-in-Spark-Streaming-JAVA-tp15479p20647.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org