You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by touchdown <yu...@gmail.com> on 2014/12/18 02:14:10 UTC
Re: java.io.NotSerializableException:
org.apache.avro.mapred.AvroKey using spark with avro
Yeah, I have the same problem with 1.1.0, but not 1.0.0.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/java-io-NotSerializableException-org-apache-avro-mapred-AvroKey-using-spark-with-avro-tp15165p20752.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: java.io.NotSerializableException:
org.apache.avro.mapred.AvroKey using spark with avro
Posted by anish <an...@sigmoidanalytics.com>.
Hi, I had the same problem.
One option (starting with Spark 1.2, which is currently in preview) is to
use the Avro library for Spark SQL.
Other is using Kryo Serialization.
by default spark uses Java Serialization, you can specify kryo serialization
while creating spark context.
val conf = new SparkConf().set("spark.serializer",
"org.apache.spark.serializer.KryoSerializer")
val sc = new SparkContext(conf)
This worked for me.
Regards,
Anish
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/java-io-NotSerializableException-org-apache-avro-mapred-AvroKey-using-spark-with-avro-tp15165p20761.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: java.io.NotSerializableException: org.apache.avro.mapred.AvroKey
using spark with avro
Posted by Anish Haldiya <an...@sigmoidanalytics.com>.
Hi, I had the same problem.
One option (starting with Spark 1.2, which is currently in preview) is to
use the Avro library for Spark SQL.
Other is using Kryo Serialization.
by default spark uses Java Serialization, you can specify kryo
serialization while creating spark context.
val conf = new SparkConf().set("spark.serializer",
"org.apache.spark.serializer.KryoSerializer")
val sc = new SparkContext(conf)
This worked for me.
Regards,
Anish
Re: java.io.NotSerializableException: org.apache.avro.mapred.AvroKey
using spark with avro
Posted by "M. Dale" <me...@yahoo.com.INVALID>.
I did not encounter this with my Avro records using Spark 1.10 (see
https://github.com/medale/spark-mail/blob/master/analytics/src/main/scala/com/uebercomputing/analytics/basic/UniqueSenderCounter.scala).
I do use the default Java serialization but all the fields in my Avro
object are Serializable (no bytes/ByteBuffer). Does your Avro schema use
bytes? If so, it seems that is wrapped in ByteBuffer, which is not
Serializable. A quick search has a fix here:
https://groups.google.com/forum/#!topic/spark-users/6HQPuxsCe0c
Hope this helps,
Markus
On 12/17/2014 08:14 PM, touchdown wrote:
> Yeah, I have the same problem with 1.1.0, but not 1.0.0.
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/java-io-NotSerializableException-org-apache-avro-mapred-AvroKey-using-spark-with-avro-tp15165p20752.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org