[jira] [Resolved] (SPARK-2200) breeze DenseVector not serializable with KryoSerializer

["Xiangrui Meng (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-2200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Xiangrui Meng resolved SPARK-2200.
----------------------------------

    Resolution: Later

Close this JIRA since this is caused by a non-default kryo setting.

> breeze DenseVector not serializable with KryoSerializer
> -------------------------------------------------------
>
>                 Key: SPARK-2200
>                 URL: https://issues.apache.org/jira/browse/SPARK-2200
>             Project: Spark
>          Issue Type: Bug
>          Components: MLlib
>    Affects Versions: 1.0.0
>            Reporter: Neville Li
>            Priority: Minor
>
> Spark 1.0.0 depends on breeze 0.7 and for some reason serializing DenseVector with KryoSerializer throws the following stack trace. Looks like some recursive field in the object. Upgrading to 0.8.1 solved this.
> {code}
> java.lang.StackOverflowError
> 	at java.lang.reflect.Field.getDeclaringClass(Field.java:154)
> 	at sun.reflect.UnsafeFieldAccessorImpl.ensureObj(UnsafeFieldAccessorImpl.java:54)
> 	at sun.reflect.UnsafeQualifiedObjectFieldAccessorImpl.get(UnsafeQualifiedObjectFieldAccessorImpl.java:38)
> 	at java.lang.reflect.Field.get(Field.java:379)
> 	at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:552)
> 	at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
> 	at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
> 	at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
> 	at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
> 	at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
> ...
> {code}
> Code to reproduce:
> {code}
> import breeze.linalg.DenseVector
> import org.apache.spark.SparkConf
> import org.apache.spark.serializer.KryoSerializer
> object SerializerTest {
>   def main(args: Array[String]) {
>     val conf = new SparkConf()
>       .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
>       .set("spark.kryo.registrator", classOf[MyRegistrator].getName)
>       .set("spark.kryo.referenceTracking", "false")
>       .set("spark.kryoserializer.buffer.mb", "8")
>     val serializer = new KryoSerializer(conf).newInstance()
>     serializer.serialize(DenseVector.rand(10))
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)