You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Neville Li (JIRA)" <ji...@apache.org> on 2014/06/19 17:26:24 UTC
[jira] [Created] (SPARK-2200) breeze DenseVector not serializable
with KryoSerializer
Neville Li created SPARK-2200:
---------------------------------
Summary: breeze DenseVector not serializable with KryoSerializer
Key: SPARK-2200
URL: https://issues.apache.org/jira/browse/SPARK-2200
Project: Spark
Issue Type: Bug
Components: MLlib
Affects Versions: 1.0.0
Reporter: Neville Li
Priority: Minor
Spark 1.0.0 depends on breeze 0.7 and for some reason serializing DenseVector with KryoSerializer throws the following stack trace. Looks like some recursive field in the object. Upgrading to 0.8.1 solved this.
{code}
java.lang.StackOverflowError
at java.lang.reflect.Field.getDeclaringClass(Field.java:154)
at sun.reflect.UnsafeFieldAccessorImpl.ensureObj(UnsafeFieldAccessorImpl.java:54)
at sun.reflect.UnsafeQualifiedObjectFieldAccessorImpl.get(UnsafeQualifiedObjectFieldAccessorImpl.java:38)
at java.lang.reflect.Field.get(Field.java:379)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:552)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
...
{code}
Code to reproduce:
{code}
import breeze.linalg.DenseVector
import org.apache.spark.SparkConf
import org.apache.spark.serializer.KryoSerializer
object SerializerTest {
def main(args: Array[String]) {
val conf = new SparkConf()
.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.set("spark.kryo.registrator", classOf[MyRegistrator].getName)
.set("spark.kryo.referenceTracking", "false")
.set("spark.kryoserializer.buffer.mb", "8")
val serializer = new KryoSerializer(conf).newInstance()
serializer.serialize(DenseVector.rand(10))
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)