You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Simone Franzini <ca...@gmail.com> on 2014/11/25 21:38:27 UTC

Kryo NPE with Array

I am running into the following NullPointerException:

com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException
Serialization trace:
underlying (scala.collection.convert.Wrappers$JListWrapper)
myArrayField (MyCaseClass)
at
com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)

I have been running into similar issues when using avro classes, that I was
able to resolve by registering them with a Kryo serializer that uses
chill-avro. However, in this case the field is in a case class and it seems
that registering the class does not help.

I found this stack overflow that seems to be relevant:
http://stackoverflow.com/questions/23962796/kryo-readobject-cause-nullpointerexception-with-arraylist
I have this line of code translated to Scala, that supposedly solves the
issue:

val kryo = new Kryo()
kryo.getInstantiatorStrategy().asInstanceOf[Kryo.DefaultInstantiatorStrategy].setFallbackInstantiatorStrategy(new
StdInstantiatorStrategy())

However, I am not sure where this line should be placed to take effect.

I already have the following, should it go somewhere in here?
class MyRegistrator extends KryoRegistrator {
    override def registerClasses(kryo: Kryo) {
        kryo.register(...)
    }
}


Simone Franzini, PhD

http://www.linkedin.com/in/simonefranzini

Re: Kryo NPE with Array

Posted by Simone Franzini <ca...@gmail.com>.

I finally solved this issue. The problem was that:
1. I defined a case class with a Buffer[MyType] field.
2. I instantiated the class with the field set to the value given by an
implicit conversion from a Java list, which is supposedly a Buffer.
3. However, the underlying type of that field was instead
scala.collection.convert.Wrappers.JListWrapper, as noted in the exception
above. This type was not registered with Kryo and so that's why I got the
exception.

Registering the type did not solve the problem. However, an additional call
to .toBuffer did solve the problem, since the Buffer class is registered
through the Chill AllScalaRegistrar which is called by the Spark Kryo
serializer.

I thought I'd document this in case somebody else is running into a similar
issue.

Simone Franzini, PhD

http://www.linkedin.com/in/simonefranzini

On Wed, Nov 26, 2014 at 7:40 PM, Simone Franzini <ca...@gmail.com>
wrote:

> I guess I already have the answer of what I have to do here, which is to
> configure the kryo object with the strategy as above.
> Now the question becomes: how can I pass this custom kryo configuration to
> the spark kryo serializer / kryo registrator?
> I've had a look at the code but I am still fairly new to Scala and I can't
> see how I would do this. In the worst case, could I override the newKryo
> method and put my configuration there? It appears to me that method is the
> one where the kryo instance is created.
>
> Simone Franzini, PhD
>
> http://www.linkedin.com/in/simonefranzini
>
> On Tue, Nov 25, 2014 at 2:38 PM, Simone Franzini <ca...@gmail.com>
> wrote:
>
>> I am running into the following NullPointerException:
>>
>> com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException
>> Serialization trace:
>> underlying (scala.collection.convert.Wrappers$JListWrapper)
>> myArrayField (MyCaseClass)
>> at
>> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
>>
>> I have been running into similar issues when using avro classes, that I
>> was able to resolve by registering them with a Kryo serializer that uses
>> chill-avro. However, in this case the field is in a case class and it seems
>> that registering the class does not help.
>>
>> I found this stack overflow that seems to be relevant:
>>
>> http://stackoverflow.com/questions/23962796/kryo-readobject-cause-nullpointerexception-with-arraylist
>> I have this line of code translated to Scala, that supposedly solves the
>> issue:
>>
>> val kryo = new Kryo()
>> kryo.getInstantiatorStrategy().asInstanceOf[Kryo.DefaultInstantiatorStrategy].setFallbackInstantiatorStrategy(new
>> StdInstantiatorStrategy())
>>
>> However, I am not sure where this line should be placed to take effect.
>>
>> I already have the following, should it go somewhere in here?
>> class MyRegistrator extends KryoRegistrator {
>>     override def registerClasses(kryo: Kryo) {
>>         kryo.register(...)
>>     }
>> }
>>
>>
>> Simone Franzini, PhD
>>
>> http://www.linkedin.com/in/simonefranzini
>>
>
>

Re: Kryo NPE with Array

Posted by Simone Franzini <ca...@gmail.com>.

I guess I already have the answer of what I have to do here, which is to
configure the kryo object with the strategy as above.
Now the question becomes: how can I pass this custom kryo configuration to
the spark kryo serializer / kryo registrator?
I've had a look at the code but I am still fairly new to Scala and I can't
see how I would do this. In the worst case, could I override the newKryo
method and put my configuration there? It appears to me that method is the
one where the kryo instance is created.

Simone Franzini, PhD

http://www.linkedin.com/in/simonefranzini

On Tue, Nov 25, 2014 at 2:38 PM, Simone Franzini <ca...@gmail.com>
wrote:

> I am running into the following NullPointerException:
>
> com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException
> Serialization trace:
> underlying (scala.collection.convert.Wrappers$JListWrapper)
> myArrayField (MyCaseClass)
> at
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
>
> I have been running into similar issues when using avro classes, that I
> was able to resolve by registering them with a Kryo serializer that uses
> chill-avro. However, in this case the field is in a case class and it seems
> that registering the class does not help.
>
> I found this stack overflow that seems to be relevant:
>
> http://stackoverflow.com/questions/23962796/kryo-readobject-cause-nullpointerexception-with-arraylist
> I have this line of code translated to Scala, that supposedly solves the
> issue:
>
> val kryo = new Kryo()
> kryo.getInstantiatorStrategy().asInstanceOf[Kryo.DefaultInstantiatorStrategy].setFallbackInstantiatorStrategy(new
> StdInstantiatorStrategy())
>
> However, I am not sure where this line should be placed to take effect.
>
> I already have the following, should it go somewhere in here?
> class MyRegistrator extends KryoRegistrator {
>     override def registerClasses(kryo: Kryo) {
>         kryo.register(...)
>     }
> }
>
>
> Simone Franzini, PhD
>
> http://www.linkedin.com/in/simonefranzini
>