You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Felix Neutatz <ne...@googlemail.com> on 2015/07/09 21:42:26 UTC

serialization issue

Hi,

I want to use t-digest by Ted Dunning (
https://github.com/tdunning/t-digest/blob/master/src/main/java/com/tdunning/math/stats/ArrayDigest.java)
on Flink.

Locally that works perfectly. But on the cluster I get the following error:

java.lang.Exception: Call to registerInputOutput() of invokable failed
        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:504)
        at java.lang.Thread.run(Thread.java:853)
Caused by: java.lang.RuntimeException: Initializing the output handlers
failed: Could not load deserializer from the configuration.
        at
org.apache.flink.runtime.operators.RegularPactTask.registerInputOutput(RegularPactTask.java:256)
        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:501)
        ... 1 more
Caused by: java.lang.RuntimeException: Could not load deserializer from the
configuration.
        at
org.apache.flink.api.java.typeutils.runtime.RuntimeSerializerFactory.readParametersFromConfig(RuntimeSerializerFactory.java:83)
        at
org.apache.flink.runtime.operators.util.TaskConfig.getTypeSerializerFactory(TaskConfig.java:1085)
        at
org.apache.flink.runtime.operators.util.TaskConfig.getOutputSerializer(TaskConfig.java:543)
        at
org.apache.flink.runtime.operators.RegularPactTask.getOutputCollector(RegularPactTask.java:1267)
        at
org.apache.flink.runtime.operators.RegularPactTask.initOutputs(RegularPactTask.java:1375)
        at
org.apache.flink.runtime.operators.RegularPactTask.initOutputs(RegularPactTask.java:1076)
        at
org.apache.flink.runtime.operators.RegularPactTask.registerInputOutput(RegularPactTask.java:254)
        ... 2 more
Caused by: java.io.InvalidClassException: java.lang.Integer; local class
incompatible: stream classdesc serialVersionUID = -8644826526760479189,
local class serialVersion
UID = 1360826667806852920
        at
java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:698)
        at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1705)
        at
java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1600)
        at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1873)
        at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1432)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:428)
        at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:302)
        at
org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:264)
        at
org.apache.flink.api.java.typeutils.runtime.RuntimeSerializerFactory.readParametersFromConfig(RuntimeSerializerFactory.java:76)
        ... 8 more

Is this an issue of Kryo in Flink? Or do I have to implement a custom
serializer?

Thank you for your help,

Felix

Re: serialization issue

Posted by Stephan Ewen <se...@apache.org>.
Hmmm, this seams like the stream is read differently than it is written.

Maybe one of the involved classes overrides the readObject() and
writeObject() methods, and implements them inconsistently. And you see if
that is the case in one of your data types?

On Wed, Jul 22, 2015 at 11:02 AM, Robert Metzger <rm...@apache.org>
wrote:

> Were you able to resolve the issue?
>
> I first thought its an JVM-version incompatibility, but this thread
> suggests a serialization issue:
>
> http://stackoverflow.com/questions/851491/java-deserialization-of-java-lang-integer-exception
>
> By the way: your emails to the dev@ list sometimes end up in the "Spam"
> folder of GMail. I'm not sure how many people actually saw your email ;)
>
> On Thu, Jul 9, 2015 at 9:42 PM, Felix Neutatz <ne...@googlemail.com>
> wrote:
>
> > Hi,
> >
> > I want to use t-digest by Ted Dunning (
> >
> >
> https://github.com/tdunning/t-digest/blob/master/src/main/java/com/tdunning/math/stats/ArrayDigest.java
> > )
> > on Flink.
> >
> > Locally that works perfectly. But on the cluster I get the following
> error:
> >
> > java.lang.Exception: Call to registerInputOutput() of invokable failed
> >         at org.apache.flink.runtime.taskmanager.Task.run(Task.java:504)
> >         at java.lang.Thread.run(Thread.java:853)
> > Caused by: java.lang.RuntimeException: Initializing the output handlers
> > failed: Could not load deserializer from the configuration.
> >         at
> >
> >
> org.apache.flink.runtime.operators.RegularPactTask.registerInputOutput(RegularPactTask.java:256)
> >         at org.apache.flink.runtime.taskmanager.Task.run(Task.java:501)
> >         ... 1 more
> > Caused by: java.lang.RuntimeException: Could not load deserializer from
> the
> > configuration.
> >         at
> >
> >
> org.apache.flink.api.java.typeutils.runtime.RuntimeSerializerFactory.readParametersFromConfig(RuntimeSerializerFactory.java:83)
> >         at
> >
> >
> org.apache.flink.runtime.operators.util.TaskConfig.getTypeSerializerFactory(TaskConfig.java:1085)
> >         at
> >
> >
> org.apache.flink.runtime.operators.util.TaskConfig.getOutputSerializer(TaskConfig.java:543)
> >         at
> >
> >
> org.apache.flink.runtime.operators.RegularPactTask.getOutputCollector(RegularPactTask.java:1267)
> >         at
> >
> >
> org.apache.flink.runtime.operators.RegularPactTask.initOutputs(RegularPactTask.java:1375)
> >         at
> >
> >
> org.apache.flink.runtime.operators.RegularPactTask.initOutputs(RegularPactTask.java:1076)
> >         at
> >
> >
> org.apache.flink.runtime.operators.RegularPactTask.registerInputOutput(RegularPactTask.java:254)
> >         ... 2 more
> > Caused by: java.io.InvalidClassException: java.lang.Integer; local class
> > incompatible: stream classdesc serialVersionUID = -8644826526760479189,
> > local class serialVersion
> > UID = 1360826667806852920
> >         at
> > java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:698)
> >         at
> > java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1705)
> >         at
> > java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1600)
> >         at
> > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1873)
> >         at
> > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1432)
> >         at
> java.io.ObjectInputStream.readObject(ObjectInputStream.java:428)
> >         at
> >
> >
> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:302)
> >         at
> >
> >
> org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:264)
> >         at
> >
> >
> org.apache.flink.api.java.typeutils.runtime.RuntimeSerializerFactory.readParametersFromConfig(RuntimeSerializerFactory.java:76)
> >         ... 8 more
> >
> > Is this an issue of Kryo in Flink? Or do I have to implement a custom
> > serializer?
> >
> > Thank you for your help,
> >
> > Felix
> >
>

Re: serialization issue

Posted by Robert Metzger <rm...@apache.org>.
Were you able to resolve the issue?

I first thought its an JVM-version incompatibility, but this thread
suggests a serialization issue:
http://stackoverflow.com/questions/851491/java-deserialization-of-java-lang-integer-exception

By the way: your emails to the dev@ list sometimes end up in the "Spam"
folder of GMail. I'm not sure how many people actually saw your email ;)

On Thu, Jul 9, 2015 at 9:42 PM, Felix Neutatz <ne...@googlemail.com>
wrote:

> Hi,
>
> I want to use t-digest by Ted Dunning (
>
> https://github.com/tdunning/t-digest/blob/master/src/main/java/com/tdunning/math/stats/ArrayDigest.java
> )
> on Flink.
>
> Locally that works perfectly. But on the cluster I get the following error:
>
> java.lang.Exception: Call to registerInputOutput() of invokable failed
>         at org.apache.flink.runtime.taskmanager.Task.run(Task.java:504)
>         at java.lang.Thread.run(Thread.java:853)
> Caused by: java.lang.RuntimeException: Initializing the output handlers
> failed: Could not load deserializer from the configuration.
>         at
>
> org.apache.flink.runtime.operators.RegularPactTask.registerInputOutput(RegularPactTask.java:256)
>         at org.apache.flink.runtime.taskmanager.Task.run(Task.java:501)
>         ... 1 more
> Caused by: java.lang.RuntimeException: Could not load deserializer from the
> configuration.
>         at
>
> org.apache.flink.api.java.typeutils.runtime.RuntimeSerializerFactory.readParametersFromConfig(RuntimeSerializerFactory.java:83)
>         at
>
> org.apache.flink.runtime.operators.util.TaskConfig.getTypeSerializerFactory(TaskConfig.java:1085)
>         at
>
> org.apache.flink.runtime.operators.util.TaskConfig.getOutputSerializer(TaskConfig.java:543)
>         at
>
> org.apache.flink.runtime.operators.RegularPactTask.getOutputCollector(RegularPactTask.java:1267)
>         at
>
> org.apache.flink.runtime.operators.RegularPactTask.initOutputs(RegularPactTask.java:1375)
>         at
>
> org.apache.flink.runtime.operators.RegularPactTask.initOutputs(RegularPactTask.java:1076)
>         at
>
> org.apache.flink.runtime.operators.RegularPactTask.registerInputOutput(RegularPactTask.java:254)
>         ... 2 more
> Caused by: java.io.InvalidClassException: java.lang.Integer; local class
> incompatible: stream classdesc serialVersionUID = -8644826526760479189,
> local class serialVersion
> UID = 1360826667806852920
>         at
> java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:698)
>         at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1705)
>         at
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1600)
>         at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1873)
>         at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1432)
>         at java.io.ObjectInputStream.readObject(ObjectInputStream.java:428)
>         at
>
> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:302)
>         at
>
> org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:264)
>         at
>
> org.apache.flink.api.java.typeutils.runtime.RuntimeSerializerFactory.readParametersFromConfig(RuntimeSerializerFactory.java:76)
>         ... 8 more
>
> Is this an issue of Kryo in Flink? Or do I have to implement a custom
> serializer?
>
> Thank you for your help,
>
> Felix
>