You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Dmitry Golubets <dg...@gmail.com> on 2017/02/10 17:26:46 UTC

How important is 'registerType'?

The docs say that it may improve performance.

How true is it, when custom serializers are provided?
There is also 'disableAutoTypeRegistration' method in the config class,
implying Flink registers types automatically.

So, given that I have an hierarchy:
trait A
class B extends A
class C extends A

and I do addDefaultKryoSerializer(classOf[A], classOf[ASerializer])

should I care about registering B and C with 'registerType' method?

It worth mentioning that when I registered my message class hierarchies, I
got:
java.lang.IllegalStateException: Could not initialize keyed state backend.
java.io.StreamCorruptedException: invalid type code: 00
on restoring from savepoint

After some debugging I found that 'registerType' was the cause.
It might be possible that my code called registerType in different order.
Could it be a problem?

Best regards,
Dmitry

Re: How important is 'registerType'?

Posted by Aljoscha Krettek <al...@apache.org>.
Hi Dmitry,
do you maybe have a more complete stack trace? I have a suspicion but I
would like to confirm that.

Best,
Aljoscha

On Fri, 17 Feb 2017 at 20:24 Dmitry Golubets <dg...@gmail.com> wrote:

> Hi Till,
>
> It happened during deserialization of a savepoint.
>
> Best regards,
> Dmitry
>
> On Fri, Feb 17, 2017 at 2:48 PM, Till Rohrmann <tr...@apache.org>
> wrote:
>
> Hi Dmitry,
>
> curious to know when exactly you observed the IllegalStateException. Did
> it happen after resuming from a savepoint or did it already happen during
> the first run of the program? If the latter is the case, then this might
> indicate a bug where we don’t use the correct ExecutionConfig to
> instantiate the serializers.
>
> Concerning the addDefaultKryoSerializer method, this basically register a
> serializer for a specific type but it does not register the type with Kryo.
> Thus, it should still be beneficial to call registerType for the type for
> which you’ve registered a default serializer. But you can also call
> registerTypeWithKryoSerializer which does both for you.
>
> Cheers,
> Till
> ​
>
> On Fri, Feb 17, 2017 at 12:38 PM, Dmitry Golubets <dg...@gmail.com>
> wrote:
>
> Hi,
>
> I was using ```cs.knownDirectSubclasses``` recursively to find and
> register subclasses, which may have resulted in order mess.
> Later I changed that to
> ````cs.knownDirectSubclasses.toList.sortBy(_.fullName)``` which should have
> fixed the order.
> But either it didn't or there was another problem, I was getting the error
> anyway.
> Interesting, it happend only on a KeyedStream after window, without window
> it was fine.
> I didn't change anything else in the job.
>
> However I removed ```registerType``` calls completely. Because I didn't
> notice any performance difference.
> Do you know if ```registerType``` has any effect at all if I use it
> together with ```addDefaultKryoSerializer``` for that type?
>
>
> Best regards,
> Dmitry
>
> On Thu, Feb 16, 2017 at 10:40 AM, Aljoscha Krettek <al...@apache.org>
> wrote:
>
> Hi,
> are you changing anything on your job between performing the savepoint and
> restoring the savepoint? Flink upgrade, Job upgrade, changing Kryo version,
> changing order in which you register Kryo serialisers?
>
> Best,
> Aljoscha
>
> On Fri, 10 Feb 2017 at 18:26 Dmitry Golubets <dg...@gmail.com> wrote:
>
> The docs say that it may improve performance.
>
> How true is it, when custom serializers are provided?
> There is also 'disableAutoTypeRegistration' method in the config class,
> implying Flink registers types automatically.
>
> So, given that I have an hierarchy:
> trait A
> class B extends A
> class C extends A
>
> and I do addDefaultKryoSerializer(classOf[A], classOf[ASerializer])
>
> should I care about registering B and C with 'registerType' method?
>
> It worth mentioning that when I registered my message class hierarchies, I
> got:
> java.lang.IllegalStateException: Could not initialize keyed state backend.
> java.io.StreamCorruptedException: invalid type code: 00
> on restoring from savepoint
>
> After some debugging I found that 'registerType' was the cause.
> It might be possible that my code called registerType in different order.
> Could it be a problem?
>
> Best regards,
> Dmitry
>
>
>
>
>

Re: How important is 'registerType'?

Posted by Dmitry Golubets <dg...@gmail.com>.
Hi Till,

It happened during deserialization of a savepoint.

Best regards,
Dmitry

On Fri, Feb 17, 2017 at 2:48 PM, Till Rohrmann <tr...@apache.org> wrote:

> Hi Dmitry,
>
> curious to know when exactly you observed the IllegalStateException. Did
> it happen after resuming from a savepoint or did it already happen during
> the first run of the program? If the latter is the case, then this might
> indicate a bug where we don’t use the correct ExecutionConfig to
> instantiate the serializers.
>
> Concerning the addDefaultKryoSerializer method, this basically register a
> serializer for a specific type but it does not register the type with Kryo.
> Thus, it should still be beneficial to call registerType for the type for
> which you’ve registered a default serializer. But you can also call
> registerTypeWithKryoSerializer which does both for you.
>
> Cheers,
> Till
> ​
>
> On Fri, Feb 17, 2017 at 12:38 PM, Dmitry Golubets <dg...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I was using ```cs.knownDirectSubclasses``` recursively to find and
>> register subclasses, which may have resulted in order mess.
>> Later I changed that to ````cs.knownDirectSubclasses.toList.sortBy(_.fullName)```
>> which should have fixed the order.
>> But either it didn't or there was another problem, I was getting the
>> error anyway.
>> Interesting, it happend only on a KeyedStream after window, without
>> window it was fine.
>> I didn't change anything else in the job.
>>
>> However I removed ```registerType``` calls completely. Because I didn't
>> notice any performance difference.
>> Do you know if ```registerType``` has any effect at all if I use it
>> together with ```addDefaultKryoSerializer``` for that type?
>>
>>
>> Best regards,
>> Dmitry
>>
>> On Thu, Feb 16, 2017 at 10:40 AM, Aljoscha Krettek <al...@apache.org>
>> wrote:
>>
>>> Hi,
>>> are you changing anything on your job between performing the savepoint
>>> and restoring the savepoint? Flink upgrade, Job upgrade, changing Kryo
>>> version, changing order in which you register Kryo serialisers?
>>>
>>> Best,
>>> Aljoscha
>>>
>>> On Fri, 10 Feb 2017 at 18:26 Dmitry Golubets <dg...@gmail.com>
>>> wrote:
>>>
>>>> The docs say that it may improve performance.
>>>>
>>>> How true is it, when custom serializers are provided?
>>>> There is also 'disableAutoTypeRegistration' method in the config class,
>>>> implying Flink registers types automatically.
>>>>
>>>> So, given that I have an hierarchy:
>>>> trait A
>>>> class B extends A
>>>> class C extends A
>>>>
>>>> and I do addDefaultKryoSerializer(classOf[A], classOf[ASerializer])
>>>>
>>>> should I care about registering B and C with 'registerType' method?
>>>>
>>>> It worth mentioning that when I registered my message class
>>>> hierarchies, I got:
>>>> java.lang.IllegalStateException: Could not initialize keyed state
>>>> backend.
>>>> java.io.StreamCorruptedException: invalid type code: 00
>>>> on restoring from savepoint
>>>>
>>>> After some debugging I found that 'registerType' was the cause.
>>>> It might be possible that my code called registerType in different
>>>> order.
>>>> Could it be a problem?
>>>>
>>>> Best regards,
>>>> Dmitry
>>>>
>>>
>>
>

Re: How important is 'registerType'?

Posted by Till Rohrmann <tr...@apache.org>.
Hi Dmitry,

curious to know when exactly you observed the IllegalStateException. Did it
happen after resuming from a savepoint or did it already happen during the
first run of the program? If the latter is the case, then this might
indicate a bug where we don’t use the correct ExecutionConfig to
instantiate the serializers.

Concerning the addDefaultKryoSerializer method, this basically register a
serializer for a specific type but it does not register the type with Kryo.
Thus, it should still be beneficial to call registerType for the type for
which you’ve registered a default serializer. But you can also call
registerTypeWithKryoSerializer which does both for you.

Cheers,
Till
​

On Fri, Feb 17, 2017 at 12:38 PM, Dmitry Golubets <dg...@gmail.com>
wrote:

> Hi,
>
> I was using ```cs.knownDirectSubclasses``` recursively to find and
> register subclasses, which may have resulted in order mess.
> Later I changed that to ````cs.knownDirectSubclasses.toList.sortBy(_.fullName)```
> which should have fixed the order.
> But either it didn't or there was another problem, I was getting the error
> anyway.
> Interesting, it happend only on a KeyedStream after window, without window
> it was fine.
> I didn't change anything else in the job.
>
> However I removed ```registerType``` calls completely. Because I didn't
> notice any performance difference.
> Do you know if ```registerType``` has any effect at all if I use it
> together with ```addDefaultKryoSerializer``` for that type?
>
>
> Best regards,
> Dmitry
>
> On Thu, Feb 16, 2017 at 10:40 AM, Aljoscha Krettek <al...@apache.org>
> wrote:
>
>> Hi,
>> are you changing anything on your job between performing the savepoint
>> and restoring the savepoint? Flink upgrade, Job upgrade, changing Kryo
>> version, changing order in which you register Kryo serialisers?
>>
>> Best,
>> Aljoscha
>>
>> On Fri, 10 Feb 2017 at 18:26 Dmitry Golubets <dg...@gmail.com> wrote:
>>
>>> The docs say that it may improve performance.
>>>
>>> How true is it, when custom serializers are provided?
>>> There is also 'disableAutoTypeRegistration' method in the config class,
>>> implying Flink registers types automatically.
>>>
>>> So, given that I have an hierarchy:
>>> trait A
>>> class B extends A
>>> class C extends A
>>>
>>> and I do addDefaultKryoSerializer(classOf[A], classOf[ASerializer])
>>>
>>> should I care about registering B and C with 'registerType' method?
>>>
>>> It worth mentioning that when I registered my message class hierarchies,
>>> I got:
>>> java.lang.IllegalStateException: Could not initialize keyed state
>>> backend.
>>> java.io.StreamCorruptedException: invalid type code: 00
>>> on restoring from savepoint
>>>
>>> After some debugging I found that 'registerType' was the cause.
>>> It might be possible that my code called registerType in different
>>> order.
>>> Could it be a problem?
>>>
>>> Best regards,
>>> Dmitry
>>>
>>
>

Re: How important is 'registerType'?

Posted by Dmitry Golubets <dg...@gmail.com>.
Hi,

I was using ```cs.knownDirectSubclasses``` recursively to find and register
subclasses, which may have resulted in order mess.
Later I changed that to
````cs.knownDirectSubclasses.toList.sortBy(_.fullName)``` which should have
fixed the order.
But either it didn't or there was another problem, I was getting the error
anyway.
Interesting, it happend only on a KeyedStream after window, without window
it was fine.
I didn't change anything else in the job.

However I removed ```registerType``` calls completely. Because I didn't
notice any performance difference.
Do you know if ```registerType``` has any effect at all if I use it
together with ```addDefaultKryoSerializer``` for that type?


Best regards,
Dmitry

On Thu, Feb 16, 2017 at 10:40 AM, Aljoscha Krettek <al...@apache.org>
wrote:

> Hi,
> are you changing anything on your job between performing the savepoint and
> restoring the savepoint? Flink upgrade, Job upgrade, changing Kryo version,
> changing order in which you register Kryo serialisers?
>
> Best,
> Aljoscha
>
> On Fri, 10 Feb 2017 at 18:26 Dmitry Golubets <dg...@gmail.com> wrote:
>
>> The docs say that it may improve performance.
>>
>> How true is it, when custom serializers are provided?
>> There is also 'disableAutoTypeRegistration' method in the config class,
>> implying Flink registers types automatically.
>>
>> So, given that I have an hierarchy:
>> trait A
>> class B extends A
>> class C extends A
>>
>> and I do addDefaultKryoSerializer(classOf[A], classOf[ASerializer])
>>
>> should I care about registering B and C with 'registerType' method?
>>
>> It worth mentioning that when I registered my message class hierarchies,
>> I got:
>> java.lang.IllegalStateException: Could not initialize keyed state
>> backend.
>> java.io.StreamCorruptedException: invalid type code: 00
>> on restoring from savepoint
>>
>> After some debugging I found that 'registerType' was the cause.
>> It might be possible that my code called registerType in different order.
>> Could it be a problem?
>>
>> Best regards,
>> Dmitry
>>
>

Re: How important is 'registerType'?

Posted by Aljoscha Krettek <al...@apache.org>.
Hi,
are you changing anything on your job between performing the savepoint and
restoring the savepoint? Flink upgrade, Job upgrade, changing Kryo version,
changing order in which you register Kryo serialisers?

Best,
Aljoscha

On Fri, 10 Feb 2017 at 18:26 Dmitry Golubets <dg...@gmail.com> wrote:

> The docs say that it may improve performance.
>
> How true is it, when custom serializers are provided?
> There is also 'disableAutoTypeRegistration' method in the config class,
> implying Flink registers types automatically.
>
> So, given that I have an hierarchy:
> trait A
> class B extends A
> class C extends A
>
> and I do addDefaultKryoSerializer(classOf[A], classOf[ASerializer])
>
> should I care about registering B and C with 'registerType' method?
>
> It worth mentioning that when I registered my message class hierarchies, I
> got:
> java.lang.IllegalStateException: Could not initialize keyed state backend.
> java.io.StreamCorruptedException: invalid type code: 00
> on restoring from savepoint
>
> After some debugging I found that 'registerType' was the cause.
> It might be possible that my code called registerType in different order.
> Could it be a problem?
>
> Best regards,
> Dmitry
>