You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Vishal Santoshi <vi...@gmail.com> on 2022/09/15 23:04:48 UTC
A question about restoring state with an additional variable with kryo
I have state in rocksDB that represents say
class A {
String a
}
I now change my class and add another variable
Class A {
String a;
long b = 0;
}
How do I make sure that when reconstituting the state, kryo does not
complain? It tries to map the previous state to the new definition of Class
A and complains that it cannot read the value for `String b`.
Unfortunately the state is not using POJO serializer.
Thanks and Regards.
Vishal
Re: A question about restoring state with an additional variable with kryo
Posted by Vishal Santoshi <vi...@gmail.com>.
Thanks, I'll check it out.
On Sun, Sep 18, 2022 at 7:33 PM David Anderson <da...@apache.org> wrote:
> Vishal,
>
> If you decide you can't live with dropping that state, [1] is a complete
> example showing how to migrate from Kryo by using the state processor API.
>
> David
>
> [1]
> https://www.docs.immerok.cloud/docs/cookbook/migrating-state-away-from-kryo/
>
>
> On Fri, Sep 16, 2022 at 8:32 AM Vishal Santoshi <vi...@gmail.com>
> wrote:
>
>> Thank you for the clarification. I thought so to,
>>
>> Unfortunately my state are generics based and those are definitely not
>> treated as a POJO , though it has all the constructs ( no arg constructor,
>> getters/setters etc ). I will likely take an at least once hit by
>>
>> Changing the uid of that specific Operator, and restart with Allow
>> non-restored state ... This will ignore state that cannot be restored (
>> for the previous uid ) , construct state for the new uid and not affect
>> other operators ( including the kafka consumer operators ). I can live with
>> it, I think.
>>
>> On Fri, Sep 16, 2022 at 2:55 AM Schwalbe Matthias <
>> Matthias.Schwalbe@viseca.ch> wrote:
>>
>>> Hi Vishal,
>>>
>>>
>>>
>>> Good news and bad news đ:
>>>
>>>
>>>
>>> - Bad: Kryo serializer cannot be used for schema evolution, see [1]
>>> - Good: not all is lost here,
>>> - If you happen to have state that you cannot afford to lose, you
>>> can transcode it by means of the savepoint API [2],
>>> - However, this takes quite some effort
>>> - In general, if you ever plan to migrate/extend your schemas,
>>> choose a data type that supports schema migration [1],
>>> - In your case, PoJo types would be the closest to your original
>>> implementation
>>> - You can disable Kryo in configuration to avoid this situation in
>>> the future, by the way,
>>> - Kryo serializer is quite slow compared to the other options and I
>>> believe it is only there as a (emergency) fallback solution: [3]
>>>
>>>
>>>
>>> Feel free to ask for clarification đ
>>>
>>>
>>>
>>> Thias
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> [1]
>>> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/schema_evolution/#kryo-cannot-be-used-for-schema-evolution
>>>
>>> [2]
>>> https://nightlies.apache.org/flink/flink-docs-master/docs/libs/state_processor_api/
>>>
>>> [3]
>>> https://flink.apache.org/news/2020/04/15/flink-serialization-tuning-vol-1.html
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *From:* Vishal Santoshi <vi...@gmail.com>
>>> *Sent:* Friday, September 16, 2022 1:17 AM
>>> *To:* user <us...@flink.apache.org>
>>> *Subject:* Re: A question about restoring state with an additional
>>> variable with kryo
>>>
>>>
>>>
>>> â *EXTERNAL MESSAGE â **CAUTION: Think Before You Click *â
>>>
>>>
>>>
>>> The exception thrown is as follows. I realize that it is trying to read
>>> the long value. How do I signal to kryo that it is OK and that he object
>>> can have a default value
>>>
>>>
>>>
>>> Caused by: java.io.EOFException: No more bytes left.
>>>
>>> at org.apache.flink.api.java.typeutils.runtime.NoFetchingInput
>>> .require(NoFetchingInput.java:80)
>>>
>>> at com.esotericsoftware.kryo.io.Input.readVarLong(Input.java:690)
>>>
>>> at com.esotericsoftware.kryo.io.Input.readLong(Input.java:685)
>>>
>>> at com.esotericsoftware.kryo.serializers.
>>> DefaultSerializers$LongSerializer.read(DefaultSerializers.java:133)
>>>
>>> at com.esotericsoftware.kryo.serializers.
>>> DefaultSerializers$LongSerializer.read(DefaultSerializers.java:123)
>>>
>>> at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
>>>
>>> at com.esotericsoftware.kryo.serializers.ObjectField.read(
>>> ObjectField.java:113)
>>>
>>> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(
>>> FieldSerializer.java:528)
>>>
>>> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>>>
>>> at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer
>>> .deserialize(KryoSerializer.java:354)
>>>
>>> at org.apache.flink.api.common.typeutils.CompositeSerializer
>>> .deserialize(CompositeSerializer.java:156)
>>>
>>> at org.apache.flink.contrib.streaming.state.RocksDBValueState.value(
>>> RocksDBValueState.java:89)
>>>
>>>
>>>
>>> On Thu, Sep 15, 2022 at 7:10 PM Vishal Santoshi <
>>> vishal.santoshi@gmail.com> wrote:
>>>
>>> << How do I make sure that when reconstituting the state, kryo does not
>>> complain? It tries to map the previous state to the new definition of Class
>>> A and complains that it cannot read the value for `String b`.
>>>
>>>
>>>
>>> >> How do I make sure that when reconstituting the state, kryo does not
>>> complain? It tries to map the previous state to the new definition of Class
>>> A and complains that it cannot read the value for `long b`.
>>>
>>>
>>>
>>> Sorry a typo
>>>
>>>
>>>
>>>
>>>
>>> On Thu, Sep 15, 2022 at 7:04 PM Vishal Santoshi <
>>> vishal.santoshi@gmail.com> wrote:
>>>
>>> I have state in rocksDB that represents say
>>>
>>>
>>>
>>> class A {
>>>
>>> String a
>>>
>>> }
>>>
>>>
>>>
>>> I now change my class and add another variable
>>>
>>>
>>>
>>>
>>> Class A {
>>>
>>> String a;
>>>
>>> long b = 0;
>>>
>>> }
>>>
>>>
>>>
>>> How do I make sure that when reconstituting the state, kryo does not
>>> complain? It tries to map the previous state to the new definition of Class
>>> A and complains that it cannot read the value for `String b`.
>>>
>>>
>>>
>>> Unfortunately the state is not using POJO serializer.
>>>
>>>
>>>
>>> Thanks and Regards.
>>>
>>>
>>>
>>> Vishal
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Diese Nachricht ist ausschliesslich fĂŒr den Adressaten bestimmt und
>>> beinhaltet unter UmstÀnden vertrauliche Mitteilungen. Da die
>>> Vertraulichkeit von e-Mail-Nachrichten nicht gewÀhrleistet werden kann,
>>> ĂŒbernehmen wir keine Haftung fĂŒr die GewĂ€hrung der Vertraulichkeit und
>>> Unversehrtheit dieser Mitteilung. Bei irrtĂŒmlicher Zustellung bitten wir
>>> Sie um Benachrichtigung per e-Mail und um Löschung dieser Nachricht sowie
>>> eventueller AnhÀnge. Jegliche unberechtigte Verwendung oder Verbreitung
>>> dieser Informationen ist streng verboten.
>>>
>>> This message is intended only for the named recipient and may contain
>>> confidential or privileged information. As the confidentiality of email
>>> communication cannot be guaranteed, we do not accept any responsibility for
>>> the confidentiality and the intactness of this message. If you have
>>> received it in error, please advise the sender by return e-mail and delete
>>> this message and any attachments. Any unauthorised use or dissemination of
>>> this information is strictly prohibited.
>>>
>>
Re: A question about restoring state with an additional variable with kryo
Posted by David Anderson <da...@apache.org>.
Vishal,
If you decide you can't live with dropping that state, [1] is a complete
example showing how to migrate from Kryo by using the state processor API.
David
[1]
https://www.docs.immerok.cloud/docs/cookbook/migrating-state-away-from-kryo/
On Fri, Sep 16, 2022 at 8:32 AM Vishal Santoshi <vi...@gmail.com>
wrote:
> Thank you for the clarification. I thought so to,
>
> Unfortunately my state are generics based and those are definitely not
> treated as a POJO , though it has all the constructs ( no arg constructor,
> getters/setters etc ). I will likely take an at least once hit by
>
> Changing the uid of that specific Operator, and restart with Allow
> non-restored state ... This will ignore state that cannot be restored (
> for the previous uid ) , construct state for the new uid and not affect
> other operators ( including the kafka consumer operators ). I can live with
> it, I think.
>
> On Fri, Sep 16, 2022 at 2:55 AM Schwalbe Matthias <
> Matthias.Schwalbe@viseca.ch> wrote:
>
>> Hi Vishal,
>>
>>
>>
>> Good news and bad news đ:
>>
>>
>>
>> - Bad: Kryo serializer cannot be used for schema evolution, see [1]
>> - Good: not all is lost here,
>> - If you happen to have state that you cannot afford to lose, you
>> can transcode it by means of the savepoint API [2],
>> - However, this takes quite some effort
>> - In general, if you ever plan to migrate/extend your schemas, choose
>> a data type that supports schema migration [1],
>> - In your case, PoJo types would be the closest to your original
>> implementation
>> - You can disable Kryo in configuration to avoid this situation in
>> the future, by the way,
>> - Kryo serializer is quite slow compared to the other options and I
>> believe it is only there as a (emergency) fallback solution: [3]
>>
>>
>>
>> Feel free to ask for clarification đ
>>
>>
>>
>> Thias
>>
>>
>>
>>
>>
>>
>>
>> [1]
>> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/schema_evolution/#kryo-cannot-be-used-for-schema-evolution
>>
>> [2]
>> https://nightlies.apache.org/flink/flink-docs-master/docs/libs/state_processor_api/
>>
>> [3]
>> https://flink.apache.org/news/2020/04/15/flink-serialization-tuning-vol-1.html
>>
>>
>>
>>
>>
>>
>>
>> *From:* Vishal Santoshi <vi...@gmail.com>
>> *Sent:* Friday, September 16, 2022 1:17 AM
>> *To:* user <us...@flink.apache.org>
>> *Subject:* Re: A question about restoring state with an additional
>> variable with kryo
>>
>>
>>
>> â *EXTERNAL MESSAGE â **CAUTION: Think Before You Click *â
>>
>>
>>
>> The exception thrown is as follows. I realize that it is trying to read
>> the long value. How do I signal to kryo that it is OK and that he object
>> can have a default value
>>
>>
>>
>> Caused by: java.io.EOFException: No more bytes left.
>>
>> at org.apache.flink.api.java.typeutils.runtime.NoFetchingInput
>> .require(NoFetchingInput.java:80)
>>
>> at com.esotericsoftware.kryo.io.Input.readVarLong(Input.java:690)
>>
>> at com.esotericsoftware.kryo.io.Input.readLong(Input.java:685)
>>
>> at com.esotericsoftware.kryo.serializers.
>> DefaultSerializers$LongSerializer.read(DefaultSerializers.java:133)
>>
>> at com.esotericsoftware.kryo.serializers.
>> DefaultSerializers$LongSerializer.read(DefaultSerializers.java:123)
>>
>> at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
>>
>> at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField
>> .java:113)
>>
>> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(
>> FieldSerializer.java:528)
>>
>> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>>
>> at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer
>> .deserialize(KryoSerializer.java:354)
>>
>> at org.apache.flink.api.common.typeutils.CompositeSerializer
>> .deserialize(CompositeSerializer.java:156)
>>
>> at org.apache.flink.contrib.streaming.state.RocksDBValueState.value(
>> RocksDBValueState.java:89)
>>
>>
>>
>> On Thu, Sep 15, 2022 at 7:10 PM Vishal Santoshi <
>> vishal.santoshi@gmail.com> wrote:
>>
>> << How do I make sure that when reconstituting the state, kryo does not
>> complain? It tries to map the previous state to the new definition of Class
>> A and complains that it cannot read the value for `String b`.
>>
>>
>>
>> >> How do I make sure that when reconstituting the state, kryo does not
>> complain? It tries to map the previous state to the new definition of Class
>> A and complains that it cannot read the value for `long b`.
>>
>>
>>
>> Sorry a typo
>>
>>
>>
>>
>>
>> On Thu, Sep 15, 2022 at 7:04 PM Vishal Santoshi <
>> vishal.santoshi@gmail.com> wrote:
>>
>> I have state in rocksDB that represents say
>>
>>
>>
>> class A {
>>
>> String a
>>
>> }
>>
>>
>>
>> I now change my class and add another variable
>>
>>
>>
>>
>> Class A {
>>
>> String a;
>>
>> long b = 0;
>>
>> }
>>
>>
>>
>> How do I make sure that when reconstituting the state, kryo does not
>> complain? It tries to map the previous state to the new definition of Class
>> A and complains that it cannot read the value for `String b`.
>>
>>
>>
>> Unfortunately the state is not using POJO serializer.
>>
>>
>>
>> Thanks and Regards.
>>
>>
>>
>> Vishal
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Diese Nachricht ist ausschliesslich fĂŒr den Adressaten bestimmt und
>> beinhaltet unter UmstÀnden vertrauliche Mitteilungen. Da die
>> Vertraulichkeit von e-Mail-Nachrichten nicht gewÀhrleistet werden kann,
>> ĂŒbernehmen wir keine Haftung fĂŒr die GewĂ€hrung der Vertraulichkeit und
>> Unversehrtheit dieser Mitteilung. Bei irrtĂŒmlicher Zustellung bitten wir
>> Sie um Benachrichtigung per e-Mail und um Löschung dieser Nachricht sowie
>> eventueller AnhÀnge. Jegliche unberechtigte Verwendung oder Verbreitung
>> dieser Informationen ist streng verboten.
>>
>> This message is intended only for the named recipient and may contain
>> confidential or privileged information. As the confidentiality of email
>> communication cannot be guaranteed, we do not accept any responsibility for
>> the confidentiality and the intactness of this message. If you have
>> received it in error, please advise the sender by return e-mail and delete
>> this message and any attachments. Any unauthorised use or dissemination of
>> this information is strictly prohibited.
>>
>
Re: A question about restoring state with an additional variable with kryo
Posted by Vishal Santoshi <vi...@gmail.com>.
Thank you for the clarification. I thought so to,
Unfortunately my state are generics based and those are definitely not
treated as a POJO , though it has all the constructs ( no arg constructor,
getters/setters etc ). I will likely take an at least once hit by
Changing the uid of that specific Operator, and restart with Allow
non-restored state ... This will ignore state that cannot be restored ( for
the previous uid ) , construct state for the new uid and not affect other
operators ( including the kafka consumer operators ). I can live with it, I
think.
On Fri, Sep 16, 2022 at 2:55 AM Schwalbe Matthias <
Matthias.Schwalbe@viseca.ch> wrote:
> Hi Vishal,
>
>
>
> Good news and bad news đ:
>
>
>
> - Bad: Kryo serializer cannot be used for schema evolution, see [1]
> - Good: not all is lost here,
> - If you happen to have state that you cannot afford to lose, you
> can transcode it by means of the savepoint API [2],
> - However, this takes quite some effort
> - In general, if you ever plan to migrate/extend your schemas, choose
> a data type that supports schema migration [1],
> - In your case, PoJo types would be the closest to your original
> implementation
> - You can disable Kryo in configuration to avoid this situation in the
> future, by the way,
> - Kryo serializer is quite slow compared to the other options and I
> believe it is only there as a (emergency) fallback solution: [3]
>
>
>
> Feel free to ask for clarification đ
>
>
>
> Thias
>
>
>
>
>
>
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/schema_evolution/#kryo-cannot-be-used-for-schema-evolution
>
> [2]
> https://nightlies.apache.org/flink/flink-docs-master/docs/libs/state_processor_api/
>
> [3]
> https://flink.apache.org/news/2020/04/15/flink-serialization-tuning-vol-1.html
>
>
>
>
>
>
>
> *From:* Vishal Santoshi <vi...@gmail.com>
> *Sent:* Friday, September 16, 2022 1:17 AM
> *To:* user <us...@flink.apache.org>
> *Subject:* Re: A question about restoring state with an additional
> variable with kryo
>
>
>
> â *EXTERNAL MESSAGE â **CAUTION: Think Before You Click *â
>
>
>
> The exception thrown is as follows. I realize that it is trying to read
> the long value. How do I signal to kryo that it is OK and that he object
> can have a default value
>
>
>
> Caused by: java.io.EOFException: No more bytes left.
>
> at org.apache.flink.api.java.typeutils.runtime.NoFetchingInput
> .require(NoFetchingInput.java:80)
>
> at com.esotericsoftware.kryo.io.Input.readVarLong(Input.java:690)
>
> at com.esotericsoftware.kryo.io.Input.readLong(Input.java:685)
>
> at com.esotericsoftware.kryo.serializers.
> DefaultSerializers$LongSerializer.read(DefaultSerializers.java:133)
>
> at com.esotericsoftware.kryo.serializers.
> DefaultSerializers$LongSerializer.read(DefaultSerializers.java:123)
>
> at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
>
> at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField
> .java:113)
>
> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(
> FieldSerializer.java:528)
>
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>
> at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer
> .deserialize(KryoSerializer.java:354)
>
> at org.apache.flink.api.common.typeutils.CompositeSerializer
> .deserialize(CompositeSerializer.java:156)
>
> at org.apache.flink.contrib.streaming.state.RocksDBValueState.value(
> RocksDBValueState.java:89)
>
>
>
> On Thu, Sep 15, 2022 at 7:10 PM Vishal Santoshi <vi...@gmail.com>
> wrote:
>
> << How do I make sure that when reconstituting the state, kryo does not
> complain? It tries to map the previous state to the new definition of Class
> A and complains that it cannot read the value for `String b`.
>
>
>
> >> How do I make sure that when reconstituting the state, kryo does not
> complain? It tries to map the previous state to the new definition of Class
> A and complains that it cannot read the value for `long b`.
>
>
>
> Sorry a typo
>
>
>
>
>
> On Thu, Sep 15, 2022 at 7:04 PM Vishal Santoshi <vi...@gmail.com>
> wrote:
>
> I have state in rocksDB that represents say
>
>
>
> class A {
>
> String a
>
> }
>
>
>
> I now change my class and add another variable
>
>
>
>
> Class A {
>
> String a;
>
> long b = 0;
>
> }
>
>
>
> How do I make sure that when reconstituting the state, kryo does not
> complain? It tries to map the previous state to the new definition of Class
> A and complains that it cannot read the value for `String b`.
>
>
>
> Unfortunately the state is not using POJO serializer.
>
>
>
> Thanks and Regards.
>
>
>
> Vishal
>
>
>
>
>
>
>
>
>
> Diese Nachricht ist ausschliesslich fĂŒr den Adressaten bestimmt und
> beinhaltet unter UmstÀnden vertrauliche Mitteilungen. Da die
> Vertraulichkeit von e-Mail-Nachrichten nicht gewÀhrleistet werden kann,
> ĂŒbernehmen wir keine Haftung fĂŒr die GewĂ€hrung der Vertraulichkeit und
> Unversehrtheit dieser Mitteilung. Bei irrtĂŒmlicher Zustellung bitten wir
> Sie um Benachrichtigung per e-Mail und um Löschung dieser Nachricht sowie
> eventueller AnhÀnge. Jegliche unberechtigte Verwendung oder Verbreitung
> dieser Informationen ist streng verboten.
>
> This message is intended only for the named recipient and may contain
> confidential or privileged information. As the confidentiality of email
> communication cannot be guaranteed, we do not accept any responsibility for
> the confidentiality and the intactness of this message. If you have
> received it in error, please advise the sender by return e-mail and delete
> this message and any attachments. Any unauthorised use or dissemination of
> this information is strictly prohibited.
>
RE: A question about restoring state with an additional variable with kryo
Posted by Schwalbe Matthias <Ma...@viseca.ch>.
Hi Vishal,
Good news and bad news đ:
* Bad: Kryo serializer cannot be used for schema evolution, see [1]
* Good: not all is lost here,
* If you happen to have state that you cannot afford to lose, you can transcode it by means of the savepoint API [2],
* However, this takes quite some effort
* In general, if you ever plan to migrate/extend your schemas, choose a data type that supports schema migration [1],
* In your case, PoJo types would be the closest to your original implementation
* You can disable Kryo in configuration to avoid this situation in the future, by the way,
* Kryo serializer is quite slow compared to the other options and I believe it is only there as a (emergency) fallback solution: [3]
Feel free to ask for clarification đ
Thias
[1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/schema_evolution/#kryo-cannot-be-used-for-schema-evolution
[2] https://nightlies.apache.org/flink/flink-docs-master/docs/libs/state_processor_api/
[3] https://flink.apache.org/news/2020/04/15/flink-serialization-tuning-vol-1.html
From: Vishal Santoshi <vi...@gmail.com>
Sent: Friday, September 16, 2022 1:17 AM
To: user <us...@flink.apache.org>
Subject: Re: A question about restoring state with an additional variable with kryo
â EXTERNAL MESSAGE â CAUTION: Think Before You Click â
The exception thrown is as follows. I realize that it is trying to read the long value. How do I signal to kryo that it is OK and that he object can have a default value
Caused by: java.io<http://java.io>.EOFException: No more bytes left.
at org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.require(NoFetchingInput.java:80)
at com.esotericsoftware.kryo.io<http://com.esotericsoftware.kryo.io>.Input.readVarLong(Input.java:690)
at com.esotericsoftware.kryo.io<http://com.esotericsoftware.kryo.io>.Input.readLong(Input.java:685)
at com.esotericsoftware.kryo.serializers.DefaultSerializers$LongSerializer.read(DefaultSerializers.java:133)
at com.esotericsoftware.kryo.serializers.DefaultSerializers$LongSerializer.read(DefaultSerializers.java:123)
at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:354)
at org.apache.flink.api.common.typeutils.CompositeSerializer.deserialize(CompositeSerializer.java:156)
at org.apache.flink.contrib.streaming.state.RocksDBValueState.value(RocksDBValueState.java:89)
On Thu, Sep 15, 2022 at 7:10 PM Vishal Santoshi <vi...@gmail.com>> wrote:
<< How do I make sure that when reconstituting the state, kryo does not complain? It tries to map the previous state to the new definition of Class A and complains that it cannot read the value for `String b`.
>> How do I make sure that when reconstituting the state, kryo does not complain? It tries to map the previous state to the new definition of Class A and complains that it cannot read the value for `long b`.
Sorry a typo
On Thu, Sep 15, 2022 at 7:04 PM Vishal Santoshi <vi...@gmail.com>> wrote:
I have state in rocksDB that represents say
class A {
String a
}
I now change my class and add another variable
Class A {
String a;
long b = 0;
}
How do I make sure that when reconstituting the state, kryo does not complain? It tries to map the previous state to the new definition of Class A and complains that it cannot read the value for `String b`.
Unfortunately the state is not using POJO serializer.
Thanks and Regards.
Vishal
Diese Nachricht ist ausschliesslich fĂŒr den Adressaten bestimmt und beinhaltet unter UmstĂ€nden vertrauliche Mitteilungen. Da die Vertraulichkeit von e-Mail-Nachrichten nicht gewĂ€hrleistet werden kann, ĂŒbernehmen wir keine Haftung fĂŒr die GewĂ€hrung der Vertraulichkeit und Unversehrtheit dieser Mitteilung. Bei irrtĂŒmlicher Zustellung bitten wir Sie um Benachrichtigung per e-Mail und um Löschung dieser Nachricht sowie eventueller AnhĂ€nge. Jegliche unberechtigte Verwendung oder Verbreitung dieser Informationen ist streng verboten.
This message is intended only for the named recipient and may contain confidential or privileged information. As the confidentiality of email communication cannot be guaranteed, we do not accept any responsibility for the confidentiality and the intactness of this message. If you have received it in error, please advise the sender by return e-mail and delete this message and any attachments. Any unauthorised use or dissemination of this information is strictly prohibited.
Re: A question about restoring state with an additional variable with kryo
Posted by Vishal Santoshi <vi...@gmail.com>.
The exception thrown is as follows. I realize that it is trying to read the
long value. How do I signal to kryo that it is OK and that he object can
have a default value
Caused by: java.io.EOFException: No more bytes left.
at org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.require(
NoFetchingInput.java:80)
at com.esotericsoftware.kryo.io.Input.readVarLong(Input.java:690)
at com.esotericsoftware.kryo.io.Input.readLong(Input.java:685)
at com.esotericsoftware.kryo.serializers.
DefaultSerializers$LongSerializer.read(DefaultSerializers.java:133)
at com.esotericsoftware.kryo.serializers.
DefaultSerializers$LongSerializer.read(DefaultSerializers.java:123)
at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField
.java:113)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(
FieldSerializer.java:528)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer
.deserialize(KryoSerializer.java:354)
at org.apache.flink.api.common.typeutils.CompositeSerializer
.deserialize(CompositeSerializer.java:156)
at org.apache.flink.contrib.streaming.state.RocksDBValueState.value(
RocksDBValueState.java:89)
On Thu, Sep 15, 2022 at 7:10 PM Vishal Santoshi <vi...@gmail.com>
wrote:
> << How do I make sure that when reconstituting the state, kryo does not
> complain? It tries to map the previous state to the new definition of Class
> A and complains that it cannot read the value for `String b`.
>
> >> How do I make sure that when reconstituting the state, kryo does not
> complain? It tries to map the previous state to the new definition of Class
> A and complains that it cannot read the value for `long b`.
>
> Sorry a typo
>
>
> On Thu, Sep 15, 2022 at 7:04 PM Vishal Santoshi <vi...@gmail.com>
> wrote:
>
>> I have state in rocksDB that represents say
>>
>> class A {
>> String a
>> }
>>
>> I now change my class and add another variable
>>
>>
>> Class A {
>> String a;
>> long b = 0;
>> }
>>
>> How do I make sure that when reconstituting the state, kryo does not
>> complain? It tries to map the previous state to the new definition of Class
>> A and complains that it cannot read the value for `String b`.
>>
>> Unfortunately the state is not using POJO serializer.
>>
>> Thanks and Regards.
>>
>> Vishal
>>
>>
>>
>>
>>
Re: A question about restoring state with an additional variable with kryo
Posted by Vishal Santoshi <vi...@gmail.com>.
<< How do I make sure that when reconstituting the state, kryo does not
complain? It tries to map the previous state to the new definition of Class
A and complains that it cannot read the value for `String b`.
>> How do I make sure that when reconstituting the state, kryo does not
complain? It tries to map the previous state to the new definition of Class
A and complains that it cannot read the value for `long b`.
Sorry a typo
On Thu, Sep 15, 2022 at 7:04 PM Vishal Santoshi <vi...@gmail.com>
wrote:
> I have state in rocksDB that represents say
>
> class A {
> String a
> }
>
> I now change my class and add another variable
>
>
> Class A {
> String a;
> long b = 0;
> }
>
> How do I make sure that when reconstituting the state, kryo does not
> complain? It tries to map the previous state to the new definition of Class
> A and complains that it cannot read the value for `String b`.
>
> Unfortunately the state is not using POJO serializer.
>
> Thanks and Regards.
>
> Vishal
>
>
>
>
>