You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Cliff Resnick <cr...@gmail.com> on 2019/03/18 03:20:23 UTC
State Migration with RocksDB MapState
After trying out state migration in 1.8 rc2 I ran into this hard stop
below. The comment does not give an indication why rocksdb map state cannot
be migrated, and I'm wondering what the status is, since we do need this
functionality and would like to know if this is a long-term blocker or not.
Anyone know?
https://github.com/apache/flink/blob/953a5ffcbdae4115f7d525f310723cf8770779df/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java#L542
Re: State Migration with RocksDB MapState
Posted by Cliff Resnick <cr...@gmail.com>.
Great news! Thanks
On Thu, Apr 25, 2019, 2:59 AM Tzu-Li (Gordon) Tai <tz...@apache.org>
wrote:
> Hi Cliff,
>
> Thanks for bringing this up again.
>
> I think it would make sense to at least move this forward be only
> exclusively checking the schema for user keys in MapState, and allow value
> schema evolution.
> I'll comment on the JIRA about this, and also make it a blocker for 1.9.0
> to make sure it will be resolved by then.
>
> Concerning your question on GenericRecord:
> The actual schema resolution is still performed using the Avro schema, so
> you will still bump into the same issue.
>
> Best,
> Gordon
>
> On Wed, Apr 24, 2019 at 7:45 PM Cliff Resnick <cr...@gmail.com> wrote:
>
>> Hi Gordon,
>>
>> I noticed there has been no movement on this issue and I'm wondering if I
>> can find some way to work around this.
>> My MapState value is a case class container of Avro-generated
>> SpecificRecords. If one SpecificRecord changes I am stuck.
>>
>> From the issue It seems like the blocker is around evolving the MapState
>> key type. That may be a nasty problem, but my key type is stable and will
>> never change. What do you think the level of difficulty would be to add
>> support for evolving only the value?
>>
>> Also, if I use GenericRecord instead of SpecificRecord will the need for
>> schema evolution still be triggered? Or does it actually go down to the
>> avro schema rather than just the class serialVersionUID?
>>
>>
>>
>>
>>
>>
>> On Mon, Mar 18, 2019 at 1:10 AM Tzu-Li (Gordon) Tai <tz...@apache.org>
>> wrote:
>>
>>> Hi Cliff,
>>>
>>> Thanks for bringing this up!
>>> AFAIK, this wouldn't be a long-term blocker. I've just opened a JIRA to
>>> track this [1].
>>>
>>> As explained in the JIRA ticket, the main reason this is disallowed in
>>> the initial support for state schema evolution was due to how migration was
>>> implemented in the RocksDB state backend.
>>> Technically speaking, enabling this in the future is definitely possible.
>>>
>>> Cheers,
>>> Gordon
>>>
>>> [1] https://issues.apache.org/jira/browse/FLINK-11947
>>>
>>> On Mon, Mar 18, 2019 at 11:20 AM Cliff Resnick <cr...@gmail.com> wrote:
>>>
>>>> After trying out state migration in 1.8 rc2 I ran into this hard stop
>>>> below. The comment does not give an indication why rocksdb map state cannot
>>>> be migrated, and I'm wondering what the status is, since we do need this
>>>> functionality and would like to know if this is a long-term blocker or not.
>>>> Anyone know?
>>>>
>>>>
>>>> https://github.com/apache/flink/blob/953a5ffcbdae4115f7d525f310723cf8770779df/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java#L542
>>>>
>>>
Re: State Migration with RocksDB MapState
Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
Hi Cliff,
Thanks for bringing this up again.
I think it would make sense to at least move this forward be only
exclusively checking the schema for user keys in MapState, and allow value
schema evolution.
I'll comment on the JIRA about this, and also make it a blocker for 1.9.0
to make sure it will be resolved by then.
Concerning your question on GenericRecord:
The actual schema resolution is still performed using the Avro schema, so
you will still bump into the same issue.
Best,
Gordon
On Wed, Apr 24, 2019 at 7:45 PM Cliff Resnick <cr...@gmail.com> wrote:
> Hi Gordon,
>
> I noticed there has been no movement on this issue and I'm wondering if I
> can find some way to work around this.
> My MapState value is a case class container of Avro-generated
> SpecificRecords. If one SpecificRecord changes I am stuck.
>
> From the issue It seems like the blocker is around evolving the MapState
> key type. That may be a nasty problem, but my key type is stable and will
> never change. What do you think the level of difficulty would be to add
> support for evolving only the value?
>
> Also, if I use GenericRecord instead of SpecificRecord will the need for
> schema evolution still be triggered? Or does it actually go down to the
> avro schema rather than just the class serialVersionUID?
>
>
>
>
>
>
> On Mon, Mar 18, 2019 at 1:10 AM Tzu-Li (Gordon) Tai <tz...@apache.org>
> wrote:
>
>> Hi Cliff,
>>
>> Thanks for bringing this up!
>> AFAIK, this wouldn't be a long-term blocker. I've just opened a JIRA to
>> track this [1].
>>
>> As explained in the JIRA ticket, the main reason this is disallowed in
>> the initial support for state schema evolution was due to how migration was
>> implemented in the RocksDB state backend.
>> Technically speaking, enabling this in the future is definitely possible.
>>
>> Cheers,
>> Gordon
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-11947
>>
>> On Mon, Mar 18, 2019 at 11:20 AM Cliff Resnick <cr...@gmail.com> wrote:
>>
>>> After trying out state migration in 1.8 rc2 I ran into this hard stop
>>> below. The comment does not give an indication why rocksdb map state cannot
>>> be migrated, and I'm wondering what the status is, since we do need this
>>> functionality and would like to know if this is a long-term blocker or not.
>>> Anyone know?
>>>
>>>
>>> https://github.com/apache/flink/blob/953a5ffcbdae4115f7d525f310723cf8770779df/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java#L542
>>>
>>
Re: State Migration with RocksDB MapState
Posted by Cliff Resnick <cr...@gmail.com>.
Hi Gordon,
I noticed there has been no movement on this issue and I'm wondering if I
can find some way to work around this.
My MapState value is a case class container of Avro-generated
SpecificRecords. If one SpecificRecord changes I am stuck.
From the issue It seems like the blocker is around evolving the MapState
key type. That may be a nasty problem, but my key type is stable and will
never change. What do you think the level of difficulty would be to add
support for evolving only the value?
Also, if I use GenericRecord instead of SpecificRecord will the need for
schema evolution still be triggered? Or does it actually go down to the
avro schema rather than just the class serialVersionUID?
On Mon, Mar 18, 2019 at 1:10 AM Tzu-Li (Gordon) Tai <tz...@apache.org>
wrote:
> Hi Cliff,
>
> Thanks for bringing this up!
> AFAIK, this wouldn't be a long-term blocker. I've just opened a JIRA to
> track this [1].
>
> As explained in the JIRA ticket, the main reason this is disallowed in the
> initial support for state schema evolution was due to how migration was
> implemented in the RocksDB state backend.
> Technically speaking, enabling this in the future is definitely possible.
>
> Cheers,
> Gordon
>
> [1] https://issues.apache.org/jira/browse/FLINK-11947
>
> On Mon, Mar 18, 2019 at 11:20 AM Cliff Resnick <cr...@gmail.com> wrote:
>
>> After trying out state migration in 1.8 rc2 I ran into this hard stop
>> below. The comment does not give an indication why rocksdb map state cannot
>> be migrated, and I'm wondering what the status is, since we do need this
>> functionality and would like to know if this is a long-term blocker or not.
>> Anyone know?
>>
>>
>> https://github.com/apache/flink/blob/953a5ffcbdae4115f7d525f310723cf8770779df/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java#L542
>>
>
Re: State Migration with RocksDB MapState
Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
Hi Cliff,
Thanks for bringing this up!
AFAIK, this wouldn't be a long-term blocker. I've just opened a JIRA to
track this [1].
As explained in the JIRA ticket, the main reason this is disallowed in the
initial support for state schema evolution was due to how migration was
implemented in the RocksDB state backend.
Technically speaking, enabling this in the future is definitely possible.
Cheers,
Gordon
[1] https://issues.apache.org/jira/browse/FLINK-11947
On Mon, Mar 18, 2019 at 11:20 AM Cliff Resnick <cr...@gmail.com> wrote:
> After trying out state migration in 1.8 rc2 I ran into this hard stop
> below. The comment does not give an indication why rocksdb map state cannot
> be migrated, and I'm wondering what the status is, since we do need this
> functionality and would like to know if this is a long-term blocker or not.
> Anyone know?
>
>
> https://github.com/apache/flink/blob/953a5ffcbdae4115f7d525f310723cf8770779df/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java#L542
>