You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Jose Cisneros <ja...@googlemail.com> on 2018/10/24 13:58:39 UTC

Fail to recover Keyed State afeter ReinterpretAsKeyedStream

Hi,


To avoid reshuffling in my job, I started using  DataStreamUtils.
reinterpretAsKeyedStream to avoid having to do another keyBy for the same
key.  The BackEndState is RocksDB.


When the job recovers after a failure, the ProcessFunction after the keyBy
restores its Keyed State correctly, while the Process function
after reinterpretAsKeyedStream does not recover the Keyed State.


I have checked the data written by the checkpoints and there is a reference
to the sate.


If I change and use keyBy instead of DataStreamUtils.
reinterpretAsKeyedStream  the Keyed State is recovered as expected.


Is the DataStreamUtils.reinterpretAsKeyedStream function not intended to
use Keyed State?


Thank you.

Regards,


Jose

Re: Fail to recover Keyed State afeter ReinterpretAsKeyedStream

Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.

Hi Jose,

As far as I know, you should be able to use keyed state on a stream returned by DataStreamUtils.reinterpretAsKeyedStream function. That shouldn’t be the issue here.

Have you looked into the logs for any meaningful exceptions of why the restore failed?
That would be helpful here to understand whether or not this is a bug.

Cheers,
Gordon


On 24 October 2018 at 9:58:54 PM, Jose Cisneros (jacisneros33@googlemail.com) wrote:

Hi,

To avoid reshuffling in my job, I started using  DataStreamUtils.reinterpretAsKeyedStream to avoid having to do another keyBy for the same key.  The BackEndState is RocksDB.

When the job recovers after a failure, the ProcessFunction after the keyBy restores its Keyed State correctly, while the Process function after reinterpretAsKeyedStream does not recover the Keyed State.

I have checked the data written by the checkpoints and there is a reference to the sate.

If I change and use keyBy instead of DataStreamUtils.reinterpretAsKeyedStream  the Keyed State is recovered as expected.

Is the DataStreamUtils.reinterpretAsKeyedStream function not intended to use Keyed State? 

Thank you.
Regards,

Jose