You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Andrea Gallina <ga...@yahoo.it> on 2019/08/12 11:44:55 UTC

Changing the way keys are defined breaks savepoints

Hi everyone,

I have a job running in production whose structure is approximately this;

stream
 ?? .filter(inboundData -> inboundData.hasToBeFiltered())
 ?? .keyBy("myKey")
 ?? .process(doSomething());

I've recently decided to test the extent to which I can change a job's 
structure without breaking backward compatibility; more specifically, 
I've tried to change the way the key is defined in the keyBy() operator 
by defining it as a lambda function rather than by field expression. The 
modified structure would therefore look like this:

stream
 ?? .filter(inboundData -> inboundData.hasToBeFiltered())
 ?? .keyBy(inboundData -> inboundData.getMyKey())
 ?? .process(doSomething());

I then tried to run the new job by restoring the savepoint taken with 
the old structure, but I get a state migration exception:

org.apache.flink.util.StateMigrationException: The new key serializer 
must be compatible

Now this was a bit unexpected since changing the way a key is defined 
does not seem like a breaking change (unlike changing the actual key 
used for partitioning).

Is this an expected behavior or am I missing something?

Thanks


---
Questa e-mail ? stata controllata per individuare virus con Avast antivirus.
https://www.avast.com/antivirus


Re: Changing the way keys are defined breaks savepoints

Posted by Congxian Qiu <qc...@gmail.com>.
Hi,

From what you described, does the key have the same type before and after
the change? and for using Lambda expression, maybe the doc[1] can be
helpful(especially the limitation)

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/java_lambdas.html#examples-and-limitations
Best,
Congxian


Andrea Gallina <ga...@yahoo.it> 于2019年8月12日周一 下午7:45写道:

> Hi everyone,
>
> I have a job running in production whose structure is approximately this;
>
> stream
>  ?? .filter(inboundData -> inboundData.hasToBeFiltered())
>  ?? .keyBy("myKey")
>  ?? .process(doSomething());
>
> I've recently decided to test the extent to which I can change a job's
> structure without breaking backward compatibility; more specifically,
> I've tried to change the way the key is defined in the keyBy() operator
> by defining it as a lambda function rather than by field expression. The
> modified structure would therefore look like this:
>
> stream
>  ?? .filter(inboundData -> inboundData.hasToBeFiltered())
>  ?? .keyBy(inboundData -> inboundData.getMyKey())
>  ?? .process(doSomething());
>
> I then tried to run the new job by restoring the savepoint taken with
> the old structure, but I get a state migration exception:
>
> org.apache.flink.util.StateMigrationException: The new key serializer
> must be compatible
>
> Now this was a bit unexpected since changing the way a key is defined
> does not seem like a breaking change (unlike changing the actual key
> used for partitioning).
>
> Is this an expected behavior or am I missing something?
>
> Thanks
>
>
> ---
> Questa e-mail ? stata controllata per individuare virus con Avast
> antivirus.
> https://www.avast.com/antivirus
>
>