Re: Important (proposed) CEP changes for Flink 1.5.

[Vishal Santoshi <vi...@gmail.com>]

That should be fine by us. We do not plan to do a SP and restore on our
prod CEP pipe, when we move to 1.5 from 1.4. I do hope that all other pipes
will remain back ward compatible ( 1.4 to 1.5 ).

On Wed, Feb 21, 2018 at 5:06 AM, Kostas Kloudas <kk...@gmail.com> wrote:

> Hi all,
>
> Currently due to backwards compatibility there are some issues that seem
> to be affecting CEP users that use RocksDB. As an example you can see this:
>
> https://issues.apache.org/jira/browse/FLINK-7756
>
> We suspect that the issues have mainly to do with the way state was kept
> internally in previous versions (using java serialization) and got
> inherited to the newer ones.
>
> Given this, we would like to introduce a revamped state serialization
> strategy which solves the sources of these problems and also reduces the
> size of the state kept at each checkpoint. The problem with this, is that
> the state format changes so already existing jobs will not be able to
> restart from a savepoint taken from a previous version of the library.
>
> I am writing this email to see if there are any objections to merging
> these changes in the 1.5 release which is about to be released. I suggest
> to wait for responses until Friday 5pm UTC+1, and if there is non, then we
> merge. If there are any, then we have to see how to proceed.
>
> You can try out the changes in this branch: https://github.com/aljoscha/
> flink/commits/fix-flink-cep-serialization and report here if you have any
> problems or suggestions.
>
> Also feel free to forward this to other users, if I forgot someone.
>
> Cheers,
> Kostas
>
>

Re: Important (proposed) CEP changes for Flink 1.5.

[Kostas Kloudas <kk...@gmail.com>]

Hi Vishal,

Yes these changes will only affect CEP.

Cheers,
Kostas

> On Feb 21, 2018, at 5:42 PM, Vishal Santoshi <vi...@gmail.com> wrote:
> 
> That should be fine by us. We do not plan to do a SP and restore on our prod CEP pipe, when we move to 1.5 from 1.4. I do hope that all other pipes will remain back ward compatible ( 1.4 to 1.5 ).
> 
> On Wed, Feb 21, 2018 at 5:06 AM, Kostas Kloudas <kkloudas@gmail.com <ma...@gmail.com>> wrote:
> Hi all,
> 
> Currently due to backwards compatibility there are some issues that seem to be affecting CEP users that use RocksDB. As an example you can see this:
> 
> https://issues.apache.org/jira/browse/FLINK-7756 <https://issues.apache.org/jira/browse/FLINK-7756>
> 
> We suspect that the issues have mainly to do with the way state was kept internally in previous versions (using java serialization) and got inherited to the newer ones.
> 
> Given this, we would like to introduce a revamped state serialization strategy which solves the sources of these problems and also reduces the size of the state kept at each checkpoint. The problem with this, is that the state format changes so already existing jobs will not be able to restart from a savepoint taken from a previous version of the library.
> 
> I am writing this email to see if there are any objections to merging these changes in the 1.5 release which is about to be released. I suggest to wait for responses until Friday 5pm UTC+1, and if there is non, then we merge. If there are any, then we have to see how to proceed.
> 
> You can try out the changes in this branch: https://github.com/aljoscha/flink/commits/fix-flink-cep-serialization <https://github.com/aljoscha/flink/commits/fix-flink-cep-serialization> and report here if you have any problems or suggestions.
> 
> Also feel free to forward this to other users, if I forgot someone.
> 
> Cheers,
> Kostas
> 
>