You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/15 17:31:17 UTC
[GitHub] [beam] omkardeshpande8 opened a new issue, #21897: [Feature Request]: Flink runner savepoint backward compatibility
omkardeshpande8 opened a new issue, #21897:
URL: https://github.com/apache/beam/issues/21897
### What would you like to happen?
Flink savepoints are backward compatible. But when using beam flink runner, this is not the case. It prevents seamless migration versions for stateful applications.
### Issue Priority
Priority: 1
### Issue Component
Component: runner-flink
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Flink runner savepoint backward compatibility [beam]
Posted by "je-ik (via GitHub)" <gi...@apache.org>.
je-ik commented on issue #21897:
URL: https://github.com/apache/beam/issues/21897#issuecomment-2058713604
Related: https://github.com/apache/beam/issues/30385
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] je-ik commented on issue #21897: [Feature Request]: Flink runner savepoint backward compatibility
Posted by GitBox <gi...@apache.org>.
je-ik commented on issue #21897:
URL: https://github.com/apache/beam/issues/21897#issuecomment-1177221561
Generally this is problem of Java serialization. Beam could (and should) pay more attention to not breaking it, but it nevertheless can happen from time to time. Moreover, the PTransform expansion can change and a lot of other things can go wrong, so - currently - the ability to recreate the pipeline into meaningful state from scratch is vital.
I *think* it could be theoretically possible to create a runner and beam-version (into some extent) agnostic snapshot of a pipeline, which could then be used to overcome such restrictions and even port a (running) pipeline between runners. But that would be a large project so even if something like this will come into existence in the future it cannot be expected soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] je-ik commented on issue #21897: [Feature Request]: Flink runner savepoint backward compatibility
Posted by GitBox <gi...@apache.org>.
je-ik commented on issue #21897:
URL: https://github.com/apache/beam/issues/21897#issuecomment-1171041116
What do you mean by backward compatible savepoint in Flink? Beam's Flink runner uses Flink's checkpoint/savepoint mechanism, thus it should be compatible under the same circumstances as native Flink. The problem is that Beam adds additional layer of abstraction that can cause state incompatibility - e.g. when a translation of a Beam PTransform changes, when a Coder changes, etc.
There currently is not any mechanism that can transfer incompatible savepoints between versions, the only option is to run a "bootstrap" version of your job, that creates the state from some historical data.
Reading the a savepoint and converting it to preserve compatibility across beam versions *could* be possible under some circumstances, but is not possible in general.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] jainikvora commented on issue #21897: [Feature Request]: Flink runner savepoint backward compatibility
Posted by GitBox <gi...@apache.org>.
jainikvora commented on issue #21897:
URL: https://github.com/apache/beam/issues/21897#issuecomment-1171552162
@je-ik From our experience, just beam version upgrade without any processor code change is not compatible and this is Beam's restriction, not Flink's. [Here's a mailing thread](https://lists.apache.org/thread/05yl9lsdmnd5lxknzkdtsvv836o6hx38) with more details. Any plans to support this in Beam? Also interested to know if this restriction exists for dataflow as well or only non-dataflow runners?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org