Automatic deployment of new version of streaming stateful job

[Maxim Parkachov <la...@gmail.com>]

Hi,

I'm trying to bring my first stateful streaming Flink job to production and
have trouble understanding how to integrate it with CI/CD pipeline. I can
cancel the job with savepoint, but in order to start new version of
application I need to specify savepoint path manually ?

So, basically my question, what is best practice of automatically
restarting or deploying new version of stateful streaming application ?
Every tip is greatly appreciated.

Thanks,
Maxim.

Re: Automatic deployment of new version of streaming stateful job

[Maxim Parkachov <la...@gmail.com>]

Hi Marc,

thanks a lot for the tool. Unfortunately, I could not direcly use it, but I
will take couple of ideas and will implement my own script.

Nevertherless, I'm really surprised that such functionality doesn't exist
out of the box.

Regards,
Maxim.

On Tue, Jul 16, 2019 at 9:22 AM Marc Rooding <ma...@webresource.nl> wrote:

> Hi Maxim
>
> You could write a script yourself which triggers the cancel with savepoint
> and then starts a new version using the savepoint that was created during
> the cancel.
>
> However, I’ve built a tool that allows you to perform these steps more
> easily: https://github.com/ing-bank/flink-deployer. The deployer will
> allow you to deploy or upgrade your jobs. All you need to do is integrate
> it into your CI/CD.
>
> Kind regards
>
> Marc
> On 16 Jul 2019, 02:46 +0200, Maxim Parkachov <la...@gmail.com>,
> wrote:
>
> Hi,
>
> I'm trying to bring my first stateful streaming Flink job to production
> and have trouble understanding how to integrate it with CI/CD pipeline. I
> can cancel the job with savepoint, but in order to start new version of
> application I need to specify savepoint path manually ?
>
> So, basically my question, what is best practice of automatically
> restarting or deploying new version of stateful streaming application ?
> Every tip is greatly appreciated.
>
> Thanks,
> Maxim.
>
>

Re: Automatic deployment of new version of streaming stateful job

[Marc Rooding <ma...@webresource.nl>]

Hi Maxim

You could write a script yourself which triggers the cancel with savepoint and then starts a new version using the savepoint that was created during the cancel.

However, I’ve built a tool that allows you to perform these steps more easily: https://github.com/ing-bank/flink-deployer. The deployer will allow you to deploy or upgrade your jobs. All you need to do is integrate it into your CI/CD.

Kind regards

Marc
On 16 Jul 2019, 02:46 +0200, Maxim Parkachov <la...@gmail.com>, wrote:
> Hi,
>
> I'm trying to bring my first stateful streaming Flink job to production and have trouble understanding how to integrate it with CI/CD pipeline. I can cancel the job with savepoint, but in order to start new version of application I need to specify savepoint path manually ?
>
> So, basically my question, what is best practice of automatically restarting or deploying new version of stateful streaming application ? Every tip is greatly appreciated.
>
> Thanks,
> Maxim.