partial upgrade

[Chen Qin <qi...@gmail.com>]

Hi there,

So far, flink job is interpreted and deployed during bootstrap phase. Once
pipeline runs, it's very hard to do partial upgrade without stop execution.
(like savepoint is heavy) Is there any plan to allow upload annotated jar
package which hints which stream tasks implementation CAN BE partial
upgraded after next checkpoint succeed without worry about backfill.


Thanks,
Chen

Re: partial upgrade

[Chen Qin <qi...@gmail.com>]

Hi Timo,

Thanks for update. Flink pipeline deployment is very ad hoc and hard to
maintain as a service platform.
Each team may update their pipeline frequently which was completely
blackbox and managed. Idea situation would be have a repo where all code
lives and mapped to running pipelines in configuration. When new code
landed ( without break topology compatibility), job manager should be able
notified to pick up and load new class.

Is there any doc I can follow to wire userclassloader up to prototype :)

Thanks,
Chen

On Mon, Oct 2, 2017 at 2:20 AM, Timo Walther <tw...@apache.org> wrote:

> Hi Chen,
>
> I think in a long-term perspective it makes sense to support things like
> this. The next big step is dynamic scaling without stopping the execution.
> Partial upgrades could be addressed afterwards, but I'm not aware of any
> plans.
>
> Until then, I would recommend a different architecture by using connect()
> and stream in a new logic dynamically. This is especially interesting for
> ML models etc.
>
> Regards,
> Timo
>
>
> Am 10/1/17 um 3:03 AM schrieb Chen Qin:
>
>> Hi there,
>>
>> So far, flink job is interpreted and deployed during bootstrap phase. Once
>> pipeline runs, it's very hard to do partial upgrade without stop
>> execution.
>> (like savepoint is heavy) Is there any plan to allow upload annotated jar
>> package which hints which stream tasks implementation CAN BE partial
>> upgraded after next checkpoint succeed without worry about backfill.
>>
>>
>> Thanks,
>> Chen
>>
>>
>

Re: partial upgrade

[Timo Walther <tw...@apache.org>]

Hi Chen,

I think in a long-term perspective it makes sense to support things like 
this. The next big step is dynamic scaling without stopping the 
execution. Partial upgrades could be addressed afterwards, but I'm not 
aware of any plans.

Until then, I would recommend a different architecture by using 
connect() and stream in a new logic dynamically. This is especially 
interesting for ML models etc.

Regards,
Timo


Am 10/1/17 um 3:03 AM schrieb Chen Qin:
> Hi there,
>
> So far, flink job is interpreted and deployed during bootstrap phase. Once
> pipeline runs, it's very hard to do partial upgrade without stop execution.
> (like savepoint is heavy) Is there any plan to allow upload annotated jar
> package which hints which stream tasks implementation CAN BE partial
> upgraded after next checkpoint succeed without worry about backfill.
>
>
> Thanks,
> Chen
>