Atomic savepint and cancel

[Or Sher <or...@gmail.com>]

Hi,

I'm a bit new to Flink and I'm trying to figure out what's the best way to
make an upgrade for my current running topology without having duplicate
messages being sent by the the sink. (One time prior the upgrade and one
time after).

I thought that the "atomic" part of the savepoint & cancel suggested that i
can just take a savepoint and cancel the job at the same time and later on
start from that savepoint and that would be it.

Having tried that, it seems that I got many duplicated messages sent by the
kafka producer sink again after the restore from savepoint.

Is that suppose to happen?
Did I misunderstood the "atomic" meaning?

Thanks,
Or.

Re: Atomic savepint and cancel

[Chesnay Schepler <ch...@apache.org>]

Hello,

savepoint&cancel is in general not an atomic operation, it only 
guarantees that no other checkpoint will be completed between the 
savepoint and the job cancellation.

You can only guarantee that no messages are sent out if you used a sink 
that supports exactly-once, which as far as i know, in the case of 
Kafka, is only possible with the upcoming 0.11 connector (PR). 
<https://github.com/apache/flink/pull/4239>

On 29.08.2017 14:11, Or Sher wrote:
> Hi,
>
> I'm a bit new to Flink and I'm trying to figure out what's the best 
> way to make an upgrade for my current running topology without having 
> duplicate messages being sent by the the sink. (One time prior the 
> upgrade and one time after).
>
> I thought that the "atomic" part of the savepoint & cancel suggested 
> that i can just take a savepoint and cancel the job at the same time 
> and later on start from that savepoint and that would be it.
>
> Having tried that, it seems that I got many duplicated messages sent 
> by the kafka producer sink again after the restore from savepoint.
>
> Is that suppose to happen?
> Did I misunderstood the "atomic" meaning?
>
> Thanks,
> Or.