Flink Checkpointing in production

[Ahmad Hassan <ah...@gmail.com>]

Hi All,

We need two clarifications for using Flink 1.6.0. We have flink jobs
running to handle 100's of tenants with sliding window of 24hrs and slide
by 5 minutes.

1) If checkpointing is enabled and flink job crashes in the middle of
spitting out results to kafka producer. Then if the job resumes from the
previously known checkpoint, then what happens to the partial results that
kafka producer has already sent out. Do we end up sending duplicates for
the same window after recovery from checkpoint. Or this window never gets
triggered again as we have forward in time ?

2) If one input event flink, produces 100 messages out in kafka producer
after window expiry, then how will setFlushOnCheckpoint will work in kafka
producer. I am confused how will it ensure that all records before
checkpoint have been sent out because we are creating 100 output events
from single input event.

Thanks for the help.

Best Regards,

Re: Flink Checkpointing in production

[vino yang <ya...@gmail.com>]

Hi Ahmad,

Answer your question：

Ahmad Hassan <ah...@gmail.com> 于2018年9月12日周三 下午4:39写道：

> Hi All,
>
> We need two clarifications for using Flink 1.6.0. We have flink jobs
> running to handle 100's of tenants with sliding window of 24hrs and slide
> by 5 minutes.
>
> 1) If checkpointing is enabled and flink job crashes in the middle of
> spitting out results to kafka producer. Then if the job resumes from the
> previously known checkpoint, then what happens to the partial results that
> kafka producer has already sent out. Do we end up sending duplicates for
> the same window after recovery from checkpoint. Or this window never gets
> triggered again as we have forward in time ?
>


*Which version of Kafka do you use? Kafka has supported producer
transactions since version 0.11. If you use this version of the connector
and choose the checkpoint mode to be exact-once, the entire pipeline will
guarantee end-to-end consistency.*


>
> 2) If one input event flink, produces 100 messages out in kafka producer
> after window expiry, then how will setFlushOnCheckpoint will work in
> kafka producer. I am confused how will it ensure that all records before
> checkpoint have been sent out because we are creating 100 output events
> from single input event.
>

*After the first reply, if your source connector can also guarantee the
end-to-end exact once semantics, then the whole process is no problem.
Because Flink's checkpoint mechanism is a snapshot of global consistency.
It is not just an independent snapshot of a particular operator.*


>
> Thanks for the help.
>
> Best Regards,
>