Re: EOS with KafkaIO Writes on Flink

[Raghu Angadi <ra...@google.com>]

On Wed, Aug 22, 2018 at 9:37 AM Abdul Qadeer <qu...@gmail.com> wrote:

> Hi!
>
> I came across this comment for KafkaIO.Write.withEOS in Beam 2.6.0:
>
> "Flink runner is
> * one of the runners whose checkpoint semantics are not compatible with current
> * implementation (hope to provide a solution in near future)"
>
> I would like to know if the Java docs are up to date and this support is
> still absent? If yes, then is there a way to get EOS for end to end
> applications written with Beam and Flink runner using Kafka sources/sinks?
> Is there a contribution possible for this support?
>

It is up to date. We would certainly like to have a contribution. Given
Beam model and Flink's horizontal checkpointing, we could not think of a
way to implement EOS. See previous discussions this on dev@ list around Aug
2017.