Writing a reliable Flink source for a NON-replay-able queue/protocol that supports message ACKs

[Martin Eden <ma...@gmail.com>]

Hi everyone,

Are there any examples of how to implement a reliable (zero data loss)
Flink source reading from a system that is not replay-able but supports
acknowledging messages?

Or any pointers of how one can achieve this and how Flink can help?

I imagine it should involve a write ahead log but not yet clear of how to
implement it and how to integrate with the Flink fault tolerance mechanism.
Can Flink maintain the write ahead log for me?

Also, does it make sense to start implementing this in the current stable
Flink release 1.2 or is there any advantage in implementing it directly in
Flink 1.3 since it is coming up soon anyway?

Thanks,
M

Re: Writing a reliable Flink source for a NON-replay-able queue/protocol that supports message ACKs

[Kostas Kloudas <k....@data-artisans.com>]

Hi Martin! 

For an example of a source that acknowledges received messages, you could check the MessageAcknowledgingSourceBase 
and the MultipleIdsMessageAcknowledgingSourceBase that ship with Flink. I hope this will give you some ideas.

Now for the Flink version on top of which to implement your source, I would suggest the Flink 1.3. The reason is that it will 
come out soon (~1 month) and it will include a lot of new features and bug-fixes. Until then, it may change a bit, but the APIs
that you will be using, will not change. 

So why not going straight for the more future-proof way?

Thanks,
Kostas

> On Apr 24, 2017, at 11:20 PM, Martin Eden <ma...@gmail.com> wrote:
> 
> Hi everyone,
> 
> Are there any examples of how to implement a reliable (zero data loss) Flink source reading from a system that is not replay-able but supports acknowledging messages?
> 
> Or any pointers of how one can achieve this and how Flink can help?
> 
> I imagine it should involve a write ahead log but not yet clear of how to implement it and how to integrate with the Flink fault tolerance mechanism. Can Flink maintain the write ahead log for me?
> 
> Also, does it make sense to start implementing this in the current stable Flink release 1.2 or is there any advantage in implementing it directly in Flink 1.3 since it is coming up soon anyway?
> 
> Thanks,
> M