You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by "Aggarwal, Ajay" <Aj...@netapp.com> on 2019/03/04 20:08:33 UTC
Checkpoint recovery and state external to flink
What happens when the flink job interacts with a user managed database and hence has some state outside of flink? In these situations when a flink job is recovered from last successful checkpoint, this external state will not be in sync with the recovered flink state. In most cases it will be ahead of the recovered flink state. Any recommendations or best practices to follow here? I am assuming it must be very common for flink applications to interact with external systems (databases, message systems etc.)
Re: Checkpoint recovery and state external to flink
Posted by "Aggarwal, Ajay" <Aj...@netapp.com>.
Hi Yun,
This is good information. Thank you.
However looks like it only applies to SinkFunction. Any thoughts for when intermediate operators are also interacting with external systems?
Thanks.
Ajay
From: Yun Tang <my...@live.com>
Date: Tuesday, March 5, 2019 at 4:04 AM
To: "Aggarwal, Ajay" <Aj...@netapp.com>, "user@flink.apache.org" <us...@flink.apache.org>
Subject: Re: Checkpoint recovery and state external to flink
Hi Ajay
I think two phase commit protocol could solve your concern for the exactly-once external system, Flink already support this feature in some sinks [1], e.g. you could refer to [2] to know which version of Kafaka producer could support exactly-once.
[1] https://flink.apache.org/features/2018/03/01/end-to-end-exactly-once-apache-flink.html
[2] https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html
Best
Yun Tang
________________________________
From: Aggarwal, Ajay <Aj...@netapp.com>
Sent: Tuesday, March 5, 2019 4:08
To: user@flink.apache.org
Subject: Checkpoint recovery and state external to flink
What happens when the flink job interacts with a user managed database and hence has some state outside of flink? In these situations when a flink job is recovered from last successful checkpoint, this external state will not be in sync with the recovered flink state. In most cases it will be ahead of the recovered flink state. Any recommendations or best practices to follow here? I am assuming it must be very common for flink applications to interact with external systems (databases, message systems etc.)
Re: Checkpoint recovery and state external to flink
Posted by Yun Tang <my...@live.com>.
Hi Ajay
I think two phase commit protocol could solve your concern for the exactly-once external system, Flink already support this feature in some sinks [1], e.g. you could refer to [2] to know which version of Kafaka producer could support exactly-once.
[1] https://flink.apache.org/features/2018/03/01/end-to-end-exactly-once-apache-flink.html
[2] https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html
Best
Yun Tang
________________________________
From: Aggarwal, Ajay <Aj...@netapp.com>
Sent: Tuesday, March 5, 2019 4:08
To: user@flink.apache.org
Subject: Checkpoint recovery and state external to flink
What happens when the flink job interacts with a user managed database and hence has some state outside of flink? In these situations when a flink job is recovered from last successful checkpoint, this external state will not be in sync with the recovered flink state. In most cases it will be ahead of the recovered flink state. Any recommendations or best practices to follow here? I am assuming it must be very common for flink applications to interact with external systems (databases, message systems etc.)