You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Antoine Philippot (JIRA)" <ji...@apache.org> on 2017/10/20 12:37:00 UTC
[jira] [Updated] (FLINK-7883) Stop fetching source before a cancel
with savepoint
[ https://issues.apache.org/jira/browse/FLINK-7883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Antoine Philippot updated FLINK-7883:
-------------------------------------
Description:
For a cancel with savepoint command, the JobManager trigger the cancel call once the savepoint is finished, but during the savepoint execution, kafka source continue to poll new messages which will not be part of the savepoint and will be replayed on the next application start.
A solution could be to stop fetching the source stream task before triggering the savepoint.
I suggest to add an interface {{StoppableFetchingSourceFunction}} with a method {{stopFetching}} that existant SourceFunction implementations could implement.
We can add a {{stopFetchingSource}} property in
{{CheckpointOptions}} class to pass the desired behaviour from {{JobManager.handleMessage(CancelJobWithSavepoint)}} to {{SourceStreamTask.triggerCheckpoint}}
was:
For a cancel with savepoint command, the JobManager trigger the cancel call once the savepoint is finished, but during the savepoint execution, kafka source continue to poll new messages which will not be part of the savepoint and will be replayed on the next application start.
A solution could be to stop fetching the source stream task before triggering the savepoint.
I suggest to add an interface {{StoppableFetchingSourceFunction}} with a method {{stopFetching}} that existant SourceFunction implementations could implement.
We can add a {{stopFetchingSource` property in {{CheckpointProperties}} and
{{CheckpointOptions}} class to pass the desired behaviour from {{JobManager.handleMessage(CancelJobWithSavepoint)}} to {{SourceStreamTask.triggerCheckpoint}}
> Stop fetching source before a cancel with savepoint
> ---------------------------------------------------
>
> Key: FLINK-7883
> URL: https://issues.apache.org/jira/browse/FLINK-7883
> Project: Flink
> Issue Type: Improvement
> Components: DataStream API, Kafka Connector, State Backends, Checkpointing
> Affects Versions: 1.4.0, 1.3.2
> Reporter: Antoine Philippot
>
> For a cancel with savepoint command, the JobManager trigger the cancel call once the savepoint is finished, but during the savepoint execution, kafka source continue to poll new messages which will not be part of the savepoint and will be replayed on the next application start.
> A solution could be to stop fetching the source stream task before triggering the savepoint.
> I suggest to add an interface {{StoppableFetchingSourceFunction}} with a method {{stopFetching}} that existant SourceFunction implementations could implement.
> We can add a {{stopFetchingSource}} property in
> {{CheckpointOptions}} class to pass the desired behaviour from {{JobManager.handleMessage(CancelJobWithSavepoint)}} to {{SourceStreamTask.triggerCheckpoint}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)