You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flink.apache.org by Maximilian Michels <mx...@apache.org> on 2019/03/13 17:04:45 UTC

Status of FLINK-2491 (Checkpointing of shutdown sources)

Hi,

Has there been any progress on 
https://issues.apache.org/jira/browse/FLINK-2491?

For the Flink Runner in Apache Beam we keep operators alive to avoid 
checkpointing to stop working [1]. Users of Flink's native API have to 
take care of this themselves.

To fix FLINK-2491 we have to:

   1) Remove shutdown operators from the list of to-be-checkpointed
      operators.

   2) Persist the shutdown operators in checkpoints to be able to 
restore
      the job correctly afterwards.

It would be great to fix this long-standing issue. Apart from removing 
the need for workarounds it would also simplify some of the test setup 
which relies on checkpointing to continue working when operators shut down.

Do you think we can make progress on this matter?

Cheers,
Max

[1] 
https://github.com/apache/beam/blob/6e89b6c7a8191429fc422228c5b8d5c75c9caa05/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java#L198