You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Maximilian Michels (Jira)" <ji...@apache.org> on 2020/05/06 15:37:00 UTC
[jira] [Created] (BEAM-9900) Remove the need for
shutdownSourcesOnFinalWatermark flag
Maximilian Michels created BEAM-9900:
----------------------------------------
Summary: Remove the need for shutdownSourcesOnFinalWatermark flag
Key: BEAM-9900
URL: https://issues.apache.org/jira/browse/BEAM-9900
Project: Beam
Issue Type: Improvement
Components: runner-flink
Reporter: Maximilian Michels
Assignee: Maximilian Michels
The {{shutdownSourcesOnFinalWatermark}} has caused some confusion in the past. It is generally used for testing pipelines to ensure that the pipeline and the testing cluster shuts down at the end of the job. Without it, the pipeline will run forever in streaming mode, regardless of whether the input is finite or not.
We didn't want to enable the flag by default because shutting down any operators including sources in Flink will prevent checkpointing from working. If we have side input, for example, that may be the case even for long-running pipelines. However, we can detect whether checkpointing is enabled and set the flag automatically.
The only situation where we may want the flag to be disabled is when users do not have checkpointing enabled but want to take a savepoint. This should be rare and users can mitigate by setting the flag to false to prevent operators from shutting down.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)