You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "Aviem Zur (JIRA)" <ji...@apache.org> on 2017/09/06 13:36:00 UTC
[jira] [Created] (BEAM-2847) Stop streaming pipeline when global
watermark reaches +INF
Aviem Zur created BEAM-2847:
-------------------------------
Summary: Stop streaming pipeline when global watermark reaches +INF
Key: BEAM-2847
URL: https://issues.apache.org/jira/browse/BEAM-2847
Project: Beam
Issue Type: Improvement
Components: runner-spark
Reporter: Aviem Zur
Assignee: Amit Sela
Stop streaming pipeline when global watermark reaches +INF as suggested in BEAM-849 comments.
We can do this similarly to the solution in [TestSparkRunner#awaitWatermarksOrTimeout|https://github.com/apache/beam/blob/v2.1.0/runners/spark/src/main/java/org/apache/beam/runners/spark/TestSparkRunner.java#L158-L179] by polling {code}
SparkTimerInternals.global(GlobalWatermarkHolder.get(...))
{code}
We can also remove this code from {{TestSparkRunner}} if we make sure all sources in test behave in a way which shifts the watermark to +INF at the end of the test.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)