You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "Aviem Zur (JIRA)" <ji...@apache.org> on 2017/09/06 13:36:00 UTC

[jira] [Created] (BEAM-2847) Stop streaming pipeline when global watermark reaches +INF

Aviem Zur created BEAM-2847:
-------------------------------

             Summary: Stop streaming pipeline when global watermark reaches +INF 
                 Key: BEAM-2847
                 URL: https://issues.apache.org/jira/browse/BEAM-2847
             Project: Beam
          Issue Type: Improvement
          Components: runner-spark
            Reporter: Aviem Zur
            Assignee: Amit Sela


Stop streaming pipeline when global watermark reaches +INF as suggested in BEAM-849 comments.

We can do this similarly to the solution in [TestSparkRunner#awaitWatermarksOrTimeout|https://github.com/apache/beam/blob/v2.1.0/runners/spark/src/main/java/org/apache/beam/runners/spark/TestSparkRunner.java#L158-L179] by polling {code}
SparkTimerInternals.global(GlobalWatermarkHolder.get(...))
{code}

We can also remove this code from {{TestSparkRunner}} if we make sure all sources in test behave in a way which shifts the watermark to +INF at the end of the test.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)