[jira] [Resolved] (BEAM-769) Spark streaming tests fail on "nothing processed" if runtime env. is slow because timeout is hit before processing is done.

["Amit Sela (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BEAM-769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amit Sela resolved BEAM-769.
----------------------------
       Resolution: Fixed
    Fix Version/s: 0.4.0-incubating

This is generally resolved.
Flakes in streaming tests could still occur, but providing a truly robust solution is a part of implementing triggers.

> Spark streaming tests fail on "nothing processed" if runtime env. is slow because timeout is hit before processing is done.
> ---------------------------------------------------------------------------------------------------------------------------
>
>                 Key: BEAM-769
>                 URL: https://issues.apache.org/jira/browse/BEAM-769
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-spark
>    Affects Versions: Not applicable
>            Reporter: Daniel Halperin
>            Assignee: Amit Sela
>             Fix For: 0.4.0-incubating
>
>
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/1586/
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/1587/
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/1588/
> {code}
> org.apache.beam.runners.spark.translation.streaming.FlattenStreamingTest.testFlattenUnbounded
> org.apache.beam.runners.spark.translation.streaming.KafkaStreamingTest.testRun
> org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest.testFixedWindows
> {code}
> The above tests use a hard-timeout (ungraceful stop) so if the runtime env. is slow enough so that the batch is not done, it'll stop anyway and assert and rightfully fail.
> It's difficult to create locally because I never had trouble on my laptop.
> Since Jenkins will be slow from time to time, it is reasonable enough to have a more robust solution here :
> # don't use checkpoint (Spark) if not necessary - only really necessary for one test in {{KafkaStreamingTest}} and {{ResumeFromCheckpointStreamingTest}} I think.
> #  allow for graceful stop - will take longer for each test, but should allow the test to finish even if runtime env. is slow.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)