[jira] [Commented] (BEAM-769) Spark streaming tests fail on "nothing processed" if runtime env. is slow because timeout is hit before processing is done.

["Amit Sela (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593215#comment-15593215 ] 

Amit Sela commented on BEAM-769:
--------------------------------

Got this ready: https://github.com/amitsela/incubator-beam/tree/BEAM-769 but it might require further attention after https://github.com/apache/incubator-beam/pull/1143 is merged so I'll hold off on it.

> Spark streaming tests fail on "nothing processed" if runtime env. is slow because timeout is hit before processing is done.
> ---------------------------------------------------------------------------------------------------------------------------
>
>                 Key: BEAM-769
>                 URL: https://issues.apache.org/jira/browse/BEAM-769
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-spark
>    Affects Versions: Not applicable
>            Reporter: Daniel Halperin
>            Assignee: Amit Sela
>
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/1586/
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/1587/
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/1588/
> {code}
> org.apache.beam.runners.spark.translation.streaming.FlattenStreamingTest.testFlattenUnbounded
> org.apache.beam.runners.spark.translation.streaming.KafkaStreamingTest.testRun
> org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest.testFixedWindows
> {code}
> The above tests use a hard-timeout (ungraceful stop) so if the runtime env. is slow enough so that the batch is not done, it'll stop anyway and assert and rightfully fail.
> It's difficult to create locally because I never had trouble on my laptop.
> Since Jenkins will be slow from time to time, it is reasonable enough to have a more robust solution here :
> # don't use checkpoint (Spark) if not necessary - only really necessary for one test in {{KafkaStreamingTest}} and {{ResumeFromCheckpointStreamingTest}} I think.
> #  allow for graceful stop - will take longer for each test, but should allow the test to finish even if runtime env. is slow.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)