[jira] [Commented] (BEAM-2506) Consider bundling multiple ValidatesRunner tests into one pipeline

["Beam JIRA Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-2506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378711#comment-17378711 ] 

Beam JIRA Bot commented on BEAM-2506:
-------------------------------------

This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3.

Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean.


> Consider bundling multiple ValidatesRunner tests into one pipeline
> ------------------------------------------------------------------
>
>                 Key: BEAM-2506
>                 URL: https://issues.apache.org/jira/browse/BEAM-2506
>             Project: Beam
>          Issue Type: Improvement
>          Components: testing
>            Reporter: Eugene Kirpichov
>            Priority: P2
>              Labels: stale-P2
>
> Currently ValidatesRunner test suites run 1 pipeline per unit test. That's a lot of small pipelines, and consumes a lot of resources especially in case of a pretty heavyweight runner like Dataflow, so tests take a long time and can't be run in parallel due to quota issues, etc.
> [~jasonkuster] says he and [~davor] discussed that we could execute multiple unit tests in a single TestPipeline.
> This JIRA is to track that idea.
> To further develop it: in case of Java, we could create a custom JUnit Runner http://junit.org/junit4/javadoc/4.12/org/junit/runner/Runner.html that would apply all the transforms and PAsserts in unit tests to a single instance of TestPipeline (per class, rather than per method), and run the whole thing at the end. PAssert captures the source location of its application, so we could still report which particular test failed.
> This obviously has fewer isolation between unit test methods, cause they effectively run in parallel instead of in sequence, so things like per-method setup and teardown will no longer be applicable. There'll probably be other issues.
> Anyway, this seems doable and high-impact.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)