[jira] [Commented] (SPARK-3431) Parallelize Scala/Java test execution

["Nicholas Chammas (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270324#comment-14270324 ] 

Nicholas Chammas commented on SPARK-3431:
-----------------------------------------

Generic update:
* For those not following [the open PR|https://github.com/apache/spark/pull/3564], I got through the problems with sbt not passing configs properly to forked JVMs, and [~joshrosen] has been methodically updating unit tests that don't play nice when run in parallel. Only a few tests remain.
* [~srowen] has been looking into using Maven + Surefire and has fixed up our execution of Java unit tests.
* I've created a separate issue, [SPARK-5161], to track parallelizing the Python unit tests. This issue here will focus on Scala/Java test execution only.

> Parallelize Scala/Java test execution
> -------------------------------------
>
>                 Key: SPARK-3431
>                 URL: https://issues.apache.org/jira/browse/SPARK-3431
>             Project: Spark
>          Issue Type: Improvement
>          Components: Build
>            Reporter: Nicholas Chammas
>            Assignee: Nicholas Chammas
>         Attachments: SPARK-3431-srowen-attempt.patch
>
>
> Running all the tests in {{dev/run-tests}} takes up to 2 hours. A common strategy to cut test time down is to parallelize the execution of the tests. Doing that may in turn require some prerequisite changes to be made to how certain tests run.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org