[GitHub] [spark] gengliangwang opened a new pull request #24373: [WIP][SPARK-27460][TESTS] Running slowest test suites in their own forked JVMs for higher parallelism

[GitBox <gi...@apache.org>]

gengliangwang opened a new pull request #24373: [WIP][SPARK-27460][TESTS] Running slowest test suites in their own forked JVMs for higher parallelism
URL: https://github.com/apache/spark/pull/24373
 
 
   ## What changes were proposed in this pull request?
   
   This patch modifies SparkBuild so that the largest / slowest test suites (or collections of suites) can run in their own forked JVMs, allowing them to be run in parallel with each other. This opt-in / whitelisting approach allows us to increase parallelism without having to fix a long-tail of flakiness / brittleness issues in tests which aren't performance bottlenecks.
   
   See comments in SparkBuild.scala for information on the details, including a summary of why we sometimes opt to run entire groups of tests in a single forked JVM .
   
   ## How was this patch tested?
   
   Unit test
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org