[jira] [Created] (SUREFIRE-1760) Unbalanced number of test executions in forks (forkCount>1) results in no performance gain for jupiter/JUnit5 tests

["Vlad Skarzhevskyy (Jira)" <ji...@apache.org>]

Vlad Skarzhevskyy created SUREFIRE-1760:
-------------------------------------------

             Summary: Unbalanced number of test executions in forks (forkCount>1) results in no performance gain for jupiter/JUnit5 tests
                 Key: SUREFIRE-1760
                 URL: https://issues.apache.org/jira/browse/SUREFIRE-1760
             Project: Maven Surefire
          Issue Type: Bug
          Components: JUnit 5.x support
            Reporter: Vlad Skarzhevskyy


Running JUnit5 tests using maven-surefire-plugin with  <reuseForks>true</reuseForks>  e.g. mvn -DforkCount=8
Results: number of tests in each fork is significantly unbalanced, *+some forks do not get tests at all+*..  
The end results is that on CI server the total test execution will take as long as longest fork and only 50% of the available resources are used. 
In our case we observed that our 3000 integration tests after migration to junit5 take 30 minutes insteads of 18 minutes as it was with junit 4.
The balance is restored when running with reuseForks=false  but this will cause additional overhead for complex integrations tests
     e.g. tests requiring additional setup , classpath scanning and so on..  
The results are similar with maven-surefire-plugin: 2.22.1, 2.22.2, 3.0.0-M4  
Maven 3.6.3 , openjdk version "1.8.0_242"  on Linux
 
BTW If junit-vintage-engine is used the same results observed for JUnit3 and JUnit4 tests.
 
Consider example of 100 tests  (10 classes 10 tests in each)
[https://github.com/skarzhevskyy/range-junit/tree/master/range-junit5-surefire-fork]
 
When running mvn -DforkCount=8  with extension that prints the number of tests executed in each fork getting following:
{quote}Tests completed  JVM Fork 8; total tests: 0; classes: 0
Tests completed  JVM Fork 4; total tests: 0; classes: 0
Tests completed  JVM Fork 1; total tests: 0; classes: 0
Tests completed  JVM Fork 6; total tests: 0; classes: 0
Tests completed  JVM Fork 3; total tests: 0; classes: 0
Tests completed  JVM Fork 2; total tests: 0; classes: 0
Tests completed  JVM Fork 7; total tests: 30; classes: 3
Tests completed  JVM Fork 5; total tests: 70; classes: 7{quote}
 
mvn -DforkCount=4
{quote}Tests completed  JVM Fork 2; total tests: 0; classes: 0
Tests completed  JVM Fork 3; total tests: 10; classes: 1
Tests completed  JVM Fork 4; total tests: 40; classes: 4
Tests completed  JVM Fork 1; total tests: 50; classes: 5{quote}
 
The distribution is also random,  results may vary between maven runs:
mvn -DforkCount=4
{quote}Tests completed  JVM Fork 4; total tests: 0; classes: 0
Tests completed  JVM Fork 2; total tests: 10; classes: 1
Tests completed  JVM Fork 1; total tests: 30; classes: 3
Tests completed  JVM Fork 3; total tests: 60; classes: 6{quote}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)