[jira] [Closed] (SUREFIRE-1760) Unbalanced number of test executions in forks (forkCount>1) results in no performance gain for jupiter/JUnit5 tests

["Tibor Digana (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SUREFIRE-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tibor Digana closed SUREFIRE-1760.
----------------------------------
      Assignee: Tibor Digana
    Resolution: Duplicate

> Unbalanced number of test executions in forks (forkCount>1) results in no performance gain for jupiter/JUnit5 tests
> -------------------------------------------------------------------------------------------------------------------
>
>                 Key: SUREFIRE-1760
>                 URL: https://issues.apache.org/jira/browse/SUREFIRE-1760
>             Project: Maven Surefire
>          Issue Type: Bug
>          Components: JUnit 5.x support
>    Affects Versions: 2.22.1, 2.22.2, 3.0.0-M4
>            Reporter: Vlad Skarzhevskyy
>            Assignee: Tibor Digana
>            Priority: Major
>
> Running JUnit5 tests using maven-surefire-plugin with  <reuseForks>true</reuseForks>  e.g. mvn -DforkCount=8
> Results: number of tests in each fork is significantly unbalanced, *+some forks do not get tests at all+*..  
> The end results is that on CI server the total test execution will take as long as longest fork and only 50% of the available resources are used. 
> In our case we observed that our 3000 integration tests after migration to junit5 take 30 minutes insteads of 18 minutes as it was with junit 4.
> The balance is restored when running with reuseForks=false  but this will cause additional overhead for complex integrations tests
>       e.g. tests requiring additional setup , classpath scanning and so on..  
> The results are similar with maven-surefire-plugin: 2.22.1, 2.22.2, 3.0.0-M4  
>  Maven 3.6.3 , openjdk version "1.8.0_242"  on Linux
>   
>  BTW If junit-vintage-engine is used the same results observed for JUnit3 and JUnit4 tests.
>   
>  Consider example of 100 tests  (10 classes 10 tests in each)
>  [https://github.com/skarzhevskyy/range-junit/tree/master/range-junit5-surefire-fork]
>   
>  When running mvn -DforkCount=8  with extension that prints the number of tests executed in each fork getting following:
> {quote}Tests completed  JVM Fork 8; total tests: 0; classes: 0
>  Tests completed  JVM Fork 4; total tests: 0; classes: 0
>  Tests completed  JVM Fork 1; total tests: 0; classes: 0
>  Tests completed  JVM Fork 6; total tests: 0; classes: 0
>  Tests completed  JVM Fork 3; total tests: 0; classes: 0
>  Tests completed  JVM Fork 2; total tests: 0; classes: 0
>  Tests completed  JVM Fork 7; total tests: 30; classes: 3
>  Tests completed  JVM Fork 5; total tests: 70; classes: 7
> {quote}
>  
>  mvn -DforkCount=4
> {quote}Tests completed  JVM Fork 2; total tests: 0; classes: 0
>  Tests completed  JVM Fork 3; total tests: 10; classes: 1
>  Tests completed  JVM Fork 4; total tests: 40; classes: 4
>  Tests completed  JVM Fork 1; total tests: 50; classes: 5
> {quote}
>  
>  The distribution is also random,  results may vary between maven runs:
>  mvn -DforkCount=4
> {quote}Tests completed  JVM Fork 4; total tests: 0; classes: 0
>  Tests completed  JVM Fork 2; total tests: 10; classes: 1
>  Tests completed  JVM Fork 1; total tests: 30; classes: 3
>  Tests completed  JVM Fork 3; total tests: 60; classes: 6
> {quote}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)