[GitHub] [maven-surefire] jon-bell opened a new pull request #253: Fixes [SUREFIRE-1516]: Poor performance in reuseForks=false

[GitBox <gi...@apache.org>]

jon-bell opened a new pull request #253: Fixes [SUREFIRE-1516]: Poor performance in reuseForks=false
URL: https://github.com/apache/maven-surefire/pull/253
 
 
   Hi,
   This PR resolves the performance bug noted in [SUREFIRE-1516](https://issues.apache.org/jira/browse/SUREFIRE-1516), which appears when using the `reuseForks=false` configuration. The root-cause of the observed performance problem comes from forked JVM teardown time.
   
   The issue is that the forked JVM should not block reading IO to read more from the host JVM after it sends BYE_ACK. Threads blocking on `read` may not be interruptable until they poll for interrupts (every 350msec for stdin), which can introduce significant latency for short-lived processes, such as surefire forked JVMs which are running just a single test at a time. This 350msec overhead can add up on projects that have thousands of test classes, each of which might take only several hundred msec to run.
   
   To measure the scope of the problem and confirm its resolution, I created a simple benchmark test suite, which consists of 100 JUnit test classes, each with a single test that calls `Thread.sleep(250)`. I instrumented the JVM to record the time that each JVM starts, the time that the test starts (as measured by JUnit), the time that the test ends (as measured by JUnit), and the time that the JVM terminates. For comparison, I also did the same experiment with ant and gradle.
   
   The table below shows the results, which represent the average time for each test (over the 100 samples):
   
   **Configuration**|**Time to start forked JVM**|**Time to run test**|**Time to tear down forked JVM**
   :-----:|:-----:|:-----:|:-----:
   ant 1.10.6|250.42|252.81|8.75
   gradle 5.6.1|394.91|253.12|16.9
   mvn (b97df5a)|250.21|252.59|358.59
   mvn (2fbe44f)|216.66|252.32|16.9
   
   Overall, most build systems took similar amounts of time to spin up the JVM, and all took the expected 250ms to run each test. You can see that the current `master` version of surefire (b97df5a) takes an unusually high amount of time to tear down the forked JVM (in fact, 350 msec more, which is exactly the time for the JVM to interrupt a thread reading from `stdin` [explained in this fantastic Stack Overflow post]((https://stackoverflow.com/questions/48951611/blocking-on-stdin-makes-java-process-take-350ms-more-to-exit))). This is an easy fix though: after receiving the `BYE_ACK` message, the forked JVM can stop reading commands from the main surefire process, since it's shutting down. After implementing this fix, the overhead goes away (as shown in 2fbe44f).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services