[jira] [Comment Edited] (SUREFIRE-1821) Broken junit report when parallel and rerunFailingTestsCount configureation is used

["Pavel Pustovoyt (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SUREFIRE-1821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17325120#comment-17325120 ] 

Pavel Pustovoyt edited comment on SUREFIRE-1821 at 4/19/21, 5:36 PM:
---------------------------------------------------------------------

[~tibordigana]

After debugging 3.0.0-M4 that's the sequence I see that make it fail:
 First run:
 1) First test run is completed and _TestSetRunListner.testSetCompleted_ is called
 2) _DeferredFileOutputStream_ is associated with this test run and because there is a lot of output _tempFile_ is created
 3) _StatelessXmlReporter_ writes one _WrappedReportEntry_ into xml results
 4) _DeferredFileOutputStream_ is freed and temporary file is deleted

Rerun:
 5) Rerun is completed and _TestSetRunListner.testSetCompleted_ is called again
 6) New DeferredFileOutputStream is created and associated with "rerun" _WrappedReportEntry_

7) _StatelessXmlReporter_ has 2 _WrappedReportEntry_ one from the original run and the second from the rerun and tries to uses them to generate new xml report

8) It tries to use _DeferredFileOutputStream_ from first test run entry but the file been freed/deleted after the first run
 9) We get _java.io.FileNotFoundException_

In our project we fixed it by releasing patched version where we mark __DeferredFileOutputStream.outputFile.deletOnExit().__
 In case of a parallel run this works fine since a fork exits pretty fast, but in a big project that runs sequentially on the same JVM it can create a lot of temporary files before JVM exits so not sure if its a good general solution.

I haven't checked the code of later version, but if temp file gets delete on test set complete event it should have the same problem. 

IMO the best solution could be having "allRerunsFinished" event for this test set and only then free resources. 


was (Author: himos):
[~tibordigana] 

After debugging 3.0.0-M4 that's the sequence I see that make it fail:
First run:
1) First test run is completed and _TestSetRunListner.testSetCompleted_ is called
2) _DeferredFileOutputStream_ is associated with this test run and because there is a lot of output _tempFile_ is created
3) _StatelessXmlReporter_ writes one _WrappedReportEntry_ into xml results
4) _DeferredFileOutputStream_ is freed and temporary file is deleted

Rerun:
5) Rerun is completed and _TestSetRunListner.testSetCompleted_ is called again
6) New DeferredFileOutputStream is created and associated with "rerun" _WrappedReportEntry_

7) _StatelessXmlReporter_ has 2 _WrappedReportEntry_ one from the original run and the second from the rerun and sees them to generate new xml report

8) It tries to use _DeferredFileOutputStream_ from first test run entry but the file been freed/deleted after the first run
9) We get _java.io.FileNotFoundException_

In our project we fixed it by releasing patched version where we mark __DeferredFileOutputStream.outputFile.deletOnExit().__
In case of a parallel run this works fine since a fork exits pretty fast, but in a big project that runs sequentially on the same JVM it can create a lot of temporary files before JVM exits so not sure if its a good general solution.

I haven't checked the code of later version, but if temp file gets delete on test set complete event it should have the same problem. 

IMO the best solution could be having "allRerunsFinished" event for this test set and only then free resources. 

> Broken junit report when parallel and rerunFailingTestsCount configureation is used
> -----------------------------------------------------------------------------------
>
>                 Key: SUREFIRE-1821
>                 URL: https://issues.apache.org/jira/browse/SUREFIRE-1821
>             Project: Maven Surefire
>          Issue Type: Bug
>          Components: Junit 4.7+ (parallel) support, Maven Failsafe Plugin, Maven Surefire Plugin
>    Affects Versions: 2.20.1, 2.22.2, 3.0.0-M5
>            Reporter: Pavel Pustovoyt
>            Priority: Major
>
> *Description:*
> When using *parallel* configuration with *rerunFailingTestsCount* bad xml report is generated for a failing test.
> *How to reproduce:*
> 1. Configure forkCount, parallel and rerun count:
> {code:java}
> <plugin>
>   <groupId>org.apache.maven.plugins</groupId>
>   <artifactId>maven-surefire-plugin</artifactId>
>   <version>3.0.0-M5</version>
>   <executions>
>     <execution>
>       <goals>
>         <goal>test</goal>
>       </goals>
>     </execution>
>   </executions>
>   <configuration>
>     <rerunFailingTestsCount>1</rerunFailingTestsCount>
>     <forkCount>1</forkCount>
>     <threadCount>1</threadCount>
>     <parallel>all</parallel>
>     <redirectTestOutputToFile>true</redirectTestOutputToFile>
>   </configuration>
> </plugin>
> {code}
> 2. Create a failing test that outputs more than 1m characters:
> {code:java}
> public class AppTest {
>     @Test
>     public void testBug() {
>         for(int i = 0;  i < 100000; i++){
>             System.out.println("Some output longer than 10 character");
>         }
>         throw new NullPointerException();
>     }
> }
> {code}
> 3. Run the test and check the report - you will see unexpected end of xml file:
> {code:java}
>     ...
>     <property name="sun.io.unicode.encoding" value="UnicodeBig"/>
>     <property name="java.class.version" value="52.0"/>
>   </properties>
>   <testcase name="testBug" classname="com.mycompany.app.AppTest" time="0.015">
>     <error type="java.lang.NullPointerException"><![CDATA[java.lang.NullPointerException
>         at com.mycompany.app.AppTest.testBug(AppTest.java:12)
> ]]></error>
>     <system-out><![CDATA[  <-- EOF
> {code}
> *Possible cause:*
> It seems that Junit 4.7 Runner considers each rerun as a separate test set, so sends _testSetCompleted_ event each time, but after the first event _DeferredFileOutputStream_ and associated with it temp file gets freed(deleted), however _testSetCompleted_ event for rerun still tries to write something there. If temp file isn't yet created (still in memory) then there will be no bug and it gets created only after 1m characters were written to the stream. I wasn't able to understand wether multiple _testSetCompleted_ events is a bug here or a "rerun" trying to write to _DeferredFileOutputStream_ of initial test run.
>  
> *Project to reproduce:*
> *[https://github.com/himos/surefire-report-bug]*



--
This message was sent by Atlassian Jira
(v8.3.4#803005)