You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@maven.apache.org by "Debraj Manna (Jira)" <ji...@apache.org> on 2023/09/14 11:26:00 UTC
[jira] [Updated] (SUREFIRE-2195) Surefire reports are not correct when running tests in parallel using JUnit5
[ https://issues.apache.org/jira/browse/SUREFIRE-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Debraj Manna updated SUREFIRE-2195:
-----------------------------------
Description:
The issue has been discussed in the [Maven mailing list|https://lists.apache.org/thread/6k10kmxzz75rz8mqn3pl1ltvfltw87m9]. I am creating a bug here as suggested in the mailing list
Cross-posting it here for completeness
My {{junit-platform.properties}} looks like below
{code:java}
junit.jupiter.execution.parallel.enabled=true
junit.jupiter.execution.parallel.config.strategy=fixed
junit.jupiter.execution.parallel.config.fixed.parallelism=8
junit.jupiter.execution.parallel.mode.default=same_thread
junit.jupiter.execution.parallel.mode.classes.default=concurrent
{code}
I am observing the surefire reports that are getting generated are not correct
I have a test class named {{com.spotnana.servicetests.backoffice.invoicing.AirInvoiceServiceTest}} contains just a single test, {{testAirTicketing()}}.
But in the report file, TEST-com.spotnana.servicetests.backoffice.invoicing.AirInvoiceServiceTest.xml.
I see like below
{code:xml}
<testsuite xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report-3.0.xsd" version="3.0" name="com.spotnana.servicetests.backoffice.invoicing.AirInvoiceServiceTest" time="44.711" tests="2" errors="1" skipped="0" failures="0”>
…
<testcase name="testCrossPnrRefund" classname="com.spotnana.servicetests.analytics.ingestion.pnr.AirServiceTest" time="5.789"/>
<testcase name="testAirTicketing" classname="com.spotnana.servicetests.backoffice.invoicing.AirInvoiceServiceTest" time="33.331”>
{code}
It seems to be picking tests from some other class like {{AirServiceTest}} as in this case.
The console logs are also not correct. Please check the attached screenshot. In the attached screenshot of the console logs, it is saying {{Test Run 5}} but the {{AirInvoiceServiceTest}} contains only 1 test. And also the report shows random data. For example, sometimes it says {{Test Run 5}} , sometimes it says {{Test Run 2}}, etc.
I am observing it as part of a big project consistently with every run. I am trying to reproduce it in a small example. If I am able to do I will share it. In the mean time if some debug logs are needed, let me know I will be happy to share them.
Junit5 Version - 5.9.3
I am observing the issue with Junit5 Version 5.10.0 also. I tried reducing the {{junit.jupiter.execution.parallel.config.fixed.parallelism}} to 3 but there also I am getting the same problem.
was:
The issue has been discussed in the [Maven mailing list|https://lists.apache.org/thread/6k10kmxzz75rz8mqn3pl1ltvfltw87m9]. I am creating a bug here as suggested in the mailing list
Cross-posting it here for completeness
My {{junit-platform.properties}} looks like below
{code:java}
junit.jupiter.execution.parallel.enabled=true
junit.jupiter.execution.parallel.config.strategy=fixed
junit.jupiter.execution.parallel.config.fixed.parallelism=8
junit.jupiter.execution.parallel.mode.default=same_thread
junit.jupiter.execution.parallel.mode.classes.default=concurrent
{code}
I am observing the surefire reports that are getting generated are not correct
I have a test class named {{com.spotnana.servicetests.backoffice.invoicing.AirInvoiceServiceTest}} contains just a single test, {{testAirTicketing()}}.
But in the report file, TEST-com.spotnana.servicetests.backoffice.invoicing.AirInvoiceServiceTest.xml.
I see like below
{code:xml}
<testsuite xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report-3.0.xsd" version="3.0" name="com.spotnana.servicetests.backoffice.invoicing.AirInvoiceServiceTest" time="44.711" tests="2" errors="1" skipped="0" failures="0”>
…
<testcase name="testCrossPnrRefund" classname="com.spotnana.servicetests.analytics.ingestion.pnr.AirServiceTest" time="5.789"/>
<testcase name="testAirTicketing" classname="com.spotnana.servicetests.backoffice.invoicing.AirInvoiceServiceTest" time="33.331”>
{code}
It seems to be picking tests from some other class like {{AirServiceTest}} as in this case.
The console logs are also not correct. Please check the attached screenshot. In the attached screenshot of the console logs, it is saying {{Test Run 5}} but the {{AirInvoiceServiceTest}} contains only 1 test. And also the report shows random data. For example, sometimes it says {{Test Run 5}} , sometimes it says {{Test Run 2}}, etc.
I am observing it as part of a big project consistently with every run. I am trying to reproduce it in a small example. If I am able to do I will share it. In the mean time if some debug logs are needed, let me know I will be happy to share them.
Junit5 Version - 5.9.3
I am observing the issue with Junit5 Version 5.10.0 also. I tried reducing the junit.jupiter.execution.parallel.config.fixed.parallelism to 3 but there also I am getting the same problem.
> Surefire reports are not correct when running tests in parallel using JUnit5
> ----------------------------------------------------------------------------
>
> Key: SUREFIRE-2195
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2195
> Project: Maven Surefire
> Issue Type: Bug
> Components: JUnit 5.x support
> Affects Versions: 3.0.0-M7, 3.1.2
> Reporter: Debraj Manna
> Priority: Major
> Attachments: image.png
>
>
> The issue has been discussed in the [Maven mailing list|https://lists.apache.org/thread/6k10kmxzz75rz8mqn3pl1ltvfltw87m9]. I am creating a bug here as suggested in the mailing list
> Cross-posting it here for completeness
> My {{junit-platform.properties}} looks like below
> {code:java}
> junit.jupiter.execution.parallel.enabled=true
> junit.jupiter.execution.parallel.config.strategy=fixed
> junit.jupiter.execution.parallel.config.fixed.parallelism=8
> junit.jupiter.execution.parallel.mode.default=same_thread
> junit.jupiter.execution.parallel.mode.classes.default=concurrent
> {code}
> I am observing the surefire reports that are getting generated are not correct
> I have a test class named {{com.spotnana.servicetests.backoffice.invoicing.AirInvoiceServiceTest}} contains just a single test, {{testAirTicketing()}}.
> But in the report file, TEST-com.spotnana.servicetests.backoffice.invoicing.AirInvoiceServiceTest.xml.
> I see like below
> {code:xml}
> <testsuite xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report-3.0.xsd" version="3.0" name="com.spotnana.servicetests.backoffice.invoicing.AirInvoiceServiceTest" time="44.711" tests="2" errors="1" skipped="0" failures="0”>
> …
> <testcase name="testCrossPnrRefund" classname="com.spotnana.servicetests.analytics.ingestion.pnr.AirServiceTest" time="5.789"/>
> <testcase name="testAirTicketing" classname="com.spotnana.servicetests.backoffice.invoicing.AirInvoiceServiceTest" time="33.331”>
> {code}
> It seems to be picking tests from some other class like {{AirServiceTest}} as in this case.
> The console logs are also not correct. Please check the attached screenshot. In the attached screenshot of the console logs, it is saying {{Test Run 5}} but the {{AirInvoiceServiceTest}} contains only 1 test. And also the report shows random data. For example, sometimes it says {{Test Run 5}} , sometimes it says {{Test Run 2}}, etc.
> I am observing it as part of a big project consistently with every run. I am trying to reproduce it in a small example. If I am able to do I will share it. In the mean time if some debug logs are needed, let me know I will be happy to share them.
> Junit5 Version - 5.9.3
> I am observing the issue with Junit5 Version 5.10.0 also. I tried reducing the {{junit.jupiter.execution.parallel.config.fixed.parallelism}} to 3 but there also I am getting the same problem.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)