You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@maven.apache.org by "Cam Morris (Jira)" <ji...@apache.org> on 2020/06/07 07:04:00 UTC
[jira] [Commented] (SUREFIRE-1228) rerunFailingTestsCount + TestNG
+ @DataProvider = failed tests pass
[ https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17127532#comment-17127532 ]
Cam Morris commented on SUREFIRE-1228:
--------------------------------------
I just discovered that testng has the ability to retry tests and has for a looooong time. It's implemented by pointing a specific test to a retry "analyzer" really a retry policy. I suppose this could be implemented in surefire by pointing each test to a default retry analyzer. It might conflict with tests that already have a retry analyzer. Here is the reference: [https://testng.org/doc/documentation-main.html#rerunning]
> rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
> -------------------------------------------------------------------
>
> Key: SUREFIRE-1228
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1228
> Project: Maven Surefire
> Issue Type: Bug
> Components: TestNG support
> Affects Versions: 2.19.1
> Reporter: Cam Morris
> Assignee: Tibor Digana
> Priority: Major
> Attachments: TestRandomFail.java, pom.xml
>
>
> Surefire confuses tests run with a data-provider with rerun tests so only one of the data-provided tests need to pass.
> Steps to reproduce:
> # Create a testNG test with a data provider that passes with some of the data and fails with others
> {code:title=TestNG example|language=java}
> @Test(dataProvider = "succeed")
> public void fail(boolean succeed) {
> if (!succeed) {
> Assert.fail("nope");
> }
> }
> @DataProvider(name="succeed")
> public Object[][] arrayIndex() {
> return new Object[][]{ {true}, {false}};
> }
> {code}
> # Configure surefire to rerun failed tests
> {code:title=pom.xml example|language=xml}
> <?xml version="1.0" encoding="UTF-8"?>
> <project xmlns="http://maven.apache.org/POM/4.0.0"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
> <modelVersion>4.0.0</modelVersion>
> <groupId>cam.test</groupId>
> <artifactId>testng-retry</artifactId>
> <version>1.0-SNAPSHOT</version>
> <dependencies>
> <dependency>
> <groupId>org.testng</groupId>
> <artifactId>testng</artifactId>
> <version>6.9.10</version>
> <scope>test</scope>
> </dependency>
> </dependencies>
> <build>
> <plugins>
> <plugin>
> <groupId>org.apache.maven.plugins</groupId>
> <artifactId>maven-surefire-plugin</artifactId>
> <version>2.19.1</version>
> <configuration>
> <rerunFailingTestsCount>1</rerunFailingTestsCount>
> </configuration>
> </plugin>
> </plugins>
> </build>
> </project>
> {code}
> - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not pass.
> - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1" The rerun confuses test runs with different data for retry attempts. Just one of the data provider tests needs to succeed for the test to be deemed a success.
> {code}mvn test
> ...
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Running TestRandomFail
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<< FAILURE! - in TestRandomFail
> fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
> java.lang.AssertionError: nope
> at TestRandomFail.fail(TestRandomFail.java:14)
> Results :
> Flaked tests:
> TestRandomFail.fail(TestRandomFail)
> Run 1: PASS
> Run 2: TestRandomFail.fail:14 nope
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 2.251 s
> [INFO] Finished at: 2016-02-10T07:39:21-07:00
> [INFO] Final Memory: 9M/244M
> [INFO] ------------------------------------------------------------------------
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)