[jira] (SUREFIRE-654) Surefire does not recognize successful tests using TestNG with the invocationCount and successPercentage parameters on the @Test annotation

["Jon Todd (JIRA)" <ji...@codehaus.org>]

    [ https://jira.codehaus.org/browse/SUREFIRE-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=351698#comment-351698 ] 

Jon Todd commented on SUREFIRE-654:
-----------------------------------

I'm running into this issue as well and its quit frustrating. This makes it very cumbersome to test probabilistic code with surefire. This is an example I'd expect to pass: 

{code:java}
    @Test(invocationCount = 100, successPercentage = 25)
    void testRandom() {
        // ~50% chance of generating 0 & 1
        int num = new Random().nextInt(2);
        Assert.assertEquals(num, 1);
    }
{code}

Though there are no test failures, each of the test cases where this ends up with an assertion failure surefire counts this as an error and fails the run.

I'd be happy to code up a fix if the patch provided on this isn't the desired way to handle this but, as it's stands there's is very useful functionality in TestNG which we simply can't use via Surefire. 


> Surefire does not recognize successful tests using TestNG with the invocationCount and successPercentage parameters on the @Test annotation
> -------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SUREFIRE-654
>                 URL: https://jira.codehaus.org/browse/SUREFIRE-654
>             Project: Maven Surefire
>          Issue Type: Bug
>          Components: TestNG support
>    Affects Versions: 2.6
>         Environment: Apache Maven 2.2.1 (rdebian-4)
> Java version: 1.6.0-18
> Java home: /usr/lib/jvm/java-6-openjdk/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux" version: "2.6.35-22-generic" arch: "amd64" Family: "unix"
> Ubuntu 10.10
>            Reporter: Steve Stodola
>         Attachments: bugtest.tar.gz, surefire654.diff
>
>
> Surefire does not recognize successful tests using TestNG with the invocationCount and successPercentage parameters on the @Test annotation. BugTest.java in the attached file has an invocationCount = 100 with a successPercentage = 1. 1 test in 100 needs to pass for the test to be considered successful. However when running *{{mvn clean test}}* Surefire reports the tests as failed. The generated report shows the tests as passing. Surefire should be identifying this test as passing.



--
This message was sent by Atlassian JIRA
(v6.1.6#6162)