You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@maven.apache.org by jontodd <gi...@git.apache.org> on 2014/09/03 03:14:04 UTC
[GitHub] maven-surefire pull request: Support for TestNG successPercentage
GitHub user jontodd opened a pull request:
https://github.com/apache/maven-surefire/pull/47
Support for TestNG successPercentage
TestNG has the concept of "successPercentage" where a probabalistic test can be run some number of innvoations and then if the successPercentage is met, the test is considered passing.
Example code might look like:
```java
@Test(invocationCount = 100, successPercentage = 25)
void testRandom() {
// ~50% chance of generating 0 & 1
int num = new Random().nextInt(2);
Assert.assertEquals(num, 1);
}
```
As detailed in SUREFIRE-654, the problem is that surefire is currently mapping each invocation to a new test run, and any failure, regardless of whether we're within the allowed success percentage or not will result in this test failing. This renders any other value for successPercentage aside from 100 useless.
Digging into the code it looks like this was just an oversight in the inital implementation of the TestNGReporter. The TestNG ITestListener#onTestFailedButWithinSuccessPercentage method API stipulates that it will only be called when we're still within the passing limit:
Invoked each time a method fails but has been annotated with successPercentage and this failure still keeps it within the success percentage requested.
This change maps tests which pass within the successPercentage as passing which is safe because if we go beyond the passing threshold the test will be marked failed as desired.
__What about tests?__
Yes, there are no test changes. It looks like the providers aren't being tested. I'd be happy to add coverage that asserts the correct stubbed reporter method is called but that seems like overkill given the existing testing conventions.
__What about breaking existing users?__
As mentioned earlier, essentially successPercentage wasn't even usable within surefire before, so this should only provide net-new functionality to users. In the example above, before my change that test would fail every time and would result in ~50 failures. After this change the above example would pass so long as there are only 25 or fewer failures.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jontodd/maven-surefire testng_success_pct_support
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/maven-surefire/pull/47.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #47
----
commit a70bbb8f043f88ccdc1ec7db6810b813688562c2
Author: Jon Todd <jo...@gmail.com>
Date: 2014-09-03T00:29:08Z
Support for TestNG successPercentage
TestNG has the concept of "successPercentage" where a probabalistic test can be run some number of innvoations and then if the successPercentage is met, the test is considered passing.
Example code might look like:
```java
@Test(invocationCount = 100, successPercentage = 25)
void testRandom() {
// ~50% chance of generating 0 & 1
int num = new Random().nextInt(2);
Assert.assertEquals(num, 1);
}
```
As detailed in SUREFIRE-654, the problem is that surefire is currently mapping each invocation to a new test run, and any failure, regardless of whether we're within the allowed success percentage or not will result in this test failing. This renders any other value for successPercentage aside from 100 useless.
Digging into the code it looks like this was just an oversight in the inital implementation of the TestNGReporter. The TestNG ITestListener#onTestFailedButWithinSuccessPercentage method API stipulates that it will only be called when we're still within the passing limit:
Invoked each time a method fails but has been annotated with successPercentage and this failure still keeps it within the success percentage requested.
This change maps tests which pass within the successPercentage as passing which is safe because if we go beyond the passing threshold the test will be marked failed once as desired.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
[GitHub] maven-surefire pull request: Support for TestNG successPercentage
Posted by jontodd <gi...@git.apache.org>.
Github user jontodd closed the pull request at:
https://github.com/apache/maven-surefire/pull/47
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org