You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@maven.apache.org by "Benoit Lagae (JIRA)" <ji...@codehaus.org> on 2015/02/06 10:30:18 UTC
[jira] (SUREFIRE-1113) Build does not fail when successPercentage
for @org.testng.annotations.Test() is not met
[ https://jira.codehaus.org/browse/SUREFIRE-1113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=362669#comment-362669 ]
Benoit Lagae commented on SUREFIRE-1113:
----------------------------------------
Some final follow-up: the fix by Vlad has been added to the master branch in TestNG, and releases from 6.8.17 onwards should return the correct behavior.
> Build does not fail when successPercentage for @org.testng.annotations.Test() is not met
> ----------------------------------------------------------------------------------------
>
> Key: SUREFIRE-1113
> URL: https://jira.codehaus.org/browse/SUREFIRE-1113
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Surefire Plugin, TestNG support
> Affects Versions: 2.18
> Environment: TestNG 6.8.8, maven-surefire-plugin 2.19-SNAPSHOT (53a40eef48ea)
> Apache Maven 3.2.3 (33f8c3e1027c3ddde99d3cdebad2656a31e8fdf4; 2014-08-11T22:58:10+02:00)
> Maven home: C:\Program Files\apache-maven-3.2.3
> Java version: 1.7.0_67, vendor: Oracle Corporation
> Java home: C:\Program Files\Java\jdk1.7.0_67\jre
> Default locale: en_US, platform encoding: Cp1252
> OS name: "windows 7", version: "6.1", arch: "amd64", family: "windows"
> Reporter: Benoit Lagae
> Assignee: Andreas Gudian
> Attachments: surefiretest.zip
>
>
> When a TestNG test with annotation methods invocationCount = x and successPercentage = y fails, this does not break the Maven build of the project. I have attached a minimal Maven project with only the pom and a single test class, which contains three dummy tests which should all fail:
> {code}
> @Test
> public void testFailsCorrectly() { fail("dummy"); }
> @Test(invocationCount = x)
> public void testCounterWorksAndTestFails() { fail("dummy"); }
> @Test(invocationCount = x, successPercentage = y)
> public void testShouldFailButDoesnt() { fail("dummy"); }
> {code}
> The first two comply, but the third one doesn't, as is implied in the name of the method. I created this project from scratch, i.e. not from an IDE: manual creation of folders, code and pom in Notepad++, ...
> I have attached both the project and the info I thought could be relevant from my test runs of the project ('diagnostics' folder). Below are the results of the individual method runs:
> {code}
> <test-method status="FAIL" signature="testCounterWorks()[pri:0, instance:blagae.StatisticsTest@575598a]" name="testCounterWorks"/>
> <test-method status="FAIL" signature="testCounterWorks()[pri:0, instance:blagae.StatisticsTest@575598a]" name="testCounterWorks"/>
> <test-method status="FAIL" signature="testCounterWorks()[pri:0, instance:blagae.StatisticsTest@575598a]" name="testCounterWorks"/>
> <test-method status="FAIL" signature="testCounterWorks()[pri:0, instance:blagae.StatisticsTest@575598a]" name="testCounterWorks"/>
> <test-method status="FAIL" signature="testFailsCorrectly()[pri:0, instance:blagae.StatisticsTest@575598a]" name="testFailsCorrectly"/>
> <test-method status="SUCCESS_PERCENTAGE_FAILURE" signature="testShouldFailButDoesnt()[pri:0, instance:blagae.StatisticsTest@575598a]" name="testShouldFailButDoesnt"/>
> <test-method status="SUCCESS_PERCENTAGE_FAILURE" signature="testShouldFailButDoesnt()[pri:0, instance:blagae.StatisticsTest@575598a]" name="testShouldFailButDoesnt"/>
> <test-method status="FAIL" signature="testShouldFailButDoesnt()[pri:0, instance:blagae.StatisticsTest@575598a]" name="testShouldFailButDoesnt"/>
> <test-method status="FAIL" signature="testShouldFailButDoesnt()[pri:0, instance:blagae.StatisticsTest@575598a]" name="testShouldFailButDoesnt"/>
> {code}
> If we comment out the other tests, and only make testShouldFailButDoesnt run, then the build succeeds while it shouldn't. See the results in surefiretest/diagnostics/only_faulty_test/console_output.txt
--
This message was sent by Atlassian JIRA
(v6.1.6#6162)