You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@maven.apache.org by "Krishnan Mahadevan (JIRA)" <ji...@apache.org> on 2016/05/18 02:58:12 UTC
[jira] [Updated] (SUREFIRE-1248) Surefire plugin includes
configuration skips when reporting test method skips
[ https://issues.apache.org/jira/browse/SUREFIRE-1248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Krishnan Mahadevan updated SUREFIRE-1248:
-----------------------------------------
Description:
When a TestNG test execution has configuration skips, surefire plugin counts the configuration skips along with the test method skips and reports them incorrectly.
Here's a sample test
{code}
import org.testng.SkipException;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;
import java.lang.reflect.Method;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
public class GroupGames {
List<String> groupsToBeSkippedList = new ArrayList<String>() {{
add("skip");
}};
@BeforeMethod (alwaysRun = true)
public void preCheck(Method method) {
List<String> testMethodGroups = Arrays.asList(method.getAnnotation(Test.class).groups());
if (! Collections.disjoint(groupsToBeSkippedList, testMethodGroups)) {
String message =
"Skipping " + method.getName() + " test because " + groupsToBeSkippedList + " was not available";
System.err.println("[DEBUGGER] :" + message);
throw new SkipException(message);
}
}
@Test (groups = {"skip", "regression"})
public void verifyServiceIsUp() {
System.err.println("Hey this is a test method");
}
}
{/code|
Here's the output when the above test is executed in an IDE [ I am using IntelliJ]
```
[TestNG] Running:
/Users/krmahadevan/Library/Caches/IntelliJIdea2016.1/temp-testng-customsuite.xml
[DEBUGGER] :Skipping verifyServiceIsUp test because [skip] was not available
Test ignored.
===============================================
Default Suite
Total tests run: 1, Failures: 0, Skips: 1
Configuration Failures: 0, Skips: 1
===============================================
```
Here's the output when the test is executed using Surefire plugin
```
[INFO] --- maven-surefire-plugin:2.19.1:test (default-test) @ testbed ---
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running organized.chaos.forums.GroupGames
[DEBUGGER] :Skipping verifyServiceIsUp test because [skip] was not available
Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 0.562 sec - in organized.chaos.forums.GroupGames
Results :
Tests run: 2, Failures: 0, Errors: 0, Skipped: 2
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 10.849 s
[INFO] Finished at: 2016-05-18T08:22:25+05:30
[INFO] Final Memory: 21M/155M
[INFO] ------------------------------------------------------------------------
```
I am able to recreate this problem with 2.19.1
was:
When a TestNG test execution has configuration skips, surefire plugin counts the configuration skips along with the test method skips and reports them incorrectly.
Here's a sample test
```
import org.testng.SkipException;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;
import java.lang.reflect.Method;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
public class GroupGames {
List<String> groupsToBeSkippedList = new ArrayList<String>() {{
add("skip");
}};
@BeforeMethod (alwaysRun = true)
public void preCheck(Method method) {
List<String> testMethodGroups = Arrays.asList(method.getAnnotation(Test.class).groups());
if (! Collections.disjoint(groupsToBeSkippedList, testMethodGroups)) {
String message =
"Skipping " + method.getName() + " test because " + groupsToBeSkippedList + " was not available";
System.err.println("[DEBUGGER] :" + message);
throw new SkipException(message);
}
}
@Test (groups = {"skip", "regression"})
public void verifyServiceIsUp() {
System.err.println("Hey this is a test method");
}
}
```
Here's the output when the above test is executed in an IDE [ I am using IntelliJ]
```
[TestNG] Running:
/Users/krmahadevan/Library/Caches/IntelliJIdea2016.1/temp-testng-customsuite.xml
[DEBUGGER] :Skipping verifyServiceIsUp test because [skip] was not available
Test ignored.
===============================================
Default Suite
Total tests run: 1, Failures: 0, Skips: 1
Configuration Failures: 0, Skips: 1
===============================================
```
Here's the output when the test is executed using Surefire plugin
```
[INFO] --- maven-surefire-plugin:2.19.1:test (default-test) @ testbed ---
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running organized.chaos.forums.GroupGames
[DEBUGGER] :Skipping verifyServiceIsUp test because [skip] was not available
Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 0.562 sec - in organized.chaos.forums.GroupGames
Results :
Tests run: 2, Failures: 0, Errors: 0, Skipped: 2
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 10.849 s
[INFO] Finished at: 2016-05-18T08:22:25+05:30
[INFO] Final Memory: 21M/155M
[INFO] ------------------------------------------------------------------------
```
I am able to recreate this problem with 2.19.1
> Surefire plugin includes configuration skips when reporting test method skips
> ------------------------------------------------------------------------------
>
> Key: SUREFIRE-1248
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1248
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Surefire Plugin
> Affects Versions: 2.19
> Reporter: Krishnan Mahadevan
>
> When a TestNG test execution has configuration skips, surefire plugin counts the configuration skips along with the test method skips and reports them incorrectly.
> Here's a sample test
> {code}
> import org.testng.SkipException;
> import org.testng.annotations.BeforeMethod;
> import org.testng.annotations.Test;
> import java.lang.reflect.Method;
> import java.util.ArrayList;
> import java.util.Arrays;
> import java.util.Collections;
> import java.util.List;
> public class GroupGames {
> List<String> groupsToBeSkippedList = new ArrayList<String>() {{
> add("skip");
> }};
> @BeforeMethod (alwaysRun = true)
> public void preCheck(Method method) {
> List<String> testMethodGroups = Arrays.asList(method.getAnnotation(Test.class).groups());
> if (! Collections.disjoint(groupsToBeSkippedList, testMethodGroups)) {
> String message =
> "Skipping " + method.getName() + " test because " + groupsToBeSkippedList + " was not available";
> System.err.println("[DEBUGGER] :" + message);
> throw new SkipException(message);
> }
> }
> @Test (groups = {"skip", "regression"})
> public void verifyServiceIsUp() {
> System.err.println("Hey this is a test method");
> }
> }
> {/code|
> Here's the output when the above test is executed in an IDE [ I am using IntelliJ]
> ```
> [TestNG] Running:
> /Users/krmahadevan/Library/Caches/IntelliJIdea2016.1/temp-testng-customsuite.xml
> [DEBUGGER] :Skipping verifyServiceIsUp test because [skip] was not available
> Test ignored.
> ===============================================
> Default Suite
> Total tests run: 1, Failures: 0, Skips: 1
> Configuration Failures: 0, Skips: 1
> ===============================================
> ```
> Here's the output when the test is executed using Surefire plugin
> ```
> [INFO] --- maven-surefire-plugin:2.19.1:test (default-test) @ testbed ---
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Running organized.chaos.forums.GroupGames
> [DEBUGGER] :Skipping verifyServiceIsUp test because [skip] was not available
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 0.562 sec - in organized.chaos.forums.GroupGames
> Results :
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 2
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 10.849 s
> [INFO] Finished at: 2016-05-18T08:22:25+05:30
> [INFO] Final Memory: 21M/155M
> [INFO] ------------------------------------------------------------------------
> ```
> I am able to recreate this problem with 2.19.1
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)