[jira] [Created] (SUREFIRE-1248) Surefire plugin includes configuration skips when reporting test method skips

["Krishnan Mahadevan (JIRA)" <ji...@apache.org>]

Krishnan Mahadevan created SUREFIRE-1248:
--------------------------------------------

             Summary: Surefire plugin includes configuration skips when reporting test method skips 
                 Key: SUREFIRE-1248
                 URL: https://issues.apache.org/jira/browse/SUREFIRE-1248
             Project: Maven Surefire
          Issue Type: Bug
          Components: Maven Surefire Plugin
    Affects Versions: 2.19
            Reporter: Krishnan Mahadevan


When a TestNG test execution has configuration skips, surefire plugin counts the configuration skips along with the test method skips and reports them incorrectly.

Here's a sample test 

```
import org.testng.SkipException;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;

import java.lang.reflect.Method;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;

public class GroupGames {
    List<String> groupsToBeSkippedList = new ArrayList<String>() {{
        add("skip");
    }};

    @BeforeMethod (alwaysRun = true)
    public void preCheck(Method method) {
        List<String> testMethodGroups = Arrays.asList(method.getAnnotation(Test.class).groups());
        if (! Collections.disjoint(groupsToBeSkippedList, testMethodGroups)) {
            String message =
                "Skipping " + method.getName() + " test because " + groupsToBeSkippedList + " was not available";
            System.err.println("[DEBUGGER] :" + message);
            throw new SkipException(message);
        }
    }


    @Test (groups = {"skip", "regression"})
    public void verifyServiceIsUp() {
        System.err.println("Hey this is a test method");
    }

}
```

Here's the output when the above test is executed in an IDE [ I am using IntelliJ]

```
[TestNG] Running:
  /Users/krmahadevan/Library/Caches/IntelliJIdea2016.1/temp-testng-customsuite.xml

[DEBUGGER] :Skipping verifyServiceIsUp test because [skip] was not available

Test ignored.
===============================================
Default Suite
Total tests run: 1, Failures: 0, Skips: 1
Configuration Failures: 0, Skips: 1
===============================================
```

Here's the output when the test is executed using Surefire plugin
```
[INFO] --- maven-surefire-plugin:2.19.1:test (default-test) @ testbed ---

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running organized.chaos.forums.GroupGames
[DEBUGGER] :Skipping verifyServiceIsUp test because [skip] was not available
Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 0.562 sec - in organized.chaos.forums.GroupGames

Results :

Tests run: 2, Failures: 0, Errors: 0, Skipped: 2

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 10.849 s
[INFO] Finished at: 2016-05-18T08:22:25+05:30
[INFO] Final Memory: 21M/155M
[INFO] ------------------------------------------------------------------------
```

I am able to recreate this problem with 2.19.1




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)