[jira] [Comment Edited] (SUREFIRE-1248) Surefire plugin includes configuration skips when reporting test method skips

["Tibor Digana (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SUREFIRE-1248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15303996#comment-15303996 ] 

Tibor Digana edited comment on SUREFIRE-1248 at 5/27/16 12:28 PM:
------------------------------------------------------------------

[~juherr]
I think the issue is with handling TestNG callbacks in surefire.
I do not know if we can fix this specific issue.
The problem is that a test is skipped once with group "skip" but run with "regression".
[~juherr] is this the issue?

So I registered interface org.testng.ITestListener and I have got the following callbacks.
How can we distinguish between these two variants regarding TestNG groups in Surefire?

Running forkMode.Test1
onStart org.testng.TestRunner@6b2fad11
[DEBUGGER] :Skipping verifyServiceIsUp test because [skip] was not available
onTestStart [TestResult name=verifyServiceIsUp status=STARTED method=Test1.verifyServiceIsUp()[pri:0, instance:forkMode.Test1@96532d6] output={null}]
onTestSkipped [TestResult name=verifyServiceIsUp status=SKIP method=Test1.verifyServiceIsUp()[pri:0, instance:forkMode.Test1@96532d6] output={null}]
onFinish org.testng.TestRunner@6b2fad11
Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 0.468 sec - in forkMode.Test1


was (Author: tibor17):
[~juherr]
I think the issue is with handling TestNG callbacks in surefire.
I do not know if we can fix this specific issue.
The problem is that a test is skipped once with group "skip" but run with "regression".
[~juherr] is this the issue?

So I registered interface org.testng.ITestListener and I have go the following callbacks.
How can we distinguish between these two variations in Surefire?

Running forkMode.Test1
onStart org.testng.TestRunner@6b2fad11
[DEBUGGER] :Skipping verifyServiceIsUp test because [skip] was not available
onTestStart [TestResult name=verifyServiceIsUp status=STARTED method=Test1.verifyServiceIsUp()[pri:0, instance:forkMode.Test1@96532d6] output={null}]
onTestSkipped [TestResult name=verifyServiceIsUp status=SKIP method=Test1.verifyServiceIsUp()[pri:0, instance:forkMode.Test1@96532d6] output={null}]
onFinish org.testng.TestRunner@6b2fad11
Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 0.468 sec - in forkMode.Test1

> Surefire plugin includes configuration skips when reporting test method skips 
> ------------------------------------------------------------------------------
>
>                 Key: SUREFIRE-1248
>                 URL: https://issues.apache.org/jira/browse/SUREFIRE-1248
>             Project: Maven Surefire
>          Issue Type: Bug
>          Components: Maven Surefire Plugin
>    Affects Versions: 2.19
>            Reporter: Krishnan Mahadevan
>            Assignee: Tibor Digana
>
> When a TestNG test execution has configuration skips, surefire plugin counts the configuration skips along with the test method skips and reports them incorrectly.
> Here's a sample test 
> {code}
> import org.testng.SkipException;
> import org.testng.annotations.BeforeMethod;
> import org.testng.annotations.Test;
> import java.lang.reflect.Method;
> import java.util.ArrayList;
> import java.util.Arrays;
> import java.util.Collections;
> import java.util.List;
> public class GroupGames {
>     List<String> groupsToBeSkippedList = new ArrayList<String>() {{
>         add("skip");
>     }};
>     @BeforeMethod (alwaysRun = true)
>     public void preCheck(Method method) {
>         List<String> testMethodGroups = Arrays.asList(method.getAnnotation(Test.class).groups());
>         if (! Collections.disjoint(groupsToBeSkippedList, testMethodGroups)) {
>             String message =
>                 "Skipping " + method.getName() + " test because " + groupsToBeSkippedList + " was not available";
>             System.err.println("[DEBUGGER] :" + message);
>             throw new SkipException(message);
>         }
>     }
>     @Test (groups = {"skip", "regression"})
>     public void verifyServiceIsUp() {
>         System.err.println("Hey this is a test method");
>     }
> }
> {code}
> Here's the output when the above test is executed in an IDE [ I am using IntelliJ]
> {code}
> [TestNG] Running:
>   /Users/krmahadevan/Library/Caches/IntelliJIdea2016.1/temp-testng-customsuite.xml
> [DEBUGGER] :Skipping verifyServiceIsUp test because [skip] was not available
> Test ignored.
> ===============================================
> Default Suite
> Total tests run: 1, Failures: 0, Skips: 1
> Configuration Failures: 0, Skips: 1
> ===============================================
> {code}
> Here's the output when the test is executed using Surefire plugin
> {code}
> [INFO] --- maven-surefire-plugin:2.19.1:test (default-test) @ testbed ---
> -------------------------------------------------------
>  T E S T S
> -------------------------------------------------------
> Running organized.chaos.forums.GroupGames
> [DEBUGGER] :Skipping verifyServiceIsUp test because [skip] was not available
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 0.562 sec - in organized.chaos.forums.GroupGames
> Results :
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 2
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 10.849 s
> [INFO] Finished at: 2016-05-18T08:22:25+05:30
> [INFO] Final Memory: 21M/155M
> [INFO] ------------------------------------------------------------------------
> {code}
> I am able to recreate this problem with 2.19.1



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)