[jira] Closed: (SUREFIRE-476) Skipped test should not be considered as "failed test"

["Dan Fabulich (JIRA)" <ji...@codehaus.org>]

     [ http://jira.codehaus.org/browse/SUREFIRE-476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dan Fabulich closed SUREFIRE-476.
---------------------------------

    Resolution: Won't Fix

I don't think this patch is logically correct.  Logically, a test can have three results: pass, fail or skip.  Unfortunately, most people assume that there are only two possible results (pass or fail); they furthermore assume that "fail" means the same thing as "did not pass."

I don't think we want an option to decide whether "skip" means "fail" or whether "skip" means "pass."  Skip is separate; it's not a success, and it's not a failure.

If a test is skipped, then it is not a success.  That doesn't mean it's a failure, but it also means that you don't have 100% success.  Perhaps what you'd want is a separate "failure percentage."  If all tests are skipped, the "pass" percentage is 0 and the failure percentage is 0.

Regardless, I'm pretty sure it's wrong to make the "successful percentage" be 100% when there are skipped tests, because those tests did not succeed.

> Skipped test should not be considered as "failed test"
> ------------------------------------------------------
>
>                 Key: SUREFIRE-476
>                 URL: http://jira.codehaus.org/browse/SUREFIRE-476
>             Project: Maven Surefire
>          Issue Type: Improvement
>          Components: report plugin
>    Affects Versions: 2.4.2
>            Reporter: Baptiste MATHUS
>         Attachments: surefire-report-skippedIsSuccess.patch
>
>
> I guess the plugin should allow people to decide whether skipped test is the same as a failed test, as it is at the moment in the report plugin.
> As you know, with the surefire plugin (not the report one), a skipped test is not considered as a failed one, as it doesn't make the whole test phase fail, isn't it? So, I guess there should be at least an option to choose how to treat skipped tests when reporting.
> As a small modification, I'm attaching a small patch to not lower the "success rate" (for example, with two tests one successful, one skipped, the success rate stays 100%). This patch uses an option -DskippedIsFailed=false that lets the current behaviour be if you don't use this switch (the testcase was also updated to test this new switch).
> I think there could be a good idea to introduce a specific part in the report showing the skipped tests. What do you think? If you agree, I could try and propose a patch for it.
> Cheers.
> PS: the patch was applied against the 2.4.2 tag. If you want me to update it against another source version, please let me know.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira