[jira] [Commented] (SUREFIRE-1963) Detecting multiple test-frameworks

["Robert James Oxspring (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SUREFIRE-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17497779#comment-17497779 ] 

Robert James Oxspring commented on SUREFIRE-1963:
-------------------------------------------------

Is it really healthy that the default behaviour will be to continue to silently skip some tests? That way the problem of "silently skipping" tests by default is entirely unresolved by this issue.

Many people won't know of the {{multipleFrameworks}} option, and with this default behaviour won't even know if they're suffering from it. I can understand that failing their build might be considered harsh and surprising, but I'd have thought it should be a warning at a minimum.

> Detecting multiple test-frameworks
> ----------------------------------
>
>                 Key: SUREFIRE-1963
>                 URL: https://issues.apache.org/jira/browse/SUREFIRE-1963
>             Project: Maven Surefire
>          Issue Type: New Feature
>            Reporter: Slawomir Jaranowski
>            Assignee: Slawomir Jaranowski
>            Priority: Major
>
> When multiple test frameworks are present on project classpath surefire chooses one framework for running test.
> During automatically provider discover some of type test can be *skipped* without any knowledge and information for developer.
> It is danger situation than developer can add *second* test framework to project and in the result some of test will be *skipped silently.*
> We can force usage one or more needed providers by adding those to plugin dependency and we can have control on executing test.
> h3. Proposition
>  - add options {{multipleFrameworks}} *empty* (not set) by default, with possible value *warn* or *fail*
> h3. Scenario 1 - default
> h4. when
>  - {{multipleFrameworks = *empty* - not set}}
>  - many tests framework on classpath
>  - no providers on plugin dependency
> h4. then
>  - current behavior - choose first provider without warning about skipped test
> h3. Scenario 2
> h4. when
>  - {{multipleFrameworks = *warn*}}
>  - many tests framework on classpath
>  - no providers on plugin dependency
> h4. then
>  - print warn about multiple providers
> h3. Scenario 3
> h4. when
>  - {{multipleFrameworks = *fail*}}
>  - many tests framework on classpath
>  - no providers on plugin dependency
> h4. then
>  - print error about multiple providers and break build
> h3. Scenario 4
> h4. when
>  - many tests framework on classpath
>  - selected providers on plugin dependency
> h4. then
>  - ignore {{multipleFrameworks}}
>  - use providers from dependency list, no warning



--
This message was sent by Atlassian Jira
(v8.20.1#820001)