You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@maven.apache.org by Slawomir Jaranowski <s....@gmail.com> on 2021/11/02 18:45:53 UTC
Surefire - multiple tests framework on classpath
Hi,
When multiple test frameworks are present on project classpath surefire
chooses one framework for running the tests.
In such situations, some tests are silently skipped.
I've created issue [1] for it with a simple proposition.
After discussion on slack another proposition was in place to detect the
test framework with other dependencies and choose a proper provider.
Or use multiple providers instead of only one.
It will be good to choose how to solve this problem.
[1] https://issues.apache.org/jira/browse/SUREFIRE-1954
--
Sławomir Jaranowski
https://twitter.com/SlawekJaran
https://github.com/slawekjaranowski
https://linkedin.com/in/slawomirjaranowski
Re: Surefire - multiple tests framework on classpath
Posted by Romain Manni-Bucau <rm...@gmail.com>.
Hi all,
to complete Sławomir description, the rational of Slack proposal to run it
instead of adding a new toggle to just fail is that the logic to detect the
described case is exactly the same so when we have detected that, either we
fail with the new toggle + ignore it by default or we try to run tests as
expected by the user since he put them here with the right dependencies -
at least it is the rational, run once we know instead of letting the user
tune and have a complex configuration.
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>
Le mar. 2 nov. 2021 à 19:46, Slawomir Jaranowski <s....@gmail.com> a
écrit :
> Hi,
>
> When multiple test frameworks are present on project classpath surefire
> chooses one framework for running the tests.
>
> In such situations, some tests are silently skipped.
>
>
> I've created issue [1] for it with a simple proposition.
>
> After discussion on slack another proposition was in place to detect the
> test framework with other dependencies and choose a proper provider.
>
> Or use multiple providers instead of only one.
>
>
> It will be good to choose how to solve this problem.
>
>
> [1] https://issues.apache.org/jira/browse/SUREFIRE-1954
>
> --
> Sławomir Jaranowski
>
> https://twitter.com/SlawekJaran
> https://github.com/slawekjaranowski
> https://linkedin.com/in/slawomirjaranowski
>
Re: Surefire - multiple tests framework on classpath
Posted by Tibor Digana <ti...@apache.org>.
Hi Slawomir,
Pls follow this page
https://maven.apache.org/surefire/maven-surefire-plugin/examples/junit-platform.html
There is a chapter "How to run multiple APIs or Engines". There are also
some links pointing to our integration test in the Apache project.
Pls have a look.
Cheers
Tibor
On Wed, Nov 3, 2021 at 6:22 PM Slawomir Jaranowski <s....@gmail.com>
wrote:
> Hi,
>
> I understand the difference between framework and providers.
> I also know that we can add specific providers as dependency to the plugin
> and in the result we have manually selected providers.
>
> When we don't configure any dependency the provider is chosen automatically
> depending on what we have on classpath, what framework.
>
> eg. When we have on classpath junit4 and junit5, but without junit5 vintage
> - junit4 tests are silently skipped.
>
> In most cases in the project we use one testing framework so
> automatically choosing one provider is ok.
>
> When we use more that one framework it is probably during migration, so we
> should define proper dependency.
> It is ok when we have control on what we do in a project, but there are
> some situations when we can make mistakes, like
> - user add second testing framework and forgot about configuration
> - we change project dependency and in transitive dependency we got next
> testing framework
>
> So I want to mitigate the risk of skipped tests by warning / failing that
> we have many frameworks and automatically provider selection takes place.
>
> We can also try to choose many providers (not one) automatically, but I
> think it will be more complicated and in standard usage is not needed.
> For automatic detection we have many corner cases, like testng + junit4 -
> testng by default runs junit4 tests, so without special configuration for
> testng we run some of the tests twice.
>
>
> śr., 3 lis 2021 o 15:09 Tibor Digana <ti...@apache.org> napisał(a):
>
> > Hi Slawomir,
> >
> > The reality is different.
> > It's not the frameworks but providers.
> > So the plugin iterates over providers.
> > You may choose whether you will define the provider artifacts in plugin
> > deps or you let the plugin to select the providers upon dependencies
> which
> > means that there is some logic and JUnit5 deps may activate one provider
> > for both the junit4 and 5.
> >
> > T
> >
> > On Tue, Nov 2, 2021 at 7:46 PM Slawomir Jaranowski <
> s.jaranowski@gmail.com
> > >
> > wrote:
> >
> > > Hi,
> > >
> > > When multiple test frameworks are present on project classpath surefire
> > > chooses one framework for running the tests.
> > >
> > > In such situations, some tests are silently skipped.
> > >
> > >
> > > I've created issue [1] for it with a simple proposition.
> > >
> > > After discussion on slack another proposition was in place to detect
> the
> > > test framework with other dependencies and choose a proper provider.
> > >
> > > Or use multiple providers instead of only one.
> > >
> > >
> > > It will be good to choose how to solve this problem.
> > >
> > >
> > > [1] https://issues.apache.org/jira/browse/SUREFIRE-1954
> > >
> > > --
> > > Sławomir Jaranowski
> > >
> > > https://twitter.com/SlawekJaran
> > > https://github.com/slawekjaranowski
> > > https://linkedin.com/in/slawomirjaranowski
> > >
> >
>
>
> --
> Sławomir Jaranowski
>
Re: Surefire - multiple tests framework on classpath
Posted by Tibor Digana <ti...@apache.org>.
Usually you will endup with one engine and one or more test APIs.
The documentation above solves all combinations with junit4, junit5 and
testng.
If somebody uses one @Test annotation and wants to say that f/w1 should run
test class 1 but f/w2 should run test class 2, then the plugin should be
configured with two executions within one phase default-test.
I don't think we should rewrap entire plugin config params per provider
since it can be solved with Maven's <executions/> in POM.
T
On Wed, Nov 3, 2021 at 6:22 PM Slawomir Jaranowski <s....@gmail.com>
wrote:
> Hi,
>
> I understand the difference between framework and providers.
> I also know that we can add specific providers as dependency to the plugin
> and in the result we have manually selected providers.
>
> When we don't configure any dependency the provider is chosen automatically
> depending on what we have on classpath, what framework.
>
> eg. When we have on classpath junit4 and junit5, but without junit5 vintage
> - junit4 tests are silently skipped.
>
> In most cases in the project we use one testing framework so
> automatically choosing one provider is ok.
>
> When we use more that one framework it is probably during migration, so we
> should define proper dependency.
> It is ok when we have control on what we do in a project, but there are
> some situations when we can make mistakes, like
> - user add second testing framework and forgot about configuration
> - we change project dependency and in transitive dependency we got next
> testing framework
>
> So I want to mitigate the risk of skipped tests by warning / failing that
> we have many frameworks and automatically provider selection takes place.
>
> We can also try to choose many providers (not one) automatically, but I
> think it will be more complicated and in standard usage is not needed.
> For automatic detection we have many corner cases, like testng + junit4 -
> testng by default runs junit4 tests, so without special configuration for
> testng we run some of the tests twice.
>
>
> śr., 3 lis 2021 o 15:09 Tibor Digana <ti...@apache.org> napisał(a):
>
> > Hi Slawomir,
> >
> > The reality is different.
> > It's not the frameworks but providers.
> > So the plugin iterates over providers.
> > You may choose whether you will define the provider artifacts in plugin
> > deps or you let the plugin to select the providers upon dependencies
> which
> > means that there is some logic and JUnit5 deps may activate one provider
> > for both the junit4 and 5.
> >
> > T
> >
> > On Tue, Nov 2, 2021 at 7:46 PM Slawomir Jaranowski <
> s.jaranowski@gmail.com
> > >
> > wrote:
> >
> > > Hi,
> > >
> > > When multiple test frameworks are present on project classpath surefire
> > > chooses one framework for running the tests.
> > >
> > > In such situations, some tests are silently skipped.
> > >
> > >
> > > I've created issue [1] for it with a simple proposition.
> > >
> > > After discussion on slack another proposition was in place to detect
> the
> > > test framework with other dependencies and choose a proper provider.
> > >
> > > Or use multiple providers instead of only one.
> > >
> > >
> > > It will be good to choose how to solve this problem.
> > >
> > >
> > > [1] https://issues.apache.org/jira/browse/SUREFIRE-1954
> > >
> > > --
> > > Sławomir Jaranowski
> > >
> > > https://twitter.com/SlawekJaran
> > > https://github.com/slawekjaranowski
> > > https://linkedin.com/in/slawomirjaranowski
> > >
> >
>
>
> --
> Sławomir Jaranowski
>
Re: Surefire - multiple tests framework on classpath
Posted by Slawomir Jaranowski <s....@gmail.com>.
Hi,
I understand the difference between framework and providers.
I also know that we can add specific providers as dependency to the plugin
and in the result we have manually selected providers.
When we don't configure any dependency the provider is chosen automatically
depending on what we have on classpath, what framework.
eg. When we have on classpath junit4 and junit5, but without junit5 vintage
- junit4 tests are silently skipped.
In most cases in the project we use one testing framework so
automatically choosing one provider is ok.
When we use more that one framework it is probably during migration, so we
should define proper dependency.
It is ok when we have control on what we do in a project, but there are
some situations when we can make mistakes, like
- user add second testing framework and forgot about configuration
- we change project dependency and in transitive dependency we got next
testing framework
So I want to mitigate the risk of skipped tests by warning / failing that
we have many frameworks and automatically provider selection takes place.
We can also try to choose many providers (not one) automatically, but I
think it will be more complicated and in standard usage is not needed.
For automatic detection we have many corner cases, like testng + junit4 -
testng by default runs junit4 tests, so without special configuration for
testng we run some of the tests twice.
śr., 3 lis 2021 o 15:09 Tibor Digana <ti...@apache.org> napisał(a):
> Hi Slawomir,
>
> The reality is different.
> It's not the frameworks but providers.
> So the plugin iterates over providers.
> You may choose whether you will define the provider artifacts in plugin
> deps or you let the plugin to select the providers upon dependencies which
> means that there is some logic and JUnit5 deps may activate one provider
> for both the junit4 and 5.
>
> T
>
> On Tue, Nov 2, 2021 at 7:46 PM Slawomir Jaranowski <s.jaranowski@gmail.com
> >
> wrote:
>
> > Hi,
> >
> > When multiple test frameworks are present on project classpath surefire
> > chooses one framework for running the tests.
> >
> > In such situations, some tests are silently skipped.
> >
> >
> > I've created issue [1] for it with a simple proposition.
> >
> > After discussion on slack another proposition was in place to detect the
> > test framework with other dependencies and choose a proper provider.
> >
> > Or use multiple providers instead of only one.
> >
> >
> > It will be good to choose how to solve this problem.
> >
> >
> > [1] https://issues.apache.org/jira/browse/SUREFIRE-1954
> >
> > --
> > Sławomir Jaranowski
> >
> > https://twitter.com/SlawekJaran
> > https://github.com/slawekjaranowski
> > https://linkedin.com/in/slawomirjaranowski
> >
>
--
Sławomir Jaranowski
Re: Surefire - multiple tests framework on classpath
Posted by Tibor Digana <ti...@apache.org>.
Hi Slawomir,
The reality is different.
It's not the frameworks but providers.
So the plugin iterates over providers.
You may choose whether you will define the provider artifacts in plugin
deps or you let the plugin to select the providers upon dependencies which
means that there is some logic and JUnit5 deps may activate one provider
for both the junit4 and 5.
T
On Tue, Nov 2, 2021 at 7:46 PM Slawomir Jaranowski <s....@gmail.com>
wrote:
> Hi,
>
> When multiple test frameworks are present on project classpath surefire
> chooses one framework for running the tests.
>
> In such situations, some tests are silently skipped.
>
>
> I've created issue [1] for it with a simple proposition.
>
> After discussion on slack another proposition was in place to detect the
> test framework with other dependencies and choose a proper provider.
>
> Or use multiple providers instead of only one.
>
>
> It will be good to choose how to solve this problem.
>
>
> [1] https://issues.apache.org/jira/browse/SUREFIRE-1954
>
> --
> Sławomir Jaranowski
>
> https://twitter.com/SlawekJaran
> https://github.com/slawekjaranowski
> https://linkedin.com/in/slawomirjaranowski
>