You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@maven.apache.org by Emond Papegaaij <em...@gmail.com> on 2021/09/10 10:41:41 UTC
JUnit Platform, forkCount > 1 and the TestPlan
Hi all,
First of all, sorry for the lengthy post. I decided to add some context to
explain things a bit, but it resulted in quite a long e-mail. For the past
few weeks I've been trying to come up with a solution for the issue I
filled under SUREFIRE-1935, but I'm getting stuck and starting to feel like
the issue cannot be solved with the current JUnit Platform API. To give
some perspective into why this issue is important for us, I first have to
explain a bit about our setup.
We write our tests in the Spock framework and use Arquillian to run the
tests in the application container. Some of our tests, especially the
Selenium based tests, require quite some setup and tear down. Prior to
starting the test, Arquillian cube builds and starts several docker
containers and the tests are run against these containers. We currently use
the JUnit 4 based Spock 1.3 with Groovy 2.5, but we like to upgrade to
Spock 2 and Groovy 3, which runs on top of the JUnit Platform. For this,
I've already started working on a Spock extension that integrates Spock 2
in the Arquillian test life cycle [1]. This extension is inspired by the
(currently Alpha) JUnit5 module for Arquillian [2]. Both use a global
registration to keep track of the state managed by Arquilllian. This will
end up somewhere at the root of the TestPlan (for example see [3]).
Because our tests are quite extensive, with a total run time of 6 hours, we
run them with a forkCount of 8, greatly reducing the total duration.
However, this is where SUREFIRE-1935 comes into play. With a forkCount > 1,
the entire test life cycle is started over and over again for every test
class. This happens in JUnitPlatformProvider at line 197 [4]. This results
in the entire Arquillian suite being torn down and setup for every class,
in our case adding several minutes to the execution of every test class
because the docker setup is done over and over again.
To overcome this issue, the state from one test class execution has to be
carried to the next. It seems the LauncherSession (introduced in JUnit 5.8)
is meant to close this gap. However, this would mean my extension would
also need to implement a LauncherSessionListener, and I'm not sure if
extensions are supposed to integrate with the launcher as well. Also, for
this surefire would need to start a session prior to the tests, and close
it when done. I think this is a good idea anyway when running on platform
1.8 or higher.
Another solution could be a (sort of) dynamic test that produces the tests
to be run one by one. However, here my knowledge of JUnit really falls
short. I've got no idea of this is even possible.
I hope someone can help me out on this one and point me in the right
direction, as we like to upgrade our test frameworks and this is blocking
us at the moment.
Best regards,
Emond Papegaaij
[1] Arquillian extension for Spock Framework 2:
https://github.com/topicusonderwijs/arquillian-testrunner-spock/tree/spock-2.0-junit5
[2] Arquillian module for JUnit 5:
https://github.com/arquillian/arquillian-core/tree/master/junit5
[3] Registration in root store:
https://github.com/arquillian/arquillian-core/blob/master/junit5/core/src/main/java/org/jboss/arquillian/junit5/JUnitJupiterTestClassLifecycleManager.java#L20
[4] JUnitPlatformProvider launching the tests:
https://github.com/apache/maven-surefire/blob/master/surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/JUnitPlatformProvider.java#L197
Re: JUnit Platform, forkCount > 1 and the TestPlan
Posted by Emond Papegaaij <em...@gmail.com>.
I've migrated the Arquillian Spock Extension to use a
SessionLauncherListener (see the latest commits at
https://github.com/topicusonderwijs/arquillian-testrunner-spock/tree/spock-2.0-junit5
). This now works perfectly with the updated maven-surefire-plugin.
Arquillian is now only started once per fork.
I've submitted the PR against surefire:
https://github.com/apache/maven-surefire/pull/389
Best regards,
Emond
On Sun, Sep 12, 2021 at 9:14 PM Emond Papegaaij <em...@gmail.com>
wrote:
> Hi Tibor,
>
> I've reworked the commit to keep the JUnit version at 1.3 and use
> reflection to get the session. Also, I've fixed the issue with the session
> used for scanning tests in main thread that remained open. The formatting
> is now in line with the checkstyle rules and all tests pass.
>
> https://github.com/topicusonderwijs/maven-surefire/commit/f30888922b8030ea039980aff413ed5282dca819
>
> Best regards,
> Emond
>
> On Sat, Sep 11, 2021 at 12:25 PM Tibor Digana <ti...@apache.org>
> wrote:
>
>> I do not want to stick to the latest version 1.8.
>> We have to stick to version 1.3 as it is right now.
>> We do not have a critical issue which could not be solved by reflection.
>> This reflection call, I proposed in GH, is made only once, no performance
>> penalty.
>>
>> T
>>
>> On Sat, Sep 11, 2021 at 7:52 AM Marc Philipp <ma...@gradle.com> wrote:
>>
>> > Hi Emond and Tibor,
>> >
>> > I’m glad you discovered the new LauncherSession API which was added for
>> > this purpose. The JUnit 5.8 GA release will come in the next few days.
>> >
>> > As you mentrioned, the official documentation does not (yet!) do a good
>> > job of explaining its intended use case:
>> >
>> >
>> https://junit.org/junit5/docs/5.8.0-RC1/user-guide/#launcher-api-launcher-session-listeners-custom
>> >
>> > Here’s a more complete example that I wrote for Gradle’s test
>> distribution
>> > plugin:
>> >
>> >
>> https://docs.gradle.com/enterprise/test-distribution-gradle-plugin/#junit_5_8_and_later
>> >
>> > It demonstrates how to initialize a “fixture” only once for an entire
>> > session and only if tests are actually going to be executed not just
>> > discovered. I’ll make sure to update the official JUnit docs to include
>> a
>> > similar example before the release.
>> >
>> > To only differentiate between the different versions of JUnit in one
>> place
>> > and stay backwards compatible, you could create an adapter class like
>> this:
>> >
>> > public class BackwardsCompatibleLauncherSession implements
>> AutoCloseable {
>> >
>> > public static BackwardsCompatibleLauncherSession open() {
>> > try {
>> > LauncherSession launcherSession =
>> > LauncherFactory.openSession();
>> > return new
>> > BackwardsCompatibleLauncherSession(launcherSession.getLauncher(),
>> > launcherSession::close);
>> > } catch (NoSuchMethodError ignore) {
>> > // JUnit Platform version on test classpath does not yet
>> > support launcher sessions
>> > return new
>> > BackwardsCompatibleLauncherSession(LauncherFactory.create(), () -> {});
>> > }
>> > }
>> >
>> > private final Launcher launcher;
>> > private final Runnable onClose;
>> >
>> > private BackwardsCompatibleLauncherSession(Launcher launcher,
>> Runnable
>> > onClose) {
>> > this.launcher = launcher;
>> > this.onClose = onClose;
>> > }
>> >
>> > Launcher getLauncher() {
>> > return launcher;
>> > }
>> >
>> > @Override
>> > public void close() {
>> > onClose.run();
>> > }
>> > }
>> >
>> > Cheers,
>> >
>> > Marc
>> >
>>
>
Re: JUnit Platform, forkCount > 1 and the TestPlan
Posted by Emond Papegaaij <em...@gmail.com>.
Hi Tibor,
I've reworked the commit to keep the JUnit version at 1.3 and use
reflection to get the session. Also, I've fixed the issue with the session
used for scanning tests in main thread that remained open. The formatting
is now in line with the checkstyle rules and all tests pass.
https://github.com/topicusonderwijs/maven-surefire/commit/f30888922b8030ea039980aff413ed5282dca819
Best regards,
Emond
On Sat, Sep 11, 2021 at 12:25 PM Tibor Digana <ti...@apache.org>
wrote:
> I do not want to stick to the latest version 1.8.
> We have to stick to version 1.3 as it is right now.
> We do not have a critical issue which could not be solved by reflection.
> This reflection call, I proposed in GH, is made only once, no performance
> penalty.
>
> T
>
> On Sat, Sep 11, 2021 at 7:52 AM Marc Philipp <ma...@gradle.com> wrote:
>
> > Hi Emond and Tibor,
> >
> > I’m glad you discovered the new LauncherSession API which was added for
> > this purpose. The JUnit 5.8 GA release will come in the next few days.
> >
> > As you mentrioned, the official documentation does not (yet!) do a good
> > job of explaining its intended use case:
> >
> >
> https://junit.org/junit5/docs/5.8.0-RC1/user-guide/#launcher-api-launcher-session-listeners-custom
> >
> > Here’s a more complete example that I wrote for Gradle’s test
> distribution
> > plugin:
> >
> >
> https://docs.gradle.com/enterprise/test-distribution-gradle-plugin/#junit_5_8_and_later
> >
> > It demonstrates how to initialize a “fixture” only once for an entire
> > session and only if tests are actually going to be executed not just
> > discovered. I’ll make sure to update the official JUnit docs to include a
> > similar example before the release.
> >
> > To only differentiate between the different versions of JUnit in one
> place
> > and stay backwards compatible, you could create an adapter class like
> this:
> >
> > public class BackwardsCompatibleLauncherSession implements AutoCloseable
> {
> >
> > public static BackwardsCompatibleLauncherSession open() {
> > try {
> > LauncherSession launcherSession =
> > LauncherFactory.openSession();
> > return new
> > BackwardsCompatibleLauncherSession(launcherSession.getLauncher(),
> > launcherSession::close);
> > } catch (NoSuchMethodError ignore) {
> > // JUnit Platform version on test classpath does not yet
> > support launcher sessions
> > return new
> > BackwardsCompatibleLauncherSession(LauncherFactory.create(), () -> {});
> > }
> > }
> >
> > private final Launcher launcher;
> > private final Runnable onClose;
> >
> > private BackwardsCompatibleLauncherSession(Launcher launcher,
> Runnable
> > onClose) {
> > this.launcher = launcher;
> > this.onClose = onClose;
> > }
> >
> > Launcher getLauncher() {
> > return launcher;
> > }
> >
> > @Override
> > public void close() {
> > onClose.run();
> > }
> > }
> >
> > Cheers,
> >
> > Marc
> >
>
Re: JUnit Platform, forkCount > 1 and the TestPlan
Posted by Tibor Digana <ti...@apache.org>.
LauncherSessionListener can be obviously used as a hack by the end users
but I do not see the reason why we should use it.
We do not want to explicitly call fixture methods because this would lead
to maintenance if Junit5 introduces new fixture mechanisms.
On Sat, Sep 11, 2021 at 12:25 PM Tibor Digana <ti...@apache.org>
wrote:
> I do not want to stick to the latest version 1.8.
> We have to stick to version 1.3 as it is right now.
> We do not have a critical issue which could not be solved by reflection.
> This reflection call, I proposed in GH, is made only once, no performance
> penalty.
>
> T
>
> On Sat, Sep 11, 2021 at 7:52 AM Marc Philipp <ma...@gradle.com> wrote:
>
>> Hi Emond and Tibor,
>>
>> I’m glad you discovered the new LauncherSession API which was added for
>> this purpose. The JUnit 5.8 GA release will come in the next few days.
>>
>> As you mentrioned, the official documentation does not (yet!) do a good
>> job of explaining its intended use case:
>>
>> https://junit.org/junit5/docs/5.8.0-RC1/user-guide/#launcher-api-launcher-session-listeners-custom
>>
>> Here’s a more complete example that I wrote for Gradle’s test
>> distribution plugin:
>>
>> https://docs.gradle.com/enterprise/test-distribution-gradle-plugin/#junit_5_8_and_later
>>
>> It demonstrates how to initialize a “fixture” only once for an entire
>> session and only if tests are actually going to be executed not just
>> discovered. I’ll make sure to update the official JUnit docs to include a
>> similar example before the release.
>>
>> To only differentiate between the different versions of JUnit in one
>> place and stay backwards compatible, you could create an adapter class like
>> this:
>>
>> public class BackwardsCompatibleLauncherSession implements AutoCloseable {
>>
>> public static BackwardsCompatibleLauncherSession open() {
>> try {
>> LauncherSession launcherSession =
>> LauncherFactory.openSession();
>> return new
>> BackwardsCompatibleLauncherSession(launcherSession.getLauncher(),
>> launcherSession::close);
>> } catch (NoSuchMethodError ignore) {
>> // JUnit Platform version on test classpath does not yet
>> support launcher sessions
>> return new
>> BackwardsCompatibleLauncherSession(LauncherFactory.create(), () -> {});
>> }
>> }
>>
>> private final Launcher launcher;
>> private final Runnable onClose;
>>
>> private BackwardsCompatibleLauncherSession(Launcher launcher,
>> Runnable onClose) {
>> this.launcher = launcher;
>> this.onClose = onClose;
>> }
>>
>> Launcher getLauncher() {
>> return launcher;
>> }
>>
>> @Override
>> public void close() {
>> onClose.run();
>> }
>> }
>>
>> Cheers,
>>
>> Marc
>>
>
Re: JUnit Platform, forkCount > 1 and the TestPlan
Posted by Tibor Digana <ti...@apache.org>.
I do not want to stick to the latest version 1.8.
We have to stick to version 1.3 as it is right now.
We do not have a critical issue which could not be solved by reflection.
This reflection call, I proposed in GH, is made only once, no performance
penalty.
T
On Sat, Sep 11, 2021 at 7:52 AM Marc Philipp <ma...@gradle.com> wrote:
> Hi Emond and Tibor,
>
> I’m glad you discovered the new LauncherSession API which was added for
> this purpose. The JUnit 5.8 GA release will come in the next few days.
>
> As you mentrioned, the official documentation does not (yet!) do a good
> job of explaining its intended use case:
>
> https://junit.org/junit5/docs/5.8.0-RC1/user-guide/#launcher-api-launcher-session-listeners-custom
>
> Here’s a more complete example that I wrote for Gradle’s test distribution
> plugin:
>
> https://docs.gradle.com/enterprise/test-distribution-gradle-plugin/#junit_5_8_and_later
>
> It demonstrates how to initialize a “fixture” only once for an entire
> session and only if tests are actually going to be executed not just
> discovered. I’ll make sure to update the official JUnit docs to include a
> similar example before the release.
>
> To only differentiate between the different versions of JUnit in one place
> and stay backwards compatible, you could create an adapter class like this:
>
> public class BackwardsCompatibleLauncherSession implements AutoCloseable {
>
> public static BackwardsCompatibleLauncherSession open() {
> try {
> LauncherSession launcherSession =
> LauncherFactory.openSession();
> return new
> BackwardsCompatibleLauncherSession(launcherSession.getLauncher(),
> launcherSession::close);
> } catch (NoSuchMethodError ignore) {
> // JUnit Platform version on test classpath does not yet
> support launcher sessions
> return new
> BackwardsCompatibleLauncherSession(LauncherFactory.create(), () -> {});
> }
> }
>
> private final Launcher launcher;
> private final Runnable onClose;
>
> private BackwardsCompatibleLauncherSession(Launcher launcher, Runnable
> onClose) {
> this.launcher = launcher;
> this.onClose = onClose;
> }
>
> Launcher getLauncher() {
> return launcher;
> }
>
> @Override
> public void close() {
> onClose.run();
> }
> }
>
> Cheers,
>
> Marc
>
Re: JUnit Platform, forkCount > 1 and the TestPlan
Posted by Emond Papegaaij <em...@gmail.com>.
Hi Marc,
With forkCount = 1 it works fine, the problem is with forkCount > 1. The
Arquillian extension registers its state in the store at the root of the
TestPlan. When running with forkCount > 1, surefire sets up a load balancer
in the main process. This load balancer streams tests one by one to the
forks. The forks run a test, and request the next one when done. This is
implemented by creating a LauncherDiscoveryRequest per test. This in turn
results in a separate TestPlan per test, causing the state of Arquillian to
be rebuilt over and over again. When running with forkCount = 1, only a
single LauncherDiscoveryRequest is created and all tests are part of the
same TestPlan.
The Arquillian extension can be modified to use a LauncherSessionListener,
but this will require changes to the code base. The same is probably true
for other extensions.
Best regards,
Emond
On Mon, Sep 13, 2021 at 7:51 AM Marc Philipp <ma...@gradle.com> wrote:
> Hi Emond,
>
> could you please elaborate why the LauncherSessionListener approach
> doesn’t work for forkCount=1 for Arquillian?
>
> Cheers,
>
> Marc
>
> On Sunday, Sep 12, 2021 at 9:37 PM, Emond Papegaaij <
> emond.papegaaij@gmail.com> wrote:
> Hi Marc,
>
> Thanks for the links to the documentation. This really helps for my
> Arquillian-Spock extension. It's good to know the LauncherSession was
> introduced to solve this issue.
>
> The problem I see with this new API is that it requires the extension to
> explicitly make use of it. For example, the JUnit 5 Arquillian extension
> will need to be adapted to this new API to work correctly with a forkCount
>
> 1. The same will probably hold for other extensions. The cause of this is
>
> that separate invocations of the launcher do not really share a life
> cycle.
> LauncherSession is able to bind these invocations together, but the engine
> will be run over and over again. JUnit does support streaming tests in a
> lazy way via the DynamicTest API. It would be great if this could somehow
> be exposed via the LauncherDiscoveryRequest. That would IMHO really solve
> this issue.
>
> Best regards,
> Emond
>
> On Sat, Sep 11, 2021 at 7:52 AM Marc Philipp <ma...@gradle.com> wrote:
>
> Hi Emond and Tibor,
>
> I’m glad you discovered the new LauncherSession API which was added for
> this purpose. The JUnit 5.8 GA release will come in the next few days.
>
> As you mentrioned, the official documentation does not (yet!) do a good
> job of explaining its intended use case:
>
>
> https://junit.org/junit5/docs/5.8.0-RC1/user-guide/#launcher-api-launcher-session-listeners-custom
>
> Here’s a more complete example that I wrote for Gradle’s test distribution
> plugin:
>
>
> https://docs.gradle.com/enterprise/test-distribution-gradle-plugin/#junit_5_8_and_later
>
> It demonstrates how to initialize a “fixture” only once for an entire
> session and only if tests are actually going to be executed not just
> discovered. I’ll make sure to update the official JUnit docs to include a
> similar example before the release.
>
> To only differentiate between the different versions of JUnit in one place
> and stay backwards compatible, you could create an adapter class like
> this:
>
> public class BackwardsCompatibleLauncherSession implements AutoCloseable {
>
> public static BackwardsCompatibleLauncherSession open() {
> try {
> LauncherSession launcherSession =
> LauncherFactory.openSession();
> return new
> BackwardsCompatibleLauncherSession(launcherSession.getLauncher(),
> launcherSession::close);
> } catch (NoSuchMethodError ignore) {
> // JUnit Platform version on test classpath does not yet
> support launcher sessions
> return new
> BackwardsCompatibleLauncherSession(LauncherFactory.create(), () -> {});
> }
> }
>
> private final Launcher launcher;
> private final Runnable onClose;
>
> private BackwardsCompatibleLauncherSession(Launcher launcher, Runnable
> onClose) {
> this.launcher = launcher;
> this.onClose = onClose;
> }
>
> Launcher getLauncher() {
> return launcher;
> }
>
> @Override
> public void close() {
> onClose.run();
> }
> }
>
> Cheers,
>
> Marc
>
>
Re: JUnit Platform, forkCount > 1 and the TestPlan
Posted by Marc Philipp <ma...@gradle.com>.
Hi Emond,
could you please elaborate why the LauncherSessionListener approach doesn’t work for forkCount=1 for Arquillian?
Cheers,
Marc
> On Sunday, Sep 12, 2021 at 9:37 PM, Emond Papegaaij <emond.papegaaij@gmail.com (mailto:emond.papegaaij@gmail.com)> wrote:
> Hi Marc,
>
> Thanks for the links to the documentation. This really helps for my
> Arquillian-Spock extension. It's good to know the LauncherSession was
> introduced to solve this issue.
>
> The problem I see with this new API is that it requires the extension to
> explicitly make use of it. For example, the JUnit 5 Arquillian extension
> will need to be adapted to this new API to work correctly with a forkCount
> > 1. The same will probably hold for other extensions. The cause of this is
> that separate invocations of the launcher do not really share a life cycle.
> LauncherSession is able to bind these invocations together, but the engine
> will be run over and over again. JUnit does support streaming tests in a
> lazy way via the DynamicTest API. It would be great if this could somehow
> be exposed via the LauncherDiscoveryRequest. That would IMHO really solve
> this issue.
>
> Best regards,
> Emond
>
> On Sat, Sep 11, 2021 at 7:52 AM Marc Philipp <ma...@gradle.com> wrote:
>
> > Hi Emond and Tibor,
> >
> > I’m glad you discovered the new LauncherSession API which was added for
> > this purpose. The JUnit 5.8 GA release will come in the next few days.
> >
> > As you mentrioned, the official documentation does not (yet!) do a good
> > job of explaining its intended use case:
> >
> > https://junit.org/junit5/docs/5.8.0-RC1/user-guide/#launcher-api-launcher-session-listeners-custom
> >
> > Here’s a more complete example that I wrote for Gradle’s test distribution
> > plugin:
> >
> > https://docs.gradle.com/enterprise/test-distribution-gradle-plugin/#junit_5_8_and_later
> >
> > It demonstrates how to initialize a “fixture” only once for an entire
> > session and only if tests are actually going to be executed not just
> > discovered. I’ll make sure to update the official JUnit docs to include a
> > similar example before the release.
> >
> > To only differentiate between the different versions of JUnit in one place
> > and stay backwards compatible, you could create an adapter class like this:
> >
> > public class BackwardsCompatibleLauncherSession implements AutoCloseable {
> >
> > public static BackwardsCompatibleLauncherSession open() {
> > try {
> > LauncherSession launcherSession =
> > LauncherFactory.openSession();
> > return new
> > BackwardsCompatibleLauncherSession(launcherSession.getLauncher(),
> > launcherSession::close);
> > } catch (NoSuchMethodError ignore) {
> > // JUnit Platform version on test classpath does not yet
> > support launcher sessions
> > return new
> > BackwardsCompatibleLauncherSession(LauncherFactory.create(), () -> {});
> > }
> > }
> >
> > private final Launcher launcher;
> > private final Runnable onClose;
> >
> > private BackwardsCompatibleLauncherSession(Launcher launcher, Runnable
> > onClose) {
> > this.launcher = launcher;
> > this.onClose = onClose;
> > }
> >
> > Launcher getLauncher() {
> > return launcher;
> > }
> >
> > @Override
> > public void close() {
> > onClose.run();
> > }
> > }
> >
> > Cheers,
> >
> > Marc
> >
Re: JUnit Platform, forkCount > 1 and the TestPlan
Posted by Emond Papegaaij <em...@gmail.com>.
Hi Marc,
Thanks for the links to the documentation. This really helps for my
Arquillian-Spock extension. It's good to know the LauncherSession was
introduced to solve this issue.
The problem I see with this new API is that it requires the extension to
explicitly make use of it. For example, the JUnit 5 Arquillian extension
will need to be adapted to this new API to work correctly with a forkCount
> 1. The same will probably hold for other extensions. The cause of this is
that separate invocations of the launcher do not really share a life cycle.
LauncherSession is able to bind these invocations together, but the engine
will be run over and over again. JUnit does support streaming tests in a
lazy way via the DynamicTest API. It would be great if this could somehow
be exposed via the LauncherDiscoveryRequest. That would IMHO really solve
this issue.
Best regards,
Emond
On Sat, Sep 11, 2021 at 7:52 AM Marc Philipp <ma...@gradle.com> wrote:
> Hi Emond and Tibor,
>
> I’m glad you discovered the new LauncherSession API which was added for
> this purpose. The JUnit 5.8 GA release will come in the next few days.
>
> As you mentrioned, the official documentation does not (yet!) do a good
> job of explaining its intended use case:
>
> https://junit.org/junit5/docs/5.8.0-RC1/user-guide/#launcher-api-launcher-session-listeners-custom
>
> Here’s a more complete example that I wrote for Gradle’s test distribution
> plugin:
>
> https://docs.gradle.com/enterprise/test-distribution-gradle-plugin/#junit_5_8_and_later
>
> It demonstrates how to initialize a “fixture” only once for an entire
> session and only if tests are actually going to be executed not just
> discovered. I’ll make sure to update the official JUnit docs to include a
> similar example before the release.
>
> To only differentiate between the different versions of JUnit in one place
> and stay backwards compatible, you could create an adapter class like this:
>
> public class BackwardsCompatibleLauncherSession implements AutoCloseable {
>
> public static BackwardsCompatibleLauncherSession open() {
> try {
> LauncherSession launcherSession =
> LauncherFactory.openSession();
> return new
> BackwardsCompatibleLauncherSession(launcherSession.getLauncher(),
> launcherSession::close);
> } catch (NoSuchMethodError ignore) {
> // JUnit Platform version on test classpath does not yet
> support launcher sessions
> return new
> BackwardsCompatibleLauncherSession(LauncherFactory.create(), () -> {});
> }
> }
>
> private final Launcher launcher;
> private final Runnable onClose;
>
> private BackwardsCompatibleLauncherSession(Launcher launcher, Runnable
> onClose) {
> this.launcher = launcher;
> this.onClose = onClose;
> }
>
> Launcher getLauncher() {
> return launcher;
> }
>
> @Override
> public void close() {
> onClose.run();
> }
> }
>
> Cheers,
>
> Marc
>
Re: JUnit Platform, forkCount > 1 and the TestPlan
Posted by Marc Philipp <ma...@gradle.com>.
Hi Emond and Tibor,
I’m glad you discovered the new LauncherSession API which was added for this purpose. The JUnit 5.8 GA release will come in the next few days.
As you mentrioned, the official documentation does not (yet!) do a good job of explaining its intended use case:
https://junit.org/junit5/docs/5.8.0-RC1/user-guide/#launcher-api-launcher-session-listeners-custom
Here’s a more complete example that I wrote for Gradle’s test distribution plugin:
https://docs.gradle.com/enterprise/test-distribution-gradle-plugin/#junit_5_8_and_later
It demonstrates how to initialize a “fixture” only once for an entire session and only if tests are actually going to be executed not just discovered. I’ll make sure to update the official JUnit docs to include a similar example before the release.
To only differentiate between the different versions of JUnit in one place and stay backwards compatible, you could create an adapter class like this:
public class BackwardsCompatibleLauncherSession implements AutoCloseable {
public static BackwardsCompatibleLauncherSession open() {
try {
LauncherSession launcherSession = LauncherFactory.openSession();
return new BackwardsCompatibleLauncherSession(launcherSession.getLauncher(), launcherSession::close);
} catch (NoSuchMethodError ignore) {
// JUnit Platform version on test classpath does not yet support launcher sessions
return new BackwardsCompatibleLauncherSession(LauncherFactory.create(), () -> {});
}
}
private final Launcher launcher;
private final Runnable onClose;
private BackwardsCompatibleLauncherSession(Launcher launcher, Runnable onClose) {
this.launcher = launcher;
this.onClose = onClose;
}
Launcher getLauncher() {
return launcher;
}
@Override
public void close() {
onClose.run();
}
}
Cheers,
Marc
Re: JUnit Platform, forkCount > 1 and the TestPlan
Posted by Tibor Digana <ti...@apache.org>.
Hi Emond,
This section of code is executed for the forkCount > 1
https://github.com/apache/maven-surefire/blob/master/surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/JUnitPlatformProvider.java#L194-L202
The above part runs if the forkCount is 1.
Do you think that using the session together with the try-with-resource in
the selected section of code makes the trick?
Pls check it your local in your local branch and your project.
T
On Fri, Sep 10, 2021 at 11:33 PM Emond Papegaaij <em...@gmail.com>
wrote:
> Hi Tibor,
>
> That's what I implemented, although I couldn't use the fancy try, because
> of the way the code is structured. The LauncherSession is started by
> LazyLauncher. This will allow registering a listener for opening and
> closing the session, given a place for pre and post fixtures.
>
> Best regards,
> Emond
>
> Op vr 10 sep. 2021 23:00 schreef Tibor Digana <ti...@apache.org>:
>
> > Hi Emond,
> >
> > Are you looking for this?
> >
> >
> https://github.com/junit-team/junit5/blob/main/documentation/src/test/java/example/UsingTheLauncherDemo.java#L86-L96
> >
> > On Fri, Sep 10, 2021 at 10:49 PM Emond Papegaaij <
> > emond.papegaaij@gmail.com>
> > wrote:
> >
> > > On Fri, Sep 10, 2021 at 9:15 PM Emond Papegaaij <
> > emond.papegaaij@gmail.com
> > > >
> > > wrote:
> > >
> > > > On Fri, Sep 10, 2021 at 8:41 PM Christian Stein <so...@gmail.com>
> > > > wrote:
> > > >
> > > >> On Fri, Sep 10, 2021 at 8:27 PM Emond Papegaaij <
> > > >> emond.papegaaij@gmail.com>
> > > >> wrote:
> > > >>
> > > >> For now, I think the LauncherSession is the best way to at least
> > provide
> > > >> > some hooks for pre and post fixtures. It shouldn't be too hard to
> > get
> > > >> this
> > > >> > in the current code base with backwards compatibility for JUnit
> > > Platform
> > > >> > 1.7 and older. I'll see if I can create a pull request for this
> > > >> somewhere
> > > >> > next week.
> > > >> >
> > > >>
> > > >> Before investing too much energy into a PR, please start a
> discussion
> > or
> > > >> open a feature request first - chances are that what you seek to
> > achieve
> > > >> is already possible, or somebody else has filed a similar request.
> > > >
> > > >
> > > > You mean at JUnit? I'm pretty sure that it's currently not possible.
> > I've
> > > > been stepping through that code for quite some time now, and the
> whole
> > > > TestPlan construction is very static. There doesn't seem to be an
> open
> > > > request for this at the moment.
> > > >
> > > > The API for a LauncherSession is very simple and I think it's a good
> > idea
> > > > to use it anyway. The hardest part will be to only start the session
> > when
> > > > using 1.8 or higher.
> > > >
> > >
> > > That was easier than I thought. I've already got a working POC at
> > >
> > >
> >
> https://github.com/topicusonderwijs/maven-surefire/commit/b112130170c244846d5e35353a4c90afaf42aff1
> > > . The only problems at the moment are that some tests fail because a
> > > TestPlan is immutable in 1.8, the main process creates a
> LauncherSession
> > > that is never closed and I screwed up the formatting in some places.
> With
> > > this change, a LauncherSession is created when JUnit Platform 1.8 is
> > used.
> > > This session is closed when all tests have been executed. When running
> on
> > > 1.7 or lower, it falls back to the old behavior. I've verified that
> only
> > > one LauncherSession is created per fork when running with forkCount >
> 1.
> > >
> > > Best regards,
> > > Emond
> > >
> >
>
Re: JUnit Platform, forkCount > 1 and the TestPlan
Posted by Emond Papegaaij <em...@gmail.com>.
Hi Tibor,
That's what I implemented, although I couldn't use the fancy try, because
of the way the code is structured. The LauncherSession is started by
LazyLauncher. This will allow registering a listener for opening and
closing the session, given a place for pre and post fixtures.
Best regards,
Emond
Op vr 10 sep. 2021 23:00 schreef Tibor Digana <ti...@apache.org>:
> Hi Emond,
>
> Are you looking for this?
>
> https://github.com/junit-team/junit5/blob/main/documentation/src/test/java/example/UsingTheLauncherDemo.java#L86-L96
>
> On Fri, Sep 10, 2021 at 10:49 PM Emond Papegaaij <
> emond.papegaaij@gmail.com>
> wrote:
>
> > On Fri, Sep 10, 2021 at 9:15 PM Emond Papegaaij <
> emond.papegaaij@gmail.com
> > >
> > wrote:
> >
> > > On Fri, Sep 10, 2021 at 8:41 PM Christian Stein <so...@gmail.com>
> > > wrote:
> > >
> > >> On Fri, Sep 10, 2021 at 8:27 PM Emond Papegaaij <
> > >> emond.papegaaij@gmail.com>
> > >> wrote:
> > >>
> > >> For now, I think the LauncherSession is the best way to at least
> provide
> > >> > some hooks for pre and post fixtures. It shouldn't be too hard to
> get
> > >> this
> > >> > in the current code base with backwards compatibility for JUnit
> > Platform
> > >> > 1.7 and older. I'll see if I can create a pull request for this
> > >> somewhere
> > >> > next week.
> > >> >
> > >>
> > >> Before investing too much energy into a PR, please start a discussion
> or
> > >> open a feature request first - chances are that what you seek to
> achieve
> > >> is already possible, or somebody else has filed a similar request.
> > >
> > >
> > > You mean at JUnit? I'm pretty sure that it's currently not possible.
> I've
> > > been stepping through that code for quite some time now, and the whole
> > > TestPlan construction is very static. There doesn't seem to be an open
> > > request for this at the moment.
> > >
> > > The API for a LauncherSession is very simple and I think it's a good
> idea
> > > to use it anyway. The hardest part will be to only start the session
> when
> > > using 1.8 or higher.
> > >
> >
> > That was easier than I thought. I've already got a working POC at
> >
> >
> https://github.com/topicusonderwijs/maven-surefire/commit/b112130170c244846d5e35353a4c90afaf42aff1
> > . The only problems at the moment are that some tests fail because a
> > TestPlan is immutable in 1.8, the main process creates a LauncherSession
> > that is never closed and I screwed up the formatting in some places. With
> > this change, a LauncherSession is created when JUnit Platform 1.8 is
> used.
> > This session is closed when all tests have been executed. When running on
> > 1.7 or lower, it falls back to the old behavior. I've verified that only
> > one LauncherSession is created per fork when running with forkCount > 1.
> >
> > Best regards,
> > Emond
> >
>
Re: JUnit Platform, forkCount > 1 and the TestPlan
Posted by Tibor Digana <ti...@apache.org>.
Hi Emond,
Are you looking for this?
https://github.com/junit-team/junit5/blob/main/documentation/src/test/java/example/UsingTheLauncherDemo.java#L86-L96
On Fri, Sep 10, 2021 at 10:49 PM Emond Papegaaij <em...@gmail.com>
wrote:
> On Fri, Sep 10, 2021 at 9:15 PM Emond Papegaaij <emond.papegaaij@gmail.com
> >
> wrote:
>
> > On Fri, Sep 10, 2021 at 8:41 PM Christian Stein <so...@gmail.com>
> > wrote:
> >
> >> On Fri, Sep 10, 2021 at 8:27 PM Emond Papegaaij <
> >> emond.papegaaij@gmail.com>
> >> wrote:
> >>
> >> For now, I think the LauncherSession is the best way to at least provide
> >> > some hooks for pre and post fixtures. It shouldn't be too hard to get
> >> this
> >> > in the current code base with backwards compatibility for JUnit
> Platform
> >> > 1.7 and older. I'll see if I can create a pull request for this
> >> somewhere
> >> > next week.
> >> >
> >>
> >> Before investing too much energy into a PR, please start a discussion or
> >> open a feature request first - chances are that what you seek to achieve
> >> is already possible, or somebody else has filed a similar request.
> >
> >
> > You mean at JUnit? I'm pretty sure that it's currently not possible. I've
> > been stepping through that code for quite some time now, and the whole
> > TestPlan construction is very static. There doesn't seem to be an open
> > request for this at the moment.
> >
> > The API for a LauncherSession is very simple and I think it's a good idea
> > to use it anyway. The hardest part will be to only start the session when
> > using 1.8 or higher.
> >
>
> That was easier than I thought. I've already got a working POC at
>
> https://github.com/topicusonderwijs/maven-surefire/commit/b112130170c244846d5e35353a4c90afaf42aff1
> . The only problems at the moment are that some tests fail because a
> TestPlan is immutable in 1.8, the main process creates a LauncherSession
> that is never closed and I screwed up the formatting in some places. With
> this change, a LauncherSession is created when JUnit Platform 1.8 is used.
> This session is closed when all tests have been executed. When running on
> 1.7 or lower, it falls back to the old behavior. I've verified that only
> one LauncherSession is created per fork when running with forkCount > 1.
>
> Best regards,
> Emond
>
Re: JUnit Platform, forkCount > 1 and the TestPlan
Posted by Emond Papegaaij <em...@gmail.com>.
On Fri, Sep 10, 2021 at 9:15 PM Emond Papegaaij <em...@gmail.com>
wrote:
> On Fri, Sep 10, 2021 at 8:41 PM Christian Stein <so...@gmail.com>
> wrote:
>
>> On Fri, Sep 10, 2021 at 8:27 PM Emond Papegaaij <
>> emond.papegaaij@gmail.com>
>> wrote:
>>
>> For now, I think the LauncherSession is the best way to at least provide
>> > some hooks for pre and post fixtures. It shouldn't be too hard to get
>> this
>> > in the current code base with backwards compatibility for JUnit Platform
>> > 1.7 and older. I'll see if I can create a pull request for this
>> somewhere
>> > next week.
>> >
>>
>> Before investing too much energy into a PR, please start a discussion or
>> open a feature request first - chances are that what you seek to achieve
>> is already possible, or somebody else has filed a similar request.
>
>
> You mean at JUnit? I'm pretty sure that it's currently not possible. I've
> been stepping through that code for quite some time now, and the whole
> TestPlan construction is very static. There doesn't seem to be an open
> request for this at the moment.
>
> The API for a LauncherSession is very simple and I think it's a good idea
> to use it anyway. The hardest part will be to only start the session when
> using 1.8 or higher.
>
That was easier than I thought. I've already got a working POC at
https://github.com/topicusonderwijs/maven-surefire/commit/b112130170c244846d5e35353a4c90afaf42aff1
. The only problems at the moment are that some tests fail because a
TestPlan is immutable in 1.8, the main process creates a LauncherSession
that is never closed and I screwed up the formatting in some places. With
this change, a LauncherSession is created when JUnit Platform 1.8 is used.
This session is closed when all tests have been executed. When running on
1.7 or lower, it falls back to the old behavior. I've verified that only
one LauncherSession is created per fork when running with forkCount > 1.
Best regards,
Emond
Re: JUnit Platform, forkCount > 1 and the TestPlan
Posted by Emond Papegaaij <em...@gmail.com>.
On Fri, Sep 10, 2021 at 8:41 PM Christian Stein <so...@gmail.com> wrote:
> On Fri, Sep 10, 2021 at 8:27 PM Emond Papegaaij <emond.papegaaij@gmail.com
> >
> wrote:
>
> For now, I think the LauncherSession is the best way to at least provide
> > some hooks for pre and post fixtures. It shouldn't be too hard to get
> this
> > in the current code base with backwards compatibility for JUnit Platform
> > 1.7 and older. I'll see if I can create a pull request for this somewhere
> > next week.
> >
>
> Before investing too much energy into a PR, please start a discussion or
> open a feature request first - chances are that what you seek to achieve
> is already possible, or somebody else has filed a similar request.
You mean at JUnit? I'm pretty sure that it's currently not possible. I've
been stepping through that code for quite some time now, and the whole
TestPlan construction is very static. There doesn't seem to be an open
request for this at the moment.
The API for a LauncherSession is very simple and I think it's a good idea
to use it anyway. The hardest part will be to only start the session when
using 1.8 or higher.
Best regards,
Emond
Re: JUnit Platform, forkCount > 1 and the TestPlan
Posted by Christian Stein <so...@gmail.com>.
On Fri, Sep 10, 2021 at 8:27 PM Emond Papegaaij <em...@gmail.com>
wrote:
> [...]
For now, I think the LauncherSession is the best way to at least provide
> some hooks for pre and post fixtures. It shouldn't be too hard to get this
> in the current code base with backwards compatibility for JUnit Platform
> 1.7 and older. I'll see if I can create a pull request for this somewhere
> next week.
>
Before investing too much energy into a PR, please start a discussion or
open a feature request first - chances are that what you seek to achieve
is already possible, or somebody else has filed a similar request.
Cheers,
Christian
Re: JUnit Platform, forkCount > 1 and the TestPlan
Posted by Emond Papegaaij <em...@gmail.com>.
On Fri, Sep 10, 2021 at 6:18 PM Tibor Digana <ti...@apache.org> wrote:
> We have to dig into Junit 5. Surefire is a streamer of classes across the
> forks which is our load balancer. Therefore each class is running
> separately, pre and post fixtures. If the Junit 5 used Java Streamer
> including dome kind of control of fixtures then web would have this issue.
>
I totally agree. To me JUnit 5 looks overly complicated. Everything can be
customized, but the overall result is that even the simple things are hard
or impossible. I've looked at various ways to change the TestPlan, but the
only way I could think of to support streaming test classes is to have some
sort of streaming TestEngine in between. I'll file a feature request at
JUnit for support for this.
For now, I think the LauncherSession is the best way to at least provide
some hooks for pre and post fixtures. It shouldn't be too hard to get this
in the current code base with backwards compatibility for JUnit Platform
1.7 and older. I'll see if I can create a pull request for this somewhere
next week.
Best regards,
Emond
Re: JUnit Platform, forkCount > 1 and the TestPlan
Posted by Tibor Digana <ti...@apache.org>.
Sorry for typos, I sent it from my mobile.
T
On Fri, Sep 10, 2021 at 6:18 PM Tibor Digana <ti...@apache.org> wrote:
> We have to dig into Junit 5. Surefire is a streamer of classes across the
> forks which is our load balancer. Therefore each class is running
> separately, pre and post fixtures. If the Junit 5 used Java Streamer
> including dome kind of control of fixtures then web would have this issue.
>
> Dňa pi 10. 9. 2021, 18:03 Emond Papegaaij <em...@gmail.com>
> napísal(a):
>
>> On Fri, Sep 10, 2021 at 5:06 PM Tibor Digana <ti...@apache.org>
>> wrote:
>>
>> > If you use forkCount > 1, the Surefire loads test classes via load
>> > balancer.
>> > If you use default forkCount = 0, all the classes are run eagerly as a
>> > suite via JUnit5 Launcher in one shot.
>> >
>>
>> Yes, this is what I saw in the JUnitPlatformProvider. The cause of my
>> issues lies in the difference in events triggered by Arquillian due to the
>> classes executed one by one. The demo-project attached to the ticket
>> demonstrates this in just a few lines of code.
>>
>> If you are aiming for Arquillian, testing the applications in the
>> > application server, you should use maven-failsafe-plugin which has
>> another
>> > testing model.
>> > See the plugin goals of Failsafe plugin. Maven Failsafe Plugin – Plugin
>> > Documentation (apache.org)
>> > <
>> https://maven.apache.org/surefire/maven-failsafe-plugin/plugin-info.html>
>> >
>> https://maven.apache.org/surefire/maven-failsafe-plugin/plugin-info.html
>> > There are Maven phases:
>> > pre-integration-test
>> > integration-test
>> > post-integration-test
>> > verify
>> >
>> > Therefore you should start the application server in the phase
>> > pre-integration-test.
>> > Accordingly, you should stop it in the phase post-integration-test.
>> >
>> > Then use Failsafe plugin in the phases integration-test and verify.
>> >
>>
>> Unfortunately, this is not how arquillian is setup. You can run tests
>> against running servers (which you can then start in
>> pre-integration-test),
>> but using arquillian cube, these containers can be started as part of the
>> test lifecycle. Also, Arquillian is just a single incarnation of the
>> underlying issue. Every extension that takes more than a few seconds to
>> start and is built to only initialize once for a suite will trigger this
>> same problem (as can be seen by the demo project). As surefire and
>> failsafe
>> use the same JUnitPlatformProvider, the problem will also be the same.
>>
>> What do you think about the solution of using a LauncherSession to bind
>> the
>> multiple executions together? This will require JUnit Platform 1.8 and
>> changes in the JUnitPlatformProvider. However, I'm not sure what the
>> intended audience of LauncherSessionListener is.
>>
>> Best regards,
>> Emond
>>
>> On Fri, Sep 10, 2021 at 12:42 PM Emond Papegaaij <
>> emond.papegaaij@gmail.com>
>> > wrote:
>> >
>> > > Hi all,
>> > >
>> > > First of all, sorry for the lengthy post. I decided to add some
>> context
>> > to
>> > > explain things a bit, but it resulted in quite a long e-mail. For the
>> > past
>> > > few weeks I've been trying to come up with a solution for the issue I
>> > > filled under SUREFIRE-1935, but I'm getting stuck and starting to feel
>> > like
>> > > the issue cannot be solved with the current JUnit Platform API. To
>> give
>> > > some perspective into why this issue is important for us, I first
>> have to
>> > > explain a bit about our setup.
>> > >
>> > > We write our tests in the Spock framework and use Arquillian to run
>> the
>> > > tests in the application container. Some of our tests, especially the
>> > > Selenium based tests, require quite some setup and tear down. Prior to
>> > > starting the test, Arquillian cube builds and starts several docker
>> > > containers and the tests are run against these containers. We
>> currently
>> > use
>> > > the JUnit 4 based Spock 1.3 with Groovy 2.5, but we like to upgrade to
>> > > Spock 2 and Groovy 3, which runs on top of the JUnit Platform. For
>> this,
>> > > I've already started working on a Spock extension that integrates
>> Spock 2
>> > > in the Arquillian test life cycle [1]. This extension is inspired by
>> the
>> > > (currently Alpha) JUnit5 module for Arquillian [2]. Both use a global
>> > > registration to keep track of the state managed by Arquilllian. This
>> will
>> > > end up somewhere at the root of the TestPlan (for example see [3]).
>> > >
>> > > Because our tests are quite extensive, with a total run time of 6
>> hours,
>> > we
>> > > run them with a forkCount of 8, greatly reducing the total duration.
>> > > However, this is where SUREFIRE-1935 comes into play. With a
>> forkCount >
>> > 1,
>> > > the entire test life cycle is started over and over again for every
>> test
>> > > class. This happens in JUnitPlatformProvider at line 197 [4]. This
>> > results
>> > > in the entire Arquillian suite being torn down and setup for every
>> class,
>> > > in our case adding several minutes to the execution of every test
>> class
>> > > because the docker setup is done over and over again.
>> > >
>> > > To overcome this issue, the state from one test class execution has
>> to be
>> > > carried to the next. It seems the LauncherSession (introduced in JUnit
>> > 5.8)
>> > > is meant to close this gap. However, this would mean my extension
>> would
>> > > also need to implement a LauncherSessionListener, and I'm not sure if
>> > > extensions are supposed to integrate with the launcher as well. Also,
>> for
>> > > this surefire would need to start a session prior to the tests, and
>> close
>> > > it when done. I think this is a good idea anyway when running on
>> platform
>> > > 1.8 or higher.
>> > >
>> > > Another solution could be a (sort of) dynamic test that produces the
>> > tests
>> > > to be run one by one. However, here my knowledge of JUnit really falls
>> > > short. I've got no idea of this is even possible.
>> > >
>> > > I hope someone can help me out on this one and point me in the right
>> > > direction, as we like to upgrade our test frameworks and this is
>> blocking
>> > > us at the moment.
>> > >
>> > > Best regards,
>> > > Emond Papegaaij
>> > >
>> > > [1] Arquillian extension for Spock Framework 2:
>> > >
>> > >
>> >
>> https://github.com/topicusonderwijs/arquillian-testrunner-spock/tree/spock-2.0-junit5
>> > > [2] Arquillian module for JUnit 5:
>> > > https://github.com/arquillian/arquillian-core/tree/master/junit5
>> > > [3] Registration in root store:
>> > >
>> > >
>> >
>> https://github.com/arquillian/arquillian-core/blob/master/junit5/core/src/main/java/org/jboss/arquillian/junit5/JUnitJupiterTestClassLifecycleManager.java#L20
>> > > [4] JUnitPlatformProvider launching the tests:
>> > >
>> > >
>> >
>> https://github.com/apache/maven-surefire/blob/master/surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/JUnitPlatformProvider.java#L197
>> > >
>> >
>>
>
Re: JUnit Platform, forkCount > 1 and the TestPlan
Posted by Tibor Digana <ti...@apache.org>.
We have to dig into Junit 5. Surefire is a streamer of classes across the
forks which is our load balancer. Therefore each class is running
separately, pre and post fixtures. If the Junit 5 used Java Streamer
including dome kind of control of fixtures then web would have this issue.
Dňa pi 10. 9. 2021, 18:03 Emond Papegaaij <em...@gmail.com>
napísal(a):
> On Fri, Sep 10, 2021 at 5:06 PM Tibor Digana <ti...@apache.org>
> wrote:
>
> > If you use forkCount > 1, the Surefire loads test classes via load
> > balancer.
> > If you use default forkCount = 0, all the classes are run eagerly as a
> > suite via JUnit5 Launcher in one shot.
> >
>
> Yes, this is what I saw in the JUnitPlatformProvider. The cause of my
> issues lies in the difference in events triggered by Arquillian due to the
> classes executed one by one. The demo-project attached to the ticket
> demonstrates this in just a few lines of code.
>
> If you are aiming for Arquillian, testing the applications in the
> > application server, you should use maven-failsafe-plugin which has
> another
> > testing model.
> > See the plugin goals of Failsafe plugin. Maven Failsafe Plugin – Plugin
> > Documentation (apache.org)
> > <
> https://maven.apache.org/surefire/maven-failsafe-plugin/plugin-info.html>
> > https://maven.apache.org/surefire/maven-failsafe-plugin/plugin-info.html
> > There are Maven phases:
> > pre-integration-test
> > integration-test
> > post-integration-test
> > verify
> >
> > Therefore you should start the application server in the phase
> > pre-integration-test.
> > Accordingly, you should stop it in the phase post-integration-test.
> >
> > Then use Failsafe plugin in the phases integration-test and verify.
> >
>
> Unfortunately, this is not how arquillian is setup. You can run tests
> against running servers (which you can then start in pre-integration-test),
> but using arquillian cube, these containers can be started as part of the
> test lifecycle. Also, Arquillian is just a single incarnation of the
> underlying issue. Every extension that takes more than a few seconds to
> start and is built to only initialize once for a suite will trigger this
> same problem (as can be seen by the demo project). As surefire and failsafe
> use the same JUnitPlatformProvider, the problem will also be the same.
>
> What do you think about the solution of using a LauncherSession to bind the
> multiple executions together? This will require JUnit Platform 1.8 and
> changes in the JUnitPlatformProvider. However, I'm not sure what the
> intended audience of LauncherSessionListener is.
>
> Best regards,
> Emond
>
> On Fri, Sep 10, 2021 at 12:42 PM Emond Papegaaij <
> emond.papegaaij@gmail.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > First of all, sorry for the lengthy post. I decided to add some context
> > to
> > > explain things a bit, but it resulted in quite a long e-mail. For the
> > past
> > > few weeks I've been trying to come up with a solution for the issue I
> > > filled under SUREFIRE-1935, but I'm getting stuck and starting to feel
> > like
> > > the issue cannot be solved with the current JUnit Platform API. To give
> > > some perspective into why this issue is important for us, I first have
> to
> > > explain a bit about our setup.
> > >
> > > We write our tests in the Spock framework and use Arquillian to run the
> > > tests in the application container. Some of our tests, especially the
> > > Selenium based tests, require quite some setup and tear down. Prior to
> > > starting the test, Arquillian cube builds and starts several docker
> > > containers and the tests are run against these containers. We currently
> > use
> > > the JUnit 4 based Spock 1.3 with Groovy 2.5, but we like to upgrade to
> > > Spock 2 and Groovy 3, which runs on top of the JUnit Platform. For
> this,
> > > I've already started working on a Spock extension that integrates
> Spock 2
> > > in the Arquillian test life cycle [1]. This extension is inspired by
> the
> > > (currently Alpha) JUnit5 module for Arquillian [2]. Both use a global
> > > registration to keep track of the state managed by Arquilllian. This
> will
> > > end up somewhere at the root of the TestPlan (for example see [3]).
> > >
> > > Because our tests are quite extensive, with a total run time of 6
> hours,
> > we
> > > run them with a forkCount of 8, greatly reducing the total duration.
> > > However, this is where SUREFIRE-1935 comes into play. With a forkCount
> >
> > 1,
> > > the entire test life cycle is started over and over again for every
> test
> > > class. This happens in JUnitPlatformProvider at line 197 [4]. This
> > results
> > > in the entire Arquillian suite being torn down and setup for every
> class,
> > > in our case adding several minutes to the execution of every test class
> > > because the docker setup is done over and over again.
> > >
> > > To overcome this issue, the state from one test class execution has to
> be
> > > carried to the next. It seems the LauncherSession (introduced in JUnit
> > 5.8)
> > > is meant to close this gap. However, this would mean my extension would
> > > also need to implement a LauncherSessionListener, and I'm not sure if
> > > extensions are supposed to integrate with the launcher as well. Also,
> for
> > > this surefire would need to start a session prior to the tests, and
> close
> > > it when done. I think this is a good idea anyway when running on
> platform
> > > 1.8 or higher.
> > >
> > > Another solution could be a (sort of) dynamic test that produces the
> > tests
> > > to be run one by one. However, here my knowledge of JUnit really falls
> > > short. I've got no idea of this is even possible.
> > >
> > > I hope someone can help me out on this one and point me in the right
> > > direction, as we like to upgrade our test frameworks and this is
> blocking
> > > us at the moment.
> > >
> > > Best regards,
> > > Emond Papegaaij
> > >
> > > [1] Arquillian extension for Spock Framework 2:
> > >
> > >
> >
> https://github.com/topicusonderwijs/arquillian-testrunner-spock/tree/spock-2.0-junit5
> > > [2] Arquillian module for JUnit 5:
> > > https://github.com/arquillian/arquillian-core/tree/master/junit5
> > > [3] Registration in root store:
> > >
> > >
> >
> https://github.com/arquillian/arquillian-core/blob/master/junit5/core/src/main/java/org/jboss/arquillian/junit5/JUnitJupiterTestClassLifecycleManager.java#L20
> > > [4] JUnitPlatformProvider launching the tests:
> > >
> > >
> >
> https://github.com/apache/maven-surefire/blob/master/surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/JUnitPlatformProvider.java#L197
> > >
> >
>
Re: JUnit Platform, forkCount > 1 and the TestPlan
Posted by Emond Papegaaij <em...@gmail.com>.
On Fri, Sep 10, 2021 at 5:06 PM Tibor Digana <ti...@apache.org> wrote:
> If you use forkCount > 1, the Surefire loads test classes via load
> balancer.
> If you use default forkCount = 0, all the classes are run eagerly as a
> suite via JUnit5 Launcher in one shot.
>
Yes, this is what I saw in the JUnitPlatformProvider. The cause of my
issues lies in the difference in events triggered by Arquillian due to the
classes executed one by one. The demo-project attached to the ticket
demonstrates this in just a few lines of code.
If you are aiming for Arquillian, testing the applications in the
> application server, you should use maven-failsafe-plugin which has another
> testing model.
> See the plugin goals of Failsafe plugin. Maven Failsafe Plugin – Plugin
> Documentation (apache.org)
> <https://maven.apache.org/surefire/maven-failsafe-plugin/plugin-info.html>
> https://maven.apache.org/surefire/maven-failsafe-plugin/plugin-info.html
> There are Maven phases:
> pre-integration-test
> integration-test
> post-integration-test
> verify
>
> Therefore you should start the application server in the phase
> pre-integration-test.
> Accordingly, you should stop it in the phase post-integration-test.
>
> Then use Failsafe plugin in the phases integration-test and verify.
>
Unfortunately, this is not how arquillian is setup. You can run tests
against running servers (which you can then start in pre-integration-test),
but using arquillian cube, these containers can be started as part of the
test lifecycle. Also, Arquillian is just a single incarnation of the
underlying issue. Every extension that takes more than a few seconds to
start and is built to only initialize once for a suite will trigger this
same problem (as can be seen by the demo project). As surefire and failsafe
use the same JUnitPlatformProvider, the problem will also be the same.
What do you think about the solution of using a LauncherSession to bind the
multiple executions together? This will require JUnit Platform 1.8 and
changes in the JUnitPlatformProvider. However, I'm not sure what the
intended audience of LauncherSessionListener is.
Best regards,
Emond
On Fri, Sep 10, 2021 at 12:42 PM Emond Papegaaij <em...@gmail.com>
> wrote:
>
> > Hi all,
> >
> > First of all, sorry for the lengthy post. I decided to add some context
> to
> > explain things a bit, but it resulted in quite a long e-mail. For the
> past
> > few weeks I've been trying to come up with a solution for the issue I
> > filled under SUREFIRE-1935, but I'm getting stuck and starting to feel
> like
> > the issue cannot be solved with the current JUnit Platform API. To give
> > some perspective into why this issue is important for us, I first have to
> > explain a bit about our setup.
> >
> > We write our tests in the Spock framework and use Arquillian to run the
> > tests in the application container. Some of our tests, especially the
> > Selenium based tests, require quite some setup and tear down. Prior to
> > starting the test, Arquillian cube builds and starts several docker
> > containers and the tests are run against these containers. We currently
> use
> > the JUnit 4 based Spock 1.3 with Groovy 2.5, but we like to upgrade to
> > Spock 2 and Groovy 3, which runs on top of the JUnit Platform. For this,
> > I've already started working on a Spock extension that integrates Spock 2
> > in the Arquillian test life cycle [1]. This extension is inspired by the
> > (currently Alpha) JUnit5 module for Arquillian [2]. Both use a global
> > registration to keep track of the state managed by Arquilllian. This will
> > end up somewhere at the root of the TestPlan (for example see [3]).
> >
> > Because our tests are quite extensive, with a total run time of 6 hours,
> we
> > run them with a forkCount of 8, greatly reducing the total duration.
> > However, this is where SUREFIRE-1935 comes into play. With a forkCount >
> 1,
> > the entire test life cycle is started over and over again for every test
> > class. This happens in JUnitPlatformProvider at line 197 [4]. This
> results
> > in the entire Arquillian suite being torn down and setup for every class,
> > in our case adding several minutes to the execution of every test class
> > because the docker setup is done over and over again.
> >
> > To overcome this issue, the state from one test class execution has to be
> > carried to the next. It seems the LauncherSession (introduced in JUnit
> 5.8)
> > is meant to close this gap. However, this would mean my extension would
> > also need to implement a LauncherSessionListener, and I'm not sure if
> > extensions are supposed to integrate with the launcher as well. Also, for
> > this surefire would need to start a session prior to the tests, and close
> > it when done. I think this is a good idea anyway when running on platform
> > 1.8 or higher.
> >
> > Another solution could be a (sort of) dynamic test that produces the
> tests
> > to be run one by one. However, here my knowledge of JUnit really falls
> > short. I've got no idea of this is even possible.
> >
> > I hope someone can help me out on this one and point me in the right
> > direction, as we like to upgrade our test frameworks and this is blocking
> > us at the moment.
> >
> > Best regards,
> > Emond Papegaaij
> >
> > [1] Arquillian extension for Spock Framework 2:
> >
> >
> https://github.com/topicusonderwijs/arquillian-testrunner-spock/tree/spock-2.0-junit5
> > [2] Arquillian module for JUnit 5:
> > https://github.com/arquillian/arquillian-core/tree/master/junit5
> > [3] Registration in root store:
> >
> >
> https://github.com/arquillian/arquillian-core/blob/master/junit5/core/src/main/java/org/jboss/arquillian/junit5/JUnitJupiterTestClassLifecycleManager.java#L20
> > [4] JUnitPlatformProvider launching the tests:
> >
> >
> https://github.com/apache/maven-surefire/blob/master/surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/JUnitPlatformProvider.java#L197
> >
>
Re: JUnit Platform, forkCount > 1 and the TestPlan
Posted by Tibor Digana <ti...@apache.org>.
Please read the documentation
https://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html
On Fri, Sep 10, 2021 at 5:05 PM Tibor Digana <ti...@apache.org> wrote:
> If you use forkCount > 1, the Surefire loads test classes via load
> balancer.
> If you use default forkCount = 0, all the classes are run eagerly as a
> suite via JUnit5 Launcher in one shot.
>
> If you are aiming for Arquillian, testing the applications in the
> application server, you should use maven-failsafe-plugin which has another
> testing model.
> See the plugin goals of Failsafe plugin. Maven Failsafe Plugin – Plugin
> Documentation (apache.org)
> <https://maven.apache.org/surefire/maven-failsafe-plugin/plugin-info.html>
> https://maven.apache.org/surefire/maven-failsafe-plugin/plugin-info.html
> There are Maven phases:
> pre-integration-test
> integration-test
> post-integration-test
> verify
>
> Therefore you should start the application server in the phase
> pre-integration-test.
> Accordingly, you should stop it in the phase post-integration-test.
>
> Then use Failsafe plugin in the phases integration-test and verify.
>
> Cheers
> Tibor
>
>
>
> On Fri, Sep 10, 2021 at 12:42 PM Emond Papegaaij <
> emond.papegaaij@gmail.com> wrote:
>
>> Hi all,
>>
>> First of all, sorry for the lengthy post. I decided to add some context to
>> explain things a bit, but it resulted in quite a long e-mail. For the past
>> few weeks I've been trying to come up with a solution for the issue I
>> filled under SUREFIRE-1935, but I'm getting stuck and starting to feel
>> like
>> the issue cannot be solved with the current JUnit Platform API. To give
>> some perspective into why this issue is important for us, I first have to
>> explain a bit about our setup.
>>
>> We write our tests in the Spock framework and use Arquillian to run the
>> tests in the application container. Some of our tests, especially the
>> Selenium based tests, require quite some setup and tear down. Prior to
>> starting the test, Arquillian cube builds and starts several docker
>> containers and the tests are run against these containers. We currently
>> use
>> the JUnit 4 based Spock 1.3 with Groovy 2.5, but we like to upgrade to
>> Spock 2 and Groovy 3, which runs on top of the JUnit Platform. For this,
>> I've already started working on a Spock extension that integrates Spock 2
>> in the Arquillian test life cycle [1]. This extension is inspired by the
>> (currently Alpha) JUnit5 module for Arquillian [2]. Both use a global
>> registration to keep track of the state managed by Arquilllian. This will
>> end up somewhere at the root of the TestPlan (for example see [3]).
>>
>> Because our tests are quite extensive, with a total run time of 6 hours,
>> we
>> run them with a forkCount of 8, greatly reducing the total duration.
>> However, this is where SUREFIRE-1935 comes into play. With a forkCount >
>> 1,
>> the entire test life cycle is started over and over again for every test
>> class. This happens in JUnitPlatformProvider at line 197 [4]. This results
>> in the entire Arquillian suite being torn down and setup for every class,
>> in our case adding several minutes to the execution of every test class
>> because the docker setup is done over and over again.
>>
>> To overcome this issue, the state from one test class execution has to be
>> carried to the next. It seems the LauncherSession (introduced in JUnit
>> 5.8)
>> is meant to close this gap. However, this would mean my extension would
>> also need to implement a LauncherSessionListener, and I'm not sure if
>> extensions are supposed to integrate with the launcher as well. Also, for
>> this surefire would need to start a session prior to the tests, and close
>> it when done. I think this is a good idea anyway when running on platform
>> 1.8 or higher.
>>
>> Another solution could be a (sort of) dynamic test that produces the tests
>> to be run one by one. However, here my knowledge of JUnit really falls
>> short. I've got no idea of this is even possible.
>>
>> I hope someone can help me out on this one and point me in the right
>> direction, as we like to upgrade our test frameworks and this is blocking
>> us at the moment.
>>
>> Best regards,
>> Emond Papegaaij
>>
>> [1] Arquillian extension for Spock Framework 2:
>>
>> https://github.com/topicusonderwijs/arquillian-testrunner-spock/tree/spock-2.0-junit5
>> [2] Arquillian module for JUnit 5:
>> https://github.com/arquillian/arquillian-core/tree/master/junit5
>> [3] Registration in root store:
>>
>> https://github.com/arquillian/arquillian-core/blob/master/junit5/core/src/main/java/org/jboss/arquillian/junit5/JUnitJupiterTestClassLifecycleManager.java#L20
>> [4] JUnitPlatformProvider launching the tests:
>>
>> https://github.com/apache/maven-surefire/blob/master/surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/JUnitPlatformProvider.java#L197
>>
>
Re: JUnit Platform, forkCount > 1 and the TestPlan
Posted by Tibor Digana <ti...@apache.org>.
If you use forkCount > 1, the Surefire loads test classes via load balancer.
If you use default forkCount = 0, all the classes are run eagerly as a
suite via JUnit5 Launcher in one shot.
If you are aiming for Arquillian, testing the applications in the
application server, you should use maven-failsafe-plugin which has another
testing model.
See the plugin goals of Failsafe plugin. Maven Failsafe Plugin – Plugin
Documentation (apache.org)
<https://maven.apache.org/surefire/maven-failsafe-plugin/plugin-info.html>
https://maven.apache.org/surefire/maven-failsafe-plugin/plugin-info.html
There are Maven phases:
pre-integration-test
integration-test
post-integration-test
verify
Therefore you should start the application server in the phase
pre-integration-test.
Accordingly, you should stop it in the phase post-integration-test.
Then use Failsafe plugin in the phases integration-test and verify.
Cheers
Tibor
On Fri, Sep 10, 2021 at 12:42 PM Emond Papegaaij <em...@gmail.com>
wrote:
> Hi all,
>
> First of all, sorry for the lengthy post. I decided to add some context to
> explain things a bit, but it resulted in quite a long e-mail. For the past
> few weeks I've been trying to come up with a solution for the issue I
> filled under SUREFIRE-1935, but I'm getting stuck and starting to feel like
> the issue cannot be solved with the current JUnit Platform API. To give
> some perspective into why this issue is important for us, I first have to
> explain a bit about our setup.
>
> We write our tests in the Spock framework and use Arquillian to run the
> tests in the application container. Some of our tests, especially the
> Selenium based tests, require quite some setup and tear down. Prior to
> starting the test, Arquillian cube builds and starts several docker
> containers and the tests are run against these containers. We currently use
> the JUnit 4 based Spock 1.3 with Groovy 2.5, but we like to upgrade to
> Spock 2 and Groovy 3, which runs on top of the JUnit Platform. For this,
> I've already started working on a Spock extension that integrates Spock 2
> in the Arquillian test life cycle [1]. This extension is inspired by the
> (currently Alpha) JUnit5 module for Arquillian [2]. Both use a global
> registration to keep track of the state managed by Arquilllian. This will
> end up somewhere at the root of the TestPlan (for example see [3]).
>
> Because our tests are quite extensive, with a total run time of 6 hours, we
> run them with a forkCount of 8, greatly reducing the total duration.
> However, this is where SUREFIRE-1935 comes into play. With a forkCount > 1,
> the entire test life cycle is started over and over again for every test
> class. This happens in JUnitPlatformProvider at line 197 [4]. This results
> in the entire Arquillian suite being torn down and setup for every class,
> in our case adding several minutes to the execution of every test class
> because the docker setup is done over and over again.
>
> To overcome this issue, the state from one test class execution has to be
> carried to the next. It seems the LauncherSession (introduced in JUnit 5.8)
> is meant to close this gap. However, this would mean my extension would
> also need to implement a LauncherSessionListener, and I'm not sure if
> extensions are supposed to integrate with the launcher as well. Also, for
> this surefire would need to start a session prior to the tests, and close
> it when done. I think this is a good idea anyway when running on platform
> 1.8 or higher.
>
> Another solution could be a (sort of) dynamic test that produces the tests
> to be run one by one. However, here my knowledge of JUnit really falls
> short. I've got no idea of this is even possible.
>
> I hope someone can help me out on this one and point me in the right
> direction, as we like to upgrade our test frameworks and this is blocking
> us at the moment.
>
> Best regards,
> Emond Papegaaij
>
> [1] Arquillian extension for Spock Framework 2:
>
> https://github.com/topicusonderwijs/arquillian-testrunner-spock/tree/spock-2.0-junit5
> [2] Arquillian module for JUnit 5:
> https://github.com/arquillian/arquillian-core/tree/master/junit5
> [3] Registration in root store:
>
> https://github.com/arquillian/arquillian-core/blob/master/junit5/core/src/main/java/org/jboss/arquillian/junit5/JUnitJupiterTestClassLifecycleManager.java#L20
> [4] JUnitPlatformProvider launching the tests:
>
> https://github.com/apache/maven-surefire/blob/master/surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/JUnitPlatformProvider.java#L197
>