You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@maven.apache.org by Martin Lambert <ma...@ntlworld.com.INVALID> on 2020/07/01 22:12:51 UTC
Surefire - Forking - Identifying Tests that run in a particular fork
Hello,
I'm currently using the Surefire plugin with Maven 3.6.3 and would like to
know if there is any way to identify what tests were run in a given fork? I
haven't seen any option for it in the current documentation. I'm running
around ~10K test cases hence using the fork option to speed things up. The
problem I'm trying to solve is that a single test intermittently fails,
which, when run individually succeeds; I suspect this is due to a prior test
case not cleaning up properly and would therefore like to be able to
identify at a minimum what test cases are in a fork group so that I can take
a divide and conquer approach to resolving the issue, even better would be
to know which test cases ran before the failing test case.
Thanks for your help,
Martin.
RE: Surefire - Forking - Identifying Tests that run in a particular fork
Posted by Martin Lambert <ma...@ntlworld.com.INVALID>.
Hi Tibor,
I'm happy to help. Please see here for my current progress https://github.com/apache/maven-surefire/compare/master...mlambert:master.
Prior to raising a pull request I was planning to:
* Add more test cases
* Test these changes with a local maven project - I'm hoping it will be as simple as "mvn clean install" and then update the Surefire dependency in the project being used as a test harness
* Add exporting of the current fork number, possibly as a System property, to allow tests to include the fork number in their output
If you prefer I can raise a request based on what I have so far.
Kind Regards,
Martin.
-----Original Message-----
From: Tibor Digana <ti...@apache.org>
Sent: 09 July 2020 12:05
To: Maven Users List <us...@maven.apache.org>
Subject: Re: Surefire - Forking - Identifying Tests that run in a particular fork
Thank you Martin for your ideas.
I have expected this goal in the console logs.
Can you support us with implementing this feature?
To make it more simple for contributors we have the abstraction in a separate module.
Can you upgrade the class SurefireConsoleOutputReporter and open a pullrequest on GitHub?
There should be a boolean parameter which specifically enables this feature so that it stays disabled by default as we keep the backwards compatibility with the previous behavior.
The code has all data available, so this is a sandbox for you and it is quite doable i would say.
Cheers
Tibor
On Wed, Jul 8, 2020 at 11:43 PM Martin Lambert <ma...@ntlworld.com.invalid> wrote:
> Hi Tibor,
>
> Sorry it's taken a little while to get back to you.
>
> To set the scene, as basic requirements when running n tests serially
> I'd like to know...
>
> * When all pass: I'd mainly be interested in the fact that all tests
> passed, what tests ran and if any were skipped
> * If one or more tests failed: it seems reasonable to additionally be
> told which test(s) failed
> * If one or more tests failed and re-running a failed test
> individually did not reproduce a failure: it would be exceedingly
> helpful to also be able to see what tests ran before a failing test,
> including their output, in order to identify if prior tests were not
> cleaning up after themselves
>
> The reason for posting my original question was that when Surefire is
> configured to use forking, e.g. <forkCount>8</count>, then as far as
> I've been able to find out only the first 2 points above hold. To help
> visualise this, you could imagine getting console output something like:
>
> Running org.apache.maven.MyTest1
> Running org.apache.maven.MyTest32
> Running org.apache.maven.MyTest4
> Running org.apache.maven.MyTest77
> ~~~~~~ Output from test unknown
> ~~~~~~ Output from test unknown
> ~~~~~~ Output from test unknown
> ~~~~~~ Output from test unknown
> ~~~~~~ Output from test unknown
> ...
> Test finished: org.apache.maven.MyTest4 total: 6 success: 4 failure: 2
> skipped: 0
> ~~~~~~ Output from test unknown
> ~~~~~~ Output from test unknown
> Running org.apache.maven.MyTest2
> ...
>
> What I'm looking for is to be able to untangle this obfuscated log i.e.
> only look at the output for a single fork, and if necessary thread,
> thereby getting back to a similar place as running a set of tests
> serially. As an example, given the output below one could easily
> filter the output based on "[Fork 2]":
>
> [Fork 1] Running org.apache.maven.MyTest1 [Fork 2] Running
> org.apache.maven.MyTest32 [Fork 3] Running org.apache.maven.MyTest4
> [Fork 4] Running org.apache.maven.MyTest77 [Fork 2] ~~~~~~ Output from
> test unknown [Fork 2] ~~~~~~ Output from test unknown [Fork 1] ~~~~~~
> Output from test unknown [Fork 3] ~~~~~~ Output from test unknown
> [Fork 2] ~~~~~~ Output from test unknown ...
> [Fork 3] Test finished: org.apache.maven.MyTest4 total: 6 success: 4
> failure: 2 skipped: 0
> [Fork 4] ~~~~~~ Output from test unknown [Fork 4] ~~~~~~ Output from
> test unknown [Fork 3] Running org.apache.maven.MyTest2 ...
>
> My current work around achieves something similar to the last example
> and
> uses:
> * The JUnit TestExecutionListener mechanism to supplement the existing
> JUnit / Surefire log lines with test start and end messages that
> include a given fork JVM's process id, this could easily be expanded
> to include the thread if needed
> * Changed my test logging code to also include a given fork JVM's
> process id; again thread could be included if needed
>
> I did look at the examples page, however, on reflection it seems that
> if the fork functionality doesn't provide a mechanism for this out of
> the box then it would be highly desirable for it to do so based on the
> reasons outlined; of course, if the fork mechanism already supports
> doing this then it would be great to hear how to configure Surefire in
> order to get this behaviour.
>
> Cheers,
>
> Martin.
>
> -----Original Message-----
> From: Tibor Digana <ti...@apache.org>
> Sent: 05 July 2020 11:59
> To: martin.lambert2@ntlworld.com.invalid
> Cc: Maven Users List <us...@maven.apache.org>
> Subject: Re: Surefire - Forking - Identifying Tests that run in a
> particular fork
>
> Hi Martin,
>
> Can you be more concrete with your business expectations?
> What logger, what kind of notations about forked JVM id you want to
> express in the logs or somewhere else?
>
> Maybe you are not aware of new features in Surefire and Failsafe, but
> we introduced Extensions API in 3.0.0-M4.
> This way you have a chance to implement the Extensions API as you like.
> It means that the features of loggers and wrapped configuration
> parameters can be customized by adding new ones.
> See the documentation
>
> https://maven.apache.org/surefire/maven-surefire-plugin/examples/junit
> -platform.html#Surefire_Extensions_and_Reports_Configuration_for_.40Di
> splayName It is maybe easier for the user due to now it is quite
> simple for the contributor to concentrate on the abstraction which
> easily leads you to the
> (default) implementation.
>
> The extensions concerns anything, not only junit5.
>
> Cheers
> T
>
> On Fri, Jul 3, 2020 at 9:25 AM Martin Lambert <
> martin.lambert2@ntlworld.com.invalid> wrote:
>
> > Hi Falko,
> >
> > Thanks for the suggestion. In the end I created a simple Junit
> > TestExecutionListener class that overrode executionStarted() and
> > executionFinished(); both methods have a TestIdentifier parameter,
> > if this is actually an instance of MethodSource (
> > https://junit.org/junit5/docs/current/api/org.junit.platform.engine/
> > or
> > g/junit/platform/engine/support/descriptor/MethodSource.html)
> > you can downcast it, this allows you can get hold of the test class'
> > name using getClassName() on the same object; somewhere I was able
> > to call an
> > isTest() method to filter out noise - I can't remember where off the
> > top of my head. To make the whole thing work you have to create a
> > ServiceLoader file as mentioned in 6.1.4 here:
> > https://junit.org/junit5/docs/current/user-guide/#launcher-api-liste
> > ne rs-custom (it really does just involve creating the specific file
> > in the location mentioned, containing only the fully qualified name
> > of your TestExecutionListener class 😊 ).
> >
> > Cheers,
> >
> > Martin.
> >
> > -----Original Message-----
> > From: Falko Modler <f....@gmx.net>
> > Sent: 01 July 2020 23:48
> > To: users@maven.apache.org
> > Subject: Re: Surefire - Forking - Identifying Tests that run in a
> > particular fork
> >
> > Hi Martin,
> >
> > OOTB you'll probably have to resort to -X (which logs way too much)
> > and even then I am not 100% sure you'll get what you need.
> >
> > Did you have a look at |${surefire.forkNumber}| (see
> > https://maven.apache.org/surefire/maven-surefire-plugin/examples/for
> > k-
> > options-and-parallel-execution.html
> > )?
> >
> > I've been using this as a system property in my projects to:
> >
> > - write one logfile per fork (e.g. via Logback)
> > - prefix each logged line with the fork number (also via Logback,
> > especially useful for Jenins console log)
> > - log each test method in a JUnit4 listener together with the log
> > file (certainly also possible with JUnit5 etc.)
> > - separate working directories, generate tracing ids etc.
> >
> > I can really recommend to take these extra measures, otherwise
> > analyzing test failures with multiple forks and/or parallel modules
> > (-T) will be a real pain in the long run.
> >
> > Cheers,
> >
> > Falko
> >
> > Am 02.07.2020 um 00:12 schrieb Martin Lambert:
> > > Hello,
> > >
> > >
> > >
> > > I'm currently using the Surefire plugin with Maven 3.6.3 and would
> > > like to know if there is any way to identify what tests were run
> > > in a given fork? I haven't seen any option for it in the current
> > > documentation. I'm running around ~10K test cases hence using the
> > > fork option to speed things up. The problem I'm trying to solve is
> > > that a single test intermittently fails, which, when run
> > > individually succeeds; I suspect this is due to a prior test case
> > > not cleaning up properly and would therefore like to be able to
> > > identify at a minimum what test cases are in a fork group so that
> > > I can take a divide and conquer approach to resolving the issue,
> > > even better would be to know
> > which test cases ran before the failing test case.
> > >
> > >
> > >
> > > Thanks for your help,
> > >
> > >
> > >
> > > Martin.
> > >
> > >
> > >
> > >
> >
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> > For additional commands, e-mail: users-help@maven.apache.org
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Surefire - Forking - Identifying Tests that run in a particular fork
Posted by Tibor Digana <ti...@apache.org>.
Thank you Martin for your ideas.
I have expected this goal in the console logs.
Can you support us with implementing this feature?
To make it more simple for contributors we have the abstraction in a
separate module.
Can you upgrade the class SurefireConsoleOutputReporter and open a
pullrequest on GitHub?
There should be a boolean parameter which specifically enables this feature
so that it stays disabled by default as we keep the backwards compatibility
with the previous behavior.
The code has all data available, so this is a sandbox for you and it is
quite doable i would say.
Cheers
Tibor
On Wed, Jul 8, 2020 at 11:43 PM Martin Lambert
<ma...@ntlworld.com.invalid> wrote:
> Hi Tibor,
>
> Sorry it's taken a little while to get back to you.
>
> To set the scene, as basic requirements when running n tests serially I'd
> like to know...
>
> * When all pass: I'd mainly be interested in the fact that all tests
> passed, what tests ran and if any were skipped
> * If one or more tests failed: it seems reasonable to additionally be told
> which test(s) failed
> * If one or more tests failed and re-running a failed test individually
> did not reproduce a failure: it would be exceedingly helpful to also be
> able to see what tests ran before a failing test, including their output,
> in order to identify if prior tests were not cleaning up after themselves
>
> The reason for posting my original question was that when Surefire is
> configured to use forking, e.g. <forkCount>8</count>, then as far as I've
> been able to find out only the first 2 points above hold. To help visualise
> this, you could imagine getting console output something like:
>
> Running org.apache.maven.MyTest1
> Running org.apache.maven.MyTest32
> Running org.apache.maven.MyTest4
> Running org.apache.maven.MyTest77
> ~~~~~~ Output from test unknown
> ~~~~~~ Output from test unknown
> ~~~~~~ Output from test unknown
> ~~~~~~ Output from test unknown
> ~~~~~~ Output from test unknown
> ...
> Test finished: org.apache.maven.MyTest4 total: 6 success: 4 failure: 2
> skipped: 0
> ~~~~~~ Output from test unknown
> ~~~~~~ Output from test unknown
> Running org.apache.maven.MyTest2
> ...
>
> What I'm looking for is to be able to untangle this obfuscated log i.e.
> only look at the output for a single fork, and if necessary thread, thereby
> getting back to a similar place as running a set of tests serially. As an
> example, given the output below one could easily filter the output based on
> "[Fork 2]":
>
> [Fork 1] Running org.apache.maven.MyTest1
> [Fork 2] Running org.apache.maven.MyTest32
> [Fork 3] Running org.apache.maven.MyTest4
> [Fork 4] Running org.apache.maven.MyTest77
> [Fork 2] ~~~~~~ Output from test unknown
> [Fork 2] ~~~~~~ Output from test unknown
> [Fork 1] ~~~~~~ Output from test unknown
> [Fork 3] ~~~~~~ Output from test unknown
> [Fork 2] ~~~~~~ Output from test unknown
> ...
> [Fork 3] Test finished: org.apache.maven.MyTest4 total: 6 success: 4
> failure: 2 skipped: 0
> [Fork 4] ~~~~~~ Output from test unknown
> [Fork 4] ~~~~~~ Output from test unknown
> [Fork 3] Running org.apache.maven.MyTest2
> ...
>
> My current work around achieves something similar to the last example and
> uses:
> * The JUnit TestExecutionListener mechanism to supplement the existing
> JUnit / Surefire log lines with test start and end messages that include a
> given fork JVM's process id, this could easily be expanded to include the
> thread if needed
> * Changed my test logging code to also include a given fork JVM's process
> id; again thread could be included if needed
>
> I did look at the examples page, however, on reflection it seems that if
> the fork functionality doesn't provide a mechanism for this out of the box
> then it would be highly desirable for it to do so based on the reasons
> outlined; of course, if the fork mechanism already supports doing this then
> it would be great to hear how to configure Surefire in order to get this
> behaviour.
>
> Cheers,
>
> Martin.
>
> -----Original Message-----
> From: Tibor Digana <ti...@apache.org>
> Sent: 05 July 2020 11:59
> To: martin.lambert2@ntlworld.com.invalid
> Cc: Maven Users List <us...@maven.apache.org>
> Subject: Re: Surefire - Forking - Identifying Tests that run in a
> particular fork
>
> Hi Martin,
>
> Can you be more concrete with your business expectations?
> What logger, what kind of notations about forked JVM id you want to
> express in the logs or somewhere else?
>
> Maybe you are not aware of new features in Surefire and Failsafe, but we
> introduced Extensions API in 3.0.0-M4.
> This way you have a chance to implement the Extensions API as you like.
> It means that the features of loggers and wrapped configuration parameters
> can be customized by adding new ones.
> See the documentation
>
> https://maven.apache.org/surefire/maven-surefire-plugin/examples/junit-platform.html#Surefire_Extensions_and_Reports_Configuration_for_.40DisplayName
> It is maybe easier for the user due to now it is quite simple for the
> contributor to concentrate on the abstraction which easily leads you to the
> (default) implementation.
>
> The extensions concerns anything, not only junit5.
>
> Cheers
> T
>
> On Fri, Jul 3, 2020 at 9:25 AM Martin Lambert <
> martin.lambert2@ntlworld.com.invalid> wrote:
>
> > Hi Falko,
> >
> > Thanks for the suggestion. In the end I created a simple Junit
> > TestExecutionListener class that overrode executionStarted() and
> > executionFinished(); both methods have a TestIdentifier parameter, if
> > this is actually an instance of MethodSource (
> > https://junit.org/junit5/docs/current/api/org.junit.platform.engine/or
> > g/junit/platform/engine/support/descriptor/MethodSource.html)
> > you can downcast it, this allows you can get hold of the test class'
> > name using getClassName() on the same object; somewhere I was able to
> > call an
> > isTest() method to filter out noise - I can't remember where off the
> > top of my head. To make the whole thing work you have to create a
> > ServiceLoader file as mentioned in 6.1.4 here:
> > https://junit.org/junit5/docs/current/user-guide/#launcher-api-listene
> > rs-custom (it really does just involve creating the specific file in
> > the location mentioned, containing only the fully qualified name of
> > your TestExecutionListener class 😊 ).
> >
> > Cheers,
> >
> > Martin.
> >
> > -----Original Message-----
> > From: Falko Modler <f....@gmx.net>
> > Sent: 01 July 2020 23:48
> > To: users@maven.apache.org
> > Subject: Re: Surefire - Forking - Identifying Tests that run in a
> > particular fork
> >
> > Hi Martin,
> >
> > OOTB you'll probably have to resort to -X (which logs way too much)
> > and even then I am not 100% sure you'll get what you need.
> >
> > Did you have a look at |${surefire.forkNumber}| (see
> > https://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-
> > options-and-parallel-execution.html
> > )?
> >
> > I've been using this as a system property in my projects to:
> >
> > - write one logfile per fork (e.g. via Logback)
> > - prefix each logged line with the fork number (also via Logback,
> > especially useful for Jenins console log)
> > - log each test method in a JUnit4 listener together with the log file
> > (certainly also possible with JUnit5 etc.)
> > - separate working directories, generate tracing ids etc.
> >
> > I can really recommend to take these extra measures, otherwise
> > analyzing test failures with multiple forks and/or parallel modules
> > (-T) will be a real pain in the long run.
> >
> > Cheers,
> >
> > Falko
> >
> > Am 02.07.2020 um 00:12 schrieb Martin Lambert:
> > > Hello,
> > >
> > >
> > >
> > > I'm currently using the Surefire plugin with Maven 3.6.3 and would
> > > like to know if there is any way to identify what tests were run in
> > > a given fork? I haven't seen any option for it in the current
> > > documentation. I'm running around ~10K test cases hence using the
> > > fork option to speed things up. The problem I'm trying to solve is
> > > that a single test intermittently fails, which, when run
> > > individually succeeds; I suspect this is due to a prior test case
> > > not cleaning up properly and would therefore like to be able to
> > > identify at a minimum what test cases are in a fork group so that I
> > > can take a divide and conquer approach to resolving the issue, even
> > > better would be to know
> > which test cases ran before the failing test case.
> > >
> > >
> > >
> > > Thanks for your help,
> > >
> > >
> > >
> > > Martin.
> > >
> > >
> > >
> > >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> > For additional commands, e-mail: users-help@maven.apache.org
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
>
RE: Surefire - Forking - Identifying Tests that run in a particular fork
Posted by Martin Lambert <ma...@ntlworld.com.INVALID>.
Hi Tibor,
Sorry it's taken a little while to get back to you.
To set the scene, as basic requirements when running n tests serially I'd like to know...
* When all pass: I'd mainly be interested in the fact that all tests passed, what tests ran and if any were skipped
* If one or more tests failed: it seems reasonable to additionally be told which test(s) failed
* If one or more tests failed and re-running a failed test individually did not reproduce a failure: it would be exceedingly helpful to also be able to see what tests ran before a failing test, including their output, in order to identify if prior tests were not cleaning up after themselves
The reason for posting my original question was that when Surefire is configured to use forking, e.g. <forkCount>8</count>, then as far as I've been able to find out only the first 2 points above hold. To help visualise this, you could imagine getting console output something like:
Running org.apache.maven.MyTest1
Running org.apache.maven.MyTest32
Running org.apache.maven.MyTest4
Running org.apache.maven.MyTest77
~~~~~~ Output from test unknown
~~~~~~ Output from test unknown
~~~~~~ Output from test unknown
~~~~~~ Output from test unknown
~~~~~~ Output from test unknown
...
Test finished: org.apache.maven.MyTest4 total: 6 success: 4 failure: 2 skipped: 0
~~~~~~ Output from test unknown
~~~~~~ Output from test unknown
Running org.apache.maven.MyTest2
...
What I'm looking for is to be able to untangle this obfuscated log i.e. only look at the output for a single fork, and if necessary thread, thereby getting back to a similar place as running a set of tests serially. As an example, given the output below one could easily filter the output based on "[Fork 2]":
[Fork 1] Running org.apache.maven.MyTest1
[Fork 2] Running org.apache.maven.MyTest32
[Fork 3] Running org.apache.maven.MyTest4
[Fork 4] Running org.apache.maven.MyTest77
[Fork 2] ~~~~~~ Output from test unknown
[Fork 2] ~~~~~~ Output from test unknown
[Fork 1] ~~~~~~ Output from test unknown
[Fork 3] ~~~~~~ Output from test unknown
[Fork 2] ~~~~~~ Output from test unknown
...
[Fork 3] Test finished: org.apache.maven.MyTest4 total: 6 success: 4 failure: 2 skipped: 0
[Fork 4] ~~~~~~ Output from test unknown
[Fork 4] ~~~~~~ Output from test unknown
[Fork 3] Running org.apache.maven.MyTest2
...
My current work around achieves something similar to the last example and uses:
* The JUnit TestExecutionListener mechanism to supplement the existing JUnit / Surefire log lines with test start and end messages that include a given fork JVM's process id, this could easily be expanded to include the thread if needed
* Changed my test logging code to also include a given fork JVM's process id; again thread could be included if needed
I did look at the examples page, however, on reflection it seems that if the fork functionality doesn't provide a mechanism for this out of the box then it would be highly desirable for it to do so based on the reasons outlined; of course, if the fork mechanism already supports doing this then it would be great to hear how to configure Surefire in order to get this behaviour.
Cheers,
Martin.
-----Original Message-----
From: Tibor Digana <ti...@apache.org>
Sent: 05 July 2020 11:59
To: martin.lambert2@ntlworld.com.invalid
Cc: Maven Users List <us...@maven.apache.org>
Subject: Re: Surefire - Forking - Identifying Tests that run in a particular fork
Hi Martin,
Can you be more concrete with your business expectations?
What logger, what kind of notations about forked JVM id you want to express in the logs or somewhere else?
Maybe you are not aware of new features in Surefire and Failsafe, but we introduced Extensions API in 3.0.0-M4.
This way you have a chance to implement the Extensions API as you like.
It means that the features of loggers and wrapped configuration parameters can be customized by adding new ones.
See the documentation
https://maven.apache.org/surefire/maven-surefire-plugin/examples/junit-platform.html#Surefire_Extensions_and_Reports_Configuration_for_.40DisplayName
It is maybe easier for the user due to now it is quite simple for the contributor to concentrate on the abstraction which easily leads you to the
(default) implementation.
The extensions concerns anything, not only junit5.
Cheers
T
On Fri, Jul 3, 2020 at 9:25 AM Martin Lambert <ma...@ntlworld.com.invalid> wrote:
> Hi Falko,
>
> Thanks for the suggestion. In the end I created a simple Junit
> TestExecutionListener class that overrode executionStarted() and
> executionFinished(); both methods have a TestIdentifier parameter, if
> this is actually an instance of MethodSource (
> https://junit.org/junit5/docs/current/api/org.junit.platform.engine/or
> g/junit/platform/engine/support/descriptor/MethodSource.html)
> you can downcast it, this allows you can get hold of the test class'
> name using getClassName() on the same object; somewhere I was able to
> call an
> isTest() method to filter out noise - I can't remember where off the
> top of my head. To make the whole thing work you have to create a
> ServiceLoader file as mentioned in 6.1.4 here:
> https://junit.org/junit5/docs/current/user-guide/#launcher-api-listene
> rs-custom (it really does just involve creating the specific file in
> the location mentioned, containing only the fully qualified name of
> your TestExecutionListener class 😊 ).
>
> Cheers,
>
> Martin.
>
> -----Original Message-----
> From: Falko Modler <f....@gmx.net>
> Sent: 01 July 2020 23:48
> To: users@maven.apache.org
> Subject: Re: Surefire - Forking - Identifying Tests that run in a
> particular fork
>
> Hi Martin,
>
> OOTB you'll probably have to resort to -X (which logs way too much)
> and even then I am not 100% sure you'll get what you need.
>
> Did you have a look at |${surefire.forkNumber}| (see
> https://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-
> options-and-parallel-execution.html
> )?
>
> I've been using this as a system property in my projects to:
>
> - write one logfile per fork (e.g. via Logback)
> - prefix each logged line with the fork number (also via Logback,
> especially useful for Jenins console log)
> - log each test method in a JUnit4 listener together with the log file
> (certainly also possible with JUnit5 etc.)
> - separate working directories, generate tracing ids etc.
>
> I can really recommend to take these extra measures, otherwise
> analyzing test failures with multiple forks and/or parallel modules
> (-T) will be a real pain in the long run.
>
> Cheers,
>
> Falko
>
> Am 02.07.2020 um 00:12 schrieb Martin Lambert:
> > Hello,
> >
> >
> >
> > I'm currently using the Surefire plugin with Maven 3.6.3 and would
> > like to know if there is any way to identify what tests were run in
> > a given fork? I haven't seen any option for it in the current
> > documentation. I'm running around ~10K test cases hence using the
> > fork option to speed things up. The problem I'm trying to solve is
> > that a single test intermittently fails, which, when run
> > individually succeeds; I suspect this is due to a prior test case
> > not cleaning up properly and would therefore like to be able to
> > identify at a minimum what test cases are in a fork group so that I
> > can take a divide and conquer approach to resolving the issue, even
> > better would be to know
> which test cases ran before the failing test case.
> >
> >
> >
> > Thanks for your help,
> >
> >
> >
> > Martin.
> >
> >
> >
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Surefire - Forking - Identifying Tests that run in a particular fork
Posted by Tibor Digana <ti...@apache.org>.
Hi Martin,
Can you be more concrete with your business expectations?
What logger, what kind of notations about forked JVM id you want to express
in the logs or somewhere else?
Maybe you are not aware of new features in Surefire and Failsafe, but we
introduced Extensions API in 3.0.0-M4.
This way you have a chance to implement the Extensions API as you like.
It means that the features of loggers and wrapped configuration parameters
can be customized by adding new ones.
See the documentation
https://maven.apache.org/surefire/maven-surefire-plugin/examples/junit-platform.html#Surefire_Extensions_and_Reports_Configuration_for_.40DisplayName
It is maybe easier for the user due to now it is quite simple for the
contributor to concentrate on the abstraction which easily leads you to the
(default) implementation.
The extensions concerns anything, not only junit5.
Cheers
T
On Fri, Jul 3, 2020 at 9:25 AM Martin Lambert
<ma...@ntlworld.com.invalid> wrote:
> Hi Falko,
>
> Thanks for the suggestion. In the end I created a simple Junit
> TestExecutionListener class that overrode executionStarted() and
> executionFinished(); both methods have a TestIdentifier parameter, if this
> is actually an instance of MethodSource (
> https://junit.org/junit5/docs/current/api/org.junit.platform.engine/org/junit/platform/engine/support/descriptor/MethodSource.html)
> you can downcast it, this allows you can get hold of the test class' name
> using getClassName() on the same object; somewhere I was able to call an
> isTest() method to filter out noise - I can't remember where off the top of
> my head. To make the whole thing work you have to create a ServiceLoader
> file as mentioned in 6.1.4 here:
> https://junit.org/junit5/docs/current/user-guide/#launcher-api-listeners-custom
> (it really does just involve creating the specific file in the location
> mentioned, containing only the fully qualified name of your
> TestExecutionListener class 😊 ).
>
> Cheers,
>
> Martin.
>
> -----Original Message-----
> From: Falko Modler <f....@gmx.net>
> Sent: 01 July 2020 23:48
> To: users@maven.apache.org
> Subject: Re: Surefire - Forking - Identifying Tests that run in a
> particular fork
>
> Hi Martin,
>
> OOTB you'll probably have to resort to -X (which logs way too much) and
> even then I am not 100% sure you'll get what you need.
>
> Did you have a look at |${surefire.forkNumber}| (see
> https://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html
> )?
>
> I've been using this as a system property in my projects to:
>
> - write one logfile per fork (e.g. via Logback)
> - prefix each logged line with the fork number (also via Logback,
> especially useful for Jenins console log)
> - log each test method in a JUnit4 listener together with the log file
> (certainly also possible with JUnit5 etc.)
> - separate working directories, generate tracing ids etc.
>
> I can really recommend to take these extra measures, otherwise analyzing
> test failures with multiple forks and/or parallel modules (-T) will be a
> real pain in the long run.
>
> Cheers,
>
> Falko
>
> Am 02.07.2020 um 00:12 schrieb Martin Lambert:
> > Hello,
> >
> >
> >
> > I'm currently using the Surefire plugin with Maven 3.6.3 and would
> > like to know if there is any way to identify what tests were run in a
> > given fork? I haven't seen any option for it in the current
> > documentation. I'm running around ~10K test cases hence using the fork
> > option to speed things up. The problem I'm trying to solve is that a
> > single test intermittently fails, which, when run individually
> > succeeds; I suspect this is due to a prior test case not cleaning up
> > properly and would therefore like to be able to identify at a minimum
> > what test cases are in a fork group so that I can take a divide and
> > conquer approach to resolving the issue, even better would be to know
> which test cases ran before the failing test case.
> >
> >
> >
> > Thanks for your help,
> >
> >
> >
> > Martin.
> >
> >
> >
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
>
RE: Surefire - Forking - Identifying Tests that run in a particular fork
Posted by Martin Lambert <ma...@ntlworld.com.INVALID>.
Hi Falko,
Thanks for the suggestion. In the end I created a simple Junit TestExecutionListener class that overrode executionStarted() and executionFinished(); both methods have a TestIdentifier parameter, if this is actually an instance of MethodSource (https://junit.org/junit5/docs/current/api/org.junit.platform.engine/org/junit/platform/engine/support/descriptor/MethodSource.html) you can downcast it, this allows you can get hold of the test class' name using getClassName() on the same object; somewhere I was able to call an isTest() method to filter out noise - I can't remember where off the top of my head. To make the whole thing work you have to create a ServiceLoader file as mentioned in 6.1.4 here: https://junit.org/junit5/docs/current/user-guide/#launcher-api-listeners-custom (it really does just involve creating the specific file in the location mentioned, containing only the fully qualified name of your TestExecutionListener class 😊 ).
Cheers,
Martin.
-----Original Message-----
From: Falko Modler <f....@gmx.net>
Sent: 01 July 2020 23:48
To: users@maven.apache.org
Subject: Re: Surefire - Forking - Identifying Tests that run in a particular fork
Hi Martin,
OOTB you'll probably have to resort to -X (which logs way too much) and even then I am not 100% sure you'll get what you need.
Did you have a look at |${surefire.forkNumber}| (see https://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html)?
I've been using this as a system property in my projects to:
- write one logfile per fork (e.g. via Logback)
- prefix each logged line with the fork number (also via Logback, especially useful for Jenins console log)
- log each test method in a JUnit4 listener together with the log file (certainly also possible with JUnit5 etc.)
- separate working directories, generate tracing ids etc.
I can really recommend to take these extra measures, otherwise analyzing test failures with multiple forks and/or parallel modules (-T) will be a real pain in the long run.
Cheers,
Falko
Am 02.07.2020 um 00:12 schrieb Martin Lambert:
> Hello,
>
>
>
> I'm currently using the Surefire plugin with Maven 3.6.3 and would
> like to know if there is any way to identify what tests were run in a
> given fork? I haven't seen any option for it in the current
> documentation. I'm running around ~10K test cases hence using the fork
> option to speed things up. The problem I'm trying to solve is that a
> single test intermittently fails, which, when run individually
> succeeds; I suspect this is due to a prior test case not cleaning up
> properly and would therefore like to be able to identify at a minimum
> what test cases are in a fork group so that I can take a divide and
> conquer approach to resolving the issue, even better would be to know which test cases ran before the failing test case.
>
>
>
> Thanks for your help,
>
>
>
> Martin.
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Surefire - Forking - Identifying Tests that run in a particular
fork
Posted by Falko Modler <f....@gmx.net>.
Hi Martin,
OOTB you'll probably have to resort to -X (which logs way too much) and
even then I am not 100% sure you'll get what you need.
Did you have a look at |${surefire.forkNumber}| (see
https://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html)?
I've been using this as a system property in my projects to:
- write one logfile per fork (e.g. via Logback)
- prefix each logged line with the fork number (also via Logback,
especially useful for Jenins console log)
- log each test method in a JUnit4 listener together with the log file
(certainly also possible with JUnit5 etc.)
- separate working directories, generate tracing ids etc.
I can really recommend to take these extra measures, otherwise analyzing
test failures with multiple forks and/or parallel modules (-T) will be a
real pain in the long run.
Cheers,
Falko
Am 02.07.2020 um 00:12 schrieb Martin Lambert:
> Hello,
>
>
>
> I'm currently using the Surefire plugin with Maven 3.6.3 and would like to
> know if there is any way to identify what tests were run in a given fork? I
> haven't seen any option for it in the current documentation. I'm running
> around ~10K test cases hence using the fork option to speed things up. The
> problem I'm trying to solve is that a single test intermittently fails,
> which, when run individually succeeds; I suspect this is due to a prior test
> case not cleaning up properly and would therefore like to be able to
> identify at a minimum what test cases are in a fork group so that I can take
> a divide and conquer approach to resolving the issue, even better would be
> to know which test cases ran before the failing test case.
>
>
>
> Thanks for your help,
>
>
>
> Martin.
>
>
>
>