API for listening to generic test events

[Benedikt Ritter <br...@apache.org>]

Hello,

I'm currently working on a Maven extension that needs to be notified of
what happened during test execution in Surefire. Currently it is only
possible to register test framework specific listeners via the <properties>
setting [1]. However there is also the org.apache.maven.surefire.report.
RunListener interface [2]. I would like to provide my own implementation
for this interface, but it doesn't seem to be possible. I'm willing to
provide a PR to add this functionality if there are chances to get it
merged.

WDYT?

Benedikt

[1]
https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#properties
[2]
https://github.com/apache/maven-surefire/blob/master/surefire-api/src/main/java/org/apache/maven/surefire/report/RunListener.java

Re: API for listening to generic test events

[Tibor Digana <ti...@apache.org>]

You cannot expose the interface
org.apache.maven.surefire.report.RunListener outside.
The users implement native Listener of JUnit5 and use it on the classpath
very close to the runner (JUnit4 or JUnit5) and their tests.
So it must be the *org.junit.platform.launcher.TestExecutionListener*.

Surefire's listener might be loaded in another CL or anything may happen
(like breaking backwards compatibility of the listener) why the user should
not directly manipulate with it in his code or CL. The Surefire API is used
for developing new providers and plugins but not for these patches or
hacks, so it's better if users still consider it as a blackbox and use
native listener which normally appear on classpath of tests.

On Thu, May 16, 2019 at 12:51 PM Benedikt Ritter <br...@apache.org> wrote:

> Hello,
>
> Am Do., 16. Mai 2019 um 12:36 Uhr schrieb Tibor Digana <
> tibordigana@apache.org>:
>
> > Is it what Marc Philipp was asking me on the chat?
> > The purpose looked like a top secret ;-) but it was not because Mark
> needed
> > it for testing of some internal report.
> > Now it looks like Junit5 wants to remove the Surefire report out been
> dead
> > and move these features into Junit5.
> >
>
> My request is unrelated to JUnit 5 or how the test report is currently
> created.
>
>
> >
> > The RunListener could be always added. So it exists in JUnit4 and TestNG
> > provider for the purposes like custom report but not the standard report.
>
>
> > We agreed with JUnit5 developers in OpenTes4J to propose a specification
> > for a new standard report file.
> > This means the runners (e.g. JUnit or TestNG) should only run the tests
> and
> > Maven should run the build including creation of the project site
> reports.
> > The JUnit5 must not create report file because Maven controls all
> regarding
> > when/where and how to save and distribute the reports in the site.
> > Additionally, since the Maven is part of build system it may easily
> > integrate reports to CI systems, but JUnit5 must never do this; otherwise
> > it would be *chaos of responsibilities*.
> > Currently the IDEs handle events programatically from JUnit4 and then
> they
> > create GUI report, so the file report is useless for such GUI.
> >
> > The Surefire has several kinds of reports:
> > + Testset Info
> > + Console Report
> > + XML test report
> >
> > Having the test report in JUnit5 or OpenTes4J would be architecture
> > misunderstanding of responsibilities.
> > It is a perfect understanding of responsibilities when we introduce
> > extensions of these reports in Surefire and we do so as we proposed
> > together with Kristian Rosenvold and Andreas Gudian.
> > The same should happen with the new report standard (not only JUnit5)
> > initiated activities by JUni5 developers.
> > This way we will be fully consistent and users may expect clear
> > responsibility segregation between CI/Maven/JUnit5.
> > So I expect designing a specification of test report by OpenTes4J
> > contributors so that the tools, like Surefire etc, can internally
> implement
> > it according to their programming limitations and technologies they use.
> >
> > Benedikt, you can provide PR of course. It's quite simple to do - only
> > these two simple lines as follows (motivated by what exists in
> > JUnitCoreProvider).
> >
> >
> https://github.com/apache/maven-surefire/blob/master/surefire-providers/surefire-junit47/src/main/java/org/apache/maven/surefire/junitcore/JUnitCoreProvider.java#L101
> >
> >
> https://github.com/apache/maven-surefire/blob/master/surefire-providers/surefire-junit47/src/main/java/org/apache/maven/surefire/junitcore/JUnitCoreProvider.java#L158
> >
> >
> Yes, I know about this, but I'm talking
> about org.apache.maven.surefire.report.RunListener and not about test
> framework specific listeners.
>
>
> > So it's quite easy because it is these two lines to add in
> > JUnitPlatformProvider. Please improve the coverage with new unit tests
> and
> > integration tests.
> > Thx.
>
>
> >
> >
> >
> >
> > On Thu, May 16, 2019 at 11:01 AM Benedikt Ritter <br...@apache.org>
> > wrote:
> >
> > > Hello,
> > >
> > > I'm currently working on a Maven extension that needs to be notified of
> > > what happened during test execution in Surefire. Currently it is only
> > > possible to register test framework specific listeners via the
> > <properties>
> > > setting [1]. However there is also the
> org.apache.maven.surefire.report.
> > > RunListener interface [2]. I would like to provide my own
> implementation
> > > for this interface, but it doesn't seem to be possible. I'm willing to
> > > provide a PR to add this functionality if there are chances to get it
> > > merged.
> > >
> > > WDYT?
> > >
> > > Benedikt
> > >
> > > [1]
> > >
> > >
> >
> https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#properties
> > > [2]
> > >
> > >
> >
> https://github.com/apache/maven-surefire/blob/master/surefire-api/src/main/java/org/apache/maven/surefire/report/RunListener.java
> > >
> >
>

Re: API for listening to generic test events

[Benedikt Ritter <br...@apache.org>]

Hello,

Am Do., 16. Mai 2019 um 12:36 Uhr schrieb Tibor Digana <
tibordigana@apache.org>:

> Is it what Marc Philipp was asking me on the chat?
> The purpose looked like a top secret ;-) but it was not because Mark needed
> it for testing of some internal report.
> Now it looks like Junit5 wants to remove the Surefire report out been dead
> and move these features into Junit5.
>

My request is unrelated to JUnit 5 or how the test report is currently
created.


>
> The RunListener could be always added. So it exists in JUnit4 and TestNG
> provider for the purposes like custom report but not the standard report.


> We agreed with JUnit5 developers in OpenTes4J to propose a specification
> for a new standard report file.
> This means the runners (e.g. JUnit or TestNG) should only run the tests and
> Maven should run the build including creation of the project site reports.
> The JUnit5 must not create report file because Maven controls all regarding
> when/where and how to save and distribute the reports in the site.
> Additionally, since the Maven is part of build system it may easily
> integrate reports to CI systems, but JUnit5 must never do this; otherwise
> it would be *chaos of responsibilities*.
> Currently the IDEs handle events programatically from JUnit4 and then they
> create GUI report, so the file report is useless for such GUI.
>
> The Surefire has several kinds of reports:
> + Testset Info
> + Console Report
> + XML test report
>
> Having the test report in JUnit5 or OpenTes4J would be architecture
> misunderstanding of responsibilities.
> It is a perfect understanding of responsibilities when we introduce
> extensions of these reports in Surefire and we do so as we proposed
> together with Kristian Rosenvold and Andreas Gudian.
> The same should happen with the new report standard (not only JUnit5)
> initiated activities by JUni5 developers.
> This way we will be fully consistent and users may expect clear
> responsibility segregation between CI/Maven/JUnit5.
> So I expect designing a specification of test report by OpenTes4J
> contributors so that the tools, like Surefire etc, can internally implement
> it according to their programming limitations and technologies they use.
>
> Benedikt, you can provide PR of course. It's quite simple to do - only
> these two simple lines as follows (motivated by what exists in
> JUnitCoreProvider).
>
> https://github.com/apache/maven-surefire/blob/master/surefire-providers/surefire-junit47/src/main/java/org/apache/maven/surefire/junitcore/JUnitCoreProvider.java#L101
>
> https://github.com/apache/maven-surefire/blob/master/surefire-providers/surefire-junit47/src/main/java/org/apache/maven/surefire/junitcore/JUnitCoreProvider.java#L158
>
>
Yes, I know about this, but I'm talking
about org.apache.maven.surefire.report.RunListener and not about test
framework specific listeners.


> So it's quite easy because it is these two lines to add in
> JUnitPlatformProvider. Please improve the coverage with new unit tests and
> integration tests.
> Thx.


>
>
>
>
> On Thu, May 16, 2019 at 11:01 AM Benedikt Ritter <br...@apache.org>
> wrote:
>
> > Hello,
> >
> > I'm currently working on a Maven extension that needs to be notified of
> > what happened during test execution in Surefire. Currently it is only
> > possible to register test framework specific listeners via the
> <properties>
> > setting [1]. However there is also the org.apache.maven.surefire.report.
> > RunListener interface [2]. I would like to provide my own implementation
> > for this interface, but it doesn't seem to be possible. I'm willing to
> > provide a PR to add this functionality if there are chances to get it
> > merged.
> >
> > WDYT?
> >
> > Benedikt
> >
> > [1]
> >
> >
> https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#properties
> > [2]
> >
> >
> https://github.com/apache/maven-surefire/blob/master/surefire-api/src/main/java/org/apache/maven/surefire/report/RunListener.java
> >
>

Re: API for listening to generic test events

[Tibor Digana <ti...@apache.org>]

Is it what Marc Philipp was asking me on the chat?
The purpose looked like a top secret ;-) but it was not because Mark needed
it for testing of some internal report.
Now it looks like Junit5 wants to remove the Surefire report out been dead
and move these features into Junit5.

The RunListener could be always added. So it exists in JUnit4 and TestNG
provider for the purposes like custom report but not the standard report.

We agreed with JUnit5 developers in OpenTes4J to propose a specification
for a new standard report file.
This means the runners (e.g. JUnit or TestNG) should only run the tests and
Maven should run the build including creation of the project site reports.
The JUnit5 must not create report file because Maven controls all regarding
when/where and how to save and distribute the reports in the site.
Additionally, since the Maven is part of build system it may easily
integrate reports to CI systems, but JUnit5 must never do this; otherwise
it would be *chaos of responsibilities*.
Currently the IDEs handle events programatically from JUnit4 and then they
create GUI report, so the file report is useless for such GUI.

The Surefire has several kinds of reports:
+ Testset Info
+ Console Report
+ XML test report

Having the test report in JUnit5 or OpenTes4J would be architecture
misunderstanding of responsibilities.
It is a perfect understanding of responsibilities when we introduce
extensions of these reports in Surefire and we do so as we proposed
together with Kristian Rosenvold and Andreas Gudian.
The same should happen with the new report standard (not only JUnit5)
initiated activities by JUni5 developers.
This way we will be fully consistent and users may expect clear
responsibility segregation between CI/Maven/JUnit5.
So I expect designing a specification of test report by OpenTes4J
contributors so that the tools, like Surefire etc, can internally implement
it according to their programming limitations and technologies they use.

Benedikt, you can provide PR of course. It's quite simple to do - only
these two simple lines as follows (motivated by what exists in
JUnitCoreProvider).
https://github.com/apache/maven-surefire/blob/master/surefire-providers/surefire-junit47/src/main/java/org/apache/maven/surefire/junitcore/JUnitCoreProvider.java#L101
https://github.com/apache/maven-surefire/blob/master/surefire-providers/surefire-junit47/src/main/java/org/apache/maven/surefire/junitcore/JUnitCoreProvider.java#L158

So it's quite easy because it is these two lines to add in
JUnitPlatformProvider. Please improve the coverage with new unit tests and
integration tests.
Thx.






On Thu, May 16, 2019 at 11:01 AM Benedikt Ritter <br...@apache.org> wrote:

> Hello,
>
> I'm currently working on a Maven extension that needs to be notified of
> what happened during test execution in Surefire. Currently it is only
> possible to register test framework specific listeners via the <properties>
> setting [1]. However there is also the org.apache.maven.surefire.report.
> RunListener interface [2]. I would like to provide my own implementation
> for this interface, but it doesn't seem to be possible. I'm willing to
> provide a PR to add this functionality if there are chances to get it
> merged.
>
> WDYT?
>
> Benedikt
>
> [1]
>
> https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#properties
> [2]
>
> https://github.com/apache/maven-surefire/blob/master/surefire-api/src/main/java/org/apache/maven/surefire/report/RunListener.java
>